Wav2Lip-HD: Improving Wav2Lip to achieve High-Fidelity Videos

This repository contains code for achieving high-fidelity lip-syncing in videos, using the Wav2Lip algorithm for lip-syncing and the Real-ESRGAN algorithm for super-resolution. The combination of these two algorithms allows for the creation of lip-synced videos that are both highly accurate and visually stunning.

Algorithm

The algorithm for achieving high-fidelity lip-syncing with Wav2Lip and Real-ESRGAN can be summarized as follows:

1. The input video and audio are given to Wav2Lip algorithm.
2. Python script is written to extract frames from the video generated by wav2lip.
3. Frames are provided to Real-ESRGAN algorithm to improve quality.
4. Then, the high-quality frames are converted to video using ffmpeg, along with the original audio.
5. The result is a high-quality lip-syncing video.
6. The specific steps for running this algorithm are described in the Testing Model section of this README.

Testing Model

To test the "Wav2Lip-HD" model, follow these steps:

1. Clone this repository and install requirements using following command (Make sure, Python and CUDA are already installed):

   git clone https://github.com/saifhassan/Wav2Lip-HD.git
   cd Wav2Lip-HD
   pip install -r requirements.txt
   

2. Downloading weights

3. Put input video to input_videos directory and input audio to input_audios directory.

4. Open run_final.sh file and modify following parameters:

   filename=kennedy (just video file name without extension)

   input_audio=input_audios/ai.wav (audio filename with extension)

5. Execute run_final.sh using following command:

   bash run_final.sh
   

6. Outputs

• output_videos_wav2lip directory contains video output generated by wav2lip algorithm.
• frames_wav2lip directory contains frames extracted from video (generated by wav2lip algorithm).
• frames_hd directory contains frames after performing super-resolution using Real-ESRGAN algorithm.
• output_videos_hd directory contains final high quality video output generated by Wav2Lip-HD.

Results

The results produced by Wav2Lip-HD are in two forms, one is frames and other is videos. Both are shared below:

Example output frames </summary>

<table> <tr> <td>Frame by Wav2Lip</td> <td>Optimized Frame</td> </tr> <tr> <td><img src="examples/1_low.jpg" width=500></td> <td><img src="examples/1_hd.jpg" width=500></td> </tr> <tr> <td><img src="examples/kennedy_low.jpg" width=500></td> <td><img src="examples/kennedy_hd.jpg" width=500></td> </tr>

</tr> <tr> <td><img src="examples/mona_low.jpg" width=500></td> <td><img src="examples/mona_hd.jpg" width=500></td> </tr> </table> </Details>

Example output videos

Acknowledgements

We would like to thank the following repositories and libraries for their contributions to our work:

1. The Wav2Lip repository, which is the core model of our algorithm that performs lip-sync.
2. The face-parsing.PyTorch repository, which provides us with a model for face segmentation.
3. The Real-ESRGAN repository, which provides the super resolution component for our algorithm.
4. ffmpeg, which we use for converting frames to video.