Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed Data

Overview

Authors

Sungwon Kim Heeseung Kim Sungroh Yoon

Abstract

We propose Guided-TTS 2, a diffusion-based generative model for high-quality adaptive TTS using untranscribed data. Guided-TTS 2 combines a speaker-conditional diffusion model with a speaker-dependent phoneme classifier for adaptive text-to-speech. We train the speaker-conditional diffusion model on large-scale untranscribed datasets for a classifier-free guidance method and further fine-tune the diffusion model on the reference speech of the target speaker for adaptation, which only takes 40 seconds. We demonstrate that Guided-TTS 2 shows comparable performance to high-quality single-speaker TTS baselines in terms of speech quality and speaker similarity with only a ten-second untranscribed data. We further show that Guided-TTS 2 outperforms adaptive TTS baselines on multi-speaker datasets even with a zero-shot adaptation setting. Guided-TTS 2 can adapt to a wide range of voices only using untranscribed speech, which enables adaptive TTS with the voice of non-human characters such as Gollum in “The Lord of the Rings”.

Guided-TTS-2

Samples

Societal Impact

Guided-TTS 2 has an advantage in significantly reducing data required for high-quality adaptive TTS. In addition, Guided-TTS 2 can adapt to not only human voice but also non-human characters such as Gollum, which shows the possibility of extension to TTS for non-human characters in industries such as games and movies. On the other hand, 10-second untranscribed speech for the target speaker is easy to obtain through recording or YouTube clips for celebrities, and the contribution of Guided-TTS 2 that reduces the data required for high-quality adaptive TTS makes a lot of room for misuse. Guided-TTS 2 is likely to be misused as voice phishing for individuals or to have a fatal effect on the security system through voice. Given this potential misuse, we’ve decided not to release our code. Although we do not release the code, due to the adaptation ability of the diffusion-based model, we expect that the adaptive TTS technology is highly likely to be misused like Deepfake. We leave the research on anti-spoofing that distinguishes generated speech from real audio as future work, considering the potential for misuse of Guided-TTS 2.