SpeechT5 TTS Swedish

This model is a fine-tuned version of microsoft/speecht5_tts on the Common Voice dataset. It achieves the following results on the evaluation set:

Loss: 0.4621

Model description

Swedish SpeechT5 model trained on Swedish language in Common Voice. Example on how to implement the model below. Test the model yourself at https://huggingface.co/spaces/GreenCounsel/SpeechT5-sv (not possible to run pipeline inference at Huggingface).

#pip install datasets soundfile 
#pip install transformers
#pip install sentencepiece

from transformers import SpeechT5Processor, SpeechT5ForTextToSpeech, SpeechT5HifiGan, set_seed
import torch

processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_tts")
model = SpeechT5ForTextToSpeech.from_pretrained("GreenCounsel/speecht5_tts_common_voice_5_sv")
vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan")

repl = [
    ('Ä', 'ae'),
    ('Å', 'o'),
    ('Ö', 'oe'),
    ('ä', 'ae'),
    ('å', 'o'),
    ('ö', 'oe'),
    ('ô','oe'),
    ('-',''),
    ('‘',''),
    ('’',''),
    ('“',''),
    ('”',''),

]

from datasets import load_dataset
embeddings_dataset = load_dataset("Matthijs/cmu-arctic-xvectors", split="validation")

speaker_embeddings = torch.tensor(embeddings_dataset[7000]["xvector"]).unsqueeze(0)
set_seed(555)

text="Förstår du vad han menar?"
for src, dst in repl:
       text = text.replace(src, dst)
inputs = processor(text=text, return_tensors="pt")

speech = model.generate_speech(inputs["input_ids"], speaker_embeddings, vocoder=vocoder)

import soundfile as sf
sf.write("output.wav", speech.numpy(), samplerate=16000)

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 1e-05
train_batch_size: 16
eval_batch_size: 8
seed: 42
gradient_accumulation_steps: 2
total_train_batch_size: 32
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 500
training_steps: 4000
mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.5349	4.8	1000	0.4953
0.5053	9.59	2000	0.4714
0.5032	14.39	3000	0.4646
0.4958	19.18	4000	0.4621

Framework versions

Transformers 4.30.0.dev0
Pytorch 2.0.1+cu118
Datasets 2.13.1
Tokenizers 0.13.3