Zero-Shot Voice Cloning: How It Works + Best Open-Source Models (Try CosyVoice Free)
Zero-shot voice cloning copies a voice from a few seconds of audio, with no training for that speaker. Here's how models like CosyVoice do it, how the main open-source models compare on languages and licences, and how to clone a voice free in your browser.
What is zero-shot voice cloning?
Zero-shot voice cloning generates speech in a new voice from a short reference recording supplied at generation time. The model is never trained on that speaker — it picks the voice up from the reference, the way a language model follows a prompt. That is the difference from few-shot adaptation and fine-tuning:
| Approach | What you provide for a new voice | Training for that voice |
|---|---|---|
| Zero-shot | One short reference clip, given with each request | None — the voice is copied at generation time |
| Few-shot | A handful of recordings of the speaker | A short adaptation step before use |
| Fine-tuning | A larger, transcribed dataset of the speaker | Training a speaker-specific copy of the model |
How zero-shot voice cloning works (CosyVoice as the example)
Modern zero-shot systems split the job into "what is said" and "how it sounds". This is the pipeline the CosyVoice papers describe; other open-source models differ in the details.
1. Turn speech into tokens
A speech tokenizer turns audio into discrete speech tokens. CosyVoice derives its tokens from a speech recognition model, so they carry what is said and line up with the text.
2. Continue the voice with a language model
A text-speech language model reads your text together with the reference clip's transcript and tokens, then generates new speech tokens in the same speaking style. CosyVoice 2 uses the pre-trained Qwen2.5-0.5B LLM for this step.
3. Add the timbre with flow matching
A flow matching model turns the speech tokens into a mel spectrogram. Timbre enters here, from a speaker embedding and the reference speech — that's what makes the output sound like the reference speaker.
4. Render the audio
A pre-trained vocoder converts the spectrogram back into a waveform you can play.
Open-source zero-shot voice cloning models compared
Each row comes from the project's own README or model card, checked October 26, 2026. Licences differ more than quality does: several popular models publish weights for non-commercial use only.
| Model | Latest open release | Languages | Licence | Source |
|---|---|---|---|---|
| CosyVoice 3 (Fun-CosyVoice 3.0) | Fun-CosyVoice3-0.5B-2512 | 9 languages, 18+ Chinese dialects and accents; cross-lingual cloning | Apache-2.0 (the README adds an "academic purposes" disclaimer) | QwenAudio/CosyVoice README |
| CosyVoice 2 | CosyVoice2-0.5B | Chinese, English, Japanese, Korean | Apache-2.0 | Hugging Face: CosyVoice2-0.5B |
| F5-TTS | F5TTS_v1_Base | Chinese and English (base models trained on the Emilia ZH-EN data) | Code MIT; pre-trained models CC-BY-NC (non-commercial) | SWivid/F5-TTS (GitHub) |
| OpenVoice V2 | OpenVoice V2 | English, Spanish, French, Chinese, Japanese, Korean | MIT — the README says free for commercial use | myshell-ai/OpenVoice (GitHub) |
| IndexTTS | IndexTTS-2.5 | Chinese, English, Japanese, Spanish, Arabic | bilibili Model Use License Agreement | index-tts/index-tts (GitHub) |
| Spark-TTS | Spark-TTS-0.5B | Chinese and English | Code Apache-2.0; weights CC-BY-NC-SA-4.0 (non-commercial) | Hugging Face: SparkAudio/Spark-TTS-0.5B |
| VoxCPM | VoxCPM2 | 30 languages plus 9 Chinese dialects | Apache-2.0 for code and weights — the README says commercial-ready | OpenBMB/VoxCPM (GitHub) |
We don't rank these by quality: each project reports benchmarks on its own choice of test sets and settings, so the numbers aren't directly comparable. The best check is to clone your own voice with two or three of them.
For the full story on CosyVoice versions — open weights vs the hosted 3.5 models — see our CosyVoice 3 guide.
Clone a voice online in 3 steps
Don't want to install a model? CosyVoice.org runs zero-shot cloning on hosted CosyVoice models in your browser — no GPU, no Python.
- 1
Upload or record a clip
3 to 30 seconds of one person speaking, as MP3, WAV, M4A or OGG (up to 10 MB). About 10 seconds of clear speech works best.
- 2
Confirm you have permission
Only clone your own voice, or a voice whose owner has given you explicit permission.
- 3
Type your text and generate
The cloned voice reads your text, and the voice is saved to Your Voices for next time. A free account includes 2 voice clones a month.
How to record a good reference clip
The reference clip is the only thing the model knows about the voice, so its quality caps the result.
One speaker, no music
Background music, a second voice or a noisy room all leak into the clone. If your clip has them, separate the voice first.
About 10 seconds, at a normal pace
A few seconds can work, but around 10 seconds of natural, clearly spoken sentences gives the model more of the voice to copy.
Keep the transcript when self-hosting
CosyVoice's zero-shot mode takes the reference clip's transcript alongside the audio; its cross-lingual mode takes only the audio.
Cross-lingual is fine
A voice recorded in one language can speak another — the CosyVoice README lists cross-lingual zero-shot cloning as a key feature.
Use voice cloning responsibly
A cloned voice can be mistaken for the real person. Clone only your own voice or one you have explicit permission to use, don't use clones to deceive or impersonate, and label synthetic audio where people might assume it's real.
On CosyVoice.org these rules are part of our Acceptable Use Policy.
Zero-shot voice cloning FAQ
What is zero-shot voice cloning?
Generating speech in a new voice from a short reference recording, without training or fine-tuning the model on that speaker. The model copies the voice from the reference at generation time.
How much audio does zero-shot voice cloning need?
Seconds, not hours. CosyVoice.org accepts 3 to 30 seconds and works best with about 10 seconds of clear speech from one person. Self-hosted models have their own limits, so check each project's README.
What is the best open-source voice cloning model?
It depends on your languages and your licence needs. CosyVoice 3 covers 9 languages and 18+ Chinese dialects; VoxCPM2 lists 30 languages; OpenVoice V2 and VoxCPM2 are licensed for commercial use, while the F5-TTS and Spark-TTS weights are non-commercial. Each project benchmarks on different test sets, so try two or three on your own voice.
Is open-source voice cloning free for commercial use?
Not always. OpenVoice (MIT) and VoxCPM2 (Apache-2.0) say commercial use is allowed; F5-TTS and Spark-TTS publish their weights under non-commercial Creative Commons licences; IndexTTS has its own model licence; CosyVoice is Apache-2.0 but its README calls the content academic. Read the licence for the exact checkpoint you use — this is not legal advice.
Is zero-shot cloning the same as fine-tuning?
No. Fine-tuning trains a copy of the model on a larger dataset of one speaker. Zero-shot cloning uses the model as it is and takes the voice from a short clip at generation time, so a new voice is available immediately.
Can I clone a voice online without installing anything?
Yes. The CosyVoice.org voice cloning tool runs in your browser: upload or record 3 to 30 seconds of audio you have permission to use and type your text. A free account includes 2 voice clones a month; Pro removes the limit.
Sources
Facts on this page were checked against these official sources on October 26, 2026.