3.1 Overview
Seed-VC [5] is a diffusion-based zero-shot voice conversion model developed by ByteDance’s Seed Team. With 552
million parameters, it achieves state-of-the-art speaker similarity (SECS: 0.867 vs. 0.754 for OpenVoice and 0.844 forCosyVoice) while preserving linguistic content. The model requires only 1–30 seconds of reference audio and
performs conversion without any fine-tuning.
The training process involves an External Timbre Shifter (ETS) that applies random timbre perturbations to the
source audio before content encoding. This forces the content encoder to discard speaker-specific information,
producing speaker-agnostic tokens. The style encoder (CAM++) operates on the original, unperturbed source to
extract the ground-truth speaker embedding. The model is trained to reconstruct the original source from the perturbed
content tokens and the ground-truth style embedding, thereby learning to combine arbitrary content with arbitrary
style.
Key public benchmarks for Seed-VC V2: – SECS (Speaker Embedding Cosine Similarity): 0.867 – WER (Word Error
Rate, content preservation): 8.2% – MOS (Mean Opinion Score): 4.12/5.0 – Inference speed: ~0.5× real-time on A100
GPU
3.2 Content Encoder
The content encoder employs a two-level discrete representation to address the timbre leakage problem that plagues
earlier voice conversion systems:
•
Narrow level (32-token codebook): HuBERT-Large features from layer 18 are processed by a ConvNeXt
encoder and quantized via Binary Spherical Quantization. This produces a compact token sequence that captures
prosody and temporal structure, used by the AR model for duration-aware generation.
•
Wide level (2048-token codebook): A Whisper-small tokenizer on the same backbone produces a richer token
representation for fine content fidelity. These tokens condition the CFM diffusion process.
The two-level design ensures that the AR model has precise prosodic alignment while the diffusion model has
sufficient content detail for high-fidelity reconstruction.
3.3 AR Transformer
The autoregressive transformer generates content tokens conditioned on source content and target style. Architecture
details: – 12 transformer layers, 768 hidden dimension, 12 attention heads – Rotary Position Embedding (RoPE) for
sequence position encoding – KV-cache pre-allocation for efficient inference
The AR model takes the source narrow tokens and target style embedding as input, and generates target narrow tokens
autoregressively.
3.4 CFM / Diffusion Transformer
The core generation engine is a Conditional Flow Matching (CFM) [6] model built on a Diffusion Transformer
(DiT) backbone:
•
13-layer DiT with 512 hidden dimension
•
Euler ODE solver with 30 integration steps
•
Multi-condition Classifier-Free Guidance (CFG) on both content and style embeddings
•
Flow matching replaces traditional diffusion with a straight-line ODE path from noise to data, enabling faster
inference with fewer steps
3.5 BigVGAN Vocoder
The vocoder converts mel-spectrograms to waveforms using NVIDIA’s BigVGAN v2 [7], a GAN-based neural
vocoder: – 22 kHz output sampling rate – 80-band mel-spectrogram input (1024 FFT, 256 hop, 1024 window) – 256×
upsampling via transposed convolutions – HiFi-GAN discriminator for adversarial training
The vocoder is pretrained and used without fine-tuning.
