Voice-Clone Recipe & Harness — Annex

This is the reproducible-mechanics companion to The Franglais Turing Test. Read that page first for the story; read this one to rebuild the rig.

The three-gate scorer

One scorer (eval/score_voice.py) is the source of truth for every engine. A candidate differs from a rival only in which wav it produced; scoring is identical, so cross-engine comparison is apples-to-apples.

Gate	Measures	Method	Pass
Noise	Clean, not hiss	Per-frame spectral flatness geomean(power)/mean(power), silence-gated by RMS energy; fraction of voiced frames above a tonal threshold	dur < 15s and frac < 0.20
Intel	The right words	Parakeet STT :9000, bag-of-words overlap (multiset intersection / expected word count) vs expected text	overlap >= 0.70
Cosine	Sounds like Bert	resemblyzer cosine vs a real-Bert anchor (centroid of held-out real clips)	higher = closer; ceiling ~0.96

The intelligibility gate is what makes the harness trustworthy. A noise-only or noise-plus-cosine harness green-lights speech-shaped gibberish — audio that is spectrally clean and carries enough timbre to score a non-trivial cosine, yet transcribes as a degenerate loop. Identity without intelligibility is a hallucinated impostor.

Caution

Parakeet :9000 transiently false-zeroes the intel gate under rapid sequential load (empty-body responses), which would poison a whole sweep. The scorer retries three times with a short linear backoff and treats an empty body as a transient hiccup, not a silent wav. A flaky read is not a failing sample.

The Chatterbox T3 LoRA recipe

Chatterbox is a ~0.5B Llama-style T3 text-to-token backbone feeding a frozen S3Gen codec stage. The fine-tune is a LoRA on the T3 backbone only, S3Gen frozen.

• Train against the transformer (model.t3.tfmr) with a manual teacher-forced cross-entropy, not the model’s own t3.loss() — which has an axis bug. Trainer: train_t3_lora.py.
• Inference: attach the adapter with PeftModel.from_pretrained onto t3.tfmr, then prepare_conditionals(dry_ref) and generate with the dry reference as the audio prompt.
• Inference scale is the gibberish dial. A strong adapter at alpha 32 / rank 16 (effective scale 2.0) saturates into gibberish. Sweep: scale 0.5 and 0.25 → 100% word-overlap; 1.0 → 64%; 2.0 → 0%. Canonical setting near 0.5 — a stronger adapter wants a lower inference scale.
• Reference acoustics dominate identity. The model copies the reference clip’s room. A dry, close-mic reference beats a stage clip; reference-room acoustics is a bigger lever than any engine swap.

Multilingual base

Chatterbox-Multilingual is the same identity on a 23-language base (t3_mtl23ls) with a French tokenizer and a required language_id parameter. Isolated in its own venv (.venv-cbx-mtl). The text must be re-tokenized with the multilingual tokenizer, though the S3 speech tokens carry over.

Caution

A silent trap on the multilingual venv: pip pulled setuptools 82, which dropped pkg_resources, which made the PerTh watermarker silently initialize to None — clips would have gone out unwatermarked with no error. Pin setuptools==80.10.2.

Apple-Silicon notes

• Heavy generation and training run on the MacBook Pro over the TB5 link; the Mini holds the council. Run long fine-tunes off the flaky link (cloud/HF GPU, or tmux with frequent checkpoints).
• Pin mlx / mlx-audio exactly. The proven pair is mlx 0.31.1 / mlx-audio 0.4.2; a one-minor bump silently breaks synthesis. Treat a green /health as a false positive — verify with a real self-warm that emits non-zero audio bytes.
• The STT side is Parakeet (parakeet-tdt-0.6b-v3), which is multilingual and handles French; measured ~7.7 GB true footprint — the surprise memory hog. The TTS/STT stack is Python; the Rust sanctum-tts :8007 dispatcher is built but not yet deployed.

The Québécois campaign (2026-07-04 addendum)

The bilingual clone shipped English at Turing grade and French at “way too Parisian” — Bert’s native ear, the only accent instrument that matters. The campaign to fix it produced three durable artifacts: a locked best-of-Chatterbox French recipe, a calibrated accent judge, and an F5-TTS fine-tune pipeline that measurably moves a base model’s accent prior.

The locked Chatterbox French recipe

Before pivoting engines, the deployed recipe got its ceiling measured — ~24 scored candidates across every dial. The keeper, live in sanctum-bert-say.py:

Language	Model	cfg	Post
EN	cbx-bert-mlx-q8 (English base)	0.5	none — Turing-approved
FR	cbx-bert-mtl-mlx-q8 (multilingual base, language_id=fr)	0.55	no atempo, no chunking

FR cfg 0.55 (up from 0.4) killed two birds: the “réalitéééé” stutter was a low-cfg sampling degeneracy amplified by the old atempo 0.9 stretch — and higher cfg is naturally slower, so the atempo crutch got deleted outright.

The measured ceilings that forced the pivot: prosody plateaus flat/robotic (~60/100 across 19 candidates — T3-architectural, no dial breaks it) and French accent is doubly blocked — the multilingual base carries a Parisian prior no reference overrides, while the English base has no French phoneme inventory at all (it mangles “chu” into a glitch). A leading English word (“Hey,”) even primes the whole utterance anglophone — cold-start language bleed, proper nouns (“Bertrand”, “Outremont”) hit hardest.

The calibrated forced-choice judge

The old absolute-score Gemini rubric rated a clip Bert heard as flatly Parisian at joual 95/100. Twice-burned rule: an LLM ear cannot score accent on an absolute scale. It can, however, pick which of two anchors a clip sits closer to.

f5-eval/accent_judge.py: clip X judged against A = real Bert Québécois and B = a France-French TTS anchor (macOS Eddy (French (France))), three runs, majority vote, plus a resemblyzer identity cosine vs a Bert-FR anchor. Before trusting it, it must pass a calibration set with known answers — including the exact clip the old rubric blew, which it now calls Parisian, agreeing with the native ear. The judge returns cited phonetic markers (affrication, nasal quality, diphthongization, laxing), which turns it from a scoreboard into a debugging instrument.

Caution

Transcripts feed everything downstream — and Whisper launders joual: measured on 40 clips, Whisper preserved 0 joual markers where a verbatim-prompted Gemini heard 13 (“pis”→“puis”, “chu”→“je suis”, fillers dropped). F5’s ref_text must be exact or the ref-trim leaks reference audio into the output; training labels must be verbatim or the model never learns what written joual sounds like.

F5-TTS: moving the prior instead of fighting it

F5-TTS (flow-matching DiT, full fine-tune only) is the pivot because its failure is fixable: zero-shot with the only public French base (RASPIAUDIO/F5-French-MixedSpeakers-reduced, LibriVox European French, v0 arch) scored Parisian 0/9, confidence 100 even with a perfect joual reference — the reference cannot beat the base prior — but the prior itself moves with training, which is exactly what Chatterbox never offered.

Campaign v1 (886 fr + 95 franglais clips, 47 min, Whisper labels, LR 1e-5, 5280 updates): the accent climbed in textbook phonetic order — affrication [t͡sy] at update 500, diphthongization [lɑʊ̯] by 1000, vowel laxing pis→[pɪs] and the first probe-level QUEBEC verdict by 1500-2000 — peaked at 3/9 québécois votes around updates 2000-3500, then overfit back down to 0/9. Native-ear check at the peak: “still parisian, but getting there” — judge and ear agree, third time.

v2 attacks v1’s measured limits: Gemini-verbatim labels (the joual is in the text now), joual-dense clips oversampled ×2-3, the ~106 English-heavy “franglais” rows dropped (only one genuinely code-switched clip exists in the whole corpus — the diglossia strikes again), and training deliberately stopped inside v1’s peak zone (~2940 updates).

Apple-Silicon F5 traps, banked

• f5-tts-mlx is v1-arch-hardcoded (text_mask_padding=True, no pe_attn_head); a converted v0 French checkpoint loads clean and generates garbage. Torch worker until the DiT grows v0 switches. Its duration param is total (reference + generation) seconds.
• finetune_cli saves checkpoints inside the venv (files("f5_tts")/../../ckpts/), copies the --pretrain file there, and auto-resumes from model_last.pt — so a kill that lands on the every-100-update save truncates the zip and the relaunch dies on a miniz error. TERM between saves, never SIGKILL near a ×100 boundary.
• macOS default 256 fds × 16 dataloader workers respawning per epoch = EMFILE at the epoch boundary. ulimit -n 65536 in the launch line.
• Full checkpoints are 5.4 GB (model+EMA+optimizer), not the 1.3 GB EMA-only size — budget keep_last_n against real disk, and copy any peak checkpoint out before the pruner eats it (v1’s two best checkpoints died this way; v2’s eval loop preserves best-so-far automatically).