🪴 Prabhav's Log

Voxtral

Jul 22, 20264 min read

Voxtral (Mistral paper)

Source clip: Voxtral · Mistral AI · arXiv 2507.13264 (July 2025)

Technical report for Voxtral Mini (4.7B) and Small (24.3B), a pair of open-weights (Apache 2.0) audio-chat models that understand speech and text and respond in text. SOTA transcription/translation among open & closed models in their price class; competitive with GPT-4o mini / Gemini 2.5 Flash on speech QA; no text-only regression vs the LLM backbone. 32K context → ~40-min audio.

Figure 1 — Voxtral Architecture.

Figures & tables (the PDF→markdown conversion dropped all 9 figures and mangled Tables 2 & 3, so these were restored manually): all nine figures rendered from the PDF live in the raw clip Voxtral and on the wiki — Fig 1 (architecture) & Fig 3 (ASR) on Voxtral, Fig 2 (pretrain patterns) on Audio-Text Pretraining Patterns. Clean results tables (params, Table 8 understanding, Table 7 translation) reconstructed on Voxtral.

Key takeaways

  • A speech-understanding model, not a dialogue model. Audio (or text) in → text out. Comprehends audio: transcribe, translate, QA, summarize, function-call. Not full-duplex, not speech-output — a different class from Moshi / Interaction Models. → Speech Understanding.
  • Encoder → adapter → decoder cascade. Whisper large-v3 encoder (50 Hz, 30 s chunked attention, short audio padded) → MLP adapter that downsamples 4× to 12.5 Hz → Mistral LLM decoder (Ministral 3B for Mini, Mistral Small 3.1 24B for Small). The opposite of TML’s Encoder-Free Early Fusion; a third route vs Moshi’s codec.
  • Two pretraining patterns, 50/50. Repetition (<repeat>, audio→its transcript) drives transcription; cross-modal continuation (<next>, audio→next text) drives understanding. Either alone cripples the other capability; balance recovers both. → Audio-Text Pretraining Patterns.
  • Heavily synthetic post-training. SFT QA/summarization/translation generated by Mistral Large over long-form transcripts; text SFT (incl. function calling) TTS’d to audio; real-speech questions paired with LLM answers to fix TTS-only generalization. Then DPO / Online DPO ranked by a text reward model fed the audio’s transcription.
  • Open contributions: Apache-2.0 weights for both sizes; speech-synthesized GSM8K / TriviaQA / MMLU benchmarks + an internal SU benchmark; native audio function calling.

Results

  • ASR (WER ↓): Voxtral Small SOTA on English Short-Form & MCV; Voxtral Mini Transcribe (a repetition-only ASR variant) beats GPT-4o mini Transcribe and Gemini 2.5 Flash on every task group despite 4.7B size. Per-task English: Table 3; per-language FLEURS/MCV/MLS: Tables 4–6.
  • Translation (BLEU ↑, FLEURS, Table 7): Voxtral Small SOTA on every language pair tested.
  • Understanding (Table 8): competitive with closed models; Small beats GPT-4o mini Audio on 3/7 tasks.
  • Text (Fig 6): Voxtral Small ≈ Mistral Small 3.1 — a drop-in text+audio replacement.

Ablations

  • Padding (§5.1): keep Whisper’s 30 s padding (removing it costs 0.5% WER on FLEURS-fr).
  • Downsampling (§5.2): 12.5 Hz (4×) is the sweet spot — minimal ASR loss, +1.5% Llama QA over 50 Hz; 6.25 Hz costs >1% WER. (12.5 Hz also happens to match Moshi’s Mimi rate.)
  • Pattern ratio (§5.3): 50/50 essential (see above).
  • Online DPO (§5.4, Table 2): improves SU quality; shipped as default Mini checkpoint; Small default stays SFT (Online DPO slightly regressed En short-form WER).

Relation to the rest of the wiki

Voxtral is the wiki’s first Speech Understanding model, a sibling category to the real-time full-duplex cluster. Contrasts worth tracking:

Open threads to pursue

  • Could a speech-understanding encoder+adapter stack be made streaming/full-duplex, or is the batch encoder a hard ceiling (cf. the codec-vs-encoder-free question on Moshi)?
  • Online DPO with a transcription-fed text reward model — how much does ignoring the raw audio (prosody, emotion) cost on paralinguistic tasks?
  • How would Voxtral’s contributed speech-synthesized benchmarks rank Moshi / a full-duplex model on understanding (vs their native interactivity strengths)?
  • Voxtral keeps Whisper’s 30 s padding as a constraint — does the chunked-attention trick generalize cleanly to the 40-min context, or degrade at the long tail?

Graph View

Backlinks