Yatishara Blog
Paper digest: Omni-Diffusion
Nanjing University / Tencent Youtu / CASIA on masked discrete diffusion any-to-any. Text, speech, and images without an AR MLLM backbone.
Paper: Li, Long, Shen, Gao, Cao, Sun, Shan, He, Fu — Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion (Nanjing University, Tencent Youtu Lab, CASIA)
Source: arXiv:2603.06577 · PDF v1 · Project page
One line claim
You can build an any-to-any multimodal model — text, speech, and images in and out — on a masked discrete diffusion backbone alone, without parking an autoregressive LLM at the center and bolting generation heads around it.
Why this paper exists
Most multimodal large language models (MLLMs) still use a conventional autoregressive spine. Text is next-token prediction. Vision and speech are adapters, projectors, or external diffusion decoders hanging off hidden states. That design inherits the strengths of modern LLMs and also their generation order: left to right, one token after another.
Masked discrete diffusion models (MDMs) train by randomly replacing tokens with [MASK] and learning to recover the originals. At inference they start from a fully masked canvas and iteratively fill high-confidence positions. Parallel decoding is natural. So is inpainting: leave some tokens visible, mask the rest, fill the holes. Those properties are why diffusion already dominates continuous image synthesis and why people keep asking whether discrete diffusion can own language and multimodal systems too.
Omni-Diffusion’s claim is architectural primacy. It is presented as the first any-to-any multimodal language model built entirely on mask-based discrete diffusion for joint understanding and generation across text, speech, and images. Related systems such as AnyGPT stay any-to-any but AR. NExT-GPT routes through a frozen LLM plus external diffusion decoders. Discrete flow matching cousins exist; Omni-Diffusion insists on MDM mechanics: loss only on masked tokens at train time, iterative top-confidence unmasking at sample time.
If Attention Is All You Need is the AR ancestor story, this paper is one of the clearest 2026 bets that the multimodal default does not have to stay AR. Pair it with LaViDa-R1 if you care about reasoning post-training on multimodal diffusion; Omni-Diffusion is more about modality coverage and joint token modeling than RL recipes.
Method
Joint distribution over discrete multimodal tokens. Text, speech, and image are tokenized into discrete sequences, wrapped with modality begin/end markers, and concatenated into one sequence. Training corrupts that sequence with masks at a time-dependent ratio. The network predicts clean tokens for masked positions only (cross-entropy on masks). No separate modality-specific training loss is used once tokens sit in the shared sequence.
Tokenizers and backbone. Images use MAGVIT-v2 (downsample factor 16, codebook 8192) for both understanding and generation. Speech encoding uses SenseVoiceSmall plus an MLP adapter into the diffusion backbone. Speech generation predicts GLM-4-Voice discrete tokens (12.5 Hz, codebook 16384) and reconstructs waveforms with that decoder. The language backbone is Dream-7B, a pretrained discrete diffusion LM, with vocabulary expanded for speech and image codes. Aside from embeddings and output heads for the new codes, the MDM architecture stays intact.
That is the contrast with AR MLLM backbones: there is no frozen chat LLM emitting text latents that a separate image or TTS model must interpret. The diffusion model itself is supposed to hold a shared semantic space over multimodal discrete tokens.
Three-stage progressive training. Stage 1 aligns vision with language via captioning and text-to-image. Stage 2 keeps visual-text data and adds ASR/TTS for speech-text alignment. Stage 3 focuses on speech-driven visual interaction: spoken VQA and speech-to-image, plus more SQA/VQA. The authors build an SDVI dataset (~30k spoken VQA samples from filtered/rewritten LLaVA-OneVision via CosyVoice2 voice cloning, plus ~30k speech-to-image pairs from JourneyDB-style captions). Filters drop math/programming, rewrite multiple choice into open answers, and cap long spoken answers — a pragmatic bet on what spoken dialogue looks like.
Attenuated tail-pad masking. Variable-length generation needs pad tokens. Uniform masking over pads makes the model overfit pads and emit endless padding. They down-weight the mask ratio on pad tokens (γ < 1) so gradients stay dominated by semantic tokens.
Inference tricks tailored to masking. Decoding follows Dream-style entropy/confidence selection with repetition penalty and classifier-free guidance. Two Omni-specific controls:
- Position penalty for images. Early in sampling, logits toward the end of the sequence are scaled down so the model does not simultaneously grow the top and bottom of an image token grid and stamp duplicate patterns. Soft bias, not hard block-autoregression.
- Special token pre-infilling for spoken dialogue. Plant a begin-of-speech marker at ~0.25L of the masked canvas so the first quarter can become text and the rest speech, letting speech generation attend to text content.
- Adaptive token-length init for ASR/TTS (roughly 3.5× text length for TTS, 0.2× speech length for ASR) to cut wasted masks and speed sampling.
Results that moved
All figures below are as reported in the paper / project materials for the ~7B Omni-Diffusion system.
Speech
| System | Type | LibriSpeech WER ↓ | LibriTTS WER ↓ |
|---|---|---|---|
| CosyVoice | TTS expert | — | 2.89 |
| GLM-4-Voice | Speech LLM | 2.82 | 5.64 |
| AnyGPT | Any-to-any AR | 8.50 | — |
| Omni-Diffusion | Any-to-any MDM | 7.05 | 3.07 |
Versus AnyGPT, ASR improves (8.50 → 7.05). Versus GLM-4-Voice on LibriTTS, TTS WER improves sharply (5.64 → 3.07) and sits near the CosyVoice expert (2.89). This is the “one backbone, many modalities” bet looking competitive on speech, not only on VQA slideware.
Vision understanding and text-to-image
| System | Params | POPE ↑ | MME-P ↑ | Seed-2-Plus ↑ | CLIP-T ↑ | CLIP-I ↑ |
|---|---|---|---|---|---|---|
| LLaVA | 7B | 76.3 | 809.6 | 30.1 | — | — |
| InstructBLIP | 14B | 78.9 | 1212.8 | 29.2 | — | — |
| AnyGPT | 8B | 67.7* | — | — | — | 0.650 |
| NExT-GPT | 7B | — | — | 26.2 | 0.225* | 0.691* |
| Omni-Diffusion | 7B | 76.6 | 1216.7 | 34.5 | 0.235 | 0.667 |
Omni-Diffusion lands near specialized visual LLMs on POPE/MME while beating the any-to-any peers they report on understanding. Text-to-image CLIP-T is ahead of NExT-GPT’s reported 0.225; CLIP-I sits between AnyGPT and systems that lean on external pretrained image diffusion. The authors’ framing: not always the absolute best specialist, but strong for a single any-to-any diffusion model that also speaks.
Speech–vision alignment. Speech-to-image on MSCOCO captions synthesized with CosyVoice2 stays close to text-to-image: at 256 steps, speech conditioning scores about 0.225 / 0.645 CLIP-T/I versus text 0.235 / 0.667. That gap is the practical test of whether speech and text land in the same semantic neighborhood.
Sampling efficiency (parallel decode advantage)
Text-to-image latency and quality as steps drop:
| Steps | Latency (s/img) | CLIP-T / CLIP-I |
|---|---|---|
| 256 | 28.57 | 0.235 / 0.667 |
| 50 | 5.52 | 0.233 / 0.662 |
| 10 | 1.29 | 0.226 / 0.650 |
Speech-to-image similarly holds CLIP quality from 256 → 10 steps while latency falls (39.90s → 4.25s in their table). TTS WER on LibriTTS stays 3.07 at 0.5L steps and 3.74 at 0.25L; it degrades more at 0.125L (4.83) as you would expect.
Free inpainting. Because generation is mask filling, unknown image regions can be masked and completed without a dedicated inpainting finetune. That is a structural perk AR next-token models do not get for free.
Appendix spoken-VQA transfer (MME questions spoken via CosyVoice2) shows Text+Image→Text at 1216.7 MME-P versus Speech+Image→Text at 1019.9 — still workable, with an expected drop when the question arrives as audio.
Limits
The paper’s own limitation note is short: perception and generation work across modalities, but downstream tasks such as instruction-based visual editing still need more data and scale. That matters when you compare to LaViDa-R1, which spends its energy on reasoning RL for editing and grounding. Omni-Diffusion is the modality-unification story; it is not claiming R1-style math/grounding RL gains.
Other practical limits follow from the design. Quality still depends on external tokenizers and vocoders (MAGVIT-v2, SenseVoice, GLM-4-Voice). Spoken VQA numbers depend on synthetic speech conversion for both training data and some evals. ASR WER at 7.05 trails dedicated speech LLMs on LibriSpeech even while beating AnyGPT. Text-to-image CLIP-I does not dominate systems that outsource image synthesis to large continuous diffusion experts. Position penalty and pre-infilling are useful but are also evidence that naive MDM decoding is not enough for clean images and coherent spoken replies.
“First any-to-any MDM” is a literature claim; readers should treat adjacent discrete-flow multimodal work as cousins, not ignore them. And like every unified model paper, benchmark wins on POPE/MME/CLIP do not automatically transfer to messy Caribbean production audio, WhatsApp voice notes, or flyer edits with brand fonts.
Who should care
Researchers building unified multimodal systems. If your roadmap still assumes “LLM center + modality peripherals,” this paper is a concrete alternative topology: one masked diffusion prior over mixed discrete tokens, plus training/inference tactics that make variable length, images, and speech behave.
Speech+vision product teams. Spoken VQA and speech-to-image are first-class here, not demos glued after a text VLM ships. If your users ask questions with their voice while looking at a photo — support lines, field ops, accessibility — the architecture is closer to the product than a text-only chat wrapper.
Engineers hunting latency knobs. The step-ablation tables are the buying argument for discrete diffusion: drop from 256 to 10 image steps and keep most CLIP quality while cutting latency from tens of seconds toward ~1–5s in their measurement setup. That is the operational contrast with strict left-to-right AR image token streams.
Caribbean operators and small studios without a sales org. You may never fine-tune Dream-7B on SDVI. You still live in a market where clients send voice notes, product photos, and “make an ad” asks in the same WhatsApp thread. AR chatbots and separate TTS/image apps force you to stitch those modalities by hand. Omni-Diffusion is research proof that vendors will keep trying to collapse that stitch into one model. Use the paper to sharpen procurement and build decisions: Can the system take speech+image and return speech? Does quality hold when sampling steps (and billable compute) are cut? Is inpainting native or a second product? You do not need a sales deck to ask those questions — you need the benchmark names and the failure modes.
Creators evaluating “omni” marketing: demand speech WER, a VQA suite, and a speech-conditioned generation metric, not only pretty text-to-image samples. This paper’s own tables make that easy.
Bottom line
Omni-Diffusion argues that masked discrete diffusion can be the whole multimodal backbone, not a side decoder. The supporting numbers are concrete: LibriTTS WER 3.07, POPE 76.6, MME-P 1216.7, Seed-2-Plus 34.5, text-to-image CLIP-T/I 0.235/0.667, and usable quality at 10 diffusion steps. The research bet is joint multimodal tokens under one mask model. The product bet is fewer modality glue layers — if serving stacks and downstream editing catch up.
We report papers. Digests written at Yatishara Blog; not a product review.