Diffusion language models (DLMs) enable fast generation by predicting multiple tokens in parallel, yet their practical adoption remains hindered by a persistent quality gap relative to comparably sized autoregressive models. We attribute this gap to a computation–difficulty mismatch: within a partially observed sequence, some unknown tokens are readily predictable, while others require substantially more computation to resolve. Existing DLMs, however, apply uniform computational depth to every unknown position at each denoising step. We introduce ALoDLM, which replaces this uniform computation with token-adaptive latent recurrence. At each denoising step, ALoDLM iteratively refines representations in latent space, allocating computation based on token difficulty. Tokens ready to commit are fed back as discrete context, while unresolved tokens retain and refine their latent states through additional recurrent passes. To learn both token prediction and computation allocation end-to-end, we formulate token-wise computation schedules as latent variables and derive a conditional negative evidence lower bound (NELBO). We train ALoDLM at 1.7B and 8B parameter scales. Across eleven benchmarks, ALoDLM outperforms all evaluated DLMs and the corresponding autoregressive baselines in average benchmark score at both scales. Importantly, ALoDLM combines superior generation quality with fast parallel decoding, establishing a strong quality–efficiency trade-off among all evaluated autoregressive and diffusion models under optimized inference engines.
A looped architecture with dynamic recurrent depth: inside every denoising step, a shared recurrent core runs up to K times, the maximum recurrent depth. Easy tokens commit early and act as resolved context, while difficult tokens keep refining their accumulated latent states through additional recurrent passes.
Token-wise exit schedules are treated as latent variables, and a conditional NELBO learns token prediction and computation allocation end-to-end. A single-trajectory gradient estimator is unbiased for this objective; the practical recipe relaxes its KL term and adds variance reduction to stabilize training.
To the best of our knowledge, ALoDLM-8B is the largest looped DLM to date. It has the best average score among all evaluated DLMs and beats its Qwen3 AR counterparts in average score over eleven benchmarks (65.5 vs 63.8 at 1.7B, 80.3 vs 78.5 at 8B), with direct SFT on a 5B-token corpus.
On GSM8K, ALoDLM-8B delivers ≈2.7× the throughput of vLLM-served Qwen3-8B at comparable or higher accuracy. Recurrent depth is an effective test-time scaling axis, and the model learns token-dependent refinement preferences without explicit difficulty labels.
ALoDLM combines strong performance with fast inference.
ALoDLM keeps the parallel denoising of a diffusion language model and adds an inner loop to every denoising step, so that computation follows token difficulty. Part 01 shows how one step runs at inference; Part 02 shows how the model learns when to stop refining.
Within a single denoising step, masked positions can differ greatly in how hard they are to predict, because of intrinsic uncertainty, multi-hop reasoning, or dependence on other tokens that are still unresolved. Yet a standard DLM applies the same fixed-depth denoiser to every masked position, wasting computation on easy predictions and starving hard ones.
Confidence-based decoding helps only partly: it defers low-confidence predictions to later denoising steps, but a deferred token still gets the same fixed compute, and its latent computation is thrown away and recomputed from its mask embedding at the next step.
Masked positions in one denoising step
Example answer: She sells 16 − 3 − 4 = 9 eggs a day, so she makes $18. Masked in this step: sells, 9, eggs, so, makes, $ and 18; some are easy to predict, others (9 and 18) are hard.
Standard DLM: one fixed depth
Mismatch: easy positions get more than they need, hard ones less.
Illustrative bar chart. Every masked position gets the same fixed depth. sells, eggs, so and $ need less than that, so part of their depth is wasted. makes needs a little more, 9 clearly more and 18 the most, so they are starved.
ALoDLM: passes follow difficulty
Matched: a position keeps refining until it commits, for at most K = 4 recurrent passes.
Illustrative stacked bars on a separate scale of recurrent passes, with the same difficulty outlines. sells, eggs, so and $ commit after pass 1; makes after pass 2; 9 after pass 3; and 18 after pass 4, the maximum recurrent depth K = 4.
Principle. A DLM should maintain and refine persistent latent states for challenging tokens within each denoising step, so that computation scales with prediction difficulty.
A denoising step starts by encoding the partially masked sequence with the Prelude. The shared Recurrent Core then runs repeatedly; after every recurrent pass, the Coda produces a readout that feeds two heads: an LM head (a token distribution per position) and an exit gate (a halting probability).
Latent states persist only within a step: committed tokens are written into the sequence, and the next denoising step re-initializes from it.
ALoDLM-8B · from Qwen3-8B, 36 layers
ALoDLM-1.7B · from Qwen3-1.7B, 28 layers
The Prelude runs once per denoising step; the Recurrent Core and the Coda readout run at every pass, and the next pass continues from the core’s output, with committed positions replaced by their token embeddings. The exit gate reads a stop-gradient copy of the readout.
One recurrent pass
\widetilde h^{(s)} = \mathsf{Recurrent\ Core}\bigl(h^{(s-1)}\bigr), \qquad r^{(s)} = \mathsf{Coda}\bigl(\widetilde h^{(s)}\bigr), \qquad s = 1,\ldots,Kp_i^{(s)} = \operatorname{softmax}\bigl(\mathsf{LMHead}(r_i^{(s)})\bigr), \qquad
\lambda_i^{(s)} = \begin{cases} \sigma\bigl(\mathsf{ExitGate}(\operatorname{sg}[r_i^{(s)}])\bigr), & s < K, \\ 1, & s = K. \end{cases}h_i^{(s)} = \begin{cases} \operatorname{Emb}(\widehat y_i), & i \in \mathcal{C}^{(s)}, \\ \widetilde h_i^{(s)}, & \text{otherwise} \end{cases}a_i^{(s)}=a_i^{(s-1)}+\big(1-a_i^{(s-1)}\big)\lambda_i^{(s)}Algorithm 1 ALoDLM decoding
Input: masked sequence x, maximum depth K, entropy threshold τ, cumulative halt threshold q
Line 5 produces the core output \widetilde h^{(s)}; line 6 is a readout branch, and the next pass continues from the pre-Coda state. Table 1 uses a single-token quality preset: greedy token selection in place of Sample, one token committed per model pass with τ inactive, a 16-token window, and q = 0.4 (1.7B) or 0.5 (8B). Figures 1 and 3 use the parallel decoder with the entropy rule of lines 9–11.
Training learns a denoiser together with a halting policy. An exit schedule \mathbf{z} assigns every masked position i\in\mathcal{M}_t an exit depth z_i\in\{1,\ldots,K\}: token i commits after recurrent pass z_i. Together with the noisy sequence \mathbf{x}_t, a schedule fully determines one denoising step of up to K passes.
J(\theta,\phi;\mathbf{x}_t,\mathbf{y}) := \mathbb{E}_{\mathbf{z}\sim q_\phi}\Biggl[\underbrace{-\sum_{i\in\mathcal{M}_t}\log p_{\theta,i}^{(z_i)}\bigl(y_i\mid\mathbf{x}_t;\mathbf{z}\bigr)}_{\mathcal{L}_{\mathrm{traj}}(\mathbf{z})}\Biggr] + D_{\mathrm{KL}}\bigl(q_\phi\,\|\,\pi\bigr) \;\geq\; -\log p_\theta\bigl(\mathbf{y}_{\mathcal{M}_t}\mid\mathbf{x}_t\bigr)\mathcal{L}_{\mathrm{train}}(\theta,\phi) := \mathbb{E}_{\mathbf{y}\sim p_{\mathrm{data}},\; t\sim\operatorname{Unif}[0,1],\; \mathbf{x}_t\sim q_t(\cdot\mid\mathbf{y})}\left[\frac{1}{t}\,J(\theta,\phi;\mathbf{x}_t,\mathbf{y})\right]\widehat{\mathcal{L}}_{\mathrm{sur}}(\mathbf{z}) = \frac{1}{t}\Biggl\{\mathcal{L}_{\mathrm{traj}}(\mathbf{z}) + \operatorname{sg}\!\left[\mathcal{L}_{\mathrm{traj}}(\mathbf{z}) + \log\frac{q_\phi(\mathbf{z})}{\pi(\mathbf{z})}\right]\log q_\phi(\mathbf{z})\Biggr\}\mathbb{E}_{\mathbf{y},t,\mathbf{x}_t,\mathbf{z}}\!\left[\nabla_{\theta,\phi}\widehat{\mathcal{L}}_{\mathrm{sur}}(\mathbf{z})\right] = \nabla_{\theta,\phi}\mathcal{L}_{\mathrm{train}}(\theta,\phi)\mathbf{x}_t corrupted input · \mathbf{y} clean sequence · \mathcal{M}_t masked positions · q_\phi schedule distribution parameterized by the exit gate · \pi truncated-geometric prior (c = 0.4) · q_t forward masking process · \operatorname{sg} stop-gradient.
Each commitment changes the context of the tokens that are still unresolved, so exit depths are modelled jointly, and summing over all exit schedules would need exponentially many separate rollouts. Instead, the exit gate parameterizes a variational distribution qϕ over schedules, and a truncated-geometric prior π with c = 0.4 favours shallow exits.
A schedule is sampled pass by pass during a single recurrent rollout. The denoiser is trained by ordinary backpropagation; the exit gate gets a score-function term whose detached, sequence-level cost credits each halting decision for its effect on the whole sequence. By Prop. 1 this gives unbiased gradients of the objective above. The practical recipe relaxes the KL term to keep token-wise adaptivity, reweights the denoiser loss across passes and adds an auxiliary next-token loss, so the guarantee concerns the original objective.
One rollout already yields a readout at every pass, so each token is also supervised at earlier passes, weighted by its detached exit-depth probabilities; the gate’s cost subtracts the first-pass prediction cost as a control variate. On a measured subset of 4,096 denoiser normalization parameters, removing intermediate supervision raises conditional gradient variance to 1.76×, 1.49× and 1.42× the full-method value at training steps 1k, 6.5k and 17k.
ALoDLM-1.7B and ALoDLM-8B are trained from Qwen3-1.7B and Qwen3-8B by direct SFT on a 5B-token corpus, without continued pre-training, with maximum recurrent depth K = 4. All models are evaluated on eleven benchmarks with the OpenCompass protocol: native chat templates, greedy decoding, a 4,096-token generation limit, and one token per model pass under each model’s default decoding rule.
With parallel decoding on GSM8K, two thresholds set ALoDLM-8B’s operating point: the entropy threshold τ decides which tokens commit, and the exit threshold q decides how long unresolved tokens keep refining. Single-stream on one NVIDIA B200.
Citation for the arXiv preprint.
@misc{fang2026alodlmadaptivelyloopeddiffusion,
title={ALoDLM: Adaptively Looped Diffusion Language Models},
author={Liancheng Fang and Zhuowei Li and Youngeun Kim and Tianchen Zhao and Rajat Koner and Jiaye Wu and Linghan Xu and Xuanbai Chen and Xiang Xu and Zheng Zhang and Jakub Zablocki and Nishant Sankaran and Yifan Xing},
year={2026},
eprint={2610.04198},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2610.04198},
}