Multi-Token Prediction (MTP)

Multi-Token Prediction (MTP)

A training and (optionally) inference technique for LLMs: instead of training the model
to predict only the single next token at each position, MTP adds auxiliary heads that
predict several future tokens at once from the same hidden state.

Origin

Introduced in Meta’s 2024 paper “Better & Faster Large Language Models via
Multi-token Prediction” (Gloeckle et al.). The original design used a shared
transformer trunk with one output head per future-token offset, each head containing
its own transformer layers but sharing the final embedding matrix. Their headline
result: predicting 4 future tokens ahead improved code-generation benchmarks
significantly at 7B scale, without changing the underlying data.

How it works

• Standard next-token prediction: the hidden state at position t is used to
  predict only token t+1.

• MTP: the same hidden state (or a lightly-elaborated version of it) is also used
  to predict tokens t+2, t+3, … t+D, via D extra prediction heads.

• Training loss: combines the standard next-token loss with the auxiliary losses,
  typically weighted and averaged across depth:

  $L=L_{\text{next}}+\frac{\lambda }{D}{\sum }_{k=1}^D{L}_{\text{MTP}}^{(k)}$

• Implementations diverge on whether the extra heads predict in parallel or
  sequentially. DeepSeek-V3’s MTP module is causal and sequential: each MTP module
  combines the main decoder’s hidden state with the embedding of the next
  ground-truth token before predicting the one after that, preserving causal
  dependency between predicted tokens rather than firing all heads independently.

Why it helps

• Training: every sequence position now yields several supervised training
  signals instead of one, densifying the learning signal without needing more data —
  part of why newer models reach a given benchmark level with comparatively less
  training data.
• Inference (speculative decoding): the auxiliary heads can double as a built-in
  draft model. They draft a short continuation of several tokens; the main model then
  verifies that continuation in a single forward pass. Accepted tokens are emitted
  together; on a rejection, drafting restarts from that point. Realized speedup
  depends on the draft’s acceptance rate, not on the training objective alone — so MTP
  weights don’t automatically confer a speedup unless the serving stack actually
  activates and exploits them (plain transformers-style inference typically does
  not; it needs an MTP-aware serving framework).

Adoption (as of Sept 2026)

• DeepSeek — the most visible adopter. DeepSeek-V3 documents an MTP objective
  extending prediction scope to multiple future tokens per position; DeepSeek-V4,
  DeepSeek-R1 and DeepSeek V3.x variants carry MTP heads into inference for parallel
  token generation.
• GLM-5.1 and several Llama 4 variants ship MTP heads in their published
  weights.
• Qwen3 / Qwen3-Next 80B, Step 3.5 Flash (MTP-3, multi-depth), Nemotron 3
  Super (recursive shared-weight variant), and Tencent Hy4-preview have also
  adopted some form of MTP.

See also

• DeepSeek — the family most associated with production MTP use
• In Context Learning
• Better & Faster LLMs via Multi-token Prediction (Gloeckle et al., 2024) — originating paper
• Awesome-Multi-Token-Prediction — curated paper/model list
• Sebastian Raschka’s MTP architecture note