Transformer Models
(Filename keeps the original “2023” snapshot name; the note now covers the architecture and its lineage through 2026.)
What a transformer is
The Transformer was introduced in “Attention Is All You Need” (Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin; arXiv 12 June 2017), an architecture “based solely on attention mechanisms, dispensing with recurrence and convolutions”. It reached 28.4 BLEU on WMT 2014 English-German and 41.8 on English-French. Because every position attends to all others, sequences are processed in parallel, which made large-scale pre-training practical.
Core components:
- Self-attention: each token weighs every other token to build context-aware representations.
- Multi-head attention: several attention operations in parallel, each learning different relations.
- Positional encoding: injects order information (the original paper used sinusoidal encodings).
- Encoder / decoder blocks: attention plus feed-forward layers; encoder-only, decoder-only and encoder-decoder variants all exist.
Landmark models (verified against the papers)
| Model | Year | Idea |
|---|---|---|
| BERT (Devlin et al., Google) | Oct 2018 | Deep bidirectional pre-training by masked-word prediction; state of the art on 11 NLP benchmarks |
| GPT series (generative-pretrained-transformer) | 2018 onward | Left-to-right (decoder-only) language model; GPT-3 (arXiv May 2020) has 175B parameters and showed few-shot learning |
| T5 (Raffel et al., Google) | Oct 2019 | Every NLP task cast as text-to-text |
| Switch Transformer (Fedus, Zoph, Shazeer) | Jan 2021 | Simplified mixture-of-experts routing, scaling to trillion-parameter sparse models |
| Megatron-Turing NLG 530B (Microsoft and NVIDIA) | Jan 2022 | 530B-parameter dense model trained with DeepSpeed and Megatron |
2026 status: the post-2023 architecture picture
The decoder-only transformer is still the base of today’s large-language-models, but the block has been changed in four main ways.
- Mixture-of-experts (MoE): each token is routed to a few expert feed-forward blocks, so total parameters far exceed active ones. Mixtral 8x7B (Mistral, arXiv 2024-01-08) used 8 experts per layer with 2 selected per token: 47B total, 13B active, Apache 2.0. DeepSeek-V3 (arXiv 2024-12-27) has 671B total with 37B active per token. The vault’s qwen-family note lists Qwen3.8 open weights at 2.4T total / 95B active (as recorded there; not re-checked); see also deepseek-family, kimi-k3 and mixture-of-experts-in-agent-swarms (a different, agent-level use of the term).
- Attention variants: grouped-query attention (GQA, arXiv 2023-05) shares key/value heads across query heads to cut inference memory; Multi-head Latent Attention (MLA), used in DeepSeek-V3, compresses the KV cache; FlashAttention (arXiv 2022-05) computes exact attention with IO-aware tiling so long sequences fit in GPU memory.
- State-space and hybrid models: Mamba (Gu and Dao, arXiv 2023-12-01) replaces attention with selective state-space layers that scale linearly in sequence length. Jamba (AI21, arXiv 2024-03-28) interleaves Transformer and Mamba layers with MoE and reports strong results up to 256K tokens context. In the vault, nvidia-nemotron is described as a hybrid Mamba-Transformer MoE, and qwen-family records Gated DeltaNet plus Gated Attention layers in Qwen3.8 (linear-attention hybrids).
- Long context: 262K native context (extensible to about 1M) is recorded for the Qwen3.8 models in qwen-family; practical limits and degradation are covered in context-rot-and-long-context.
Pure transformers have not disappeared; in the author’s view the trend is towards sparse and hybrid blocks to cut inference cost per token.
Applications
NLU (classification, NER, QA), machine translation, text generation, code, speech recognition (e.g. Conformer, convolution plus transformer), vision and multimodal models.
Challenges
Training and inference cost grows with parameters and context length (attention cost is quadratic in sequence length); commonly used mitigations include distillation (DistilBERT, TinyBERT), pruning, quantisation, efficient or sparse attention, and MoE. Interpretability, robustness to adversarial input and bias remain open.
Related
neural-networks, artificial-neural-network, in-context-learning, massive-multitask-language-understanding.
Sources
- https://arxiv.org/abs/1706.03762 (accessed 2026-10-02)
- https://arxiv.org/abs/1810.04805 (accessed 2026-10-02)
- https://arxiv.org/abs/2005.14165 (accessed 2026-10-02)
- https://arxiv.org/abs/1910.10683 (accessed 2026-10-02)
- https://arxiv.org/abs/2101.03961 (accessed 2026-10-02)
- https://arxiv.org/abs/2201.11990 (accessed 2026-10-02)
- https://arxiv.org/abs/2401.04088 - Mixtral of Experts (accessed 2026-10-05)
- https://arxiv.org/abs/2412.19437 - DeepSeek-V3 Technical Report (accessed 2026-10-05)
- https://arxiv.org/abs/2312.00752 - Mamba (accessed 2026-10-05)
- https://arxiv.org/abs/2403.19887 - Jamba (accessed 2026-10-05)
- https://arxiv.org/abs/2305.13245 - GQA (accessed 2026-10-05)
- https://arxiv.org/abs/2205.14135 - FlashAttention (accessed 2026-10-05)
- Qwen3.8 and Nemotron details are taken from the vault notes qwen-family and nvidia-nemotron, which cite vendor model cards (not re-checked here).