Multimodal Models

Multimodal models process and/or generate more than one modality: text, images, audio, video, sometimes actions. Common designs encode images or audio into tokens that a language model reads alongside text. CLIP (2021) aligned images and text in a shared embedding space; Visual Instruction Tuning (LLaVA, 2023) attached a vision encoder to an LLM and tuned it on instructions; GPT-4 (2023) was announced as accepting image and text input.

Kinds

• Understanding: image/PDF/screen/video input, text out (vision-models, computer-use-agents rely on screenshots).
• Generation: text-to-image and text-to-video, mostly diffusion-models (flux-dot-1, nano-banana, google-veo, openai-sora).
• Speech and audio: see ai-audio notes such as speech-synthesis.
• Embodied: vision-language-action models for robots (physical-ai-and-humanoids).
• Multimodal retrieval: embedding-models and gemini-file-search.

Related: large-language-model, tokenization, computer-vision.

Sources

• https://arxiv.org/abs/2103.00020 (CLIP, 2021)
• https://arxiv.org/abs/2304.08485 (Visual Instruction Tuning, 2023)
• https://arxiv.org/abs/2303.08774 (GPT-4 Technical Report, 2023)
• https://en.wikipedia.org/wiki/Multimodal_learning (secondary)
• arXiv API 2026-10-07 confirmed titles/dates: CLIP 2021-02, Visual Instruction Tuning 2023-04, GPT-4 Technical Report 2023-03.

Open items

• Current model-by-modality capability table not included; see the model notes.