Meta Segment Anything Model 3 (SAM 3)

Meta Segment Anything Model 3 (SAM 3)

by Meta AI (meta)

Promptable Concept Segmentation: unified detection, segmentation, and tracking foundation model

See: https://ai.meta.com/blog/segment-anything-model-3/

Status update (2026-09-30)

• SAM 3 was released in 2025. The GitHub repo (facebookresearch/sam3) announced SAM 3.1 “Object Multiplex” on 2026-03-27: shared-memory joint multi-object tracking, faster without accuracy loss; users must pull the latest code to use the 3.1 checkpoints.
• Licence: the SAM License (see repo LICENSE), not a standard OSI licence. Checkpoints are on Hugging Face and require requesting access and an access token.
• ComfyUI lists SAM 3/3.1 support (comfyui).
• Re-verified 2026-10-02 against the Meta blog: SAM 3D (open-source 3D object/human reconstruction from one image, powers Facebook Marketplace “View in Room”) and the SAM 3 Agent exist; new effects in Meta’s Edits app were announced as “coming soon” (rollout status not verified).
• Size: 848M parameters; needs Python 3.12+, PyTorch 2.7+, CUDA 12.6+ (GitHub README). No SAM 4 found in searches (2026-10-02).
• SAM 3.1 claim: roughly 7x faster at 128 objects on one H100 versus SAM 3, with mixed per-benchmark accuracy changes (secondary analysis of Meta’s release notes).

Features

• Promptable Concept Segmentation (PCS): segment all instances of a concept specified via short noun-phrase text prompts (open-vocabulary).
• Exemplar prompting: provide one or a few image examples (exemplars) to define the target concept visually.
• Visual prompting (backwards-compatible): points, boxes, and masks (as in earlier SAM versions).
• Built-in open-vocabulary detector + presence head: detects whether a concept is present and localizes all instances without requiring an external detector.
• Video tracking and temporal consistency: re-identify and track concept instances across frames using a tracker + memory encoder.
• Interactive refinement workflows: combine text/exemplar prompts with visual corrections iteratively.
• SAM 3D (companion): single-image 3D reconstruction for objects, scenes and human bodies.
• Large-scale concept dataset and data engine: iterative human+AI annotation workflow scaled to millions of concepts.

Superpowers

SAM 3 is designed to move segmentation from per-object interactive workflows to concept-driven, large-scale automation.

• Who it’s for: researchers and engineers building segmentation, detection, dataset-labeling, video tracking, AR/VR, and media-editing tools; product teams adding intuitive segmentation to user-facing apps.
• What you gain:
  • Open-vocabulary segmentation: ask for “yellow school bus”, “striped cat”, or provide a single exemplar and get all matching instances across an image or video.
  • Faster annotation and dataset creation: automatically find and mask every instance of a concept, reducing manual labeling effort.
  • Unified API for image + video + 3D tasks: a single model checkpoint that can be used for many downstream workflows.
  • Better zero/few-shot generalization: improved performance on rare categories and transfer tasks compared to prior systems.

Pricing

• Weights free to download under the SAM License after requesting access on Hugging Face (see Status update).
• Product integrations: Meta announced SAM 3 effects for its Edits app and uses SAM 3D in Facebook Marketplace; availability varies by product and region.

---

Overview

Released: 2025-11-19 (Meta AI; confirmed by several secondary sources). SAM 3 is the third major iteration of the Segment Anything family and introduces “Promptable Concept Segmentation” (PCS): the ability to accept short noun-phrase or exemplar prompts and return all instances of the described concept in images and videos.

SAM 3 unifies detector, segmenter, and tracker functionality into a single architecture and extends the Segment Anything ecosystem with SAM 3D for single-image 3D reconstruction.

Technical summary

• Architecture highlights:
  • DETR-style detector backbone conditioned on prompts (text/exemplars/visual).
  • Text encoder to process noun-phrase prompts and map them into concept embeddings.
  • Exemplar encoder that converts example images into concept representations.
  • Fusion/conditioning encoder to combine image features with prompt embeddings.
  • Presence head that separates recognition (is the concept present?) from localization (where is it?), improving multi-instance detection.
  • Tracker and memory encoder adapted from prior SAM video work to enable temporal consistency across frames.
• Training & data:
  • Meta developed a scalable data engine combining AI proposal models (including captioning models), SAM variants, and human annotators in an iterative loop.
  • This pipeline enabled scaling to millions of unique concepts (Meta blog: over 4 million unique concepts; the public SA-Co benchmark has about 270K concepts per the GitHub README) while maintaining high-quality masks and text labels.
  • Training stages included perception encoder pre-training, detector pre-training, detector fine-tuning on curated high-quality datasets (SA-Co), and separate tracker training with a frozen backbone.

Benchmarks & performance

• Meta blog: “a 2x gain over existing systems in both image and video” on SA-Co, and users preferred SAM 3 outputs over the strongest baseline about 3 to 1 (vendor claims). The GitHub README says SAM 3 reaches 75-80% of human performance on SA-Co.
• Zero-shot LVIS and interactive point-to-mask numbers are in the SAM 3 paper (arXiv 2511.16719); not re-checked here.
• Latency claims in demos were not verified; measure on your own hardware.

Companion systems

• SAM 3 Agent: a multimodal agent that decomposes complex language queries into prompts SAM 3 can execute; useful for referring-expression understanding and reasoning benchmarks.
• SAM 3D: reconstructs 3D shape/pose from single images, enabling 3D-aware editing and downstream robotics/AR use cases.

Use cases & examples

• Dataset annotation: automatically mask every instance of a class (e.g., “sneakers”) across large image corpora or videos to accelerate training-data creation.
• Video analytics & wildlife monitoring: track and count all instances of species across camera-trap videos or surveillance footage.
• Creative tools & media editing: remove or edit all instances of an object (e.g., “lamp”) in a video sequence.
• Robotics & perception: open-vocabulary object discovery and tracking in robot perception pipelines.
• Human-in-the-loop labeling: use exemplar + text prompts and refine with clicks/masks.

Limitations & considerations

• Open-vocabulary models are sensitive to prompt wording; careful prompt engineering and exemplar selection help improve results.
• Failure modes: ambiguous prompts, heavy occlusion, or extreme domain shift can reduce accuracy; inspect masks and use refinement where necessary.
• Licensing & safety: review Meta’s license for model weights and any usage policies before commercial deployment. Consider privacy and safety implications when applying to sensitive domains.

References & further reading

• Official Meta AI announcement: Segment Anything Model 3 (see Meta AI blog for full paper, code, and demos).

• SAM 3D announcement and examples (Meta AI blog).

• SA-Co benchmark and dataset descriptions included in Meta’s research materials.

Sources

• https://github.com/facebookresearch/sam3 (accessed 2026-09-30)
• https://ai.meta.com/blog/segment-anything-model-3/ (accessed 2026-10-02)
• https://arxiv.org/abs/2511.16719 (paper, via search; accessed 2026-10-02)
• https://github.com/facebookresearch/sam3/blob/main/RELEASE_SAM3p1.md (via search; accessed 2026-10-02)
• https://learnopencv.com/sam-3-whats-new/ and https://blog.roboflow.com/what-is-sam3/ (secondary; accessed 2026-10-02)