ThirdBrAIn.tech

Advanced Video Search

Avatar Lip Sync with Remotion

5 min read

Explore related topics

Video topicsVideotopicsRemotion - Programmatic Video Framework with ReactRemotion -ProgrammaticVideo…Avatar LipSync withRemotion

Avatar Lip Sync with Remotion

Partly verified 2026-10-02

Verified: MuseTalk v1.5 (2025-03-28, MIT code and usable weights, 30fps+ on V100; inference tested on a 4GB RTX 3050 Ti in fp16, slow), Rhubarb mouth shapes (6 basic A-F + 3 extended G, H, X), TalkingHead.js (MIT; built-in English, German, French, Brazilian Portuguese, Finnish, Lithuanian; extendable via Azure TTS), @sage-rsc/talking-head-avatar 1.0.3 (MIT, last published 2025-10-25), @remotion/three exists (also <ThreeWebGPUCanvas/>). HeyGen now lists Avatar V, IV and III, Photo Avatar, Video Agent, and 175+ language translation. Not verified: HeyGen per-minute prices and limits (pricing page requires the app; docs moved to developers.heygen.com), SadTalker/Wav2Lip/LivePortrait/VideoReTalking/V-Express speed and VRAM figures.
Ready Player Me is gone: it was acquired by Netflix (Dec 2025) and shut its public services on 2026-01-31 (Wikipedia). Use custom or Mixamo-rigged GLB avatars (for example Avaturn, which TalkingHead lists as compatible) instead.

Remotion does not have built-in lip sync or avatar features. It’s a React video framework for programmatic composition. You combine it with external tools for the actual lip sync generation.

Decision Matrix

GoalApproachCostEffort
Fastest to production (opinion)HeyGen API → RemotionPaid (billing not verified)Low
Free open-source optionMuseTalk v1.5 (Python) → RemotionFree (GPU needed)Medium
Full creative controlRhubarb + custom GLB avatar + @remotion/threeFreeHigh
Existing footage dubbingWav2Lip or VideoReTalking → RemotionOpen-source; licence terms not verified (GPU needed)Medium
Interactive web app (not video export)@sage-rsc/talking-head-avatarFreeMedium

Approach 1: HeyGen API (Turnkey)

Call HeyGen’s API with a script + avatar, get back an MP4 with lip sync baked in, composite in Remotion.

Endpoint: POST /v2/video/generate with avatar_id, voice_id, script text.

Capabilities (March 2026 snapshot, not verified; check developers.heygen.com):

  • Stock avatars, custom photo avatars and Digital Twins (counts and tiers not verified)
  • Transparent WebM output for compositing (not verified)
  • Lip sync quality per language: not verified

Tiers (not verified; check HeyGen’s API pricing in the app): Avatar III, Avatar IV, Video Agent (prompt-based) and a TTS-only option are listed on developers.heygen.com; billing details are not verified.

Limits: not verified (an earlier draft listed 30 min, 50 scenes, 5000 characters and 1080p; removed as unconfirmed).

Remotion integration: generate the video via the API, then import it as a <Video> component. (An earlier draft cited a HeyGen skills install command for Claude Code; it is unverified and removed.)

Alternatives (unverified 2026 status): Hedra, D-ID, Synthesia.

Approach 2: DIY with Rhubarb + Remotion (Full Control)

Pipeline: Audio → Visemes → Avatar animation → Remotion render.

Step 1 — Extract Visemes

rhubarb -o output.json -f json input.wav

Outputs JSON mouthCues with timestamp + mouth shape (A-F basic, G-H-X extended = 9 shapes). A community WASM port (rhubarb-lip-sync-wasm) was listed earlier; not confirmed on the Rhubarb repo (unverified).

Alternative: WhisperX for word-level timestamps → phoneme-to-viseme mapping via g2p libraries.

Step 2 — Choose Avatar Type

TypeFormatProsCons
2D Sprite sheet9 mouth PNGsSimple, fastStatic, less natural
3D avatar (Ready Player Me shut down 2026-01-31; use Avaturn or similar)GLB with Oculus OVR visemes (15 shapes)Smooth, naturalMore setup
3D CustomMixamo-rigged GLB with ARKit/Oculus blend shapesFull controlRigging skills needed

Oculus OVR Viseme Standard (15 shapes): aa, E, I, O, U, PP, SS, TH, CH, FF, kk, nn, RR, DD, sil.

Step 3 — Animate in Remotion

const { fps } = useVideoConfig();  
const frame = useCurrentFrame();  
const timeMs = (frame / fps) * 1000;  
const cue = mouthCues.find(c => timeMs >= c.start && timeMs < c.end);  
  
// 2D: swap sprite  
return <Img src={`/mouth-${cue.value}.png`} />;  
  
// 3D: use @remotion/three with <ThreeCanvas>  
// mesh.morphTargetInfluences[visemeIndex] = 1.0;

For 3D: @remotion/three wraps React Three Fiber and syncs with useCurrentFrame().

Approach 3: AI Talking Head Models (Image-to-Video)

Feed a single photo + audio → get realistic talking video → composite in Remotion.

ModelQualitySpeedGPU VRAMLicenseBest For
MuseTalk v1.5not verified30+ FPS on V100 (per repo)runs on 4 GB fp16, slowly (tested per repo)MITOpen-source option
SadTalkernot verifiednot verifiednot verifiednot verifiedStylized talking heads (opinion)
Wav2Lipnot verifiednot verifiednot verifiednot verifiedDubbing existing footage (opinion)
LivePortraitnot verifiednot verifiednot verifiednot verifiedPortrait animation (opinion)
VideoReTalkingnot verifiednot verifiednot verifiednot verifiedEditing existing videos (opinion)
V-Expressnot verifiednot verifiednot verifiednot verifiedSelf-hosted option (opinion)

Pick (author’s opinion): MuseTalk v1.5, because it is MIT licensed, reports 30+ FPS on a V100 and runs on a 4 GB GPU in fp16 (per its repo).

Remotion integration:

<Video src={staticFile("avatar-output.mp4")} />

Generate with Python, import the result.

Approach 4: React Component Packages

@sage-rsc/talking-head-avatar (npm)

  • Viseme-based lip sync via audio worklet processing
  • Supports ARKit + Oculus OVR blend shapes
  • TTS: Azure, Google, ElevenLabs, Edge TTS, browser native
  • GLTF/GLB with Mixamo-compatible rigs
  • TypeScript, v1.0.3 (last published 2025-10-25 per the npm registry)

Remotion compatibility (author’s assessment, not tested): likely limited, since it uses real-time audio worklets that may not serialize for frame-by-frame rendering. Possible workaround: pre-render as video, import into Remotion.

TalkingHead.js

  • Open-source JavaScript class for real-time 3D avatar lip sync
  • Needs 15 Oculus viseme blend shapes (plus ARKit); Avaturn avatars compatible
  • 6 languages built in, extensible via Azure TTS
  • Streaming API (low-latency claim not verified)
  • Better suited for interactive web apps than Remotion video export (opinion)

Key Packages & Tools

PackagePurpose
@remotion/threeThree.js integration for Remotion (3D avatars)
@remotion/captionsCaption utilities (not verified in this note)
rhubarb-lip-syncCLI for audio → viseme extraction
rhubarb-lip-sync-wasmBrowser WASM port of Rhubarb (community, unverified)
whisperxWord-level timestamps from audio
@sage-rsc/talking-head-avatarReact 3D avatar with TTS + lip sync

Open items

  • HeyGen per-minute billing, limits and Remotion skills install command: unverified (pricing behind app login).
  • Speed/VRAM/licence columns for SadTalker, Wav2Lip, LivePortrait, VideoReTalking, V-Express: not re-checked (web search budget exhausted this session; only repo fetches possible).
  • Missing from the table: newer 2026 talking-head or avatar models were not surveyed.

Sources

Filter Videos

Tags

Advanced Tag Filters

Channels

Favorites

Active list: Quick Saves

This list is empty

Browse videos and save them to a list to get started

Explore related topics

Video topicsVideotopicsRemotion - Programmatic Video Framework with ReactRemotion -ProgrammaticVideo…Avatar LipSync withRemotion

Backlinks