ngosangns Knowledge Base

Video Generation

#technology#video-generation#world-models

Updated: 2026-09-24

Source

This note covers AI video tools and one research model, and the two halves are quite different. The first half is a short list of services and open projects for text-to-video, text-based video search, lip sync, portrait animation, and character animation. Entries include Runway, v-express, Emote Portrait Alive, LivePortrait, Wan 2.2 Animate, and OmniHuman-1, each with a line on what it does and a link. The second half explains Meta's V-JEPA 2, a self-supervised world model trained on video. That part walks through its capabilities in world understanding, prediction, and planning for robot control, then the two-stage training architecture and the applications it points toward. The note ends with links to the project site and a related world model note.

Video Generation Tools

  • Text to video:
  • Search videos by text: twelvelabs.io #videoSearch
  • Create videos with AI tools
  • v-express - Voice-driven lip-sync technology: GitHub #lipSync #ComfyUI
  • Emote Portrait Alive: Turns a person's image + audio into a video of that person - Website #emote
  • LivePortrait: Turns a photo of a character into a video with the expressions of another input video - Hugging Face #livePortrait
  • Wan 2.2 Animate: A large AI model with 14 billion parameters using a Mixture-of-Experts architecture, enabling high-quality character video generation from a single reference image. Supports replacing actors in a video while preserving the scene and camera motion, and accurately reproduces facial expressions and body movement - Website #video #animation #character #MoE
  • OmniHuman-1: A multimodal AI framework that generates realistic human video from a single still image and motion signals such as audio or video. Produces video with accurate lip sync and gestures, and supports dynamic full-body video generation. Based on a Diffusion Transformer - Website #video #human #multimodal #diffusion

V-JEPA 2 (Video Joint Embedding Predictive Architecture 2)

Overview

  • V-JEPA 2: A self-supervised foundation world model trained on video.
  • Achieves state-of-the-art results in visual understanding and prediction.
  • Enables zero-shot robot control in new environments.
  • A next step toward a vision of AI that uses world models to:
    • Understand physical reality
    • Predict outcomes
    • Plan effective strategies
    • All with minimal supervision

Key Capabilities

World Understanding

  • Excellent motion understanding.
  • Leading visual reasoning ability when combined with language modeling.

Prediction

  • Can predict how the world will evolve.
  • Sets a new state of the art in anticipating actions from contextual cues.

Planning for Robot Control

  • Builds on its understanding and prediction abilities.
  • Can be used for zero-shot robot planning to interact with unfamiliar objects in new environments.
  • Trained on 62 hours of robot data from the Droid dataset.
  • Deployed on a robot arm in new environments.
  • By specifying tasks as goal images, the model completes tasks such as Reaching, Grasping, and Pick-and-place.
  • Task-agnostic: Can be trained without extensive robot data or task-specific demonstrations.

Model Architecture

  1. Pre-training (Self-supervised learning): The encoder and predictor are pre-trained through self-supervised learning on visual data.
  2. Fine-tuning: Fine-tuned on a small amount of robot data, enabling effective planning without extensive expert robot demonstrations.

Vision and Applications

  • Robotic Assistants: Opens a new era for robotics, handling household chores and complex tasks.
  • Wearable Assistants: Helps people navigate busy environments, warning of obstacles and hazards.

Resources

See also: Image Generation · Content And Multimedia Tools · Code World Model