Ant Group’s Robbyant Unveils LingBot-VA 2.0: A Causal Video Action Model Built for Physical AI

Robbyant, the AI ​​unit embedded within the Ant Group, has released a LingBot-VA 2.0.The first model of the native base. It describes a video action base model for manipulating a general robot. The research team pre-trains the entire stack for simulation instead of fine-tuning the video generator.

What is LingBot-VA 2.0?

Many action video models also use two components designed for digital content creation. Another VAE focused on reconstruction. One is a bidirectional video distribution core, with an attached action module.

This creates three limitations. Pixel-reconstruction latents are still visible but carry limited body structure. The duplication of audio over video tokens is too slow for closed-loop control. Common video purposes never teach how actions reshape the world.

The fourth structural difference. Backbones use bidirectional attention, while control occurs continuously in time. LingBot VA Version 1.0 developed that stack into a causal model. Version 2.0 prepends a native DiT trigger.

Version 1: Semantic Visual-Action Tokenizer

Building on that motivation, the first phase replaces VAE’s for compression only. Following RepWAM, the tokenizer adds two objectives to the rebuild.

Semantic Alignment pulls hidden visible towards the Perception Encoder frozen teacher. A latent action target extracts the joint variable between successive latencies. The inverse dynamics model predicts each hidden action. The forward dynamics model divides it into transport and residual maps.

World regions and actions now share one hidden space. So labelless web video has action-related direction.

Version 2: Causal DiT with Sparse MoE Video Streams

On top of that gap, version 2 trains the causal DiT. Retains the Mixture-of-Transformers design of version 1.0. The video expert and the action expert share one reason for their attention. Each has a different forwarding route.

Both types measure asymmetrically. The video master replaces its dense FFN with a routed MoE layer. That layer holds 128 SwiGLU experts, top-8 route, one shared expert. Load balancing follows a lossless-helpful Balancing strategy. The action expert keeps the FFN dense in the hidden 768.

The video core has about 13.0B parameters, about 1.9B active. For action experts and MCP heads, the training includes about 15.3B parameters. About 2.5B activates each token in consideration. Training uses the optimized flow objective with the hybrid Muon plus AdamW optimizer.

Where the Coaching Sign Comes From

Beyond architecture, two goals shape what the model learns.

Multi-chunk prediction (MCP) corrects myopic surveillance. The teacher constraint controls only the next part, so that the model minimizes the loss by copying the appearance. The MCP attaches three lightweight modules that predict the next three components. At release it matched the base accuracy of 45k steps to 20k steps, 2.3x training speed.

Meanwhile, five missions are trained together instead of being staged: T2I, T2V, TI2VA, ICL, and human-robot cooperative training. Sampling follows a coarse-to-fine order, from the look to the control of the video’s action. Keeping every purpose alive avoids forgetting the previous ones.

Hierarchical planning

Chunk level control cannot be followed by long horizon goals. Above the policy therefore sits the VLM planner, LoRA configured with a frozen tower. Outputs structured JSON: generated, instruction, generation_instruction, local_scene_description. It runs at about 2 Hz behind an asynchronous shared buffer. The policy is read at each chunk boundary, so scheduler delays do not block performance.

Consultation in advance

Although there is a small backbone, the use attacks the serial bottle. When the robot is waiting, the model delay becomes the control delay.

So Pre-Consultation uses prediction and execution as an asynchronous stream. While the robot executes the chunk a_t, the video technician imagines its result. The action expert determines a_{t+1} from that.

Running ahead is dangerous to drift. So each return view is coded z_{t+1}, clearing the guess. The forward-dynamics base loss trains the video specialist for this role.

# Pseudocode for the asynchronous rollout (Sec. 2.3.7, Eq. 29).
# Not runnable: policy, executor and encode() are placeholders.

C = init_kv_cache(encode(obs_0))            # feedback-grounded cache C_t
a = policy.action_expert(C)                 # cold start: first action chunk a_0

while not done:
    executor.start(a)                       # execution stream, non-blocking

    C_tmp  = C + [a]                        # prediction stream: C_t u {a_t}
    z_hat  = policy.video_expert(C_tmp)     # forward dynamics -> imagined z_{t+1}
    a_next = policy.action_expert(C_tmp + [z_hat])

    obs = executor.wait()                   # real observation of a_t returns
    C   = overwrite(C_tmp, z_hat, encode(obs))   # re-ground: z_hat <- true z_{t+1}
    a   = a_next

Working

Therefore, testing includes simulation and real hardware. In RoboTwin 2.0, every model is trained with 2,500 clean and 25,000 random samples, for all 50 tasks.

Distillation reduces the video sample from 5 steps to 2, and the action sample from 10 to 2. FP8 TensorRT engines, paged/jammed KV cache with FlashInfer support, and side-by-side host head removal provide the rest.

# Reproduces Table 3 of the report exactly. Runnable as-is.
K = 32  # low-level control steps inside one generated chunk

stack = [("BF16 PyTorch async rollout baseline", 927),
         ("+ Consistency distillation",          466),
         ("+ Low-precision compiled execution",  369),
         ("+ Long-horizon attention optimization", 272),
         ("+ Runtime overhead reduction",        142)]

for name, ms in stack:
    print(f"{name:40s} {ms:4d} ms  {round(1000 / ms * K):4d} Hz")

print("end-to-end speedup:", round(927 / 142, 1), "x")

Version 1.0 vs Version 2.0

The tokenizer ablation splits the first row. Changing the WAN2.2 VAE semantic tokenizer raises the 1.3B model from 78.0 to 86.6.

Use Cases and Examples

Beyond the benchmarks, four use cases stand out.

• Riding a few shots: The report says that the model adapts from 10 to 15 displays. Real-world testing uses 20 mobile demos for each function. The multi-task test area covers all four tested tasks.
• A control with display mode: In-context learning allows video of a person’s demonstration to replace text instruction. After the correction of the four observed functions, the policy made an invisible innovation. One example: “put the calabash on the green plate.”
• Measuring cheap data: Human-robot cooperative training hand puts into place robotic action. Each hand becomes a virtual parallel gripper. The egocentric corpus includes 65.4k episodes.
• Active control: Shows include Air Hockey and a conveyor belt, where the policy is to expect moving objects.

Key Takeaways

• It pre-trains DiT for the causal video action from scratch instead of customizing the video generator.
• A semantic token puts world states and implicit actions into one coherent space.
• Sparse MoE video streaming: ~2.5B of 15.3B parameters per token.
• Foresight Reasoning goes beyond prediction and action, based again on all actual observations.
• Chunk latency 927 ms to 142 ms; async control 35 Hz to 225 Hz.

Interactive Dynamic Explainer

Check it out Paper again Project Page. Also, feel free to follow us Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to Our newspaper. Wait! are you on telegram? now you can join us on telegram too.

Need to work with us on developing your GitHub Repo OR Hug Face Page OR Product Release OR Webinar etc.? Connect with us

_{Note: Thanks to the Ant Research team for the thought leadership/Resources for this article. Ant Research team supported this content/article for promotion.}