The shift from chatbots to bots that follow natural language commands goes through one class of models. VLA models — vision, language, and movement models — combine visual perception, language understanding, and motion generation in a single neural network. Their power is real, but it depends almost entirely on the training data they absorb. This guide explains what VLA training data actually contains, what teams underestimate, and how to plan a dataset that produces a publishable model.
Key takeaways
- VLA models directly map vision and language inputs to robot actions in a single network.
- Training data should include concurrent visual feedback, language instructions, and procedures.
- Discrete action codes require extensive explanatory data to learn well.
- Human egocentric video is increasingly being mined as a low-cost source for VLA pre-training.
- Strong feedback loops are just as important as training data for reliable dissemination.
- VLA fine-tuning succeeds or fails in the accuracy of the annotations, not the raw size alone.
What is the VLA model?
A VLA model is a basic robotic model that takes images and natural language instructions as input and output for robot actions. Unlike traditional pipelines that separate perception, planning, and control into different modules, vision, language, and action models learn to map comprehensively into a single network.
FLA model: A neural network takes simultaneous visual feedback and natural language instructions and produces sequences of robot actions or action codes.
This unified design allows VLA models to inherit reasoning abilities from significant prior visual language training and to expand them through motor control. For deployment, this means that a single model can in principle perform many tasks, but only if its training data covers them with the right structure.
What does VLA training data actually contain?
VLA training data contains four basic components of each episode: visual feedback, natural language instructions, a course of action, and a mark of success or failure. Around these, teams add timestamps, sensitivity status, and evaluation flags.
The four mandatory classes:
- Visual notes – RGB frames, often combined with depth views or wrist camera views.
- Language instructions – Brief natural language commands such as “Pour water into the cup.”
- Work paths – Discrete or continuous action sequences set according to the robot’s degrees of freedom.
- Results labels – Explicit success, failure or partial completion marks for each episode.
An open, 7-billion-parameter VLA model was trained on over 1 million episodes derived from 22 robot incarnations (Stanford et al., 2024), demonstrating the expected diversity of generalization across tasks. Without this breadth, VLA models tend to memorize specific objects rather than generalize.
Why is action annotation more difficult than image annotation?
Action annotation is more difficult because events live in continuous, high-dimensional spaces and depend on the robot’s embodiment, not just the frame content. Marking the box surrounding the cup is simple; Marking a path that successfully grips that cup with a specific gripper at a specific point of contact is not.
Action code: A discrete representation of the robot’s motion or end-actor displacement that the VLA model can predict like a language code.
Annotation teams need to align each action code with the concurrent observation, mark moments of communication, capture failure recovery, and mark atomic boundaries of language instructions. He’s old Explanation of data Workflows address this at scale, with structured classifications set to robotic workspaces and acceptance limits for each task.
Where does selfish human video fit into VLA training?
Egocentric human video lends itself to a scalable pre-training resource that fills the gaps that real robot data cannot fill. First-person video footage of humans cooking, picking and assembling captures behaviors on a scale that remote automation will never reach.
A recent paper transformed unstructured selfish human videos into loops in VLA format—1 million clips and 26 million frames—by treating the human hand as a dexterous end-actor (Wu et al., arXiv, 2025). This type of cross-representation data is now routine in VLA pretraining prescriptions.
The catch: raw video is not training data. It needs segmentation, language descriptions, hand pose retargeting, and quality checking before it reaches the VLA pipeline. He’s old Physical artificial intelligence Data operations include selfish capture, Real2sim transformation, and VLA-aligned annotation in a single delivery.
How do you create evaluation sets that capture VLA failure modes?
Evaluation suites detect VLA failure modes when designed before, rather than after, training. Three structures are of greatest importance: success criteria in distribution, realizations of generalization outside of distribution, and risk-related safety scenarios.
Imagine a home VLA model that has been extensively trained in kitchen tasks. A reasonable evaluation battery would test: known tasks in known kitchens (under-distribution), known tasks in unfamiliar lighting (unfamiliar lighting), unknown objects with known instructions (concept generalization), and rare events such as accidental spills (safety level). Without both, deployment risks remain unmeasurable.
A useful neutral resource for organizing risk level coverage is NIST AI Risk Management Frameworkwhich separates levels of influence in a way that is clearly consistent with the design of the evaluation set.
