From Seeing to Doing: VLMs and VLA Models

Suppose I ask a robot to put the red mug next to the kettle. It has to find both objects, figure out where the mug is, move its gripper to it, pick it up, and set it down without hitting anything. A model that can describe the scene has only done the first part of the job.

That’s the difference I want to unpack here. Vision-language models (VLMs) connect images with words. Vision-language-action (VLA) models use visual and language input to choose actions for a robot. The names are similar, but making a prediction about an image and moving something in the real world are very different tasks.

What does a VLM do?

A VLM is trained to connect visual input with language. It may answer questions about an image, produce a caption, locate a named object, or compare two scenes. A common design combines an image encoder with a language model. A projection or connector maps visual features into a form the language model can use alongside text tokens.

There isn’t one standard VLM design. Some take one image; others split a high-resolution image into crops or look at several video frames. Many answer in text, while some can also return things like object coordinates or masks.

The useful question is: what does the model output? A VLM usually gives you information about the image. It might say where the mug is or describe how to pick it up. Neither answer tells the arm which direction to move.

From describing to acting

A robot policy must map observations and a task instruction to actions that its particular body can execute. The observation might include camera images, joint positions, gripper state, or force readings. The action might specify target positions, velocities, gripper commands, or a short chunk of controls.

VLA models bring vision, language, and action into a common learned policy. In the RT-2 work, for example, robot actions are represented as tokens so a model can be trained on both web-scale vision-language examples and robot demonstrations. Other systems use separate action heads or continuous action representations. The design choice matters: the action format needs to fit the robot, the task, and the control frequency.

The robot shouldn’t plan once and then blindly play back a long sequence. It moves a little, looks again, and adjusts. A long action chunk means fewer model calls, but also more time before it can correct a bad move. Short chunks make it easier to react, though they require more frequent inference.

Same ingredients, different job

A VLM can still be useful in a robot. For example, one component might ask it to identify the mug, then give that result to a conventional motion planner. That system uses a VLM, but the VLM itself isn’t controlling the robot. A VLA’s job is to produce actions from the robot’s observations and the instruction.

Where do the actions come from?

Usually, from demonstrations. Someone operates the robot, and the training data records what it saw, what task it was doing, and how it moved. The model is trained on many of these trajectories. The more variety in tasks, objects, viewpoints, and robots, the better chance it has of handling a situation beyond the exact demonstrations.

Images and captions from the web can teach a model that a whisk is a kitchen tool, but they don’t tell it how far to move a particular arm to pick one up. Robot demonstrations supply that connection. Some VLA training mixes both kinds of data; RT-2 is a well-known example.

Actions can be encoded in several ways:

• Discrete action tokens: quantize dimensions such as position, rotation, or gripper state into bins, then predict token sequences. This fits naturally into an autoregressive language model, though quantization and token-by-token generation can be limiting.
• Continuous action heads: predict real-valued controls directly, often with a specialized policy head.
• Action chunks or trajectories: generate a short sequence of future controls together. This can make execution smoother and amortize inference, but errors may persist until the next replanning step.
• Generative action policies: model a distribution over plausible motions rather than one average action, useful when several valid ways to complete a task exist.

The choice depends on the job. A robot stacking blocks slowly has different timing needs from one handling objects with both hands. There’s no single action format that fits all of them.

Why people are interested in them

Language can specify the goal flexibly

Language is a convenient way to describe a goal. A model trained on many objects and instructions might handle a new combination without needing a separate task ID or a new model for every object. “Might” is doing some work here: it depends on the training data and still has to be checked on the robot.

Shared policies can reuse experience

The other appeal is sharing experience between tasks. If the model has learned to grasp handles on several objects, that skill may carry over to a new container or tool. That’s the hope behind training one general policy instead of a separate policy for every task.

They connect semantic decisions to motor output

The catch is that a single learned policy can be harder to inspect than a system with explicit perception, planning, and control modules. And the VLA label doesn’t guarantee generalization. It depends on the data, the robot, and what was actually tested.

What makes this difficult in practice?

Robot data is expensive and uneven

Images and text are easy to find online. Robot trajectories are not. Each one depends on a particular robot, camera setup, workspace, and controller. A dataset can list lots of different tasks and still show the model the same few objects, viewpoints, and motion styles over and over.

The model also has to connect its movement to what happens next. A camera may not show depth or hidden geometry; it may be hard to tell whether a grasp succeeded until the robot tries to lift the object. A model can learn to imitate demonstrations without learning a rule that holds up in a new situation.

Embodiment matters

An action only makes sense for a particular robot. One arm may reach a spot that another can’t; grippers, calibration, and joint limits differ too. Getting a policy to work across robots means accounting for those differences, not just reusing the same command.

Good language answers do not prove good control

It might correctly answer that the cup is left of the plate and still miss the cup. Picking it up takes accurate motion, good timing, feedback, and some way to recover if the first grasp slips. A strong score on image questions doesn’t tell us whether a robot can do that safely.

Closed-loop execution has real constraints

Timing matters. If the model takes too long to respond, the scene may have changed by the time the robot acts. A grasp can slip, an object can move, or a person can enter the workspace. A real system needs collision checks, monitoring, an emergency stop, and a way to stop or hand off a task when it’s uncertain.

Evaluation can hide fragility

One success rate can hide a lot. Maybe the robot works in bright light but fails near a window, or handles cups but not bowls. Tests should include new scenes and objects, repeated attempts, small disturbances, recovery, and response time—not only the examples used in a demo.

Does the VLA have to do everything?

No. A VLA can suggest the next action while other parts of the robot stack check whether that action is safe and execute it. State estimation, joint limits, collision checking, and low-level motor control don’t disappear just because a learned model is involved.

camera + robot state + instruction
                │
                ▼
       VLM / VLA policy ───────► proposed action
                │                      │
       confidence / status       safety and limit checks
                                       │
                                       ▼
                              low-level controller
                                       │
                                       ▼
                                    robot
                                       │
                          fresh sensor observation ──┐
                                                     └──► replan

This split also makes failures easier to track: did the model suggest a bad move, did the controller fail to execute it, or did the sensor miss something? A safety layer won’t fix a poor policy, but it can reject commands that violate basic limits.

For a slow pick-and-place demo, a larger model and frequent replanning may be fine. For a fast task involving contact, latency matters much more, and a dedicated controller may need to handle the quick corrections.

The short version

The quick distinction: a VLM tells you something about what it sees; a VLA uses what it sees, along with an instruction, to choose what the robot should do next.

VLM: image + words ──► grounded information
VLA: observation + goal ──► action ──► new observation ──► next action

That last step is a big one. The model has to work with a particular body, learn from robot data, respond on time, and cope with a world that doesn’t hold still. That’s why a VLA is more than a VLM with a robot arm attached.

References

• Brohan et al., “RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control”, 2023.
• Kim et al., “OpenVLA: An Open-Source Vision-Language-Action Model”, 2024.
• Kawaharazuka et al., “Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications”, 2025.

The diagrams are simplified illustrations of common patterns; individual models differ.