Recently, there have been some interesting new results showing that some of the latest large-scale VLMs (Vision-Language Models) can directly control robots to solve a variety of real-world tasks. This has led to some claims and excitement that progress in robotics might happen as an emergent effect of scaling current multi-modal models instead of training different robotics-specific models. As someone who’s been doing research in the field for a few years now, I decided to dive into trying to understand these results and sort through the various potential implications and future directions.

What are the new results?

One widely-circulated result that also has a clear description of the testing protocol is from Robocurve. Researchers asked a few recent frontier VLMs (Claude Fable 5, Claude Fable 5.1, GPT-6 Astra) to perform some simple tasks (placing a block into a bowl, placing a puzzle piece into a groove) by controlling some YAM arms. The model takes the task description and camera images (and a history of previous images and actions if available) and outputs end-effector positions and orientation for each end-effector¹. Another related result is from the RoboDojo team², who ran GPT-6 Astra and GPT-5.5 through much the same kind of interface on their official simulation benchmark: 42 tasks, 50 episodes each, ranked against the 40 policies on their public leaderboard. Here, Astra placed first, ahead of every learned policy on the board. In both cases there is no other learned policy anywhere between the model and the robot.

The full reported results are worth looking at because they yield at least two useful conclusions:

The first thing is the trend line across model generations on the easy task(s): (e.g. 5% → 40% → 95% from Robocurve’s results), over models released within about a year of each other, none of which was built with these specific robot arms in mind. RoboDojo’s results show that GPT-5.5 and Astra scored 1.13 and 28.97 respectively on the same set of tasks with the same harness, which suggests the improvement is directly due to capability improvements in the underlying model. The second thing is that this trend does not seem to manifest on the harder task, where the newest and by far the strongest model does no better than one two generations older. In Robocurve’s testing of a contact-rich puzzle-piece insertion task, both models get the puzzle piece to the groove and then stall at the insertion — the part that requires reacting to contact rather than reasoning about geometry. RoboDojo’s per-axis scores show the same split, and show it more starkly: Astra places first overall only by dominating the generalization and open-instruction axes, and is beaten on precision (12.65 against 28.25) and long-horizon execution (21.45 against 44.12) by policies it otherwise outranks. Its success rate on the precision tasks is 4%.⁶

There have been a handful of more anecdotal results. One X user found that GPT-6 Astra is able to solve a range of tasks much more quickly (in terms of number of turns) and successfully than other recent frontier VLMs. However, they allowed the model to write functions and use helper tools (e.g. SAM for perception) instead of directly having it output end-effector commands. A number of other users demonstrated Astra performing a variety of other impressive real-world tasks (e.g. painting the golden-gate bridge, setting up a MuJoCo simulation and drawing a dove inside it, training a dexterous pen-spinning behavior or turning the knob on a real-world washing machine). However, these were largely demonstrations lacking thorough empirical results, comparisons to baselines, or clarity on the robot’s exact I/O specification or prompting setup.

It is important for context to note that this is not the first time that VLMs or LLMs have been used to directly control robots. There are a number of research works dating back to at least 2022 (SayCan⁷, Code as Policies⁸, ProgPrompt⁹, Inner Monologue¹⁰) that explore this idea, and it remains an extremely active area of investigation (e.g. this¹¹, or this¹²). Anthropic has even studied their latest models’ capabilities on robotics tasks for the past year or so (Project Fetch¹³, and more recently an evaluation across quadrupeds, humanoids, and arms¹⁴). What is noteworthy about these new results though is that the VLM is controlling the robot at a fairly low-level (end-effector targets) instead of via an intricate harness or a number of purpose-built tools, and that performance seems to have improved somewhat dramatically with recent improved model releases.

Why are these results exciting?

¹⁵ Recent prevailing wisdom in robotics has been that robotics fundamentally requires capabilities that VLMs and LLMs cannot possess because of their structure, training data¹⁶, and/or training objectives. The vast majority of recent state-of-the-art robotics demos (e.g. this, or this¹⁷, or this¹⁸) have come from training models on robotics data that are specialized to robotics. However, these new results and demos referenced above illustrate that VLMs might possess sparks of embodied intelligence and robotic control despite not being explicitly trained for these things. Moreover, it seems that newer generations of models are increasingly capable at these tasks. Given this, perhaps scaling up existing models and training techniques will lead to models that can do ever more complex robotics tasks. Thus, perhaps progress in robotics could happen as a by-product of large-scale foundation model training instead of requiring purpose-built robotics-specific models.

This is quite significant if true, and could have several important implications for current directions in the field.

1. Perhaps very little robotics data will be required. Many recent results have been built atop extremely large-scale, often proprietary, data-collection efforts, and there is significant interest and effort being expended to obtain the largest useful robotics dataset (RealOmni-Open¹⁹, Index²⁰). However, if VLMs can learn to do well on robotics tasks from being trained (largely) on internet text, image, and coding data, then perhaps we don’t need much robotics data and these data collection efforts are unnecessary.
2. Transferring across embodiments might not be hard. A frequent challenge for robotics models has been having the same model control a variety of different robot hardware form-factors. There is significant effort (e.g. described here²¹, or here²²) required to train models to exhibit cross-embodiment generalization. If VLMs are able to solve robotics problems by writing programs or outputting controls specific to the problem and environment, then perhaps cross-embodiment generalization emerges without any significant or explicit attempts to train for it.
3. Harnesses and tools might be invaluable. A significant enabler for VLMs to be useful at coding has been good tooling and harnesses for developers (e.g. Claude Code). Indeed, research in leveraging such models for robotics (which has been ongoing for a number of years) has had similar findings (Code as Policies, CaP-X¹², Waddle²³): building the right tools and abstractions around the model might determine success or failure for end-to-end robot behavior.

A related thought experiment

In some sense, building a general-purpose robot by training a model to be extremely good at programming and physical reasoning is not an altogether surprising strategy. We’ve known for a long time that it’s possible to program any specific robot to do any specific task. Assuming the environment is roughly static, and given enough time to think through the specific motions and measure out the relevant distances, a skilled robot programmer or automation integrator can write out a program that performs that specific task in that specific environment. This is, in fact, most of what industrial robotics is: the integration, programming and commissioning work wrapped around an arm routinely costs as much as the arm itself or more, and it has to be redone for every new robot and environment. The trouble has been generalizing the program to produce the correct motion in new tasks and environments.

Imagine for a moment a future where a general-purpose robot is widely and easily available. From any customer’s point of view, they can simply buy a robot, bring it to their environment of choice (e.g. a home, cafe, factory, etc.) and just ask it to start doing useful tasks. However, unbeknownst to the customer, each robot has a little (and invisible) elf that operates it. Whenever a human asks the robot to do something, the elf quickly runs around with a measuring tape and then writes out the precise program that generates the motions that accomplish that specific task in that environment.

This is an admittedly contrived setup, but I believe it is a useful lens for thinking about a particular approach to building such a general-purpose robot. Instead of trying to mimic/automate human brains (i.e., training a robotics-specific model) by having a model go directly from pixels to torques, we have a model that automates the programmer of robots, which is able to achieve tasks in a fully general way by writing hyper-specific programs for each task and environment it encounters.

What’s missing?

While current results and what they promise are certainly exciting, there are several important and substantial hurdles to be cleared before it is clearly practical to build and deply general-purpose robots by scaling VLMs.

Thorough experimentation and (strong) evidence of generalization

The bulk of current evidence is either a demonstration with no quantitative results, or a set of preliminary quantitative results that lacks scale and statistical rigor (i.e., hundreds or more trials with results over several random seeeds). Moreover, many of them do not directly compare against established baselines from the research literature: Code as Policies or more recent improvements¹² run on the same frontier models, recent VLAs like π0.7²⁴, MolmoAct2²⁵ or GR00T N1.5²⁶, and recent WAMs like DreamZero²⁷. These are currently far from the type of thorough result that could be published in a robotics research paper that would be accepted at a top conference or journal.

Additionally, a significant part of the excitement around these results comes from the idea that the underlying models were trained on a very small amount (if any) of robotics data. If in fact they were trained on large quantities of relevant robotics data (perhaps even on the tested embodiments) ²⁸, then these results do not really demonstrate that physical commonsense is coming as an emergent phenomenon of large-scale training on non-robotics data. Even if they were not trained on robotics data, it is not guaranteed that performance on robotics tasks will continue to scale with training in any meaningful way (i.e., the scaling laws²⁹ of these models on robotics tasks are entirely unclear). Thorough experimentation and some insight into the training data for these latest frontier models would help determine the extent to which there is substantial evidence of physical commonsense³⁰ emerging from scaling these models.

Task complexity

Current results demonstrate behavior on relatively simple, often short-horizon tasks with limited contact and physical interaction. However, a significant challenge in robotics is solving long-horizon, contact-rich tasks that require significant dexterity — folding laundry out of a dryer³¹, assembling and packing deformable goods, making a bed or a coffee end-to-end, or cracking + beating eggs and using them to make an omelette. There is, as of yet, limited evidence that VLMs will be able to solve such tasks on robots directly³². Indeed, VLMs - even augmented with tools and harnesses - might be simply incapable of solving certain tasks. In the above-discussed thought experiment of the elf, it is possible that even an extremely competent elf would be unable to write the correct program without precisely measuring distances involved, or without a sense of touch, which could be impossible for a VLM operating just from robot cameras without any additional sensors. However, several research works have demonstrated that such models are able to leverage tools or ML (e.g. Eureka³³, DrEureka³⁴) to perform dexterous behavior, so perhaps this could be a feasible path forward.

Reliability and safety on real hardware

There is much evidence that current models are not particularly safe or reliable at executing useful behavior on real robots. Robodojo’s results show that Astra solves their tasks with an average success rate of 28.97%, which is far from reliable completion. Moroever, the RoboDojo team had to hald their real-robot campaign for safety after Astra repeatedly issued physically unreasonable or unsafe actions, including incidents that damaged hardware. Robocurve found that several recent VLMs will execute harmful and dangerous tasks on real hardware without refusal. Model alignment remains a challenging problems even for disembodied VLMs: ensuring models will safely execute actions on hardware could be even more challenging.

Speed and cost in deployment

One issue noted as part of all the recent results is speed. Frontier VLMs take on the order of seconds (at best) to produce a turn of output. However, some tasks - such as walking even a quadruped robot - require commands to be output at a much higher frequency (e.g. 50 Hz³⁵). This control frequency is not feasible for modern large frontier VLMs ³⁶, and it is unclear that it will ever be. A related issue is the current paradigm for querying large models: it may not be feasible for every robot to assume constant connection to a large centralized model server simply because wifi might have too much latency or too little bandwidth to be reliable.

Related to the issue of speed is that of cost. Each of the trials in Robocurve’s results cost between $0.94 and $2.69 in API calls for a single pick-and-place. If frequency issues are resolved, then robots might be querying models at 50 Hz for commands, and any useful physical task might involve tens of millions of queries. At current rates, the end-user cost of doing this is likely infeasible.

It is worth noting though that inference speeds and costs for VLMs are improving. Robocurve notes that “LLM token output speed increases by 2-7x per year”, and projects that VLMs could potentially control robots at the required control frequency by the end of this year, or by 2029. Costs are falling on a similar kind of curve³⁷. Both of these concerns could additionally be addressed by distilling large models into smaller ones that can be run locally. ßHowever, it is unclear whether these trends will continue to hold in the coming months and years.

Conclusion

Recent results and demonstrations show that frontier VLMs are surprisingly capable at solving tasks on robot hardware despite seemingly not having been trained to do so. If these results and demonstrations are early sparks of physical intelligence emerging from large-scale foundation-model training, then progress in robotics might happen in a different and perhaps faster way than is currently projected. However, there is a lot of additional work to be done before this path of scaling VLMs can feasibly be used to power and ship general-purpose robots.

While this post has generally contrasted scaling VLMs for robotics against training and developing robotics-specific models, there is no reason why these approaches cannot be combined. Indeed, several recent results demonstrate that combining frontier VLMs with VLAs outperforms either approach individually. Indeed, several robotics researchers argue that arming these models with models trained specifically for robots, as well as tools and techniques from classical robotics, is a promising path forward. ³⁸

Thanks to Ryan Hoque, Zachary Siegel, and Lucas Manuelli for helpful comments on draft versions of this post, and to Chao Chen, Jonathan Tompson, and Sangbae Kim for helpful discussion on the ideas it contains.

Citation

If you found this post useful in your own work or writing, please feel free to cite it. You can use the BibTeX below, or just link to the post directly.

@misc{kumar2026vlmagents,
  title        = {Tackling Robotics with (V)LM Agents},
  author       = {Kumar, Nishanth},
  year         = {2026},
  month        = {September},
  howpublished = {Blog post},
  url          = {https://nishanthjkumar.com/blog/2026/Tackling-Robotics-with-VLM-Agents/}
}