Triton 47 › Robotics & automation › After the PhD

Robot transformer Vision Language Action development

January 06, 2025 · 1076 words · 5 min read · 1 like on WordPress idea

Index note links 24 prior posts; Open-X, OpenVLA, LeRobot; 35 links

Currently, the model can be implemented to do simple tasks like picking up, grabbing things, wiping, at a cost of poor gripping and noisy control. But the current idea is directly implementable in say kitchen to prepare food, given long list of text commands to execute in steps, while not expecting perfection.

Practical things include robotic arm itself, the requirement of good GPU for local computation of transformer AI model , large storage for dataset training (which already needs PB of data storage space). But for fine tuning, only local data is used, but it can also increase a lot in size. It takes input from an angle, much like we humans take input from a top view angle and perform the tasks through our hands. And the VLM with cross (X) embodiment can allow different structures to execute the same tasks, as current model outputs gripper target position and change in rotation per frame required to execute that task in steps. IK will be used given robotic arm constraints to achieve the motion change required, and will be executed by the mechanical control of the robot, which itself is not a complex thing than its intelligence counterpart, but however, mechanical structures can be expensive for a local student or normal person looking to afford, and data information intelligence is given for free, given that it can be processed, usually can be over cloud as the current models are very slow, for some taking only 3 frames per second to execute the action. These are currently slow (progress at mid2024) because of limited computational speed for these large models and local university setup. However, commercial development would likely to be succeeded by established companies like OpenAI, however, their models are not public and quality development will only be taken as product, focusing more on abstractness, to user, but also consumerism and money, latter of which are not the strong suits of most people.

Development is in scale of how computers have come, the robot counterparts, and mechanical shops would quickly emerge providing gears, cycloidal gears, bioform actuators, motors, and powerhouses to generate electricity more effectively will quickly emerge, along with online update system which instantiates quick upload to robot model and software quality. Only till then, when such models have been bought to same ecosystem such as app development (rate of which is basically same as food generation or any other necessities), hobby part of me would include exploring robots, their structures and intelligence to deploy as hobby scenario. Soon, this will be out the window, as commercially available quality reliable robots will come, and by then the focus would have shifted to other frontier of development, which could be health and human genetics. Then after learning to edit human life forms to cure them and support living structures, next decades would be revolution of cybernetics, meanwhile making current structures (energy generation, battery, computation processing, actuation) more efficient, cheaper, more modular and understandeable, and easy to implement, while making globe more greener sustainable and better place for all, for better life experience. However, era also has to see many other developments, but they matter only to experience of mind, and for those who like to experience them, the only thing matters is the practical reach of it, whether its affordable, or in dedicated setting, of future world. But change also is within the thought process of the minds of the masses, which would support and propel whats to come, while supporting whats exists, and not decaying it down, only making it better.

But to get started on hobby level, things seem still small and rudimentary. On research scale, it does not seem to have impact because they already have reached the researcher consumer scale, but as a researcher, with limited options, budget, and work time, only few things can be practically done.

Around 6 months ago, LeRobot, with SO ARM100 with 6 servos was made available which is an hobby small robotic arm. While it does not have much utility for real-practical tasks, the concepts used can help developers to get into this field. Since I got 3d printers, I find it bit convenient to approach these, but I’d rather would like to focus more on harmonic drive arms and full fledged humanoid or some structure robots that enable generic manipulation which aligns with state of the art development. While its a steeper curve for mind, eventually abstracting it, everything can be made a normal-person job.

Since the BoM for this follower-leader arm pair would be around 300e for me, its quite affordable for a learning toy for an engineer, but also requires work GPU. Even then, industrial or accurate arms are no use in general settings if they can’t have at least human level dexterity and locomotion, but they can have more.

Since the major developments are done by major companies and universities, for a small budget-mentor-skill limited research group, a branch of that research can only be achieved, and I would need to move on to other exciting things.

—

more stuff:
Recent simulator for robotics and physics world simulations: Genesis embodied https://genesis-embodied-ai.github.io/.

#ai #artificial-intelligence #robotics #robots #technology

Original: https://triton7777.wordpress.com/2025/01/06/robot-transformer-vision-language-action-development/

Robot transformer Vision Language Action development

Related in the archive