I Gave My OpenClaw Agent a Physical Body

I recently gave my OpenClaw a real robot arm to play with. The results just about blew my own neural network.

The AI agent was able to configure the arm, use it to see and slowly grab things, and even train another AI model to pick up and place specific objects. And they say AGI is still a few years away! (I’m joking, it probably is).

The results have me convinced that we may be on the brink of a robotics breakthrough. Training and controlling robots used to require considerable skill. Today’s AI models can make it almost easy.

“AI-powered coding is super exciting because it has the potential to bridge the gap between conventional engineering methods, which are reliable but don’t generalize, and contemporary vision-language-action models, which generalize but are not yet reliable,” says Ken Goldberg, a roboticist at UC Berkeley who is exploring the approach.

I told OpenClaw to try moving its new arm and it came up with this little wave.

I bought a prebuilt arm called a LeRobot 101. It’s part of an open-source project from HuggingFace that makes it relatively cheap to start building and experimenting with robotics.

The LeRobot comes with two arms: a controller arm that a person operates using a handle and a trigger, and a follower arm with a camera that replicates those movements. You can train an AI model by teleoperating the controller arm and having the model learn how to move the follower in response to what it sees on the camera.

Building With OpenClaw

Before using OpenClaw, I spent several hours trying to connect and calibrate the robot, at one point nearly breaking the motors by applying the wrong settings, which caused them to overheat.

Then, with help from OpenClaw and Codex, I was able to vibe code a simple program that closed the claw’s gripper when it spotted a red ball. In the terminal, Codex went through the tricky work of configuring the connections to the robot. Then, with my help, it calibrated the positions of its joints. It also wrote a Python script that used several libraries to identify and grip the ball in question. Vibe-coding isn’t perfect of course, and hallucinations can introduce bugs especially when working with different hardware, but the results were impressive.

Then, with my help, the robot-agent figured out how to identify and grip a red ball.