Enable Open-Ended Robotic Manipulation Through Real-Time Speech Commands
Issue open-ended robotic commands using speech-to-action models for instant execution.
Direct robotic manipulation systems via open-ended speech commands by pairing observation models (like Qwen-Omni) with manipulation models (like Qwen-RobotManip).
The Scenario
A warehouse worker needs to direct a robotic arm to perform a non-standard handling task quickly using voice commands.
Before & after
Robotic tasks were limited to a pre-defined list of scripts. Testing a new manipulation task required updating the code and re-deploying the model, taking 30–60 minutes.
Using Qwen-Omni in tandem with RobotManip, you can use speech to randomly propose tasks and let the AI judge and execute them in real-time. The interaction takes seconds per task.
The Prompt
[AS_SPEECH_INPUT] I need you to pick up the red block, rotate it 90 degrees, and place it inside the open container. Judge your own execution for success.
The combination of Qwen-Omni and Qwen-RobotManip allows for open-ended instruction following and real-time execution judgment, enabling robots to handle tasks not found on a pre-defined list.
Source
Qwen"Qwen-Omni observes the scene, randomly proposes manipulation tasks via speech, and judges execution in real time. Each video shows Qwen-RobotManip completing tasks..."