Automate Physical Task Planning Using Omni-Vision Models
Combine vision-language models with manipulation models for open-ended robotics.
Use Qwen-Omni to observe scenes and propose tasks dynamically for robotic models, bypassing the need for pre-defined instruction lists.
The Scenario
You are testing a robotic arm's versatility in a kitchen setting and want it to perform tasks like picking up a spoon or opening a lid without pre-programming each specific motion.
Before & after
Defining new robotic manipulation tasks requires manually scripting a list of steps and hard-coding success criteria, which takes 2–4 hours per task.
Using Qwen-Omni combined with RobotManip, you can describe a task in natural language and the model will propose and judge the physical execution in roughly 1–3 minutes.
The Prompt
Act as an orchestrator for Qwen-RobotManip. Analyze the following scene: [INSERT_IMAGE_DESCRIPTION_OR_SCENE_DATA]. Propose 3 manipulation tasks and define the judgment criteria for successful completion.
Qwen-Omni acts as a high-level observer while Qwen-RobotManip executes the physical actions. This combination allows for open-ended, speech-based task generation and real-time judgment of whether a task was completed properly.
Source
Qwen"Qwen-Omni observes the scene, randomly proposes manipulation tasks via speech, and judges execution in real time."