Formulating a multi-step robot manipulation task

WHAT THIS TESTS: Whether you can frame a long-horizon manipulation task as a learning problem with sensible perception, action, and reward design.

A GOOD ANSWER COVERS: Start with perception. The robot needs to detect and localize the red mug and the sink, likely with an object detector or segmentation on RGB-D so it gets both identity and 3D position, plus grasp-pose estimation for the mug and self state such as joint angles and gripper status. State could be a learned embedding of these observations rather than raw pixels alone. The action space spans two regimes: navigation to move the base toward the table and sink, and manipulation to reach, grasp, lift, carry, and release. A flat continuous action space over base and arm is hard to learn, so a hierarchical formulation helps: a high-level policy selects subgoals like go-to-table, grasp-mug, go-to-sink, place, and low-level controllers or motion planners execute them. For reward, a single sparse signal only on final success makes credit assignment nearly impossible over such a long horizon, so use shaped, staged rewards: progress toward the mug, a bonus for a stable grasp, progress toward the sink while holding it, and a final placement success, with small penalties for drops, collisions, or wasted time. Sim-to-real training with domain randomization, plus demonstrations to bootstrap, makes this practical.

COMMON WRONG ANSWERS: Using only one sparse terminal reward for the whole task; ignoring perception and assuming ground-truth object poses; choosing a flat low-level action space with no hierarchy; forgetting failure penalties like dropping or collisions; expecting pure RL from scratch on real hardware without simulation or demonstrations.

LIKELY FOLLOW-UPS: How do you avoid reward hacking from shaped rewards? Why decompose into subgoals, and how do options or hierarchical RL help? How would imitation learning or offline data accelerate this? How do you handle sim-to-real gaps in perception and dynamics?

ONE CONCRETE EXAMPLE: Define subgoals reach, grasp, transport, place. The policy earns dense reward for shrinking the gripper-to-mug distance, a large bonus when force sensors confirm a lift, continued reward for closing distance to the sink while the mug stays grasped, and a final success bonus on release over the sink, with penalties for collisions or drops. Trained in simulation with randomized lighting and mug positions and seeded with a few teleoperated demonstrations, the robot learns the full sequence before transfer to hardware.