The sim-to-real gap: why simulated robots fail on hardware
The sim-to-real gap is the performance lost when a simulation-trained policy runs on a real robot. What causes it and the techniques used to close it.
- The sim-to-real gap is the drop in performance when a policy trained in simulation is run on a physical robot.
- It has three sources: physics the simulator gets wrong, especially contact and friction; sensing that looks different in reality; and timing, such as latency and actuator response.
- The main fixes are domain randomization, system identification, modelling the actuators from real data, and fine-tuning on a small amount of real-world experience.
- Simulation transfers well for locomotion and less well for manipulation, where behaviour depends on contact details that are hard to simulate.
The sim-to-real gap is the performance lost when a robot policy trained in simulation is run on a physical robot. A policy that succeeds 99 times out of 100 in the simulator may drop objects, stumble or freeze on real hardware, because the simulated world it learned in was not quite the real one.
It matters because simulation is the cheapest source of robot experience there is. Thousands of simulated robots can practise in parallel, around the clock, without breaking anything. The gap is the tax on that bargain.
Where the gap comes from
Physics
Simulators approximate. They do well with rigid bodies moving through free space and badly with contact: the moment a foot meets the floor or a finger meets an object. Friction, slip, deformation of soft materials, cables and cloth, and liquids are all simplified, sometimes heavily. A policy can learn to exploit a simulator’s approximations — a grasp that works only because simulated friction is too generous — and that behaviour evaporates on hardware.
Actuators
A simulated motor delivers exactly the torque it is asked for, instantly. A real one has friction in its gearbox, backlash, torque limits that change with speed and temperature, and a response that lags the command. For dynamic movement this is often the largest single source of error.
Sensing
Rendered camera images differ from real ones in lighting, texture, noise, motion blur and lens distortion. Simulated depth sensors are clean where real ones have holes and artefacts. A vision-based policy trained only on rendered images is seeing a slightly different world.
Timing
Real systems have delays: sensor capture, computation, communication, actuation. A policy trained with zero latency will be acting on stale information in reality, and it will not know it.
How the gap is closed
No single technique solves it. Practical systems stack several.
Domain randomization. Instead of trying to make the simulator exactly right, make it varied. Randomise friction, masses, motor strength, delays, lighting and textures across training. The policy is forced to work across the whole range, and if the range is wide enough, reality falls inside it. This is the workhorse of sim-to-real transfer.
System identification. Measure the real robot — link masses, joint friction, motor constants, latency — and tune the simulator to match. This narrows the range that randomisation has to cover. Too much randomisation produces a cautious, low-performing policy, so better measurement buys better performance.
Learned actuator models. Record how the real actuators respond to commands, train a small model to reproduce that response, and use it in the simulator in place of the idealised motor. A 2019 study on the ANYmal quadruped showed that a small neural network trained this way predicted joint torque far better than an idealised motor model, and it became a standard ingredient of simulation-trained legged locomotion.
Real-to-sim. Build the simulated scene from the real one. Scanning and reconstruction methods can turn photos of an actual workspace into a simulation environment, so the policy trains on something that looks like where it will work.
Real-world fine-tuning. Train in simulation, then adapt on a small amount of real data. The simulation supplies the bulk of the experience; reality corrects the errors.
Co-training. For manipulation with vision-language-action models, simulated and real demonstrations are often mixed in a single training set. Simulation adds volume and variety; real data anchors the model to how things actually look and behave.
Where simulation works, and where it does not yet
Locomotion is the success story. Walking and running policies for quadrupeds and humanoids are now routinely trained entirely in simulation with reinforcement learning and transferred directly to hardware. The physics involved — rigid links, ground contact — is what simulators do best.
Manipulation is harder. The outcome depends on fine contact with objects of unknown shape, weight and friction, and on vision. Simulation helps, but most general-purpose manipulation systems still lean on real teleoperated demonstrations.
Deformable and granular things — cloth, food, cables, liquids — remain the hardest to simulate well enough to learn from.
Why this matters for safety and deployment
A simulation result is evidence about the simulator. It becomes evidence about the robot only to the extent the two have been shown to agree. For anyone deploying a robot, that means success rates quoted from simulation should be treated as an upper bound until measured on hardware — and that a risk assessment cannot rest on simulated testing alone.
Frequently asked questions
What is the sim-to-real gap in robotics?
It is the difference between how a robot policy performs in simulation and how it performs on real hardware. A policy can succeed nearly every time in the simulator and fail often on the robot because the simulated world differs from the real one in physics, sensing and timing.
What is domain randomization?
A training technique in which the simulator’s parameters — friction, mass, motor strength, lighting, textures, delays — are varied randomly across training runs. The policy cannot rely on any one setting, so it learns behaviour that works across a range wide enough to include the real world.
Why is sim-to-real harder for manipulation than for walking?
Walking depends mainly on rigid-body dynamics and foot contact with the ground, which simulators model reasonably well. Manipulation depends on fine contact between fingers and objects that may be soft, slippery or deformable, and on camera images of varied objects — all much harder to simulate accurately.
Can a robot be trained entirely in simulation?
For some skills, yes. Legged locomotion policies are routinely trained entirely in simulation and deployed without real-world training. For general manipulation, most current systems still depend on real demonstration data, with simulation used to add volume and variety.
Sources
- Learning agile and dynamic motor skills for legged robots — Hwangbo et al., Science Robotics, 2019