Chapter 9: Sim-to-Real Transfer and Evaluation

Summary

In Agentic Coding, the test environment and production environment are nearly identical — Docker guarantees this. In Agentic Robotics, this guarantee is impossible. Visual, physical, and control gaps exist between simulation and reality (the sim2real gap). This chapter covers research that measures and narrows this gap. SIMPLER establishes the standard for simulation-based evaluation, and Natural Language Sim2Real uses language as a tool to bridge the divide.

9.1 Introduction: The Test Environment Fidelity Problem

In software development, "passes in test, fails in production" is rare. CI/CD pipelines run tests in environments identical to production. Container technology has effectively eliminated cross-environment differences.

In robotics, "succeeds in simulation, fails in reality" is routine. Simulation rendering differs from real-world visuals, physics engines approximate contact, deformation, and fluids, and control command latency differs. These three dimensions of gap constitute the sim2real gap — this chapter's central topic.

Why this matters is clear from Chapter 8: agentic loop efficiency depends on iteration speed, and physical experiments take minutes per iteration. If the agentic loop can run in simulation and transfer results to reality, loop speed can accelerate by orders of magnitude. The sim2real gap is the key bottleneck for this acceleration.

9.2 SIMPLER: The Standard for Simulation-Based Policy Evaluation

SIMPLER [Li et al., 2024] provides an open-source suite of simulation environments for reliably evaluating real-world robot manipulation policies.

It implements simulations of Google Robot and WidowX BridgeV2 environments, systematically identifying and mitigating control disparity and visual disparity. The key contribution: reliable evaluation is possible without building a complete digital twin.

SIMPLER demonstrated strong correlation between simulation and real-world performance, and accurately reflected policy sensitivity to distribution shift. It evaluates generalist policies including RT-1, RT-1-X, and Octo, providing a comparable benchmark.

SIMPLER corresponds to Agentic Coding's staging environment. Just as code is verified in a test environment before production deployment, robot policies are verified in simulation before real-world deployment. The fundamental difference: Docker makes code test environments nearly identical to production; robot simulation always has a sim2real gap.

9.3 Natural Language Sim2Real: Bridging the Gap with Language

[UT Austin, 2024] proposes overcoming the visual gap between simulation and reality by using natural language descriptions as a common semantic representation.

Pre-training an image encoder to predict natural language descriptions teaches it domain-invariant representations — simulation images and real images look different but are described by the same language. Small amounts of real-world demos combined with large amounts of simulation demos are used jointly.

The 25-40% improvement over CLIP and R3M is impressive — pre-training with just hundreds of image-language pairs exceeded internet-scale pre-training (CLIP, R3M).

The idea of "using language as an intermediate representation to bridge domain gaps" resonates with RT-H's language motion (see Chapter 5). Whether natural language can serve as a universal tool for narrowing gaps across not just visual but also physical and control domains remains an open question.

9.4 Three Dimensions of the Sim2Real Gap

The sim2real gap decomposes into three independent dimensions:

Because the dimensions are independent, solving one still leaves others as bottlenecks. SIMPLER addresses control and visual gaps; NL Sim2Real focuses on the visual gap. The physical gap is the hardest dimension, limited by the accuracy of contact, deformation, and fluid simulation in current technology.

9.5 The Absence of Evaluation Standards: Where Is Agentic Robotics' SWE-bench?

Just as SWE-bench accelerated Agentic Coding's development, Agentic Robotics needs standardized benchmarks. The current state is fragmented — each paper evaluates on its own environment and metrics.

Four fundamental evaluation difficulties exist: environment non-reproducibility (physically identical setups are impossible), metric diversity (task success, execution time, safety violations, generalization all matter), embodiment diversity (different action spaces prevent fair comparison), and cost (large-scale physical experiments require massive time and resources — AutoRT's 77,000 episodes took 7 months).

9.6 Comparison with Agentic Coding: The Fundamental Difference in Test Fidelity

The verification gap between Agentic Coding and Agentic Robotics is one of the two most severe across the seven dimensions (5/5 severity).

Code unit tests run in milliseconds; thousands execute in minutes. The robot equivalent of a "unit test" does not exist. Each physical trial takes minutes and requires human supervision.

More fundamentally, code test fidelity is nearly 100% — test environments match production. Simulation test fidelity is limited by the sim2real gap. SIMPLER showed "strong correlation," but correlation is not identity.

This gap is irreducible in principle. No matter how precise simulation becomes, perfect simulation of contact dynamics cannot be achieved due to approximation limits in current physics modeling. The solution direction is not "perfect simulation" but "imperfect but useful simulation + efficient real-world verification."

9.7 Open Problems and Outlook

First, establishing an integrated benchmark suite that unifies SIMPLER's simulation environments, CaP-X's agentic metrics, and Open X-Embodiment's cross-embodiment protocols. SWE-bench's impact on Agentic Coding proves that such a standard would accelerate Agentic Robotics development.

Second, adaptive sim2real transfer — rather than deploying simulation-trained policies directly to reality, rapidly adapting them with small amounts of real-world experience. PragmaBot's experience-based learning (see Chapter 8) provides early clues.

Third, fleet-scale verification. As AutoRT demonstrated, operating multiple robots simultaneously increases verification throughput. However, this requires Google-scale infrastructure, raising accessibility concerns. Open-source fleet management systems are needed.

The most promising convergence direction is hierarchical verification: high-level plans verified via scene graphs (VeriGraph, Chapter 7), mid-level via code (Code-as-Symbolic-Planner, Chapters 3 and 5), low-level via simulation (SIMPLER) — a multi-layer verification pipeline where each level uses the most efficient verification method, reserving physical experiments for final confirmation only.

References

1. Li, X. et al., "Evaluating Real-World Robot Manipulation Policies in Simulation (SIMPLER)," arXiv:2405.05941, 2024. scholar
2. Lang4Sim2Real, "Natural Language Can Help Bridge the Sim2Real Gap," arXiv:2405.10020, 2024. scholar
3. Brohan, A. et al., "AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents," arXiv:2401.12963, 2024. scholar
4. Fu, M. et al., "CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation," arXiv:2603.22435, 2026. scholar
5. Open X-Embodiment Collaboration, "Open X-Embodiment: Robotic Learning Datasets and RT-X Models," arXiv:2310.08864, 2023. scholar
6. Ekpo, D. et al., "VeriGraph: Scene Graphs for Execution Verifiable Robot Planning," arXiv:2411.10446, 2024. scholar
7. Chen, Y. et al., "Code-as-Symbolic-Planner: Foundation Model-Based Robot Planning via Symbolic Code Generation," arXiv:2503.01700, 2025. scholar
8. Yardi, Y. et al., "Bridging the Sim2Real Gap: Vision Encoder Pre-Training for Visuomotor Policy Transfer," 2025. scholar