How fast is the new hyped Genesis simulator?

Original post about Genesis

Jan 8, 2025 Update: After discussion and some bug fixes there is a new comprehensive report by the Genesis Team, I highly recommend you read their new report. I have not had time to verify everything but their code is open source and the new report is more accurate and representative considering various contexts and typical robot learning workflows. The report also highlights critical issues in my benchmarking and does a good job diving deeper into simulation topics than my higher-level post. Notably, their latest simulation version achieves the claimed 43M+ FPS with just the robot arm and self-collisions enabled, and 27M FPS when random actions are added. Additionally, the simulation now features more stable grasping capabilities. Although achieving 43M FPS (430,000x real-time speed) may not be typical in most scenarios, you can still see sim speedups compared to other simulators when using the same scene and setup.

Genesis was released last week to immense public frenzy over how impressive the demo videos look and how fast it runs. I don’t doubt the demo videos (clearly cherry picked but everyone does this), but I draw the line at inflating the speed benchmark numbers. Genesis is an impressive effort from many people and it has its pros, but the way it is presented is egregiously misleading enough for me to publicly post this. Note that I am writing this post of my own volition to help others/the community fact-check and verify claims, and all code is open-sourced.

Note: For those unaware, Genesis is not a video generative model, it is a GPU parallelized rule-based physics simulator like Mujoco MJX and PhysX. The generative AI part is for generating scene layouts which are populated by off the shelf 3D models, followed by motion generation.

tldr; Genesis has magical speedups by essentially not simulating anything of substance. Corrected benchmark code shows the publicly reported numbers are off by 150x and is 3-10x slower compared to existing GPU parallelized simulators for robot manipulation.

Accurate Benchmark Speeds

NOTE these numbers are no longer accurate, see the note at the top of this blog post.

The first problem is Genesis’s speed benchmark is not representative of its actual performance in reinforcement learning, data generation, or other workflows. There are four key issues:

1. The benchmark scripts essentially use the fastest physics solver configurations, which mean they simulate less accurately (physics substeps = 1). Their own “fast” robot dog training environment uses 2 substeps and their robot manipulation examples uses 4 substeps.

2. They benchmark simulating the Franka robot arm and robot dog taking one action (moving to a fixed position) and then staying idle for > 90% of the benchmark time. Obviously, this runs fast as you aren’t simulating much.

3. Robot self collisions are disabled by default. This means the Franka arm for example can pass through itself without collision. Most sims and benchmarks don’t disable self collisions.

4. According to their website/post the best results are shown when they sleep/hibernate all objects that aren’t moving. However this is not realistic as in most situations in simulation workflows something is moving. This coincides with point 2 of not simulating anything of substance.

There are likely more but these 4 account for most of the magic “speedups”. When running the corrected code the speed (for the Franka benchmark) drops 150x from 43 million to just 0.29 million FPS (tested on a 4090 GPU), which is on par or slower than existing GPU simulators.

Corrected Franka Arm Benchmark Code: https://gist.github.com/StoneT2000/904b2f8fd8cd9d4ed43b9772ee044c1d

Corrected Robot Dog Benchmark Code: https://gist.github.com/StoneT2000/6faa41c375f20f69db6dcbb4804be8a3

State simulation FPS of Genesis Franka environment with successively added corrections

Given that the Genesis benchmark only tests locomotion/few collisions, I created a different set of code designed to test robot manipulation. Github Code here. It compares Genesis and ManiSkill/SAPIEN (my lab’s PhysX based open source GPU sim). I would like to test MJX and Isaac Lab but due to lack of time I haven’t. However since Isaac Lab runs on PhysX like ManiSkill, you can assume Isaac Lab state sim speed is about the same as ManiSkill. The FrankaPickCubeBenchmark environment spawns a Franka arm and a cube. The results show Genesis is 3-10x slower than ManiSkill in the following 2 figures when benchmarking physics only simulation:

State (physics-only) FPS with dt=0.01 (sim frequency 100Hz) and control frequency of 50Hz on a cube picking environment. Annotated GB numbers indicate GPU memory usage. The robot arm is tested by following a scripted path to pick up the cube and shake around. Not yet tested with Isaac Lab / MJX due to time constraints but quite sure they run faster than 100K FPS, especially Isaac since Isaac uses PhysX as well. FPS for both ManiSkill/Genesis do not improve beyond 8192 environments.

Same settings as above but takes random actions each step instead of scripted actions.

The next two videos show what occurs during benchmarking in this code. The blue background (left video) is Genesis and the light gray one (right video) is ManiSkill. You might also notice that for Genesis, the cube falls out of the gripper, indicating significant issues with simulation accuracy. I’m sure the Genesis team will fix the simulation bugs but when they do it is all the more likely Genesis would run even slower.

Simulation Accuracy Problems

You might have noticed already in the benchmark videos of the cube lifting motion, the cube glitches around and even falls out of the robots gripper in Genesis. The current Genesis tutorial code has this issue as well.

According to the Genesis authors, the issue stems from poorly tuned constraint solver configurations - bad enough that their tutorial code for the same motion doesn’t work either. The underlying issue may be that Genesis, despite the claims in their demo videos, is still too early in its development to reliably support fast and stable basic or dexterous manipulation. Their rigid body contact solver code is quite simple, making it relatively fast, but not flexible enough for more complex simulations like robust locomotion or manipulation (at the moment).

Does Genesis train a robot dog in 26 seconds?

It does and doesn’t. First off, the 26 seconds is not faked, but the original post on X/Twitter misleads users to think this is some groundbreaking record.

So why does it train “faster”? It’s because they only train on flat terrain (easier to simulate) and provide a very simple reward that likely doesn’t transfer as well to the real world compared to what Isaac Lab and MJX have done. If you built the same exact environment in other GPU sims with the same reward function the training code will run just as fast. One of the Genesis authors has already confirmed that the training speed for locomotion is the same in Isaac Lab in a followup post.

Simulating “Reality”

Genesis claims that they can simulate “reality” at 430,000x real time speed. However, currently their simulations essentially run blind. If you turn on a camera for the robot, the simulation speed drops to just 10x realtime speed. In comparison both Isaac Lab and ManiSkill truly simulate “reality” (simulation + rendering in parallel) at 1,000x real time speed (depending on the task being simulated and how many objects).

Furthermore, if you consider "reality" as being photorealistic rendering, just as the demo videos Genesis showed, then you need to run ray-tracing. This will be slow regardless of simulator used (in the 10-100 FPS range depending on ray tracing settings).

Conclusion

Genesis is a nice simulator with lots of cool features, and simulation indeed is something I strongly believe in when it comes to scaling up synthetic data for training and evaluating embodied AI / robotics. Genesis can be commended for its effort in having so many physical simulations (fluid, particles, rigid body etc.) integrated into one package, and it is relatively easy to use (took me 1 day to make new environments and write benchmark code. Their API and even their documentation is similar to ManiSkill/SAPIEN).

However, the main takeaway is that Genesis is not as fast as reported (it is slower by >100x than claimed), and compared to an existing GPU sim Genesis is slower by 3-10x on environments with slightly more collisions / complex dynamics. It is likely that testing a more contact-rich environment, such as Peg Insertion, would reveal a more significant difference. The original benchmarking code is too simple and inflates performance metrics by essentially benchmarking nothing. This is a significant issue, as it misleads many into believing that Genesis is faster than it actually is. Furthermore, there are no public comparisons with Mujoco MJX, the framework that Genesis is partially based on, which raises questions about its true performance.

True scientific advances in simulation/rendering speed and fidelity have been thanks to a long history of efforts from the people who make PhysX, Isaac Sim, Mujoco, SAPIEN and more.