Into the Omniverse: Open World Foundation Models Generate Synthetic Worlds for Physical AI Development

Into the Omniverse: Open World Foundation Models Generate Synthetic Worlds for Physical AI Development

Leading robotics and AI companies are already using these technologies to accelerate physical AI development.

Skild AI , which builds general-purpose robot brains, is using Cosmos Transfer to augment existing data with new variations for testing and validating robotics policies trained in NVIDIA Isaac Lab .

Skild AI uses Isaac Lab to create scalable simulation environments where its robots can train across embodiments and applications. By combining Isaac Lab robotics simulation capabilities with Cosmos’ synthetic data generation, Skild AI can train robot brains across diverse conditions without the time and cost constraints of real-world data collection.

Serve Robotics uses synthetic data generated from thousands of simulated scenarios in NVIDIA Isaac Sim. The synthetic data is then used in conjunction with real data to train physical AI models. The company has built one of the largest autonomous robot fleets operating in public spaces and has completed over 100,000 last-mile meal deliveries across urban areas. Serve’s robots collect 1 million miles of data monthly, including nearly 170 billion image-lidar samples, which are used in simulation to further improve robot models.

Learn more about how Serve Robotics uses Isaac Sim to accelerate development, testing and deployment of its sidewalk delivery robots by watching the below livestream.

Beyond bringing people meals, Serve recently used its robots to deliver compute power — dropping off brand-new NVIDIA DGX Spark personal AI supercomputers to Refik Anadol, Will.I.AM and Ollama. With 1 petaflop of AI performance, DGX Spark offers developers desktop capabilities for workflows from AI model prototyping and model fine-tuning to inference and robotics development.

Autonomous drone delivery company Zipline also participated in the DGX Spark drop, with Chief Hardware Officer Jo Mardall receiving a DGX Spark by drone at the company’s headquarters and testing facility in Half Moon Bay, California. Zipline uses the NVIDIA Jetson edge AI and robotics platform for its drone delivery systems.

Data scientist and Omniverse community member Santiago Villa is using synthetic data with Omniverse libraries and Blender software to improve mining operations by identifying large boulders that halt operations.

Undetected boulders entering crushers can cause delays of seven minutes or more per incident, costing mines up to $650,000 annually in lost production. Using Omniverse to generate thousands of automatically annotated synthetic images across varied lighting and weather conditions dramatically reduces training costs while enabling mining companies to improve boulder detection systems and avoid equipment downtime.

FS Studio partnered with a global logistics leader to improve AI-driven package detection by creating thousands of photorealistic package variations in different lighting conditions using Omniverse libraries like Replicator . The synthetic dataset dramatically improved object detection accuracy and reduced false positives, delivering measurable gains in throughput speed and system performance across the customer’s logistics network.

Robots for Humanity built a full simulation environment in Isaac Sim for an oil and gas client using Omniverse libraries to generate synthetic data, including depth, segmentation and RGB images, while collecting joint and motion data from the Unitree G1 robot through teleoperation.

Omniverse Ambassador Scott Dempsey is developing a synthetic data generation synthesizer that builds various cables from real-world manufacturer specifications, using Isaac Sim to generate synthetic data augmented with Cosmos Transfer to create photorealistic training datasets for applications that detect and handle cables.

Learn more about OpenUSD, Cosmos and synthetic data for physical AI by exploring these resources:

Train using the “ Getting Started With Isaac Sim ” learning path, which covers Isaac Sim for robot simulation, ROS 2 integration, synthetic data generation and more.

Explore the generative AI reference workflow for synthetic data generation .

Key considerations

  • Investor positioning can change fast
  • Volatility remains possible near catalysts
  • Macro rates and liquidity can dominate flows

Reference reading

More on this site

Informational only. No financial advice. Do your own research.

Leave a Comment