NVIDIA SimReady connects digital twins to AI
- Summary:
- Beau Perschall, NVIDIA’s first Director Omniverse Sim Data Ops, talks us through an important development.
Recent advances in AI have been driven by new tools for processing text and image files and the metadata for describing them at scale. But the 3D world is a bit more complex, with dozens of file formats just for shapes and structures. And things get more complicated when trying to represent the inner workings of factories, warehouses, and equipment.
Leading digital twin players are starting to standardize the Universal Scene Description (USD) format to exchange data across authoring, engineering, and tools for 3D design. Now, NVIDIA wants to build on this momentum with SimReady assets that also characterize physical properties, behavior, materials, and IoT data streams for USD-driven simulations. This should improve design workflows and AI training for warehouse robots, autonomous driving, medical devices, and marketing collateral.
Leading the effort is Beau Perschall, NVIDIA’s first Director Omniverse Sim Data Ops. He previously led business development at TurboSquid, one of the largest markets for 3D content exchange. Last year, NVIDIA tasked him with identifying ways to improve 3D simulation exchange as well. Perschall explains:
I am basically responsible for building this standard way of going from visual content only to simulation content and all that represents. I love trying to figure out what’s next. The rabbit hole is very deep for the things you might want to simulate in the future. Some people want to improve defect detection on a production line, while others want to improve autonomous vehicles. It’s hard to keep up and a lot of fun.
We are very much in our infancy at this point. But the long-term goal is to provide guide rails to allow companies to develop consistency within their libraries to make them useful across their own company and then to customers. We work with BMW, Siemens, and Kuka, and today the data that passes between is like the wild west. We’re trying to at least get them to boil their things down into whatever commonalities we can all agree upon as an industry, as opposed to saying you must use this, or you must go this route.
Adding a composable layer
To put this all in context, vendors have been developing 3D formats and standards for years, like FBX, IGES, and OBJ. These are useful for some aspects of design but can be hard to translate widely across 3D tools, particularly when you want to compose larger scenes from a collection of objects, like representing equipment in a factory.
So, Pixar developed the USD format to streamline data exchange across multiple content creation tools. NVIDIA picked up the mantle and started working with the larger 3D and industrial automation community to extend the use of USD broadly.
The glTF format is also suitable for sharing rendered 3D objects but not creating 3D designs or simulating them. Beau spent the last year figuring out how to build a new level of abstraction for 3D simulation. SimReady essentially adds a layer of metadata onto USD files for describing what the data represents, how to render the materials, or how to characterize physical behavior. Perschall says:
In the past year, I’ve been learning the ins and outs of what’s actually important for simulation and then starting to build from there. It’s understanding how you start to classify objects. How do you actually build that taxonomy? What are the kinds of simulations that people are doing for digital twins? Is it machine learning for computer vision, mobile robots, or robotic dexterity? The use cases are exceptionally broad. So, we must start small with the most atomic pieces of content we can take and understand what is valuable.
USD takes advantage of Pixar’s experience in assembling USD components into large assemblies, prototypes, and full-blown movies. SimReady adds the ability to describe how to access other connected data from within the USD file. Perschall explains:
We started with the idea that a USD file by itself is still one file, but you could package a lot of these to represent different parts of a single asset. And when you combine those, even though technically users are only working with one USD file, they have access to all of the other data that is interconnected and interwoven to make it useful inside of simulations. You can have the geometry in one USD, the materials in another, and do any number of things to give it physical properties down the road.
Simulation-as-a-Service
NVIDIA is laying the groundwork for two ways of enhancing the value of 3D data exchanged via USD formats. Connectors translate from one format to USD and back, while extensions directly process the data.
NVIDIA has developed specific extensions for simulating robots and autonomous vehicles. Down the road, Perschall says that consistent simulation formats would also make it easy for others to build extensions.
One of the first projects was to simplify the naming conventions of things. For example, a car could be called an automobile, vehicle, or sedan. The reference implementation took advantage of Wikidata, which provides an open platform for describing over 100 million names of objects.
They also plan to make it easy to describe how joints in large machinery move and computer-aided manufacturing workflows. This could help engineers identify subtle design changes that could make a big difference in manufacturing. They also want to make it easier to overlay IoT data on top of digital twins:
I don’t think we’re looking to make a walled garden around Omniverse. We want to connect all the different tools that want to see their data in one place. But we want that data to live wherever a customer needs it.
Down the road, Perschall expects this could unlock new business models. Industry experts could package their expertise into new simulations companies could use to improve their product designs, manufacturing workflows and business plans. Meanwhile, 3D and simulation designers could rent out their models to train new AI and machine learning algorithms. This could provide new business models for companies that may previously have been skittish about selling their data to rent or license it out on a per-project or annual basis.
Currently, most companies that provide 3D content develop their own naming convention to improve automation and access for their customers. NVIDIA’s outsized role in the graphics space could bring some order to the industry.
Pawel Nikiel, director of technology at CG Trader, which operates a 3D asset marketplace and automation platform that competes with TurboSquid, believes that standards can simplify the naming conventions and reuse for customers and data providers. It will also make it easier to harmonize data from different marketplaces. Nikiel says:
The more data you can feed your AI training algorithms, the bigger the probability you will get fairly trained models that are reliable, won’t crash, and won’t harm anyone.
My take
The current crop of generative AI has primarily been trained on things people have written about the world. One result is a tendency to AI hallucinations and inaccuracies. Further progress could benefit from better consistent standards for characterizing the physical properties and operation of products, factories, and equipment.
The recent progress in deep learning can trace its direct history to efforts to build a database of labeled images in a consistent format called ImageNet. These early efforts helped inspire research into new ways of designing neural net architectures and training them on relatively cheap NVIDIA GPUs.
Efforts to improve naming conventions for 3D content and simulations could spark similar kinds of innovations in creating better AI and machine learning algorithms for analyzing the world and generating 3D content.