Author | Haper Zhuo
Interview | Eric Wang
Editor | Monica He
Produced by | GOSIM

German Ros Sanchez, currently the Executive Director of the Embodied AI Foundation and Director of Simulation Ecosystem Development at NVIDIA, is a distinguished figure in machine learning and artificial intelligence. His deep academic background, groundbreaking research, and relentless drive for self-challenge have shaped his exceptional career.

At the age of six, he started programming in BASIC. At ten, he discovered Red Hat Linux, which sparked his passion for open source. Open source gave him a sense of belonging, like a home, and became the starting point of his career.

In 2017, driven by his passion for robotics, German joined the Toyota Research Institute, focusing on computer vision research for Autonomous Driving and virtual reality technologies. He established the sensor simulation team, pioneering innovative simulation tools for End-to-End evaluation of Autonomous Driving systems.

In 2018, he became the Head of 3D Vision Projects at Intel Labs, leading new solutions in 3D scene understanding and reconstruction. His efforts successfully fostered a large open source community, expanding Carla Simulator’s user base to over 200,000 and making it one of the most widely recognized Autonomous Driving simulation platforms in academia, industry, and government agencies in the U.S. Similarly, Open3D grew to over 150,000 users and was designated as a key project within the Python community, ranking in the top 1% of downloads.

In 2024, GOSIM was honored to invite German Ros Sanchez to China to engage with global tech professionals and share his invaluable experiences. At GOSIM CHINA 2024, he delivered two outstanding talks: "The Role of Open Source Simulation in the Age of Embodied Intelligence" and "Building End-to-End Autonomous Driving Solutions with CARLA." Following the event, the GOSIM Open AGI Forum hosted an exclusive interview with German, conducted by CSDN senior tech editor Eric Wang, where he shared insights on his open source journey, computer vision, and Autonomous Driving.

Key Takeaways from German Ros Sanchez:

• The door to robotics was Autonomous Driving.
• In order to make something successful, good tooling is critical, and some of the key tools in there are simulation ecosystems.
• End-to-End is a great democratization technology, because in the past, you needed to have a large team of people working on the different problems: Perception, planning, control.
• Open source technology acts as the glue that binds simulation, systems, and other related elements together.
• Many companies struggle despite having large user bases and active communities because they fail to translate this into sustainable project development. This has led to the rise of hybrid models, such as freemium or partially open source approaches.
• At a certain point, projects surpass expectations and take on a life of their own—Carla exceeded our initial vision and grew beyond our control.

The following is the full interview:

From Childhood Programming to Autonomous Driving: German’s Journey in Open Source and Robotics

Eric Wang: You describe yourself as a strong open source advocate. What's inspired your strong passion for open source?

German Ros Sanchez: Open source has been in my life since I was like a little kid. I started in computers very early. When I was six, I started programming in BASIC. Around 10 or 11, someone who knew my mother saw me using Windows and asked, “Interesting! You’re using Windows. Do you know Linux?” At the time, I had no idea what Linux was. Then he installed Red Hat on my computer and replaced Windows. I kind of fell in love. Because it gave me so much power, like I could do everything with that distribution. I could compile my own things, change everything. And since then, I became a very passionate person about open source.

I started participating in open source forums at universities. I remember when I was 16, I was teaching open source related classes in college to professors and other people. And that's where I kind of fell in love. There was a huge community around open source. It felt like home. Like finally there's a home for me, all these people that also care about open source. So that's why I have been so passionate from that day about open source.

Eric Wang: Why did you join the Toyota Research Institute before?

German Ros Sanchez: My deep interest in robotics. In 2017, I joined the Toyota Research Institute to work on computer vision research for Autonomous Driving and virtual reality technologies. I established a sensor simulation team, developing innovative simulation tools for End-to-End evaluation of Autonomous Driving systems.

For me, I saw that the door to robotics was Autonomous Driving. Autonomous Driving was something that was becoming mature enough to have an impact in the world. So that was one of the main reasons to get into Autonomous Driving.

When you get into Autonomous Driving, you realize that in order to make something successful, you need good tooling. And some of the key tools in there are simulation. However, there was a lack of high-quality simulation tools available. That's why we had to connect the pieces. We needed to create a simulation platform so that we could be good at developing Autonomous Driving solutions, which was one of my true passions.

Eric Wang: When did you decide to fully commit to Autonomous Driving as your career path, given that your PhD was in computer vision? Can you explain the connection?

German Ros Sanchez: From the start, my focus has always been on deploying robots in real-world applications. If you think about the different tasks that you have to approach to have robots out there in the real world, it becomes obvious that scene understanding is one of the first tasks that you need to figure out.

From the beginning, it has always been about how do we put robots out there. If you think about the different tasks that you have to approach to have robots out there in the real world, it becomes obvious that scene understanding is one of the first tasks that you need to figure it out.

Developing robots that can autonomously navigate requires precise environmental perception, which is why I initially entered the field of computer vision. My goal was to develop technologies that enable the robotic vehicle, to actually perceive the real world, understand what's happening, and eventually take decisions based on that understanding. This is why my thesis was very focused on computer vision.

It was always with the mindset of I want to apply it to Autonomous Driving and Robotics. Once we solve this problem, we’ll be ready to tackle the next big challenge.

Eric Wang: When did you make that clear?

German Ros Sanchez: I think it was basically towards the middle of my PhD that it became clear that eventually it wasn't just about computer vision. It was in general for me about Autonomy, Robotics, and Autonomous Vehicles, whatever it takes. If the focus needs to be computer vision, so be it. If the focus needs to be something else like planning and control, let's attack whatever problem needs to be solved in order to enable Autonomous Driving.

From Scratch to Dream: The Birth of Carla

Eric Wang: Could you share the most exciting part of creating Carla?

German Ros Sanchez: To better understand Carla, we should trace back to the technological developments before its inception. Around 2012–2013, we had the deep learning revolution. Many techniques previously deemed ineffective were revitalized, especially in computer vision and pattern recognition. However, these technologies relied heavily on vast amounts of annotated data, which were extremely difficult to obtain.

I remember that I labeled more than 10,000 images manually at the pixel level, I couldn’t take it anymore—I couldn’t spend my entire career just labeling data. This was before annotation services were common and inexpensive. So we decided to revisit the old idea of using game engines and simulation platforms to automatically generate data. We created the first platform in that space called Cynthia.

Cynthia was a huge boom, a huge success in academia. We even licensed it to industry across all games. But it was focused on just perception, training, and validation of perception.

Later, someone suggested expanding the platform and making it open source, and my dream came true. That was the origin of Carla.

As we embarked on this journey, we basically had to, as you say, build it from scratch. At the time, there was nothing like Carla, so we had to explore new ideas and technologies that no one had tackled before. During the first year, we had to redesign Carla multiple times before finally arriving at a feasible initial version. It was a very challenging time to basically learn the right path by doing and by making mistakes.

Eric Wang: In 2016, the Carla project received funding from Vladan Kotem and Intel and officially transformed into a fully open source simulator. During the development process, what challenges did you encounter, and how did you overcome them?

German Ros Sanchez: I think Vladan had a visionary outlook ahead of his time. He recognized the significance of this tool, and we agreed with that vision. We said “okay, let's work on it!”

But I think none of us really expected the success of Carla and the adoption that the community had, so that was one of the challenges from our perspective. Our original intention in creating Carla was primarily to meet our own needs. As researchers in the field, we needed this tool. But we never expected Carla to gain such a massive following and be rapidly embraced by the industry: “Hey! We want more of this!”

So that was the main difficulty that at some point, it just took off, it just had a life on its own. And we kind of realized that that was much bigger than we anticipated. We started to realize that Carla wasn’t just a short-term project—we had to continue improving it because it was bringing immense value to the industry.

As the project progressed, we had to think about how to establish a nonprofit organization to support Carla’s ongoing development. This involved figuring out how to secure funding to hire developers, attract community, and so on. That was the real challenge we hadn’t foreseen.

Eric Wang: How does Carla simulate street driving? Does the research team need to collect the behavior of the people who walk in the street or what will work on it?

German Ros Sanchez: Carla is highly flexible, allowing users to choose their own approach. High-fidelity behavior modeling is critical for some users, meaning they prefer models based on real-world data. This requires us to collect and validate large amounts of real-world data.

For users with lower complexity requirements, rule-based models are an option. Carla ultimately selects models based on different needs and the level of precision required for various use cases. Leveraging our extensive experience in Autonomous Driving simulation, Carla supports multi-layered simulation demands.

Eric Wang: Omniverse is also advancing Autonomous Driving through simulation ecosystems. As the Director of Omniverse Development at NVIDIA, do you see any obstacles to its development? How are they being addressed?

German Ros Sanchez: There are no obstacles to Omniverse’s development. It is an outstanding platform backed by a team of incredibly talented engineers. Omniverse is gradually evolving into a set of APIs that users can freely integrate with any interesting and practical functional modules.

So we are bringing new capabilities that integrate with the Omniverse APIs so that people can get access to things like high-fidelity sensor simulation through Omniverse. So if you are a Carla developer or a Carla user and want to have access to high-fidelity simulation. Very soon you'll be able to just leverage the Omniverse APIs to do so. So things are going well in that front.

Eric Wang: What applications of open source simulation have yet to emerge but might become possible in the future?

German Ros Sanchez: I think simulation is so broad. In my view, the true meaning of open source simulation is querying the world as a service or turning the world into a computational model where you can launch any query that you want.

Currently, simulation platforms mainly address sensor data and basic physics. However, as technology advances, they will be able to handle more complex tasks. We will be able to accurately simulate physical phenomena, sensor behavior, and the propagation of light and electromagnetic waves, ultimately driving advancements in AI models.

We are also exploring the application of simulation technology in space exploration. For instance, we are collaborating with NASA to study how to develop lunar habitats and how to create autonomous systems capable of constructing them. The more advanced simulation technology becomes, the broader its applications will be.

Eventually, we will be able to create highly realistic simulations of Earth and other planets, enabling a wide range of use cases. This will allow us to transform everything into a computational problem, leveraging computing power to explore fields such as autonomous lunar missions and protein structure analysis.

From Ethical Challenges in Autonomous Driving to Innovations in Smart Mobility

Eric Wang: As a deep participant in AI technology development, what are your thoughts on the ethical challenges posed by deploying AI-powered robots in public spaces?

German Ros Sanchez: When you put robots in the real world, bad things can happen. It's not like something that operates on the web. Therefore, safety is our top priority.

I remember when I started at Toyota Research that I cared mostly about putting things in production and coming up with new agents. We were making significant progress in this regard. At that time, safety wasn’t considered a critical issue—until the Uber accident occurred, which made me realize its importance.

Sometimes you have your new version of your stack in the vehicle, even with safety drivers present, unforeseen issues still arose. Some of these problems might have resulted from modifications I had personally pushed. The thought of my friends potentially being put at risk deeply concerned me. That’s when I became acutely aware that safety must never be overlooked.

So I think it's imperative for companies and developers to really consider safety first—that is the ethical approach. Establishing strict and effective validation processes is essential. Regardless of the solution being deployed, we need regulatory oversight and independent third-party evaluations to ensure the systems function properly. The more attention safety receives, the better the overall outcome. We must remain vigilant because Autonomous Driving technology is truly on its way, especially following recent news about Tesla’s autonomous taxi initiative.

Eric Wang: In your estimation, when will we see autonomous taxis or fully self-driving cars on the roads?

German Ros Sanchez: Today! So for example, where I live in the Bay Area, in California. Waymo is deploying robot taxis. So I have taken many Waymo vehicles to move throughout the area. It's great! It works great! We're going to keep seeing more of that in different cities. So what we call L4, basically robot taxis that work well within a geographical region.

It will not be just from Waymo. We're going to see this vehicle from Cruise. We'll see vehicles from Tesla and many other vendors. So this technology is here today.

I guess the question becomes, when are we going to see it everywhere? That will take a significant amount of time. The reason is that developers are extremely cautious—any misstep could have a profound impact on public perception and even change how governments view these technologies and companies. A single mistake could erase years of progress overnight. Because every new city, every new mile that you add, you need to be very careful. One mistake and everything could go away.

Eric Wang: In the Autonomous Driving field, what do you see as the most promising areas of development? Is it feasible for every vehicle to be equipped with intelligent connectivity, allowing owners to manage their cars remotely through smart devices?

German Ros Sanchez: I'm really passionate and a big believer about some of the new technologies, such as End-to-End solutions—a great democratization technology, which break down traditional barriers between different technological domains. This enables teams to focus entirely on solving driving-related problems without unnecessary distractions. I think that's gonna make newcomers to this space to have a way to leapfrog existing companies. So I'm very passionate and enthusiastic about that technology.

Another technology that is very close to my heart is Neural Rendering. With this, you can create virtual replicas of the real world while significantly lowering computational overhead. As Neural Rendering and neural reconstruction techniques continue to evolve, we will eventually have highly realistic digital twins—not necessarily of the entire world, but at least covering key regions. Users will be able to construct personalized digital twin models effortlessly, without requiring specialized knowledge.

The Art of Balancing Open Source, Business Models, and Government Roles in Technology Innovation

Eric Wang: The debate between open source and closed-source approaches is frequently discussed. What is your perspective?

German Ros Sanchez: I feel a little bit divided in that topic. On one hand, I strongly support open source. However, as someone involved in an open source nonprofit foundation, I am acutely aware of the complexities of developing open source software. That’s why you must establish a viable business model—one that can convert millions of users into resources that support the developers behind these projects. Many companies struggle because they're not able to convert that traction, that momentum that they have in the community with a large community of users to be able to self-maintain, to maintain the project.

As a result, we’ve seen the emergence of hybrid models, like freemium models or models where you have the source available but it's not fully open source. In these cases, commercial use requires licensing fees. The lack of effective financial mechanisms makes it challenging for open source organizations to thrive. Therefore, we must explore forward-thinking solutions. Ideally, more organizations will recognize the necessity of supporting open source initiatives and provide funding for their adoption and promotion.

Eric Wang: What role does open source technology play in advancing Embodied AI Integration?

German Ros Sanchez: It's important to realize what elements enable the acceleration of the field and what elements can kill the acceleration. Because we don't have problems out there yet at scale. Why not? We need to accelerate, we need to become faster, we need to put things in production faster. I really believe that open source is a key contributor to that acceleration. Especially open source in simulation. The flexibility of open source tools allows us to make necessary modifications based on specific requirements. I see open source technology as a crucial glue that binds simulations, systems, and other interconnected elements together. When creating autonomous, adaptive systems, we need scalable and flexible tools, and open source solutions offer one of the best pathways forward.

Eric Wang: Gary has suggested that governments should support open source initiatives. However, government investment in this area remains minimal despite its potentially massive benefits. What is your take on this?

German Ros Sanchez: This is a complex topic, but I support Gary's view on the government should be more participative in the founding of open source projects. As open source developers, we contribute to the broader tech community’s progress. Startups, mid-sized enterprises, governments, and various organizations all rely on open source solutions. We are driving global technological advancements. In the meantime, we need to find alternative ways to sustain open source projects. Increased government support would be the ideal scenario.

Eric Wang: How do you balance business objectives with the complexities of technological innovation?

German Ros Sanchez: It's definitely a challenging problem. So I try to be very connected to the research that happens in the world, like very active talking to professionals in academia and industry, trying to understand what are the new technologies and ideas that are emerging in the research field. I emphasize data-driven decision-making by continually questioning: "Will this work?" "Can it scale?" To establish a solid perspective, we must either experiment firsthand or collaborate with those conducting the most reliable experiments. This is my method for navigating forward: maintaining strong research ties, gathering data, and making well-informed decisions.

Of course, sometimes we have to place strategic bets on certain technologies and evaluate their commercial viability. Over time, reality will reveal the truth. We must have conviction, turn ideas into reality, gather feedback quickly, and iterate continuously.

Eric Wang: What personal challenges do you think you will face in the coming years?

German Ros Sanchez: I do face some challenges. So I think with the mindset of how do we bring Embodied AI into the real world soon. We need to solve many different problems. One problem is how do we make sure that the right simulation tooling and other type of tooling is in place. One of the key issues is how we can leverage simulation systems for business activities.

If we fail to establish a proper business model around simulation tools, it will be very difficult to keep them functional, up to date, and at the forefront of the industry. This is one of the critical problems we need to solve. A good business model should balance open source and commercial aspects while helping these tools continue to evolve. This has been one of my primary areas of focus recently.

German Ros’ Advice for Developers

Eric Wang: In this era of rapid AI advancements, how can developers and researchers stay competitive and innovative?

German Ros Sanchez: It's becoming trickier and trickier. Especially in the U.S., new regulations are emerging. In California, for instance, AI developers must have greater oversight of the data they use to train models. Now, AI developers are subject to external audit-like procedures, which complicates things.

It's hard for individual developers to keep up with industry. I think one of the positive things that has happened within industry is that some large corporations have decided to open their foundational models like Meta’s Llama and Mistral. This openness allows the community to build upon them. However, the landscape will continue to shift. Smaller developers will struggle to keep pace, so government support, such as providing access to hardware and datasets, is crucial.

Eric Wang: Many autonomous vehicle startups are emerging. Can you share some insights to help researchers and developers better plan and grow in the AGI era?

German Ros Sanchez: We are living in a time of groundbreaking innovation, with new developments happening daily. Researchers and developers should first focus on the tools available, understand how simulation technology can be integrated into their workflow, and leverage it to gain a competitive edge. Mastering these tools and continuously using simulation to accelerate progress is my key piece of advice.

Eric Wang: Thank you for these valuable insights and for today's conversation. To all our viewers, that concludes this episode of the Open AGI Forum. See you next time!