Training Your NVIDIA JetBot to Avoid Collisions Using NVIDIA Isaac Sim

Collecting a variety of data is important for AI model generalization. A good dataset consists of objects with different perspectives, backgrounds, colors, and sometimes obstructed views. The model should learn how to handle outliers or unseen scenarios. This makes the data collection and labeling process hard.

In this post, we showcase sim2real capabilities of NVIDIA Isaac Sim for the collision avoidance task on NVIDIA JetBot. Sim2real makes data collection easier using the domain randomization technique. It expedites model training without access to the physical environment. However, in sim2real, simulation accuracy is important for decreasing the gap between simulation and reality.

Picture of JetBot modeled in Isaac Sim. — *Figure 1. Waveshare JetBot in Isaac Sim.*

In JetBot, the collision avoidance task is performed using binary classification. In other words, you show model images that are considered blocked (collision) and free (no-collision). The following example images are from a real-world Waveshare JetBot perspective (Figure 2) and Isaac Sim (Figure 3) for collecting blocked and free data.

Figure shows eight real-world images from the Waveshare JetBot perspective. — *Figure 2. Images from the Waveshare JetBot perspective.* Top row: *blocked (collision) class label.* Bottom row: *free (no-collision) class label. Waveshare Jetbot has a RPi V2 camera that generates a pinkish tinge.*

Photos show eight Isaac Sim images from the camera’s perspective. — *Figure 3. Images from Isaac Sim.* Top row: *blocked (collision) class label.* Bottom row: *free (no-collision) class label. It consists of different backgrounds, objects, and colors using domain randomization.*

Using Isaac Sim for training collision avoidance

Here are the detailed steps to collect data using Isaac Sim on the Waveshare JetBot:

Import the JetBot and move it into the simulation.
Import objects and the JetBot to a simple indoor room.
Use Domain Randomization and the Synthetic Data Recorder.

Video 1. Tutorial for Isaac Sim with JetBot: Importing Jetbot and objects in the scene.

Video 2. Tutorial for Isaac Sim with JetBot: Domain randomizations.

Importing the JetBot and moving it in simulation

Install Isaac Sim 2020.2. For more information, see System Requirements.

To run Isaac Sim Local Workstation, launch /.isaac-sim.sh to run Isaac Sim in the regular mode. There is an option to run in headless mode as well, for which you must download the client on your local workstation [LINK].

When you launch the script, you should see the startup window with the following resources (Figure 4):

Top left—Viewport from which to visualize the stage or scene in simulation.
Top right—The Objects, Physics, and other details related to the simulation.
Bottom left—Access the samples pre-installed with Omniverse: omniverse://ov-isaac-dev/Isaac/Robots/Jetbot/.

Figure shows the first pop window after ./isaac-sim.sh command is executed. — *Figure 4. Isaac Sim start up window.*

To open a JetBot sample, right-click the jetbot.usd file. You see that the stage now consists of the Jetbot and the world (Figure 5). The world further consists of a Physics scene with a staticPlaneActor, which are both needed for the correct simulation of objects in the scene. We discuss these later in this post.

Figure demonstrates JetBot sample invoked using USD file. — *Figure 5. Imported JetBot file to Isaac Sim*.

To move the Jetbot, change the angular velocity of one of the joints (left/right revolute joints). In Figure 6, the right wheel joint has been set to a target angular drive velocity of 2.6 rad/sec. When you choose Play, you should see the robot move in a circle.

Figure shows right wheel joint has been set to a target angular drive velocity. — *Figure 6. Assigning an angular to the wheel joints.*

Next, investigate importing the Jetbot into a simple indoor room where you collect the data to train the model.

Importing objects and the JetBot to a simple indoor room

To find simple_room.usd, navigate to omniverse://ov-isaac-dev/Isaac/Environments/Simple_Room/. Right-click and open this scene. Figure 7 shows a simple room example.

Figure demonstrates Isaac sim’s simple room scene. — *Figure 7. Simple room scene in Isaac Sim*.

Add Physics to the scene by choosing Physics, Add Physics. Make sure that nothing is selected in the scene on the right; otherwise, Physics may be incorrectly added for the scene.

Select towel_room_floor_bottom_218 and choose Physics, Set, Collider. This makes the floor the collider or ground plane for objects in the scene. When simulation begins, objects treat this as the ground plane.

Import the JetBot into this room by navigating to omniverse://ov-isaac-dev/Isaac/Robots/Jetbot/ and dragging the jetbot.usd file into the scene. It may take some time or several attempts. When you choose Play, you should be able to see the JetBot drop onto the surface. If you get warnings similar to “physics scene not found”, make sure that you have followed the previous steps correctly.

If you do not want the camera of the JetBot to be visible on the Viewport, choose Stage, JetBot, rgb_camera and then select the ‘eye’ icon to disable the Omniverse visualization for the camera.

Figure adds JetBot to simple room scene. — *Figure 8. Adding the Jetbot into the room*.

Figure displays removing of camera visualization on JetBot. — *Figure 9. Remove camera visualization on JetBot.*

To add more objects into the scene, navigate to omniverse://ov-isaac-dev/Isaac/Props/YCB/Axis_Aligned, which contains a few common everyday objects from the YCB dataset. Drag and drop objects from the options available. After you drag a particular object into the scene, make sure that you select Physics, Physics, Set, and Rigid Body. This ensures that the object behaves properly after the simulation has started.

Figure demonstrates adding another object in simple room scene from Isaac Sim. — *Figure 10. Adding another object in the scene.*

Following the same procedure, drag and drop more objects in the scene. Figure 10 shows more objects in the scene.

You can also look at the objects from the JetBot camera view. To do this, below the Viewport, change Perspective to Camera, jetbot_camera. This is the view for gathering data. However, the resolution for the Viewport must be changed to match the actual camera of the JetBot in the real world. To do this, below the Viewport, select the gear icon and set the resolution to 1024×1024. The aspect ratio must be 1:1.

The corresponding view of the JetBot changes as well. You should be able to see more background details come into the picture. This is how the actual JetBot looks at the world.

Figure demonstrates adding multiple objects in simple room scene from Isaac Sim. — *Figure 11. Adding multiple objects in the scene.*

Figure shows the camera perspective of JetBot from Isaac Sim. — *Figure 12. JetBot camera perspective.*

Setting up camera perspective resolution to 512x512 for DL model consumption. — *Figure 13. Changing Viewport resolution to 512×512.*

Using domain randomization and the Synthetic Data Recorder

While capturing data, make sure that you cover a variety of scenarios, as the locations, sizes, colors, and lighting can keep changing in the environment for your objects of interest. Add a domain randomization component to make the model more robust and adaptable. This ensures that you have good generalization to the real- world data as well.

For this example, I set the position of the JetBot to X = 0, Y = 0, Z = 23. You can move the table out of that position, or you are free to select a position of your choice for the JetBot. The initial object, the banana, is kept at X = 37, Y = 0, Z = 22.

Choose Create, Isaac, DR, Movement Component. Make sure that no object is selected while you add this DR; otherwise, there may be unpredictable behavior.

Now choose Windows, Isaac, Relationship Editor. This open a tab in the bottom right, to the right of Details, Audio Settings.

In Stage under Root, there should now be a movement_component_0 created towards the end. Select the Relationship Editor in the tabs below and select primPaths. Add the object to randomize. For this case, select the banana.

Add the path: Root/_11_banana. You must specify the range of movement for this DR component. On the Details tab, specify the X, Y, and Z range:

X: 10 to 40
Y: -10 to 10
Z: 20 to 20

After making these changes, choose Play and you see the banana move at a random location between your specified points. You have successfully added a Domain Randomization Movement component for a banana. The data recorded in this simulation would be of the class Collision/Blocked. This is because the banana is close to the JetBot and could result in a collision with it.

Figure displays domain randomization using object movement. — *Figure 14. Adding movement component for domain randomization.*

By changing the range of the X component for movement randomization, you can gather data for the Free/No-collision class as well. In this case, there would be no object within 40cm of the JetBot. The objects beyond the range of 40cm do not cause a collision with the JetBot so you can add randomization for them. Similarly, you can add randomization for scale, color, and lighting for the objects needed.

In the Waveshare JetBot, there is a pinkish tinge when using the actual camera. This can be accounted for as well. For this, choose Create, Isaac, DR, Light Component. In the Relationship Editor, specify the Path value of the light in the room. Now, the color and effects of lighting are randomized as well.

In conclusion, you can edit the range of values for the first and second color to ensure variation in lighting, as per your real-world scenario.

Figure displays domain randomization using background light change. — *Figure 15. Lighting component for domain randomization.*

You can also record data from this simulation. To do so, choose Window, Isaac, and Synthetic Data Recorder. On the Synthetic Data Recorder tab, you can now specify the sensors to use while recording data. For this post, use RGB, as it is a classification problem in this case.

Set the output directory and the capture period in seconds to appropriate values, such as 0.7 for the capture period. After completing a recording, you should find a folder named /rgb in your output path which contains all the corresponding images. Create two separate folders for collision and no-collision and store the corresponding images stored there after applying different randomizations. You can now use these images to train a classification model and deploy it on the JetBot.

For next steps, check if JetBot is working as expected. After the dataset is collected using Isaac Sim, you can directly go to Step 2 – Train neural network.

Conclusion

In this post, we showed how you can use Isaac Sim with JetBot for the collision avoidance task. You used domain randomization for lighting glares and to perform background variations, taking advantage of the different objects available in Isaac Sim to create a dataset. You learned how to collect a dataset to build a generalized model such that it can work accurately on unseen scenarios.

For more information, see Getting Started with JetBot.