How to connect IntelliJ to Webots’ Autonomous Car Simulator (on a Mac)

Basic Setup

So you have a design for a revolutionary new A.I. that’s been stewing in your mind for months. The time is right, and now you want to put your not-fully-formed idea to the test. You start looking around for pre-built testing environments that include some common challenges. You need these to demonstrate the business value of your designs. Where do you start?

One platform you may come across is Open A.I.’s gym. This python-based library gives you the option to test your new ideas on a suite of standard A.I. and M.L. challenges.

But there are limits to what Open A.I.’s gym provides. The library has few 3D robotics simulators, and nothing close to a full driving simulator. And although Open A.I.’s gym is well-known for its Atari games, a lot more money is thrown around in the field of industrial robotics and autonomous cars. These are two challenges which, if you can address or even improve, have high value for businesses such as car manufacturers and factories.

Open A.I.’s gym also requires you to build in python, and compile its library in your own application. There are probably ways to separate the gym from your own code, but it’s more effort than I’m willing to put in. I’d prefer a language-agnostic solution that I can run separately

Advantages of Webots

1. The Webots code is open source (so is OpenAI’s gym).
2. You can connect a variety of languages to the simulation through a common interface.
3. The environments are more realistic and practical than OpenAI.
4. The driving simulator works on Mac (unlike Carla)
5. The simulation can be piped to any IDE or external source code, so you don’t need to recompile the library every time. I’ve been told the simulation and the controller code can work on separate servers too, but I’ve yet to test that.

Instructions: Running the Simulation

First, download the Webots application.

Once you have the dmg file, open it and copy it to your Application folder.

Run the Webots application.

As soon as the application loads, and before you do anything else, hit the pause button at the top. The simulation will auto-play, and your computer may not be able to handle it.

In the menu select Webots > Preferences.

In the General tab, set “Startup mode” to “Pause”. This will stop it from auto-playing when you run it next time.

In the OpenGL tab, lower the preferences to what your computer can handle. I disabled everything.

In the menu, go to File > Open Sample World.

Under vehicles, select city_night (or another if you like), then OK.

Feel free to hit play at the top, and see it run.

You need to add the path to the application to your .bash_profile.

export WEBOTS_HOME=“/Applications/Webots.app”

I assume the path above is where the app is located. If you’ve never done this before, you need to create a file called .bash_profile in your home directory (ie. ~/). Don’t forget the period at the beginning of the file name. Add the line above to the file and save it.

Your IntelliJ Project

Open IntelliJ. Create a new project based on Gradle by selecting it on the left-hand side. For the SKD, you can pick either Java or KotlinJVM.

Go back to your Application folder. Open the app folder by right-clicking on the application and selecting Show Package Contents.

Find the controller library. It is probably here: /Applications/Webots.app/lib/controller/java

Once you’re sure it’s there, go back to IntelliJ. Go to your project’s build.gradle file, and add these dependencies inside the dependencies {} brackets.

compile files(“/Applications/Webots.app/lib/controller/java/Controller.jar”)
compile files(“/Applications/Webots.app/lib/controller/java/vehicle.jar”)

Inside the project’s src folder, under src/main/java create a class called DrivingJava. Add the following code. Your package may be different.

package driving;
import com.cyberbotics.webots.controller.Camera;
import com.cyberbotics.webots.controller.vehicle.Car;

public class DrivingJava extends Car {
     private int timeStep = 128;
     private Camera camera = getCamera(“camera”);
     void run() {
          while (step(timeStep) != -1) {
               setCruisingSpeed(70.0);
          }
     }
     public static void main(String[] args) {
          new DrivingJava().run();
     }
}

In the menu, select Run > Edit Configurations…

Create a new run configuration by clicking the (+) in the top left corner and select Application. Give it a name, and add the following fields:

1. Main class: Select the path where you put DrivingJava. The IDE is looking for the main() method inside your class.
2. VM options: -Djava.library.path= ${WEBOTS_HOME}/lib/controller/java
3. Environment variables: WEBOTS_ROBOT_NAME=vehicle

The word “vehicle” in the last line comes from the name of the vehicle in the Webots app (BmwX5 > name). It may be different for different projects.

Back in Webots, find the BmwX5 on the left-hand side. Open the left arrow, select the controller, click Select below, and switch it to <extern>.

You should see the following in the console below:

[autonomous_vehicle] setting speed to 50 km/h
[autonomous_vehicle] You can drive this car!
[autonomous_vehicle] Select the 3D window and then use the cursor keys to:
[autonomous_vehicle] [LEFT]/[RIGHT] - steer
[autonomous_vehicle] [UP]/[DOWN] - accelerate/slow down
INFO: Terminating controller “autonomous_vehicle”.

Running the code

Finally, you can run your code.

In Webots, click the play button at the top. The simulation will not actually play. It’s waiting for external commands from your Java code.

Back in IntelliJ, compile and run your DrivingJava configuration. If everything worked, the animation should start and the Webots console should display the following:

[autonomous_vehicle] setting speed to 50 km/h
[autonomous_vehicle] You can drive this car!
[autonomous_vehicle] Select the 3D window and then use the cursor keys to:
[autonomous_vehicle] [LEFT]/[RIGHT] — steer
[autonomous_vehicle] [UP]/[DOWN] — accelerate/slow down
INFO: Terminating controller “autonomous_vehicle”.
INFO: Starting extern controller for robot “vehicle”.

The DrivingJava code above just slams the car into a tree. But it’s a start.

Important note: Every time you run and end your Java program, the extern controller unfortunately stays connected to your previous process. The next time you run it, it tells you it can’t find a vehicle to connect to. Y̶o̶u̶ ̶h̶a̶v̶e̶ ̶t̶o̶ ̶g̶o̶ ̶t̶o̶ ̶t̶h̶e̶ ̶c̶o̶n̶t̶r̶o̶l̶l̶e̶r̶ ̶i̶n̶ ̶W̶e̶b̶o̶t̶s̶,̶ ̶s̶w̶i̶t̶c̶h̶ ̶i̶t̶ ̶t̶o̶ ̶n̶o̶n̶e̶,̶ ̶c̶l̶i̶c̶k̶ ̶O̶K̶,̶ ̶t̶h̶e̶n̶ ̶s̶w̶i̶t̶c̶h̶ ̶i̶t̶ ̶b̶a̶c̶k̶ ̶t̶o̶ ̶<̶e̶x̶t̶e̶r̶n̶>̶.̶ ̶I̶t̶’̶s̶ ̶a̶n̶n̶o̶y̶i̶n̶g̶,̶ ̶b̶u̶t̶ ̶I̶ ̶h̶a̶v̶e̶n̶’̶t̶ ̶f̶o̶u̶n̶d̶ ̶a̶ ̶w̶a̶y̶ ̶a̶r̶o̶u̶n̶d̶ ̶i̶t̶.̶

Solution: If you hit the ‘back to beginning’ button in Webots (see image above), it will reset the extern controller and end your running program at the same time.

Troubleshooting

I see no 3D screen, only a grey screen where the 3D display should be.

I had that issue too. The only thing that worked was updating my Mac to Catalina (version 10.15).

I keep getting the following error:

WARNING: Failed to attach extern robot controller: no available “<extern>” robot controller named “vehicle” found.

As noted above, this means you ran the application once, and it has since terminated. But the terminated process still has control over <extern>. Your new process couldn’t find an “available” controller, because the only one that exists is already reserved. T̶h̶e̶ ̶o̶n̶l̶y̶ ̶w̶a̶y̶ ̶I̶ ̶f̶o̶u̶n̶d̶ ̶t̶o̶ ̶f̶i̶x̶ ̶t̶h̶i̶s̶ ̶i̶s̶ ̶t̶o̶ ̶s̶e̶t̶ ̶t̶h̶e̶ ̶c̶o̶n̶t̶r̶o̶l̶l̶e̶r̶ ̶t̶o̶ ̶n̶o̶n̶e̶,̶ ̶t̶h̶e̶n̶ ̶b̶a̶c̶k̶ ̶t̶o̶ ̶<̶e̶x̶t̶e̶r̶n̶>̶ ̶a̶g̶a̶i̶n̶. If you hit the ‘back to beginning’ button in Webots, it will reset the extern controller and end your running program at the same time.

If anything else doesn’t work, leave a comment and I’ll update the article.