A simulation system invented at MIT to train driverless cars creates a photorealistic world with infinite steering possibilities, helping the cars learn to navigate a host of worse-case scenarios before cruising down real streets.

Control systems, or “controllers,” for autonomous vehicles largely rely on real-world datasets of driving trajectories from human drivers. From these data, they learn how to emulate safe steering controls in a variety of situations. But real-world data from hazardous “edge cases,” such as nearly crashing or being forced off the road or into other lanes, are — fortunately — rare.

Some computer programs, called “simulation engines,” aim to imitate these situations by rendering detailed virtual roads to help train the controllers to recover. But the learned control from simulation has never been shown to transfer to reality on a full-scale vehicle.

The MIT researchers tackle the problem with their photorealistic simulator, called Virtual Image Synthesis and Transformation for Autonomy (VISTA). It uses only a small dataset, captured by humans driving on a road, to synthesize a practically infinite number of new viewpoints from trajectories that the vehicle could take in the real world. The controller is rewarded for the distance it travels without crashing, so it must learn by itself how to reach a destination safely. In doing so, the vehicle learns to safely navigate any situation it encounters, including regaining control after swerving between lanes or recovering from near-crashes.

In tests, a controller trained within the VISTA simulator safely was able to be safely deployed onto a full-scale driverless car and to navigate through previously unseen streets. In positioning the car at off-road orientations that mimicked various near-crash situations, the controller was also able to successfully recover the car back into a safe driving trajectory within a few seconds.

“It’s tough to collect data in these edge cases that humans don’t experience on the road,” says first author Alexander Amini, a PhD student in the Computer Science and Artificial Intelligence Laboratory (CSAIL). “In our simulation, however, control systems can experience those situations, learn for themselves to recover from them, and remain robust when deployed onto vehicles in the real world.”

The work was done in collaboration with the Toyota Research Institute. Joining Amini on the paper are Igor Gilitschenski, a postdoc in CSAIL; Jacob Phillips, Julia Moseyko, and Rohan Banerjee, all undergraduates in CSAIL and the Department of Electrical Engineering and Computer Science; Sertac Karaman, an associate professor of aeronautics and astronautics; and Daniela Rus, director of CSAIL and the Andrew and Erna Viterbi Professor of Electrical Engineering and Computer Science.

Data-driven simulation

Historically, building simulation engines for training and testing autonomous vehicles has been largely a manual task. Companies and universities often employ teams of artists and engineers to sketch virtual environments, with accurate road markings, lanes, and even detailed leaves on trees. Some engines may also incorporate the physics of a car’s interaction with its environment, based on complex mathematical models.

But since there are so many different things to consider in complex real-world environments, it’s practically impossible to incorporate everything into the simulator. For that reason, there’s usually a mismatch between what controllers learn in simulation and how they operate in the real world.

Instead, the MIT researchers created what they call a “data-driven” simulation engine that synthesizes, from real data, new trajectories consistent with road appearance, as well as the distance and motion of all objects in the scene.

They first collect video data from a human driving down a few roads and feed that into the engine. For each frame, the engine projects every pixel into a type of 3D point cloud. Then, they place a virtual vehicle inside that world. When the vehicle makes a steering command, the engine synthesizes a new trajectory through the point cloud, based on the steering curve and the vehicle’s orientation and velocity.

Then, the engine uses that new trajectory to render a photorealistic scene. To do so, it uses a convolutional neural network — commonly used for image-processing tasks — to estimate a depth map, which contains information relating to the distance of objects from the controller’s viewpoint. It then combines the depth map with a technique that estimates the camera’s orientation within a 3D scene. That all helps pinpoint the vehicle’s location and relative distance from everything within the virtual simulator.

Based on that information, it reorients the original pixels to recreate a 3D representation of the world from the vehicle’s new viewpoint. It also tracks the motion of the pixels to capture the movement of the cars and people, and other moving objects, in the scene. “This is equivalent to providing the vehicle with an infinite number of possible trajectories,” Rus says. “Because when we collect physical data, we get data from the specific trajectory the car will follow. But we can modify that trajectory to cover all possible ways of and environments of driving. That’s really powerful.”

Reinforcement learning from scratch

Traditionally, researchers have been training autonomous vehicles by either following human defined rules of driving or by trying to imitate human drivers. But the researchers make their controller learn entirely from scratch under an “end-to-end” framework, meaning it takes as input only raw sensor data — such as visual observations of the road — and, from that data, predicts steering commands at outputs.

“We basically say, ‘Here’s an environment. You can do whatever you want. Just don’t crash into vehicles, and stay inside the lanes,’” Amini says.

This requires “reinforcement learning” (RL), a trial-and-error machine-learning technique that provides feedback signals whenever the car makes an error. In the researchers’ simulation engine, the controller begins by knowing nothing about how to drive, what a lane marker is, or even other vehicles look like, so it starts executing random steering angles. It gets a feedback signal only when it crashes. At that point, it gets teleported to a new simulated location and has to execute a better set of steering angles to avoid crashing again. Over 10 to 15 hours of training, it uses these sparse feedback signals to learn to travel greater and greater distances without crashing.

After successfully driving 10,000 kilometers in simulation, the authors apply that learned controller onto their full-scale autonomous vehicle in the real world. The researchers say this is the first time a controller trained using end-to-end reinforcement learning in simulation has successful been deployed onto a full-scale autonomous car. “That was surprising to us. Not only has the controller never been on a real car before, but it’s also never even seen the roads before and has no prior knowledge on how humans drive,” Amini says.

Forcing the controller to run through all types of driving scenarios enabled it to regain control from disorienting positions — such as being half off the road or into another lane — and steer back into the correct lane within several seconds. “And other state-of-the-art controllers all tragically failed at that, because they never saw any data like this in training,” Amini says.

Next, the researchers hope to simulate all types of road conditions from a single driving trajectory, such as night and day, and sunny and rainy weather. They also hope to simulate more complex interactions with other vehicles on the road. “What if other cars start moving and jump in front of the vehicle?” Rus says. “Those are complex, real-world interactions we want to start testing.”

The post Training Driverless Cars Before They Hit the Highway appeared first on Artificial Intelligence.

Chinese autonomous vehicle startup Pony.ai said Wednesday that it has received a $400 million investment from Japanese automaker Toyota, upping the ante in its push to develop automated vehicles.

The Guangzhou-based company will use the fresh capital to boost the development and commercialization of driverless cars, which will mainly feature what’s called “Level 4 autonomy.”

The Society of Automotive Engineers divides autonomous driving technology into six levels ranging from 0 to 5. Level 4 autonomy allows a car to be in almost total control all the time without any human intervention.

The new investment marks an extension of the two companies’ partnership initially forged in August 2019, when Pony.ai and Toyota joined forces to conduct road tests of autonomous vehicles in Beijing and Shanghai using the Japanese automaker’s Lexus-branded cars.

According to a statement published on Pony.ai’s public WeChat account, the two partners will also explore a possible partnership on mobility services.

In 2018, Pony.ai started piloting its PonyPilot project in the southern city of Guangzhou, testing a fleet of 100 autonomous cars used for taxi-hailing.

Toyota is also a financial backer of Chinese ride-hailing giant Didi Chuxing. In July last year, the Japanese carmaker announced that it would invest $600 million in Didi and their new joint venture for mobility services. Didi also plans to test its robotaxi services in Shanghai this year.

Earlier this week, China’s National Development and Reform Commission and several other government agencies published an autonomous vehicle development plan, setting a goal for 2025, by which time the country should achieve mass production of vehicles with “conditional” self-driving capabilities.

The post China’s Pony.ai Secures $400 Million from Toyota to Develop Driverless Cars appeared first on Artificial Intelligence.

Autonomous cars have been recently hitting the headlines and dominating tech-talks. It’s seen as a post-Uber disruption to public commuting and transportation of goods. It is surely not a figment of imagination in the age of artificial intelligence (AI) which is being used to complement driverless cars. The combined might of AI and driverless technologies is a formidable force.

The likes of Waymo, Tesla, etc. are heavily invested in driverless cars. In fact, Waymo has been testing the driverless car in Phoenix. Tesla has already implemented a couple of “autopiloting” features in its cars.

But before we get into further details, let us discuss what autonomous means. There are various levels of vehicle automation:

• Automation for driver assistance – It is a preliminary level or starting point of car automation where the system assists the driver but does not take control of the car. E.g. parking sensors.
• Partially automated driving – the system takes partial control, but the driver is primarily responsible for the operation of the vehicle.
• Highly automated driving – allows users to let the system take control of the vehicle for a longer duration of time. E.g. on the highway.
• Fully automated driving – The system is responsible for driving the vehicle without interference from any human. However, the human presence is still needed.
• Completely automated car – the vehicle can completely navigate its way through from a point to another without any assistance from a driver.

Depending on the level of automation, the definition of autonomous varies. While automation for driver assistance and partially automated cars are in commercial use, the remaining stages are still under test conditions.

For us to achieve the remaining stages of automation or even come close, AI is the stimulus that is being used. For the purpose of this article, let us discuss the impact of AI in highly automated driving and completely or fully automated vehicles, and how the power of AI is being harnessed to bring it to reality.

The role of artificial Intelligence in complementing the use of autonomous cars

Subject to regulatory and social acceptance, the impact of completely autonomous cars is not limited to the disruption of the public transport system. From a macro level, it impacts urbanization, township planning, food delivery, and possibly shake the ever-increasing real-estate market.

For AI to work, it needs IoT devices (such as radars, ultrasound, radar, cameras, LiDAR, accelerometers, and gyroscopes) that augment real-time operating environment and positioning of the vehicle.

Having discussed the potential of AI, let’s talk about the top four areas where AI is seen as a gamechanger towards the success of autonomous vehicles.

1. AI for self-driving car safety

Before AI completely takes over the driver’s seat, it is being used as a co-pilot to gain the confidence of the users, regulators, manufacturers. By analyzing data feeds across its sensors, AI can be handy in situations where flesh and blood drivers are prone to making human errors.

AI can score very high in areas such as:

•  Emergency control of the vehicle
  • Cross-traffic detection
  • Syncing with traffic signals
  • Breaking in cases of emergencies
  • Active monitoring of blind spots
  • Altering the driver in case he or she is distracted

The quantum of processing power needed to drive a vehicle is enormous as you do not have control of your external environment which has countless variables – it needs a lot of learning. There are numerous companies testing AI’s applicability in driving, but the most noteworthy achievements have been made by Waymo and Tesla.

Waymo’s AI algorithms are fed with real-time data from sensors, GPS, radar, lidar, cameras, and cloud services. These data are processed to produce control signals that are used to operate the car.

1. Curated cloud services targeted for individuals

AI can be used to accurately gauge the physical condition of the vehicle. Data gathered from the usage can be processed for both:

• Predictive maintenance
  Prescriptive maintenance

This way, drivers will have an easier time finding a car warranty plan that is cost-effective and meets their particular needs, and that also reflects the car’s current condition.

1. Accurate feed for regulators and insurance companies

Data from automated cars can be used to determine traffic violations and claims. From an insurance perspective, AI can be of help to determine the:

• Driver risk assessment – using AI, a driver’s behavior can be accurately gauged and based on the risk profile the premium can be charged
• Ease of claim – data from the vehicle and can be used for faster processing of claims in case of accidents. Art Financial’s AI-based video app Dingsunbao 2.0 allows users to access their auto damage.

1. Monitoring the driver and user behavior

The applicability of AI in autonomous cars is not limited to stricter requirements such as safety but also fills the fun quotient. AI can be used for a host of infotainment features in the car.

AI is helpful to provide customized infotainment during the travel. Based on the data collected over time, AI can predict and prescribe preferences based on user behavior. It could include:

• Seat position adjustment
• Mirror adjustment
• Regulating the air-conditioning
• Songs to be played

AI is gaining in prominence with each passing day. Governments too have jumped into the race to woo investors to bring AI-based driverless cars for commercial use.

In August 2018, the British Government unveiled plans for an AI simulator, intended for the purpose of attracting companies as a favorable destination for testing self-driving cars. Named as OmniCAV, the simulator can recreate a virtual version of 32km of Oxfordshire roads.

The world is changing faster than imagined, and AI is getting smarter with every passing day.We are just around the corner to witness the post-Uber era so hold your breath.

The post How AI Is Paving the Way for Autonomous Cars appeared first on Artificial Intelligence.