Motivations, Opportunities and Challenges
Dr. N. Kemal Ure – Director of AI, Eatron Technologies

Part 4: Eatronian Approach to RL and Autonomous Driving

In this final part of the series of blog posts on Reinforcement Learning (RL), I would like to share our own work at Eatron in applying RL to real-world autonomous driving.

1.Overcoming Limitations of Rule Based Lane Changing in Noisy Environments

Automated lane changing is one of the most useful functionalities in self-driving cars. Good tactical decisions for lane changing enables not getting stuck behind slower drivers, while also ensuring safety by avoiding collisions. Rule based lane changing algorithms achieve this by tuning decisions to designer specified conditions, such as triggering a lane change if the speed and position of the vehicles surrounding the ego vehicle are above or below a set threshold. Our work [1] applies RL to tactical lane changing problem, and shows that RL is superior to rule based approaches, especially when the perception is noisy and the surrounding traffic is difficult to predict.

2.Learning How to Simulate Real-World Traffic for Tactical Decision Making

Most RL algorithms, including the ones we develop at Eatron, are trained on simulators first, and then fine-tuned on real-world road tests. Hence the performance of the RL agent is mostly determined by the realism of the simulator. Although the visual quality of the simulators improved greatly over the years, employing realistic traffic dynamics is still an open challenge. We solve this problem by learning generative driving models from real-world traffic data [2], and then use these models in our simulator to train RL agents in realistic simulated traffic dynamics. Results show that agents trained in this manner generalizes much better across a variety of driving scenarios. Hence the future of RL is not only about transferring simulations to real life, but also about transferring real life to simulators.

3.Ensuring Safety Through Fusion of Model Based Feedback Control and Reinforcement Learning

Stability and robustness are integral properties of all automatic control systems. Unfortunately, data-driven control methods, such as RL do not offer the same type of guarantees. That being said, it is possible to combine two approaches to get the best of both worlds. One of our earlier works at Eatron [3], shows that it is possible to design a high-level RL agent, which coordinates/supervises a number of model predictive control (MPC) based low level controllers to achieve superior performance in automated driving tasks. Since MPC controllers are stable by design, this division of labor between RL and model-based control approaches enable data-driven methods to outperform classical approaches without sacrificing stability guarantees. Our later work [4] shows that it is also possible to improve the safety of RL based lane changing by receiving feedback from safety metrics.

4.Solving Highly Complex Driving Tasks Through Curriculum Reinforcement Learning

Although training an RL agent for solving a relatively simply driving scenario is a not big hurdle anymore, the task can get daunting when we factor in various weather conditions and complex road geometry. Training an RL agent from scratch in a highly complicated environment requires humongous training times and often results in unsatisfying performance. Curriculum Reinforcement Learning (CRL), attempts to solve this problem by training the agent in simpler environments first, and then transfer the performance to harder tasks. Our most recent effort [5], demonstrates that CRL is a highly promising direction for the future of RL based self-driving. Our results demonstrate that training the agent in relatively simpler road geometries (such as straight roads vs. curvy ones) and less adversarial weather conditions (such as sunny vs. snowy weather) first and then transferring the learned behaviors to more complex tasks achieve far superior performance compared to training the agent from scratch in difficult environments. We think that all RL training in the future will involve a curriculum of some sort.

Epilogue

I hope you enjoyed this series of blog posts on applied RL. All being said, RL is still at its infancy for applications to real-world problems. Me and my colleagues at Eatron are firm believers of the potential of RL in autonomous driving and we are excited to push the limits of this technology. I am looking forward to share more results with you and talk to you more about other exciting AI work we do at Eatron in future posts.

References

[1] Alizadeh, A., Moghadam, M., Bicer, Y., Ure, N. K., Yavas, M. U., Kurtulus, C., Tactical Lane Changing with Deep Reinforcement Learning in Dynamic and Uncertain Traffic Scenarios, IEEE Intelligent Transportation Systems Conference (ITSC), 2019.
[2] Ozturk, A., Gunel, M. B., Yavas, M. U., Ure. N. K., Development of A Stochastic Traffic Environment with Generative Time-Series Models for Improving Generalization Capabilities of Autonomous Driving Agents, IEEE Intelligent Vehicles Conference (IV), 2020.
[3] Ure N. K., Yavas, M. U., Alizadeh, A., Kurtulus, C., Enhancing Situational Awareness and Performance of Adaptive Cruise Control through Model Predictive Control and Deep Reinforcement Learning, IEEE Intelligent Vehicles Conference (IV), 2019.
[4] Yavas, M. U., Kumbasar, T., Ure. N. K., Sample Efficient Deep Q Learning with a Safety Feedback Reward, IEEE Intelligent Vehicles Conference (IV), 2020.
[5] Ozturk, A., Gunel, M. B., Dagdanov, R., Vural, M. E., Yurdakul, F., Dal, M., & Ure, N. K. (2021). Investigating Value of Curriculum Reinforcement Learning in Autonomous Driving Under Diverse Road and Weather Conditions. arXiv preprint arXiv:2103.07903.