Autonomous Driving System Using Proximal Policy Optimization in Deep Reinforcement Learning

Imam Noerhenda Yazid, Ema Rachmawati


Autonomous driving is one solution that can minimize and even prevent accidents. In autonomous driving, the vehicle must know the surrounding environment and move under the provisions and situations. We build an autonomous driving system using Proximal Policy Optimization (PPO) in Deep Reinforcement Learning, with PPO acting as an instinct for the agent to choose an action. The instinct will be updated continuously until the agent reaches the destination from the initial point. We use five sensory inputs for the agent to accelerate, turn the steer, hit the brakes, avoid the walls, detect the initial point, and reach the destination point. We evaluated our proposed autonomous driving system in a simulation environment with several branching tracks, reflecting a real-world setting. For our driving simulation purpose in this research, we use the Unity3D engine to construct the dataset (in the form of a road track) and the agent model (in the form of a car). Our experimental results firmly indicate our agent can successfully control a vehicle to navigate to the destination point.


Autonomous driving; Deep reinforcement learning; Proximal policy optimization; Agent; Unity3D


Volvo Trucks Accident Research Team, “Volvo Trucks Safety Report 2017,” 2017. [Online]. Available:

K. Bengler, K. Dietmayer, B. Färber, M. Maurer, C. Stiller, and H. Winner, “Three Decades of Driver Assistance Systems Review and Future Perspectives,” IEEE Intelligent Transportation Systems Magazine, vol. 6, no. 4, pp. 6–22, 2014.

L. Fridman et al., “MIT Advanced Vehicle Technology Study: Large-Scale Naturalistic Driving Study of Driver Behavior and Interaction With Automation.,” IEEE Access, vol. 7, pp. 102021–102038, 2019, doi: 10.1109/ACCESS.2019.2926040.

S. Grigorescu, B. Trasnea, T. Cocias, and G. Macesanu, “A survey of deep learning techniques for autonomous driving,” Journal of Field Robotics, vol. 37, no. 3, 2020, doi: 10.1002/rob.21918.

Y. Tian, K. Pei, S. Jana, and B. Ray, “DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars,” Proceedings of the 40th International Conference on Software Engineering, p. 12, doi: 10.1145/3180155.

L. Li, K. Ota, and M. Dong, “Humanlike Driving: Empirical Decision-Making System for Autonomous Vehicles,” IEEE Transactions on Vehicular Technology, vol. 67, no. 8, 2018, doi: 10.1109/TVT.2018.2822762.

B. Wu, F. Iandola, P. H. Jin, and K. Keutzer, “SqueezeDet: Unified, Small, Low Power Fully Convolutional Neural Networks for Real-Time Object Detection for Autonomous Driving,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2017, vol. 2017-July. doi: 10.1109/CVPRW.2017.60.

P. J. Navarro, L. Miller, F. Rosique, C. Fernández-Isla, and A. Gila-Navarro, “End-to-end deep neural network architectures for speed and steering wheel angle prediction in autonomous driving,” Electronics (Switzerland), vol. 10, no. 11, 2021, doi: 10.3390/electronics10111266.

M. Teti, W. E. Hahn, S. Martin, C. Teti, and E. Barenholtz, “A controlled investigation of behaviorally-cloned deep neural network behaviors in an autonomous steering task,” Robotics and Autonomous Systems, vol. 142, 2021, doi: 10.1016/j.robot.2021.103780.

Y. Dai and S. G. Lee, “Perception, Planning and Control for Self-Driving System Based on On-board Sensors,” Advances in Mechanical Engineering, vol. 12, no. 9, 2020, doi: 10.1177/1687814020956494.

B. R. Kiran et al., “Deep Reinforcement Learning for Autonomous Driving: A Survey,” IEEE Transactions on Intelligent Transportation Systems, 2021, doi: 10.1109/TITS.2021.3054625.

F. Ye, S. Zhang, P. Wang, and C. Y. Chan, “A survey of deep reinforcement learning algorithms for motion planning and control of autonomous vehicles,” IEEE Intelligent Vehicles Symposium, Proceedings, vol. 2021-July, pp. 1073–1080, Jul. 2021, doi: 10.1109/IV48863.2021.9575880.

A. el Sallab, M. Abdou, E. Perot, and S. Yogamani, “Deep Reinforcement Learning framework for Autonomous Driving,” IS and T International Symposium on Electronic Imaging Science and Technology, pp. 70–76, Apr. 2017, doi: 10.2352/issn.2470-1173.2017.19.avm-023.

M. Legrand, “Deep reinforcement learning for autonomous vehicle control among human drivers,” 2017.

A. Yu, “Deep Reinforcement Learning for Simulated Autonomous Vehicle Control,” Course Project Reports, pp. 1–7, 2016, doi: 10.1016/0141-1136(95)00078-X.

E. Yurtsever, L. Capito, K. Redmill, and U. Ozgune, “Integrating Deep Reinforcement Learning with Model-based Path Planners for Automated Driving,” IEEE Intelligent Vehicles Symposium, Proceedings, pp. 1311–1316, 2020, doi: 10.1109/IV47402.2020.9304735.

S. Wang, D. Jia, and X. Weng, “Deep Reinforcement Learning for Autonomous Driving,” [RecSys2018]Proceedings of the 12th ACM conference on Recommender systems, pp. 95–103, 2018, Accessed: Feb. 25, 2022. [Online]. Available:

Ó. Pérez-Gil et al., “Deep reinforcement learning based control for Autonomous Vehicles in CARLA,” Multimedia Tools and Applications, vol. 81, no. 3, pp. 3553–3576, Jan. 2022, doi: 10.1007/S11042-021-11437-3/FIGURES/10.

Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel, “Benchmarking Deep Reinforcement Learning for Continuous Control,” 33rd International Conference on Machine Learning, ICML 2016, vol. 3, pp. 2001–2014, Apr. 2016, Accessed: Mar. 01, 2022. [Online]. Available:

J. Edu et al., “A Theoretical Analysis of Deep Q-Learning Jianqing Fan,” Proceedings of Machine Learning Research, vol. 120, pp. 1–4, 2020.

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. K. Openai, “Proximal Policy Optimization Algorithms.”

D. Q. Tran and S. H. Bae, “Proximal Policy Optimization Through a Deep Reinforcement Learning Framework for Multiple Autonomous Vehicles at a Non-Signalized Intersection,” Applied Sciences 2020, Vol. 10, Page 5722, vol. 10, no. 16, p. 5722, Aug. 2020, doi: 10.3390/APP10165722.

Y. Wu, S. Liao, X. Liu, Z. Li, and R. Lu, “Deep Reinforcement Learning on Autonomous Driving Policy With Auxiliary Critic Network,” IEEE transactions on neural networks and learning systems, vol. PP, 2021, doi: 10.1109/TNNLS.2021.3116063.

F. Ye, X. Cheng, P. Wang, C. Y. Chan, and J. Zhang, “Automated Lane Change Strategy using Proximal Policy Optimization-based Deep Reinforcement Learning,” IEEE Intelligent Vehicles Symposium, Proceedings, pp. 1746–1752, Feb. 2020, doi: 10.1109/IV47402.2020.9304668.

A. Juliani et al., “Unity: A General Platform for Intelligent Agents,” Sep. 2018, Accessed: Feb. 24, 2022. [Online]. Available:

G. Rong et al., “LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving,” 2020 IEEE 23rd International Conference on Intelligent Transportation Systems, ITSC 2020, May 2020, doi: 10.1109/ITSC45102.2020.9294422.



  • There are currently no refbacks.

View IJAI Stats

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.