Decision-Making and Control for Freeway On-Ramp Merging Using Deep Reinforcement Learning

This work studies high-speed on-ramp merging decision-making and control for an automated vehicle using deep reinforcement learning (DRL). We consider no vehicle-to-everything (V2X) wireless communication and the merging vehicle relies on its own sensors to obtain the states of other vehicles and the road information to merge from on-ramp to the main road. We consider the states of five vehicles as the environment state of the reinforcement learning training framework: the merging vehicle, and two preceding and two following vehicles within the sensing range of the merging vehicle when it is or is projected on the main road. The control action of the reinforcement learning training framework is the acceleration command for the merging vehicle. We use Deep Deterministic Policy Gradient (DDPG) as the DRL algorithm for continuous control, assuming there exists an optimal deterministic policy to match the state to the action. The DRL rewards encourage merging midway between two main-road vehicles with the same speed as the first preceding one, and penalize hard braking, stops, and collisions. When testing the trained policy, we observed 1 collision out of 16975 testing episodes (0.006% collision rate). By analyzing the merging vehicle's behaviors, we found that it learned human-like behaviors such as slowing down to merge behind or speeding up to merge ahead a main-road vehicle. For the only collision case, we found that the merging vehicle kept shifting between slowing down and speeding up, suggesting that it might be trapped at a bifurcation state.