Structured Exploration: Bootstrapping Reinforcement Learning With Sub-Optimal Policies for Autonomous Driving in Complex Traffic Scenarios

Details

• Creators:
  Redmill, Keith A. ; Zhang, Zhihao ; Yurtsever, Ekim
• Corporate Creators:
  Ohio State University. Center for Automotive Research ; Carnegie Mellon University. Traffic21 Institute. Safety21 University Transportation Center (UTC)
• Corporate Contributors:
  United States. Department of Transportation. University Transportation Centers (UTC) Program ; United States. Department of Transportation. Office of the Assistant Secretary for Research and Technology
• Subject/TRT Terms:
  • Automated vehicle control
  • Intelligent vehicles
  • Decision making
  • Optimization
  • Policy
  • Multilane highways
  • Algorithms
  • Learning
  • Training
  • Driving behavior
  More +
• Resource Type:
  Tech Report
• Geographical Coverage:
  United States
• Edition:
  Final Report: July 1, 2025 to July 31, 2026
• Corporate Publisher:
  Carnegie Mellon University. Traffic21 Institute. Safety21 University Transportation Center (UTC)
• Description:
  Automated vehicle control using reinforcement learning (RL) has attracted significant attention due to its potential to learn driving policies through environment interaction. However, RL agents often face training challenges in sample efficiency and effective exploration, especially in complex driving scenarios with delayed rewards, making it difficult to discover an optimal driving strategy that can escape myopic behaviors. To address these issues, we propose guiding the training of the RL driving agent with a demonstration policy that need not be a highly optimized or expert-level controller. Specifically, we integrate a rule-based lane-change controller, which embeds human-like driving heuristics, with the Soft Actor-Critic (SAC) algorithm to enhance exploration and learning efficiency. The suboptimal controller is employed both as a soft constraint during the early stages of policy training and as an additional source of training samples when populating the replay buffer. We demonstrate this approach in a multi-lane highway trap escape overtaking scenario with challenging traffic patterns that create strong exploration barriers, presenting results that consistently outperform a number of standard RL algorithms.
• Format:
  PDF
• Funding:
  69A3552344811
• Download URL:
  https://rosap.ntl.bts.gov/view/dot/93544/dot_93544_DS1.pdf
• File Type:
  [PDF - 7.66 MB]
• Collection(s):
  Autonomous Vehicles ; University Transportation Centers
• Main Document Checksum:
  urn:sha-512:8f4fd76c5bde604938a2b70e95666a4605f38cb2ea9c3da86bc689208ca04e20cff1682c52816570a6b6907b60afa62408258c697dae273de799e2147ec7f1ad