Reinforcement Learning With Function Approximation for Traffic Signal Control

Prashanth, LA and Bhatnagar, Shalabh (2011) Reinforcement Learning With Function Approximation for Traffic Signal Control. In: IEEE Transactions onIntelligent Transportation Systems, 12 (2, Sp.). pp. 412-421.

PDF Reinforcement.pdf - Published Version Restricted to Registered users only Download (413kB) | Request a copy

Abstract

We propose, for the first time, a reinforcement learning (RL) algorithm with function approximation for traffic signal control. Our algorithm incorporates state-action features and is easily implementable in high-dimensional settings. Prior work, e. g., the work of Abdulhai et al., on the application of RL to traffic signal control requires full-state representations and cannot be implemented, even in moderate-sized road networks, because the computational complexity exponentially grows in the numbers of lanes and junctions. We tackle this problem of the curse of dimensionality by effectively using feature-based state representations that use a broad characterization of the level of congestion as low, medium, or high. One advantage of our algorithm is that, unlike prior work based on RL, it does not require precise information on queue lengths and elapsed times at each lane but instead works with the aforementioned described features. The number of features that our algorithm requires is linear to the number of signaled lanes, thereby leading to several orders of magnitude reduction in the computational complexity. We perform implementations of our algorithm on various settings and show performance comparisons with other algorithms in the literature, including the works of Abdulhai et al. and Cools et al., as well as the fixed-timing and the longest queue algorithms. For comparison, we also develop an RL algorithm that uses full-state representation and incorporates prioritization of traffic, unlike the work of Abdulhai et al. We observe that our algorithm outperforms all the other algorithms on all the road network settings that we consider.

Item Type:	Journal Article
Publication:	IEEE Transactions onIntelligent Transportation Systems
Publisher:	IEEE
Additional Information:	Copyright 2011 IEEE. Personal use of this material is permitted.However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.
Keywords:	Q-learning with full-state representation (QTLC-FS);Q-learning with function approximation (QTLC-FA);reinforcement learning (RL);traffic signal control
Department/Centre:	Division of Electrical Sciences > Computer Science & Automation
Date Deposited:	22 Jun 2011 09:26
Last Modified:	22 Jun 2011 09:26
URI:	http://eprints.iisc.ac.in/id/eprint/38422

Actions (login required)

View Item