TRAFFIC control remains a hard problem for researchers and engineers, due to a number of difficulties. The major two are the modeling difficulty and the optimization difficulty.

First, transportation systems are usually distributed, hybrid and complex^{[1, 2, 3, 4, 5]}. How to accurately and also conveniently describe the dynamics of transportation systems still leaves not fully solved. As pointed out in ^[5] and ^[6], most recent control systems aim to predict future states of transportation systems and make appropriate signal plans in advance. This requirement highlights the importance and hardness of transportation systems' modeling.

There are mainly two kinds of approaches to solve this difficulty^[5]. One kind is the flow model based approaches, which formulate analytical models to describe the dynamics of macroscopic traffic flow measured at different locations. For example, cell transmission models (CTM) and its variations were frequently considered in reports due to its simplicity and effectiveness^[7]. However, when traffic scenarios are complex, the modeling costs and errors need to be carefully considered.

The other kind is the simulation based approaches, which estimate/predict future traffic flow states using either artificial intelligence learning or simulations^{[8, 9, 10]}. Artificial intelligence models learn and reproduce macroscopic traffic flow dynamics based on recorded traffic flow measurements. In contrast, simulations describe and reproduce the actions of individual microscopic traffic participators, which as a result provides flexible power to better describe macroscopic traffic flow dynamics. However, both artificial intelligence learning and simulation are time-consuming. The tuning of the control performance also becomes hard, since no theoretical analysis tool can be straightforwardly applied for these approaches.

Second, when traffic flow descriptions are established, how to determine the best signal plans becomes another problem. For flow model based approaches, we can use mathematical programming methods to solve the given objective functions (usually in terms of delay or queue length) with the explicitly formulated constraints derived from analytical models^{[7, 11, 12, 13]}.

Differently, for artificial intelligence learning and simulation based approaches, we will reverse the cause-effect based on the learned relationships between control actions and their effect on traffic flows. The try-and-test methods are then used to seek a (sub)optimal signal plan, based on the predicted or simulated effects of the assumed control actions. In literatures, heuristic optimization algorithms, such as genetic algorithms (GA)^[14] were often applied to accelerate the seeking process. However, the converging speeds of such algorithms are still questionable in many cases.

In this paper, we focus on reinforcement learning approach for traffic signal timing problems^{[15, 16, 17, 18]}. Reinforcement learning approach implicitly models the dynamics of complex systems by learning the control actions and the resulted changes of traffic flow. Meanwhile, it seeks the (sub)optimal signal plan from the learned input-output pairs.

The major difficulty of reinforcement learning for traffic signal timing lies in the exponentially expanding complexity of signal timing design with the number of the considered traffic flow states and control actions.

Recently, a new method is proposed to simultaneously solve the modeling and optimization problems of complex systems by using the so called deep Q network^[19]. The deep Q networks indeed combine two hot tools: reinforcement learning^[20] and deep learning^{[21, 22, 23]}. Herein, deep learning uses multiple layers of artificial neural networks to learn the implicit maximum discounted future reward when we perform a special action given a special state.

In this paper, we examine the feasibility and effectiveness of this deep reinforcement learning method in building traffic signal timing plan. Although a few approaches have been proposed to approximate the maximum discounted future reward^[16], our numerical tests show that deep networks provides a more convenient and powerful tool to reach this goal.

To give a more detailed explanation of our new designs, we will first present the new signal timing approach using deep learning in Section II; then in Section III, we provide some simulation results to show the effectiveness of the proposed new approach; finally, we conclude the paper in Section IV.

Let us define $s_t$ as the system states at time $t$, $a_t$ as the action at time $t$. $p$ denotes a policy, the rule how an action is chosen in each state. It characterizes a mapping from the state set $S$ to the action set $A$.

At each time step, the controller observes state $s_t $ and chooses an action $a_t$ to perform. The state of the system will be altered from $s_t$ to $s_{t+1}$ in the next time $t+1$, and a real-valued reward $r_t$ will be received. All the records $\left( {s_t , a_t , r_t , s_{t+1}} \right)$ will be stored into a replay memory to find the best policy that gives the highest reward.

We define the discounted future reward $R_t $ from time $t$ as

where $\gamma \in \left[{0, 1} \right]$ are the discount factor.

Suppose we always choose an action that maximizes the (discounted) future reward. Particularly, we define the so called Q-function $Q \left( {s_t , a_t } \right)$ that represents the maximum discounted future reward when we perform action $a_t$ in state $s_t$, and continue optimally from time $t$ onward. That is,

Equation (2) has the property of the Bellman equation. So we can iteratively approximate the Q-function using

This form can be explained as the maximum future reward for the current state and action is the current reward plus the maximum future reward for the next state.

If $Q \left( {s_t , a_t } \right)$ is known, we can directly solve (3) to find the optimal policy. However, $Q \left( {s_t , a_t } \right)$ is usually unknown. So, we need to estimate $Q \left( {s_t , a_t } \right)$ in the learning process. In early stages of learning, $Q \left( {s_t , a_t } \right)$ may be completely wrong. However, the estimation gets more and more accurate with every iteration of updating.

For traffic signal timing problems for intersections, traffic sensors measure the states (speed, queueing length, etc.) of traffic flows. We consider all these measures as system states. The actions here are giving rights of way for each specified traffic stream. So, after one action $a_t $ is performed, traffic states will be changed from $s_t $ to $s_{t+1}$. We also calculate a reward $r_t $ (the performance like traffic delay and queue length) after each action.

In conventional approaches, the Q-function is implemented using a table or a function approximator^{[17, 18]}. However, the state spaces of traffic signal timing problems is so huge that we can hardly solve the formulated reinforcement learning problem within a finite time with a table based Q learning method; and the traditional function approximator based Q learning method can hardly capture dynamics of traffic flow.

In contrast, we use the deep stacked autoencoders (SAE) neural network^{[23, 24]} to estimate the Q-function here. This neural network takes the state as input and outputs the Q-value for each possible action. Fig. 1 gives an illustration of its structure. As its name indicates, the SAE neural network contains multiple hidden layers of autoencoders where the outputs of each layer is wired to the inputs of the successive layer.

Fig. 1

	Download: JPG larger image
Fig. 1 The deep SAE neural network for approximating Q function.

Autoencoders are building blocks of creating the deep SAE neural network. An autoencoder is a neural network that sets the target output to be equal to the input. Fig. 2 gives an illustration of an autoencoder, which has three layers: one input layer, one hidden layer, and one output layer.

Fig. 2

	Download: JPG larger image
Fig. 2 An autoencoder.

Given an input $x$, an autoencoder first encodes the input $x$ to a hidden representation $y\left( x \right)$ based on (4), and then decodes the representation $y\left( x \right)$ back into a reconstruction $z\left( x \right)$ computed as in (5). The objective of an autoencoder is to learn a mapping that produces $z\left( x \right)\approx x$.

where $W_1 $ and $W_2 $ are weight matrices, $b$ and $c$ are bias vectors.

The model parameters $\theta _{ae}$ of an auctoencoder can be obtained by minimizing the reconstruction error $L\left( {X, Z}\right)$:

where $\beta $ is the weight of the sparsity penalty term, $H_D $ is the number of hidden units, $\rho $ is a sparsity parameter, and $\hat {\rho }_j =\frac{1}{N}\sum\nolimits_{i=1}^N {y_j \left( {x^{(i)}} \right)} $is the average activation of the hidden unit $j$ over the training set.

The Kullback-Leibler (KL) divergence $\mbox{KL}\left( {\rho \vert \vert \hat {\rho }_j } \right)$ is defined as

We take greedy layerwise approach to train each layer in turn, so that autoencoders can be ``stacked" in a greedy layerwise fashion for pretraining (initializing) the weights of a deep network. The obtained approximation of Q function is written as $Q\left( {s_t , a_t , \theta } \right)$, with $\theta $ characterizes the values of weights and other parameters of SAE network.

During Q-learning, this deep SAE is trained by minimizing the following simple loss function over samples of experience till time $t$:

where a record $\left( {{s}'_t , {a}'_t , {r}'_t , {s}'_{t+1} } \right)$ is drawn randomly from the replay memory.

Once we had set up a policy after we observe a certain number of records, we will face the explore-exploit dilemma: Should we exploit the current working policy or explore other (possibly better) policies. To solve this problem, we apply the $\varepsilon $-greedy strategy. That is, with probability $\varepsilon$ we select a random action and with probability $1-\varepsilon$ to follow the current policy.

In summary, the detailed learning algorithm is shown in Algorithm 1.

Algorithm 1

Algorithm 1 The deep reinforcement learning algorithm for traffic signal control

Algorithm 1 The deep reinforcement learning algorithm for traffic signal control

In pretraining procedure, we take greedy layerwise approach to train each layer in turn, so that autoencoders can be ``stacked" in a greedy layerwise fashion for pretraining (initializing) the weights of a deep network. The obtained approximation of Q function is written as $Q\left( {s_t, a_t, \theta } \right)$, with $\theta $ characterizes the values of weights and other parameters of SAE network.

During Q-learning, this deep SAE is trained by minimizing the following simple loss function over samples of experience till time $t$.

To compare the proposed deep reinforcement learning traffic controller and the conventional traffic controller, we design the following cases.

The geometry of the studied crossing is shown in Fig. 3. Each branch of the intersection has four lanes (two lanes for coming vehicles and two for leaving ones). In other words, there are 8 incoming lanes connected to this intersection. No left-turn, right-turn or U-turn is allowed.

Fig. 3

	Download: JPG larger image
Fig. 3 The geometry of the simulated intersection.

Suppose there are two phases here: one phase allows vehicles running either north-to-south or south-to-north to enter the intersection, the other phase allows vehicles running either east-to-west or west-to-east to enter the intersection. We assume that there is no red-clearance time here, so one phase appears immediately after another phase. The minimum green time is 15 s for both two phases.

We measure the queueing lengths of the 8 incoming lanes per 5 s, and the values are denoted $q_t^{e-w, 1} $, $q_t^{e-w, 2} $, $q_t^{w-e, 1} $, $q_t^{w-e, 2} $, $q_t^{n-s, 1} $, $q_t^{n-s, 2} $, $q_t^{s-n, 1} $, $q_t^{s-n, 2} $ at sampling time $t$. We take the measured queueing lengths during the last four samples as the state of the system at time $t$. There are only two actions: remain the current phase, change to the other phase (it must be pointed out that the second action might be forbidden sometimes if the current phase has not yet lasted for 15 s).

We use PARAMICS to simulate the traffic flow dynamics with given traffic states and actions. The reward is defined as the absolute value of the difference between north-south/south-north direction and east-west/west-east direction, i.e.

For both conventional reinforcement learning and the new deep reinforcement learning traffic controller, we pretrain the controller with randomly generated traffic demands shown in Table I. The minimum traffic demand for each approach is 100 veh/h and the maximum is 2000 veh/h.

Table Ⅰ

Table Ⅰ THE TRAFFIC DEMANDS USED FOR PRETRAINING

Table Ⅰ THE TRAFFIC DEMANDS USED FOR PRETRAINING

In both pretraining and testing cases, we set $\gamma =0.1$ and let $\varepsilon $ vary as

It should be pointed out that we had tested other settings of $\gamma $ and $\varepsilon $. Results indicate that the performance difference between conventional and new deep reinforcement learning methods remained unchanged.

For the new deep reinforcement learning traffic controller, we choose a four-layer SAE neural network which has two hidden layers. There are 32 neurons in the input layer, since we input the 8 queueing length values during the last four sampling intervals. There are just 2 neurons in the output layer, since we only have two actions. There are 16 and 4 neurons in the hidden two layers, respectively. The activation function for hidden layers is the sigmoid function.

Fig. 4 gives the traffic demands used for testing. Fig. 5 and Fig. 6 give the comparison between the delay and the number of stops queueing lengths observed when deep reinforcement learning method and conventional ordinary reinforcement learning method are used, respectively.

Fig. 4

	Download: JPG larger image
Fig. 4 The traffic demands used for testing.

Fig. 5

	Download: JPG larger image
Fig. 5 The delay observed when deep reinforcement learning method and conventional reinforcement learning method are used.

Fig. 6

	Download: JPG larger image
Fig. 6 The number of stops observed when deep reinforcement learning method and conventional reinforcement learning method are used.

Considering the uncertainty of simulation with randomness, we further compare the mean values of the obtained delay and the number of fully stopped vehicles of two reinforcement learning traffic controller during 20 times of simulations.

We find that the average delay can be reduced about 14 ${\%}$ if the deep reinforcement learning method is used instead of conventional ordinary reinforcement learning method. In morning peak hours, a vehicle may spend about 13 s less to pass the studied intersection.

Moreover, the number of fully stopped vehicles is reduced by 1020 vehicles when the deep reinforcement learning method is applied. Particularly, the number of fully stopped vehicles is reduced by 410 vehicles in morning peak hours.

Let $Qns$ denotes the queue length in the N-S direction, $Qwe$ denote the queue length in the W-E direction Fig. 7 shows the box plot of queue lengths in the N-S and W-E directions generated by the conventional reinforcement learning method and the deep reinforcement learning method proposed in this paper. Clearly, the queue length generated by the deep reinforcement learning method is less than that generated by the conventional reinforcement learning method.

Fig. 7

	Download: JPG larger image
Fig. 7 Box plots of queue lengths of the deep reinforcement learning method and the conventional reinforcement learning method in the test scenario.

Fig. 8 shows the scatter plots of queue states in the test scenario. We assess the queue balance using the linear regression $R^{2}$ of the queue states. As indicated in ^[25] and ^[26], a larger $R^{2}$ value indicates more balanced queues. We can observe that the conventional reinforcement learning method and the deep reinforcement learning method proposed in this paper can have similar effects on producing balanced queues in the N-S and W-E directions. However, as indicated in Fig. 5, the deep reinforcement learning control can maintain less queue length than the conventional reinforcement learning method. Nevertheless, the conventional reinforcement learning method sometimes generates extremely long queue lengths.

Fig. 8

	Download: JPG larger image
Fig. 8 Scatter plots of queue states of the deep reinforcement learning method and the conventional reinforcement learning method in the test scenario.

Reinforcement learning had gained increasing interests in traffic control field. Different from traffic flow model based control strategy, it simultaneously learn the dynamics of traffic systems and the optimal control plan by implicitly modeling the control actions and the change of system states.

In this paper, we apply the deep reinforcement learning method^[19] developed recently and show that it notably outperforms conventional approaches in finding a better signal timing plan. We believe the combination of such deep knowledge representation, deep reinforcement learning method and parallel intelligent transportation systems^{[8, 9, 10, 27, 28, 29, 30, 31, 32]} may have great potential to change the development course of next-generation ITS.