A Ramp Metering Method Based on Congestion Status in the Urban Freeway

In ramp metering methods, the ALINEA algorithm is a very effective way and has been applied widely. But the critical occupancy in ALINEA algorithm is often difficult to obtain and not particularly accurate. It will greatly affect the performance of ALINEA algorithm. In this paper, an improved ALINEA algorithm, named CS-ALINEA, is proposed. In this method, the traffic flow is used to replace the occupancy as the control parameter and the control rate can be selected according to the congestion status reclassified adaptively. In the existing ramp control methods, to guarantee the traffic efficiency of mainstream, the impact of ramp overflow on ground road traffic is often ignored. In order to resolve this issue, the segmented control method is adopted in this paper. When the ramp queuing length exceeds the critical queue length, the signal timing scheme is adjusted by selecting the control rate to avoid the ramp overflow. The SUMO simulation platform is used to simulate the ramp control and test the CS-ALINEA algorithm. The experimental results show that the proposed method can optimize the ramp queuing length and reduce waiting time of vehicles while the efficiency of urban freeway can be guaranteed.


I. INTRODUCTION
With the rapid development of urban transportation, traffic congestion is becoming more and more serious. Now, the main control methods that can effectively alleviate the traffic congestion problem of urban freeways are ramp metering and variable speed limit control [1]. Zhang et al. [2] proposed a cycle-based variable speed limit (CVSL) strategy. In this method, the speed limit was reduced in part of the cycle in order to create gaps on the mainstream artificially. This will increase opportunities for on-ramp vehicles to merge together. Reinforcement learning is also widely used in variable speed limit research [3]- [5].
The ramp metering is the most widely used and most effective method for urban freeways traffic control. In this method, the traffic flow can be controlled to enter the mainstream by setting up the signal lights on the ramp in order to improve the mainstream capacity [6], [7]. Many existing ramp control algorithms mainly used traffic information such as traffic flow density and occupancy as input for the algorithm [8].
The associate editor coordinating the review of this manuscript and approving it for publication was J. D. Zhao .
Papageorgiou proposed a variety of improved ALINEA algorithms including UP-ALINEA (Upstream-Occupancy Based ALINEA), FL-ALINEA (Flow-based ALINEA) and so on [9] to meet the needs of different actual scenarios. In order to resolve cyclical shock effect of the occupancy of ALINEA, and improve response speed caused by the reduced lane numbers in downstream bottleneck segments, the PI-ALINEA was proposed [10], but PI-ALINEA had two parameters needing to be adjusted because of the added proportional term. It made the control more complicated and difficult to be implemented in real scenarios. By introducing an estimator of critical occupancy in the downstream of the mainstream, Smaragdis et al. [11] proposed the AD-ALINEA to adaptively adjust the critical occupancy rate in real time in order to adapt to real-time changing traffic status better. Chi et al. [12] used iterative feedback adjustment method and used downstream real-time traffic flow density as input to adaptively adjust and optimize the control gain K r in the ALINEA according to the real-time I/O number. The algorithm had better robustness and convergence. The FF-ALINEA proposed by José Ramó n D [13] reduced traffic crashes by predicting the bottleneck road density and VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ modifying the control structure of the ALINEA. The Fuzzy control method is also an important control method. The mainstream density, speed, queuing length are fuzzed and the fuzzy rule base is established to obtain a signal control scheme according to fuzzy decision. But in fuzzy control method, the fuzzy rules and membership functions are always set by empirical values. The robustness of system is not good enough [14], [15]. Liang et al. [16] designed a traffic density controller based on LWR (Lighthill-Whitham-Richards) model and RBF neural network, which could keep the freeway traffic at a setting traffic density. Ivanjko et al. [17] proposed a Q-learning-based ramp control algorithm using downstream speed and ramp queuing length as the state space. van de Weg et al. [18] combined ALINEA with variable speed limit strategy to optimize the upstream and downstream speed boundaries and related parameters in ALINEA. The traffic flow density was taken as import parameter. The calculation time of MPC (Model Predictive Control) strategy was reduced and the throughput capacity was improved.
Lu et al. [19] proposed another reinforcement learning based system RAS, which not only improved the traffic efficiency by defining the number of vehicles on the mainstream and ramp as state space and optimizing the action and reward, but also reduced the total time spent of freeway network from uncontrolled conditions. Nowadays, there are lots of intelligent control algorithms in the field of traffic control. These algorithms have good control effects on the testing platform. But in the actual traffic control scenarios, these intelligent algorithms have much problems such as long training time and initial values relying on empirical data. It makes these algorithms rather difficult to be implemented. Although the ALINEA algorithm is simple and its stability is good. It does not need to rely on any model information of the ramp and is easy to be implemented. These characteristics make ALINEA very practical. But in the ALINEA algorithm, its efficiency depends on the setting of the critical occupancy, which is usually set through empirical value. But the critical occupancy may always change in actual scenarios. It is always difficult to obtain accurate value.
In this paper, an improved ALINEA method, named CS-ALINEA (congestion status based ALINEA), is proposed. In this method, the traffic flow is used to replace the occupancy, and the clustering method is used to reclassify the congestion status. According to different status, the corresponding control rate can be selected adaptively in order to achieve ramp control. In the actual traffic scenarios, the ground road traffic should be not affected while the ramp control is performed. In the existing ramp control methods, to guarantee the traffic efficiency of mainstream, the impact of ramp overflow on ground road traffic is often ignored. In the improved algorithm proposed in this paper, when the ramp queuing length exceeds the critical queue length, the signal timing scheme is adjusted through selecting the corresponding control rate and the maximum value of the regulation rate is adopted to control the ramp overflow.
The main content of this paper is arranged as follows: Section 2 introduces the classification method of congestion status in the urban freeway. Section 3 discusses the improved CS-ALINEA based on congestion status. Section 4 uses the SUMO simulation platform to verify the proposed algorithm. Finally, the conclusion of this paper is given.

II. CONGESTION STATUS CLASSIFICATION A. CONGESTION STATUS LEVEL
Congestion status on the mainstream of urban express can be divided into six levels (speed km/h) according to the national standards: very smooth (v > 65), smooth (50, 65], mild congestion (35, 50], moderate congestion (20,35], severe congestion (5,20], heavy congestion (0, 5]. The national standards are more universal, but considering the number of lanes in actual scenarios, lane width and other factors, the congestion status of different urban freeways should be classified according to the actual scenarios. In the actual scenarios, when the urban freeway is in heavy congestion, no more vehicles will be allowed to enter the ramp and the ramp should be closed. That means the ramp metering is not required. In the smooth and very smooth status, the mainstream traffic is in good condition. The very smooth can be regarded as a special smooth status with all-green signal release. The left four situations can be divided into four levels of speed:

B. GAUSSIAN MIXTURE CLUSTERING METHOD
Gaussian mixture clustering is based on the Gaussian mixture model, which is different from K-means and other clustering methods for calculating distance from the center point. The probability model is used to represent the clustering prototype in Gaussian mixture clustering. The probability density function of Gaussian mixture distribution is shown in equation (1): The distribution is composed of K mixed components, and each mixed component represents a Gaussian distribution. N(x|µ k , k ) is the probability density function of the Gaussian distribution, where µ k is the mean vector and k is the covariance matrix. The microwave data set used in this paper is D = {x 1 , x 2 , · · · , x m }, in which x m is a two-dimensional data, derived from the speed and flow data obtained by the microwave detector. π k is a blending factor with a value greater than 0 and K k=1 π k = 1. In the clustering algorithm, each Gaussian distribution can be regarded as a congestion status, and π k can be taken as the probability that the k th congestion status is selected. Let z j ∈ {1, 2, · · · K } represents the Gaussian mixed composition of input traffic data x j , the priori probability of z j is P(z j = k) corresponding to π k . According to Bayesian theorem, the posterior probability of z j is briefly shown as γ jk , it can be calculated through equation (2): The microwave data D can be divided into k congestion status through Gaussian mixture clusters, and the division of congestion is determined by the posterior probability. The cluster mark λ j of each sample can be calculated through equation (3).
From equation (1), it can be seen that µ k , k , π k need to be determined in advance. The maximum likelihood estimation is used to find a set of parameters that maximize the likelihoods function and these parameters can be taken as the most appropriate parameters. Firstly, the posterior probability can be calculated based on current parameters in Gaussian mixture model in EM algorithm, and then the model parameters can be updated according to equations (4), (5) and (6). The calculation process will be continuously iterated until the most suitable parameters can be obtained.

III. IMPROVEMENT RAMP METERING METHOD BASED ON CONGESTION STATUS A. ANALYSIS OF RAMP METERING ALGORITHM
The vehicles on the ramp are controlled by the signal lights installed at the junction of the ramp and the mainstream. The timing scheme of the signal lights can be selected in order to limit the on-ramp traffic confluence and ensure the mainstream traffic. The schematic diagram of on-ramp control is shown in Figure 1. The ALINEA algorithm is a closed-loop feedback control algorithm that controls the vehicles import rate through adjusting the ramp metering rate of traffic lights in actual scenarios. It is guaranteed that the downstream of the mainstream is always maintained at the expected occupancy.
Equation (7) is the control rate of the ALINEA, r(k) is the ramp metering rate of the k th cycle, k r is the regulator parameter and is usually determined according to empirical values.Ô is the expected occupancy of downstream, it is the occupancy measured when the flow is equal to the capacity, which can be obtained by analyzing the flow-occupancy graph. O(k −1) is the actual measured occupancy value of the k-1 th cycle detector.

B. IMPROVEMENT OF ALINEA ALGORITHM WITH UPSTREAM SPEED AND FLOW
The congestion status of the freeway can be judged according to the upstream speed of the ramp measured for each control cycle. Based on the analysis of congestion status, the signal timing of the ramp can be adopted the corresponding control rate. It is shown in equation (8).
Here, r(k) represents the ramp metering rate of the kth cycle ramp; K F is the regulator parameter; r max and r min are the maximum and minimum ramp metering rate of the ramp signal control respectively;q represents the expected saturated flow in the mainstream of the freeway;ṽ k−1 is the mean speed of the upstream vehicles in the k-1th cycle; q out (k − 1) is the flow coming from upstream and ramp to downstream in the k-1th cycle.
Taking the upstream flow q in and ramp flow q r from the previous cycle as input, the flow of the previous cycle entering to the downstream can be calculated in equation (9).
Through calculating the difference between this flow and the expected saturated flowq, the flow of downstream that can still be accommodated at the next cycle can be obtained. The values in the early morning or peak period may be quite different. In order to avoid large fluctuations in the control rate, K F should be set a value less than 1. The saturated flow q act can be obtained according to historical data. Firstly, taking the weather into account, traffic accidents and other interference factors, the capacity of the mainstream may be lower. Secondly, considering that congestion usually occurs in downstream and spreads to upstream, the expected saturated flowq should be set slightly less than the actual saturated flow q act , so that the ramp can be controlled in advance.
The experimental results show that whenq = 0.8q act and K F = 0.1, the control effect could be better.
When the mainstream of urban freeway is in a status of smooth flow, the downstream flow is not saturated. r(k) can be risen and the green light phase can be constantly extended. It can be explained in view of control rate that if the upper bound is not set, r(k) can continue to rise in the early morning or late at night with low traffic flow. So through setting r max , the growth of the r(k) can be controlled. The maximum r(k) occurs when the signal lights are in all green release. That means there is to no control.
When the mainstream is in the mild congested status, r(k) should be reduced in order to limit the on-ramp traffic entering the mainstream. Due to the normal distribution of the flow, when the speed continues to decrease, the flow will gradually decrease after increasing to a saturated flow. The difference between q act andq out is still positive. If the control rate is set to that of the smooth state, r(k) will continue to rise, it will be opposite to the control target at this time. Therefore, the ramp metering rate should be reduced as opposed to the processing method in the smooth status. At the same time, if the r(k) is set too small, the vehicles will stay on the ramp too long. It will be easy to cause safety accidents. When the rush hour is coming and traffic flow is increasing, r(k) should be reduced continuously. But the minimum value r min should be prevented from reducing to 0. In a moderate congestion state, the congestion increases and r(k) is set to r min directly. When the mainstream enters a status of heavy congestion, the ramp is closed directly. At this time, signal does not need to be controlled and r(k) is set to 0.

C. CONSTRAINTS ON RAMP QUEUING
Generally, the ramp queuing does not be constrained in ALINEA. But in actual traffic control scenarios, the traffic on the mainstream and the ground road will be congested at the same time during rush hours. At this time, ramp metering can alleviate the congestion of the mainstream effectively. However, the queuing length of vehicles remaining on the ramp will keep increasing because of the red light. It will cause the congestion overflow to the ground. In the actual scenarios, improving the capacity of the freeway mainstream should guarantee that the vehicles queuing on the ramp will not cause overflow to the ground and influence the ground traffic. Yang et al. [20] studied the effect of different traffic flow arrival modes on the ramp queuing and proposed a method to calculate the cumulative queuing length by measuring the arrival rate and export rate of each cycle. In this paper, the checkpoint data and ramp metering rate are used to calculate the number of vehicles arriving and moving out in each cycle. Then the cumulative queue length can be obtained. In the adjustment cycle k, the number of arriving vehicles A(k) can be obtained through the checkpoint data, the number of departing vehicles D(k) can be calculated through the previous cycle ramp metering rate, the cumulative number in k th adjustment cycle can be calculated in equation (10): The cumulative queue length L is shown in equation (11): In the equation (11), λ is the number of lanes on the ramp; µ is the headway of the vehicles on the ramp queuing. When the queue is emptied, Q(k) may be negative. Therefore the δ is set to 0 when Q(k) is negative, otherwise the δ is set to 1.
Assuming that the total queuing length of the ramp is L, in the segmented control method, the critical queue length is set to L 1 , and the value is 0.6L. The maximum queue length is set to L 2 , and the value is 0.9L [21]. When the queue length exceeds L 1 , the green light phase should be increased appropriately in advance so that more vehicles will be released on the mainstream. When the queue length is between (L 1 , L 2 ], the vehicles allowed entering ramp in the next cycle with queuing length exceeding L 1 can be calculated. The control rate is shown in equation (12).
When the queue length exceeds L 2 , the ramp queuing is near overflow and it will affect the traffic on the ground road. At this time, the ramp metering rate which takes r max is used to control the traffic. The ramp is shut down. It can effectively avoid the ramp overflow and influencing on ground road traffic. The signal control method combined with the length of the ramp queuing can be obtained. The segmentation of CS-ALINEA control rate based on the length of the ramp queuing is shown in equation (13):

IV. EXPERIMENTAL ANALYSIS A. ANALYSIS OF CLUSTERING RESULTS
In this paper, the data set comes from the microwave data between June 25, 2018 and July 6, 2018 including 10 working days in Shangtang freeway, Hangzhou. In the data set, there is no severe congestion data which will cause the entrance ramp to be closed. Here, the cluster number k is set to 3 and the data set will be clustered into three categories respectively. The blue dots will indicate smooth, green dots indicate mild congestion, and red dots indicate moderate congestion. For microwave detection speed less than minimum speed in moderately congested speed zones, it can be considered as in a severe congestion status. It can be shown that the data set can be roughly divided into three categories by speed in Figure 2.
In Table 1, the four speed intervals are corresponding to the four congestion status.

B. ANALYSIS OF CONGESTION WITH LIGHT BAND DIAGRAM
Here, the mean speeds of the historical data are used to cluster and divide the congestion zone in the adjacent     Figure 3. From the light band diagram, when and where the congestion occurred, how fast the congestion spread on the mainstream, and how the congestion dissipated. This kind of ramp control has advantage in local intersection congestion, but it is hard to solve the chain reaction caused by multi-ramp congestion at the same time. It can be taken as a useful analysis method for multi-ramp coordinated strategy in the further.

C. ALGORITHM SIMULATION
In this paper, the open source simulation platform SUMO (Simulation of Urban Mobility) is used in the experiment. The simulation experiment is divided into four groups: no-signal control, fixed timing control, ALINEA, freeway control algorithm based on Q-learning [3] and CS-ALINEA. The simulation experiment results are analyzed on the mainstream travel time, mainstream average traffic throughout, ramp waiting time and the length of the ramp queuing as the evaluation index respectively. The on-ramp from north to south in the section of Daguan intersection on the Shangtang freeway is taken as a testing object and the simulation model is built. It is shown in Figure 4. The mainstream is of two lanes in both directions and the entrance ramp is of two lanes. The vehicles will converge in the acceleration lanes after passing the traffic lights in the interwoven zone.
In SUMO, there are two simulation models. One is the car-following model and the other is the import model. The car-following model is used to analyze the driver's driving behavior. There are many car-following models in SUMO. In this paper, the Intelligent Driver Model is used. The model is shown in equations (14) and (15): where s * (V n (t), V n (t)) is the minimum expectation; V 0 is the expected speed; V n (t) is the vehicle speed at time t; V n (t) is the speed difference from the previous moment for this vehicle at time t; a is the maximum acceleration; b is the expected deceleration; s 0 is the minimum distance when the traffic gets congested; T is the response time. When the vehicle type in the SUMO is configured, the model parameters can be set in Table 2.
The number of vehicles entering the ramp is mainly related to the distance between vehicles. Equations (16) and (17) are used to calculate the distance before and after the vehicles respectively. Only when the front gap g l and rear gap g f are greater than the acceptable gaps g l,min and g f ,min , the vehicle can be allowed to enter the ramp.
where g ol is the initial front vehicle distance and g of is the initial rear vehicle distance; v s is the speed of the vehicle s; v l is the speed of the front vehicle; v f is the speed of the     Figure 5. The simulation input flow came from the microwave data on the mainstream and the passing data came from the checkpoints on the ramp. In order to guarantee the time granularity consistency between the checkpoint data and the microwave data, the traffic flow was collected every 5 minutes. The real-time interaction between the simulation environment and the algorithm was achieved through the TRACI plug-in that comes with SUMO.
The no-signal control scheme can be achieved by setting the all signal lights to green on the ramp. The fixed timing control can use the actual on-ramp signal light timing scheme. Here, the cycle was set to 60s, the green light phase was set to 30s, the red light phase was set to 30s. ALINEA and CS-ALINEA could be adjusted by the real-time monitoring data. The freeway control algorithm based on Q-learning was designed mainly to reduce the total travel time and mainstream travel time by changing the upstream speed limit. In this algorithm, the downstream density was taken as the status and the speed limit was taken as the action. It constructed the reward function through Poisson distribution. The reward was calculated according to the status in order to select the action. It can select the best action in the learning process.
It recorded the different values about mainstream travel time, ramp waiting time, average queue length, and main line traffic in Table 3.
According to the influences of four indexes on the control effect of freeway, the corresponding weight coefficients are set respectively. The main goals of freeway control are the mainstream traffic average throughout and the mainstream travel time. These weight ratios should be greater than that of others. In this paper, the importance of ramp queuing is improved. So, its weight ratio is near to the above two items. The weight ratios of these four indexes are abbreviated as T travel , T wait , L ramp , F mainline and they are set to 0.3, 0.15, 0.25, 0.3, respectively. In the four indicators, the larger the mainstream traffic is, the better the evaluation result is. This is a positive reward. That means weighted average is always positive. The other three indicators are opposite. They are negative penalties. The weighted average can be calculated in equation (18).
It is shown in Table 3, when K F is set to 0.1, the evaluation score is the largest. That means that when the evaluation effect of mainstream traffic is optimal, the ramp waiting time and the ramp average queue length are both optimal. Meanwhile, it has a longer mainstream travel time and less mainstream average traffic throughout compared to some of others. So, the appropriate value of K F will be set to 0.1.
The mean speed in the upstream and the average queuing length on the ramp can be obtained every 5 minutes by the    detectors installed on the mainstream and ramps. In Figure 6, and Figure 7, when the traffic flow is small, the upstream speed under the control of the four different schemes has no change much. During the peak periods, the no-signal control scheme makes no restrictions on vehicles entering the ramp, so the vehicles on the ramp enter the mainstream without waiting time. This will make the mainstream congestion worse and the mean speed is lower than that of other three schemes. The optimization goal of ALINEA is to ensure the mainstream traffic effect, the mean speed under ALINEA control during rush hour is the fastest, but it forms a longer queue because of the ignorance of the ramp queue.
The mainstream travel time, ramp waiting time, average queue length and mainstream traffic throughout under the four control schemes are shown in Table 4. In ALINEA algorithm, travel time is reduced by 10.44s (15.28%) compared to the uncontrolled scheme, and the mainstream traffic average throughout is increased by 38.10%, but it was at the expense of the traffic efficiency on the ramp. Compared to the uncontrolled scheme, the mainstream travel time of CS-ALINEA is reduced by 9.41s (13.77%), and the mainstream traffic average throughout is increased by 36.19%. The optimization goal of the Q-learning method is designed to reduce the travel time of the mainstream. From the results, the Q-learning algorithm achieves the best performance in the travel time. It is reduced by 15.07s (22.05%) compared to uncontrolled solution. At the meantime, the ramp queuing length and the waiting time in this method are longer than that of others. Taking the length of the ramp queue as the control target, the travel time of the mainstream has increased a little compared with ALINEA, while the average waiting time for vehicles on the ramp has been reduced by 5.82s (17.31%) and the average queue length has decreased by 20.35%. Compared with Q-learning, CS-ALINEA has improved 4.88s (14.94%) and 1.25m (14.52%) on the average waiting time and the average queue length respectively.
In summary, the CS-ALINEA method can reclassify the mainstream congestion status, and choice the corresponding control rate according to the classification standard designed. Similar to ALINEA, CS-ALINEA is easy to be implemented and has excellent effect in on-ramp metering. Unlike other algorithms, the CS-ALINEA can guarantee traffic efficiency in the mainstream while avoiding the ramp overflow. It will effectively reduce the impact on the ground traffic caused by the long queuing on the ramp.

V. CONCLUSION
In actual traffic control scenarios, the existing ramp control algorithms are often difficult to directly work on urban freeways due to the restrictions on the acquisition of traffic data and the guarantee requirements for ground road traffic. The CS-ALINEA proposed in this paper reclassifies the congestion status of the mainstream and adaptively selects the control rate to ensure the maximum throughout on the mainstream. At the same time, it can solve the problem of ramp overflow. At present, the CS-ALINEA proposed in this paper is aimed at local ramp metering control. In next step, this method will be used in the urban expressway network in order to solve the problem of coordinated control of multi-ramp. In the research work of coordinated control of multi-ramp, there are many challenging results, including a non-parametric control technology for multi-agent based on reinforcement learning proposed by Belletti et al. [22]. The algorithm based on the MWR (mutual weight regularization) algorithm to alleviate the curse of dimensionality of multi-agent control schemes by sharing experience between agents [23], [24]. In the further, we will try to combine our method with the reinforcement learning method above in the coordinated control. The congestion status can be taken as evaluation indicators in order to optimize the reward function and the coordination strategies can be improved. How to maximize the overall capacity of the urban freeway will be the main task of the next stage.

DATA AVAILABILITY
The experimental data used to support the findings of this study are included within the article. YE WU is currently pursuing the master's degree in computer science and technology with the College of Computer Science and Technology, Zhejiang University of Technology, China.
His research interest includes intelligent transportation systems.
SHIPENG CAO is currently pursuing the master's degree in computer science and technology with the College of Computer Science and Technology, Zhejiang University of Technology, China. His research interests include intelligent transportation systems and reinforcement learning.
LINAN ZHU received the Ph.D. degree in mechanical manufacturing and automation from the Zhejiang University of Technology, in 2014.
He currently works as an Associate Professor with the Zhejiang University of Technology. His research interests include cloud manufacturing, informatization of manufacturing industry, networked manufacturing, and resource scheduling. He is also a member of the Chinese Artificial Intelligence Society and the China Computer Federation.
GUOJIANG SHEN received the B.Sc. degree in control theory and control engineering and the Ph.D. degree in control science and engineering from Zhejiang University, Hangzhou, China, in 1999 and 2004, respectively.
He is currently a Professor with the College of Computer Science and Technology, Zhejiang University of Technology. His current research interests include intelligent control theory and application, advanced control technology and application, and urban road traffic modeling and control technology. VOLUME 8, 2020