Performance and Measurement Analysis of a Commercial 5G Millimeter-Wave Network

Millimeter-wave (mmW) communication has great potential in expanding channel capacity, increasing transmission rates, enhancing anti-interference capabilities, and reducing delay. Therefore, it has been the subject of much attention and research, and it has been adopted as a core technology of 5G. So far, researchers have experimentally verified that high throughput can be achieved with 5G mmWs. However, compared to those of traditional medium- and low-frequency base stations, the propagation characteristics of mmWs have caused link instability and interference problems in large-scale networks, which brings new challenges to the industrial capability of the mmW band and puts forward new requirements for the performance evaluation of mmW base stations. In this paper, based on the signal characteristics of the mmW band and applicable scenarios of mmW base stations, a performance evaluation method for mmW base stations in commercial networks is comprehensively presented. Using the evaluation system described in this paper, we conduct measurements on commercial mmW base stations, evaluate the performance of these stations, and provide comprehensive guidance for their design and actual deployment.


I. RESEARCH BACKGROUND
As an important part of the global fifth-generation (5G) spectrum strategy, the millimeter-wave (mmW) band has the advantages of large bandwidth and low delay and is an important future direction for the development of mobile communication technology [1], [2]. Major countries and regions around the world have made strategic planning for mmW spectrum resources, and countries such as the United States, South Korea, and Japan have already completed some of the mmW spectrum auctions, which have been deployed successively. The Ministry of Industry and Information Technology of China has approved 24.75 to 27.5 GHz and 37 to 42.5 GHz as the main experimental research and development bands.
The propagation characteristics of mmWs are different from those of low and medium frequencies [3], [4], and the deployment scenario of mmW base stations also has its own The associate editor coordinating the review of this manuscript and approving it for publication was Wenchi Cheng . characteristics. Compared to sub6 GHz bands, mmWs have a relatively large loss under the direct path, making it difficult to realize long-range communication. Therefore, the deployment of mmW base stations should focus on short-range hotspot coverage to provide higher data rates to meet the demand for extremely fast user experience rates and large system capacity in hotspot areas, such as those deployed in large stadiums, airports, and public squares. This type of scenario involves little or no obstruction, high user density, and large traffic demand. mmWs are easily blocked by buildings and cannot penetrate materials such as concrete walls and infrared reflective glass. Therefore, mmW base stations need to be deployed in scenarios with rich reflection paths, such as commercial streets and office buildings. mmWs can also be used for indoor coverage needs, such as fixed wireless access, as well as for backhaul to achieve high-speed data transmission [5]. Performance evaluation of mmWs in different application scenarios can greatly support their commercialization. However, existing literature usually focuses on the simulation of specific channel models, while the simulation-based performance evaluation of various realworld application scenarios has not been reported.
At present, large-scale commercialization of mmWs has not yet begun anywhere in the world, and research on the test methods for performance evaluation of mmW base stations is rather limited [6]. Comprehensive and objective evaluation of the performance indicators of commercial base stations is not only important for guiding the design of mmW base stations but also helpful to guide the deployment of mmW base stations. In addition, the evaluation results of commercial mmW base stations will be valuable in key indicator design as well as deployment scenarios and methods for mmW networks. However, performance evaluations of commercial mmW base stations in existing networks have not been reported so far.
The main contributions of the present study are as follows. The performance of mmW base stations in different application scenarios is simulated, and an evaluation method based on commercial mmW base station networks is proposed considering the design characteristics of mmW base stations. On the one hand, the evaluation method proposed in this paper is validated through simulation and field measurement; on the other hand, the field test results further reveal the characteristics of mmWs, which will be key to the design of mmW base stations and their deployment in existing networks. To the best of our knowledge, this study is the first of its kind to provide the results from an evaluation of the actual test performance of a commercial mmW base station network specifically for the practical application scenario in China.
The rest of the paper is organized as follows. First, in Section 2, based on the simulation results, the propagation characteristics of mmWs are introduced, scenarios applicable to mmW base stations are discussed, and a simulation of system performance under different application scenarios is evaluated. Section 3 examines how to evaluate the key indicators of the mmW base station hardware based on the testing of the existing network in different scenarios. The focus of Section 4 is to investigate the performance of mmW base stations through link simulation and network simulation, including the impact of different equivalent isotropically radiated power (EIRP) base stations on coverage and capacity. In Section 5, the commercial mmW base stations are tested in commercial networks under typical indoor and outdoor application scenario according to the evaluation method proposed in this paper. The test results of the existing network are analyzed and discussed, and the simulation results and the measured results of the performance are compared. Finally, the findings of this study are summarized in Section 6.

A. mmW TRANSMISSION CHARACTERISTICS
The propagation characteristics of the wireless signal in the spectral range of mmWs need to be considered first in mmW communication. The electromagnetic wave propagation model is the simplest free-space propagation model. The basic transmission loss in free-space is given by Formula (1): where d is the distance (m) and λ is the wavelength. Substitution of λ = c/f , where f is the frequency (MHz) and c is the speed of light in a vacuum (m/s), into Formula (1) gives the following Formula (2), after rearranging the terms: It can be seen from above formula that the high transmission frequency of mmWs relative to the sub6 GHz frequency bands leads to an increase in free-space loss.
Existing propagation models such as Okumura et al. [7] and Ikegami et al. [8] aim to simulate signals at frequencies below 6 GHz. However, due to the increase in frequency spectrum, the mmWs are highly susceptible to problems such as signal interruption or intermittent transmission due to shadow fading [9], so the corresponding mmW propagation models are under constant correction and improvement. In the present study, the relations between the path loss and distance of electromagnetic signals with different frequency spectra are simulated based on the free-space model and the rural macro model in the International Telecommunication Union (ITU) model, respectively. As shown in Fig. 1, regardless of topographical factors, in a line-of-sight (LOS) environment, the path loss increases with increasing frequency, and the path loss of the 28 GHz signal is approximately 21 dB higher than that of the 2.3 GHz signal. In the case of non-LOS (NLOS) transmission, the penetration loss of radio waves through obstacles increases as the frequency increases. Table 1 lists the penetration losses of signals in different frequency bands in different materials as published by the ITU [10]. The NLOS transmission losses of mmW signals are very large, and wall and glass seriously affect the propagation of mmWs, severely limiting the application scenarios of mmWs.
In 3GPP, typical 5G deployment scenarios are defined, and transmission loss models are provided for different  deployment scenarios [11], [12]. Formula (3) can be used as a general formula for the transmission loss: where a( hm ) is a function of terminal antenna height. Table 2 lists various parameters of the transmission model for different deployment scenarios.

B. APPLICATION SCENARIOS AND SIMULATION-BASED PERFORMANCE EVALUATION OF A mmW BASE STATION
As mentioned above, mmWs are suitable for dense urban, street, and indoor scenarios. In the following, we perform simulation-based evaluation of the performance of a mmW base station under these three scenarios. For the 3D model of dense urban buildings shown in Fig. 2, we simulate the downlink covered only by the 26 GHz frequency band and by the combination of the 2.6 and 26 GHz (i.e., low and high) frequency bands. The results are shown in Fig. 3. With coverage provided by a 26 GHz mmW base station alone, a very high data rate can be achieved in outdoor-area coverage; however, the indoor area is well covered only in low-loss buildings and not in high-loss buildings. Therefore, in the dense urban scenario, the deployment of low-frequency base stations should be considered to improve the coverage performance, and the mmW base station should be deployed to increase the throughput and capacity of users.
For the street scenario, the main form of the mmW base station is the street station. The building area model as shown in Fig. 4 is constructed, where the deployment locations of mmW base stations are marked in red. The simulation results in Fig. 5 show that in the micro base station scenario in a cell, the cell can be fully covered by the street-level deployment (100 m × 100 m), and the average downlink throughput of 800 Mbps can be realized for a single user of the cell. A higher capacity of the cell can be achieved by beamforming combined with massive multi-user, multi-input, multi-output (MU-MIMO) technology.
Another application scenario of mmW base stations is the indoor scenario with high human traffic (e.g., stadiums and airports). Fig. 6 shows the simulation result of the downlink throughput of a stadium. As can be seen, for a stadium with a size of 100 m × 60 m, excellent coverage can be achieved by using 690 m 2 /station, i.e., a total of eight stations. The simulation results demonstrate that the average downlink and   uplink throughputs are greater than 1 Gbps and 300 Mbps, respectively.
In summary, the main application scenarios of mmW macro base stations are streets, squares, and fixed access in dense urban areas for covering blind spots and hot spots and providing high-speed access. The main application scenario of mmW micro base stations are high-traffic areas such as exhibition halls, bus/railway stations, and ball fields for covering hot spots and providing high data rates. The main application scenarios of mmW indoor base stations are high-end indoor scenarios such as gymnasiums, waiting lounges, hospitals, workshops, and shopping malls for providing high-speed access.

III. METHOD FOR PERFORMANCE EVALUATION OF mmW BASE STATIONS BASED ON EXISTING NETWORK
Base station design requires comprehensive consideration of various complex indicators, such as 3GPP specifications, numerology selection, radio frequency (RF) performance indicators, carrier design, modulation method, power supply, power consumption, volume, heat dissipation, mechanical design, and environment-related indicators. The hardware capability of a base station is an important factor determining its performance. For example, the EIRP directly determines the coverage capability, RF performance, and capacity of the base station; the choice of subcarrier spacing determines the delay performance of the base station; the beam management and beam algorithms of the base station determine the continuous coverage capability of the beam; and the stability of the base station determines whether the base station can operate stably with good performance in different environments.
The present study proposes new test cases and methods to better investigate the performance of mmW base stations, focusing on the characteristics of mmW propagation and mmW base station design. As an example, the EIRP is jointly determined by transmit power and antenna gain (Formula 4), and the selection of EIRP determines the output power of the VOLUME 8, 2020 RF power amplifier, the antenna gain, the number of antenna elements, and the antenna array size, which in turn affect heat dissipation and volume weight of the product and impact the deployment of base stations: EIRP = 10 lg (P tx ) + G array + G elem + 10 lg N array (4) where P tx is the transmit power, N array is the number of antenna elements included in the antenna array, and G elem is the antenna element gain. The deployment of mmW base stations can be roughly divided into indoor scenarios and outdoor scenarios. In the outdoor LOS environment, the mmW path loss exponent normally has a relatively fixed relationship with the propagation distance. In the indoor environment, due to the existence of various objects and walls, the path loss indices in different experimental environments are found to be different under the same distance. In fact, the path loss in an indoor environment is the result of the combined action of many parameters, which can be expressed as Formula (5) [13]: where G t and G r are the transmit and receive antenna gains, respectively; i and T j are the reflection coefficient and transmission coefficient of the object, respectively; m 1 and m 2 are the number of reflections and transmissions, respectively; and S is the scattering factor. Therefore, the performance evaluation of mmW base station needs to be performed in outdoor and indoor scenarios, respectively. Considering the propagation characteristics of mmWs, a new method for performance evaluation of a single mmW base station is proposed in the present study, mainly including fixed-point tests, environmental tests, handover tests, and delay tests, relieving the limitations of traditional base station evaluation methods in the performance evaluation of mmW base stations.
The fixed-point tests include the fixed-point test conducted along the normal direction of the base station antenna array under indoor and outdoor scenarios, the fixed-point test carried out using wrap-around paths in the horizontal plane with the base station as the center and with different radii, and the fixed-point test performed on the vertical plane and at different angles from the horizontal plane of the base station. The remote test along the normal direction is a traditional test item for single-station coverage and can be used to examine the mmW data rate and coverage, including the coverage capabilities of the uplink and downlink control channels as well as the traffic channel, and the uplink and downlink rates. Due to their propagation characteristics, mmWs can be obstructed to varying extents in the application scenarios, severely affecting the coverage of mmW base stations. Therefore, to improve the evaluation of coverage capability, it is necessary to add the wrap-around test on the basis of the remote test of a single cell. According to the remote distance measured in the remote test, the far, middle, and near points are, respectively, selected to wrap around from 60 • on one side to 60 • on the other side of the normal direction of the base station antenna array. In addition to selection according to distance, the wrap-around path can also chosen according to the reference signal received power (RSRP). Specifically, the points of −80 dBm, −90 dBm, −100 dBm, and −110 dBm are selected using the RSRP test results along the normal direction in the remote test, and then the wrap-around test is performed with the base station as the center and a wrap-around range that should cover 60 • on each side of the normal direction of the base station antenna array. One of the important application scenarios of the mmW base station is the high-density scenario, such as commercial areas and office buildings. In these dense high-rise scenarios, a fixed-point test needs to be performed in the vertical plane to evaluate the coverage capability of the mmW base station in the vertical range.
The environmental tests include the comparison of different obstructions and the environmental change test. Obstructions can be compared to evaluate the effect of the common obstructions on the coverage capability of commercial mmW base station under typical scenarios in an existing network. The test results can, on the one hand, further verify the impact of the obstructions encountered during the wrap-around test and, on the other hand, complement the impact of the obstructions not involved in the wrap-around test or the remote test.
It is recommended that the distance from the terminal to the base station remain basically unchanged in the outdoor scenarios with and without glass, walls, and human bodies, for the comparison of terminal test results. In outdoor application scenarios, since mmW propagation can be affected by weather and seasonal variations, the performance of the mmW base station under the influence of rainfall and trees should also be evaluated. The ITU recommends that Formula (6) be used to evaluate the loss of mmWs caused by leaves: where f is the frequency (MHz), R is the leaf depth (m), and A max is the maximum loss, which has an empirical value of 40 dB. ITU uses γ R (dB/km) to characterize the rain attenuation, as shown in Formula (7), where R is the rain rate, and k and α are frequency-related parameters.
The attenuation γ R caused by signals below 10 GHz is negligible. The relations between rain attenuation and signal frequency under different rain rates are shown in Fig. 7 [14]: For environmental tests under indoor scenarios, it is recommended that drawers, partitions, glass, and human bodies be selected to conduct the obstruction comparison test under actual application scenarios.
Handover tests include intercell handover, interbeam handover, and user equipment (UE) attitude adjustment. As in the case of sub6 GHz base stations, the intercell handover and average throughput should be tested to investigate whether FIGURE 7. Relation between rain attenuation and signal frequency under different rainfall conditions [14].
the service can be maintained for a long time and whether handover can be carried out normally in a continuouscoverage network with multiple base stations. The 5G mmW beamforming and beam management algorithms both encounter new challenges, which require the testing of beam design, including the intracell beam handover. In outdoor scenarios, intracell and intercell handover tests are performed, routes are selected within the coverage area of the mmW base stations, the terminals pass through cells/beams at different movement speeds, and different speeds should characterize the typical movement speeds of pedestrians, bicycles, and vehicles, respectively. In an indoor test, only low-speed mobile terminals are required to test the intercell and interbeam handovers. Due to the strong directivity of mmW beams, the angle of terminal antennas may have a significant impact on the signal connection, so the effect of different attitudes of UE on the received signal should be tested.
Regarding the delay test, mmW base station carriers have different settings than traditional low-and medium-frequency base stations, with subcarrier spacing up to 120 kHz, which can improve the delay performance of base stations. The delay test should include user plane delay and control plane delay to investigate the delay performance of a single user at good, medium, and poor points.

IV. PERFORMANCE ANALYSIS OF mmW BASE STATIONS
This section analyzes the performance of the mmW base station, with a focus on the coverage and capacity of the mmW base station by means of link simulation and network simulation.

A. COVERAGE CAPACITY ANALYSIS
This experiment uses link simulation to investigate the coverage range of base stations with different EIRPs. The basic formula for calculating the coverage range of the uplink cell by link simulation is shown as Formula (8), which determine the coverage range of the cell by using the maximum signal attenuation at the cell edge, L p,max : where L sa,celledge is the signal attenuation at the cell edge (unit: dB), L BL is the body penetration loss (unit: dB), L BPL is the building penetration loss (unit: dB), L CPL is the vehicle penetration loss (unit: dB), G a,NB is the narrow-beam antenna gain (unit: dBi), and G a,UE is the antenna gain of the UE (unit: dBi). The link calculation of the downlink coverage range of the cell is based on Formula (9): (9) where P TX,RB is the transmit power (unit: dBm) of each resource block at the reference point TX, S UE is the sensitivity of UE (unit: dBm), and B IDL,celledge is the downlink noise increment, which is defined as: where DL is the model adjustment factor, F c is the average of the ratio of the power received from other cells at the edge of the cell to the power received by its own cell, Q n PDSCH is the physical downlink shared channel load, N RB,DL is the thermal noise of each resource block in the downlink, and L sa,celledge is the maximum attenuation of the downlink signal of the broadcast beam at the cell edge.
Based on the above principle, the coverage of a single station in a dense urban area under LOS conditions is simulated with the following configuration: single-carrier bandwidth of 400 MHz, 26 GHz band, new radio time division duplex, an uplink/downlink ratio of 1:3, and a subcarrier spacing of 120 kHz. Since mmW base stations are often used in hot spot coverage scenarios, their demand for the target rates should be higher than that of medium-and low-frequency base stations. Hence, the downlink target rate is set to 500 Mbps, and the uplink target rates are set to 5 Mbps and 1 Mbps (which are the requirements for the medium-and lowfrequency bands). Considering the deployment scenarios analyzed above, the typical movement speeds of a terminal can be divided into three types: approximately stationary (e.g., stadium bleachers), walking (e.g., large shopping malls), and cycling (e.g., office blocks). Table 3 shows the ranges of the cell that base stations with different EIRPs can reach at the target rate for the UE with each of the three speeds when the penetration loss is not considered. The data indicate that as the EIRP of the base station increases, the coverage capability is improved, so the range of the cell is expanded. When the EIRP continues to increase from 56 dBm, the limitation of the uplink rate causes the increase in the corresponding coverage distance to slow down. The coverage gain with the design of EIRP as 68 dBm is not notably different from that as 62 dBm, and excess  EIRP can substantially enlarge the antenna array, leading to increased weight, volume, and cost.
When a body penetration loss of 14 dB is considered [15], the range of the cell that the UE with each of the three speeds can reach at the target rate is shown in Table 4. Body penetration loss severely affects the coverage capability of base stations. With the same target, the range of cells with the consideration of body penetration loss is approximately one-third of that without such consideration. Considering the body penetration loss and when the terminal movement speed is very low, a 56 dBm base station can approximately cover a range of 250 m, with the downlink and uplink reaching 500 Mbps and 5 Mbps, respectively.

B. CAPACITY ANALYSIS
To study the capacity performance of base stations, base stations with three typical EIRPs (38 dBm, 47 dBm, 56 dBm) in coverage performance simulations are selected based on a network simulation to examine the effect of EIRP on the capacity. A 3GPP transmission model is used for dense urban scenarios (Fig. 8) with a station height of 10 m, station spacing of 250 m, and bandwidth of 400 MHz. The terminal has two channels of reception with a power of 23 dBm. Fig. 9 shows the simulation results of the uplink and downlink capacities of base stations with different EIRPs. As the EIRP increases from 38 dBm to 47 dBm, the capacity increases significantly. In comparison, as the EIRP continues to increase from 47 dBm to 56 dBm, the capacity further increases, but at a decreasing rate, which may be attributed to the increased intercell interference caused by excess EIRP.
The capacity testing of mmW base stations, such as the single-user throughput of a cell and the single-cell throughput, are consistent with that of medium-and low-frequency base stations.

V. PERFORMANCE TESTS OF COMMERCIAL mmW BASE STATIONS
The existing network test is conducted on the basis of commercial base stations and terminals (Fig. 10), in which the mmW base station has an operating band of 26 to 28 GHz, the maximum EIRP of 55 dBm, 8 RF chains, 512 antenna arrays, and 64 QAM as the highest modulation mode supported by uplink and downlink. The terminal uses the WNC Pocket 5G Router mmW R2 device, which operates at the  frequency range of 26-28 GHz, with a total EIRP of 23 dBm. The number of RF chains is two, and the number of antenna arrays is eight; the uplink and downlink supports the highest modulation method 64QAM. Because mmW base stations have not been deployed on a large scale in a field test, the test stations and the actual test environment are severely limited. As a result, only the evaluation items presented in Section 2 can be partially validated.

A. PERFORMANCE TESTS IN OUTDOOR SCENARIOS
Fixed-point tests, environmental tests, and handover tests are performed in outdoor scenarios. Tests are conducted based on two different single sites, respectively, which are the light pole at the intersection of Fangzhou Road and Nanxieyu Street and the pylon in Songzejiayuan District VI in the city of Suzhou, as shown in Fig. 10.
The main outdoor fixed-point tests are the remote test and the wrap-around test along the normal direction of the antenna array. Due to the limitations of the actual conditions, no fixed-point test is conducted in the vertical plane.
The downlink remote tests along the normal direction of the antenna array are carried out by using base stations with different powers. As the terminal becomes increasingly remote along the normal direction, the RSRP, signal and interference/noise ratio (SINR), and DL throughput are tested and recorded at various points along the direction. Due to traffic restrictions, only the walking speed is used to make the UE remote. The test results are shown in Fig. 11. The remote distances of the 40 dBm and 55 dBm base stations are approximately 210 m and 240 m, respectively. The peak rate on the test path of the 55 dBm base station is approximately 2.1 Gbps. It can be seen in the figure that as the EIRP of the active antenna unit increases, the coverage performance is improved, and the downlink rate increases. The test result with 52 dBm is similar to that with 55 dBm, while all indicators of the 40 dBm base station are relatively poor, verifying that the coverage capability of a base station can be enhanced by increasing EIRP. However, as EIRP increases, the coverage capability of base stations tends to stabilize. As shown in the figure, when the remote distance is approximately 190 m, the downlink throughput drops rapidly, which is mainly caused by the uplink limitation, as verified by the uplink remote test.
The wrap-around test in the fixed-point test is conducted by selecting the 55 dBm base station and using a wrap-around radius of 20 m, 50 m, and 100 m, respectively. The RSRP test results corresponding to different wrap-around radii are shown in Fig. 12.    The test results of downlink throughput in different wrap-around paths are shown in Fig. 13. The test trajectory with a wrap-around radius of 20 m covers approximately a range of 80 • . Due to the positions of buildings, the wrapping around is mainly concentrated on one side of the normal and reaches 10 • from the normal on the other side of the normal. The blockage of the buildings leads to a substantial decrease in various indicators. With a wrap-around radius of approximately 50 m, the angle between the trajectories is approximately 60 • , and the fluctuation of indicators in the wrap-around path is mainly caused by the obstruction from trees and bushes. With a wrap-around radius of 100 m, due to road restrictions, wrap-around of only approximately 20 • occurs, and the obstruction from trees and bushes is the main factor causing the decrease in indicators. It can be seen from the test results that, due to the transmission characteristics of mmWs, the coverage performance of mmW base stations is severely impaired in the environment with obstructing trees and buildings, which is an issue that should be paid attention in the actual deployment.
Environmental tests are all performed based on 55 dBm base stations, including the comparative tests with and without obstructions, as well as tree and rainfall tests.
In the comparative tests with and without glass obstruction, the distance between the test point and the antenna is 60 m, and the angle between the test site and the antenna normal is approximately 7 • . The obstructing glass is made of ordinary sheet glass with a thickness of approximately 0.5 cm. The test results are shown in Table 5. In the mmW scenario, glass obstruction leads to a level attenuation of approximately 8 dB    and a decrease of approximately 41% in the average downlink rate.
In the comparative tests with and without wall obstruction, the distance between the test point and the antenna is 60 m, and the angle between the test site and the antenna normal is approximately 7 • . As shown in Table 6, wall obstruction in the mmW scenario severely impacts the relevant indicators, leading to a level attenuation of approximately 14 dB and a significant reduction, approximately 56%, in the average downlink rate.
In the comparative tests with and without human body obstruction, the distance between the test point and the antenna is 50 m, and the angle between the test site and the antenna normal is approximately 5 • . Table 7 gives the test results with and without human body obstruction, showing that human body obstruction leads to a level attenuation of 14 dBm and a downlink rate attenuation of approximately 25%.
In the comparative tests with and without tree obstruction, the distance between the test point and the antenna is approximately 45 m, and the angle between the test site and the antenna normal is approximately 5 • . The obstructing trees are medium-sized ginkgo trees with a diameter of 30 cm at breast height, a height of approximately 15 m, and a generally dense canopy of approximately 3 m diameter. Table 8 lists the test results with and without tree obstruction, showing that the obscuration by trees leads to a level attenuation of approximately 7 dB and a reduction in the average downlink rate by 18%.
In the rain attenuation test, the above remote tests along the normal direction are repeated in light and moderate rain scenarios, respectively, to compare the effect of rain attenuation on the coverage performance of mmW base stations. In the moderate rain test, the remote distance is approximately 217 m, and the downlink rate reaches an instantaneous peak of approximately 2.0 Gbps. In the light rain test, the remote distance is approximately 240 m, and the downlink rate peaks at 2.1 Gbps. The results from the remote tests with and without rain attenuation, respectively, are shown in Fig. 14. For the remote distance in the range of 40 to 60 m, due to the obstruction by trees, the SINR indicator deteriorates and then recovers as the remote distance increases. As can be seen from the figure, the rainfall mainly causes slight deterioration of SINR and downlink throughput indicators, and as the remote distance decreases and the rain gets heavier, the impact is more severe. Comparison of the results from fixed-point tests at good, medium, and poor points in light rain reveals the following. Due to the light rain, the downlink rate and RSRP are reduced by approximately 4% and 0.1%, respectively, at the good point, by approximately 10% and  0.2 at the medium point, and by approximately 14% and 3 at the poor point. Rain attenuation has a very weak impact and a relatively large impact on the UE at the good point and the poor point, respectively, which is mainly caused by the poor quality and large fluctuation of the channel itself at the poor point. Overall, rainfall does not have a severe impact on the performance of mmW base stations.
As discussed in Section 2, to evaluate the beam management performance of mmW base stations, handover tests, including interstation handover, interbeam handover, and UE antenna angle handover, are required. Due to the limitations of terminal parameters, it is impossible to perform interbeam handover. Therefore, in the present study, intercell handover and UE antenna angle handover are tested first. The cell handover test is similar to the medium-and low-frequency base station test. After the UE enters the network, routes are selected to successfully conduct a total of 10 rounds of handover between two cells. The average delay of the handover control plane is 49 ms, which is relatively large and is mainly determined by the characteristics of the non-standalone (NSA) network.
When there is human body obstruction, the placement angle of the UE varies between 0 • , 90 • , 180 • , and 270 • , the distance between the test point and the base station is 50 m, and the angle between the test point and the antenna normal is approximately 5 • . The test results are shown in Table 9. The RSRP level does not vary much (less than 2%) with UE antenna angle. The placement angle of the UE has a small impact on the SINR (less than 4%) and has a larger impact on the downlink rate (approximately 15%). When there is no obstruction by a human body, the placement angle of the UE has even smaller effects on the test results, so all indicators basically remain unchanged. In summary, different UE placement angles lead to different UE antenna directions, and the results demonstrate that different UE antenna directions have little impact on service performance.

B. PERFORMANCE TESTS IN INDOOR SCENARIOS
The indoor tests are carried out in the exhibition hall on the fourth floor of the China Mobile in the Suzhou Industrial Park. The floor plan and site photograph of the test scenario are shown in Fig. 15. The area is a single-cell environment of a single station. Fixed-point tests, environment tests, and delay tests are conducted to evaluate the performance of a mmW base station in indoor scenarios. In the remote tests at fixed points along the normal direction of the antenna array of the base station, the remote test distance is 25 m due to site restrictions. The base station is configured with different transmit powers to evaluate its coverage performance. Fig. 16 shows the test results of various   indicators when the base station has an EIRP of 31 dBm. Among the results, the average RSRP is -88.58 dBm, corresponding to -110.44 dBm at 1% of the CDF and -75.30 dBm at 95% of the CDF, and the SINR is smaller than 25 dB over the entire remote path. The average downlink throughput is 1.44 Gbps, and rates greater than 1.5 Gbps can be achieved at positions closer to the station. Fig. 17 shows the test results from varying the EIRP to 34 dBm, 37 dBm, and 40 dBm. The coverage performance of the 31 dBm base station is significantly worse than that of the other base stations. The test results of the 37 dBm and 40 dBm cases do not differ significantly. It can be seen from Fig. 18 that as the distance between the terminal and the base station increases, the RSRP decreases linearly. That of the 31 dBm base station decreases most significantly. As the distance increases, the SINR exhibits an overall downward trend, and the test result of the 31 dBm case decreases the most. The results of the 37 dBm and 40 dBm cases show little difference. In terms of downlink throughput, the result of the 31 dBm case is significantly worse than that of other cases. At relatively close distances (within a range of 6 m), the downlink rates of the 34, 37, and 40 dBm cases are basically equal. As the distance increases, the downlink rate of the 34 dBm RF unit decreases fast, and those of the 37 dBm and 40 dBm RF units are essentially the same. In summary, as the EIRP increases, both the coverage and the downlink rate of the base station increase. When the EIRP is less than 33 dBm, the theoretical peak rate cannot be reached. As the  EIRP further increases, the peak rate obtained in the test is greater than 2.1 Gbps.
The fixed-point wrap-around tests are conducted with the 40 dBm base station as the center and selecting a wrap-around radius of 2 m (near point), 8 m (midpoint), or 17 m (far point. The coverage angle of the warp-around path is restricted due to the limitation of indoor scenarios. In the wrap-around tests with different distances, the downlink rate always peaks in the normal direction. As shown in Fig. 19, in the wraparound test with a radius of 2 m, the downlink rate of each point on the test path is maintained at 2.10 Gbps, with only small fluctuations (less than 100 Mbps). On the path of the wrap-around test with a radius of 8 m, the downlink rate remains at approximately 1.95 Gbps with relatively small fluctuation, and it drops to approximately 1.8 Gbps at 60 • from the normal. On the path of the wrap-around test with a radius of 17 m, the downlink rate in the normal direction is approximately 1.87 Gbps, which drops significantly with the deviation from the normal direction, reaching a downlink rate of approximately 1.46 Gbps at a deviation of 30 • from the normal. The wrap-around tests are conducted at RSRPs of -80 dBm and -90 dBm, which correspond to the distances of 8 m and 12 m, respectively. The test results of the two are consistent in that the downlink rate reaches the maximum near the normal direction and decreases with the deviation from the normal direction.
To evaluate the coverage capability of the base station beam in the vertical dimension, the measurement results at different angles on the vertical plane are obtained on the basis of selecting a test point at a straight-line distance of approximately 2 m from the base station and having an antenna electronic downtilt angle of 0 • . The measurement results are shown in Table 10, from which it can be seen that the base station can cover an area in the vertical range of 60 • , with deviation from the horizontal line and only slight deterioration of various indicators.
In the indoor scenario, environmental tests are conducted only to compare obstructions. Based on actual scenarios, typical concentrated obstructions, including glass, drawers, partitions, and the human body are selected to test the loss of mmWs. Table 11 summarizes the losses introduced by different obstructions. Tempered glass causes 3 dB of penetration loss, partition and drawer obstructions result in a loss of approximately 6 dB, and the human body loss is only approximately 1 dB.
Based on the current situation of the existing network, it is impossible to verify the SA network-based control plane delay. The single-user ping packet delay can be examined first based on the NSA network to test the user's ping packet delay at good, medium, and poor points, respectively.
The delay tests include the user plane delay test and the control plane delay test. Since the control plane delay is mainly determined by the type of network, the test at this stage is limited by the use of the NSA network. Therefore, the user plane delay is the main delay measured. The test results are shown in Table 12, which also lists the delay test results for the same brand terminal in the 2.6 GHz band. It can be seen that the UE is located at the 32-byte ping packet service of the good point, the average delay of 26 GHz is reduced by 28% relative to that of 2.6 GHz, and the delay of the 2000-byte ping packet service is reduced by up to 60%, verifying that the mmW base station has relatively good delay performance.

C. DISCUSSION
It can be seen from the indoor fixed-point test results that when the EIRP of the base station is 31 dBm, the downlink peak rate still cannot be reached even if the UE is at a good point. Therefore, to achieve good performance of the base station, it is recommended that the EIRP of the base station be greater than 31 dBm in indoor scenarios. However, blindly increasing the EIRP does not lead to greater gains but rather to greater interference and higher costs. Therefore, it is recommended that the EIRP of an indoor base station be no greater than 37 dBm. It can be seen from outdoor fixedpoint tests that the remote distance along the normal direction is determined by the constrained uplink distance. Therefore, increasing the EIRP by increasing the transmit power of the RF amplifier does not improve the coverage performance of the base station, and the design EIRP does not need to be greater than 56 dBm.
By comparing the results of tests with obstructions in outdoor scenarios and indoor scenarios, it is found that the human-body loss of indoor mmWs is only approximately 1 dB, and this value increases to 14 dB outdoors, which is much far greater than the indoor test result. This is because the mmWs are less impacted by obstructions due to the abundance of refraction and reflection paths in indoor scenarios.
When the indoor UE is in a completely enclosed environment such as drawers or partitions in the present tests, the reflection and refraction paths of mmWs are severely hindered, and hence the coverage capability of mmWs is greatly impaired.
In addition, the transmission characteristics of mmWs are verified through fixed-point tests, environmental tests, handover tests, and delay tests in different scenarios. Especially in outdoor scenarios, they are susceptible to obstructions and are related to season and weather; rain attenuation has little influence on good points but severely impacts poor points. These findings confirm that the performance evaluation method for the mmW base stations proposed in Section 3 can comprehensively investigate the performance of mmW base stations, as demonstrated by the evaluation test cases herein.
By comparing the results of the outdoor fixed-point remote test with the simulation results in Section 4.1, it can be seen that in all the results of base stations with different EIRPs, the actual remote distance is smaller than the cell range obtained by simulation. This is because when the cell range is evaluated through simulation, the free transmission loss based on LOS conditions cannot accurately reflect the path loss in actual scenarios. To simply estimate the remote distance from the simulation results, empirical factors can be added to the path loss expressed by Formula 3. For example, the frequency correlation coefficient C in this formula could be increased to align the measurement results and simulation results in the mmW band. The selection and verification of such empirical factors will be carried out in future research.
The glass losses are approximately 8 dB and 3 dB when tested in outdoor and indoor scenarios, respectively, which are not completely consistent with the evaluation result of 3 dB shown in Table 1. The difference between the indoor and outdoor test results is, on the one hand, caused by the difference in indoor and outdoor reflection and refraction paths and, on the other hand, related to the difference in the thickness of various types of glass. Therefore, the specific glass type, its thickness, and the environment need to be considered in the evaluation of the loss of mmWs penetrating the glass.
The wall loss according to the outdoor test is 14 dB, which is slightly smaller than the simulation result of 16 dB in Table 1. As in the test with glass obstruction, many factors affect the test results, which need to be evaluated based on the actual situation. By comparing the results of the rain attenuation evaluation model and the actual rain attenuation tests, it can be seen that the measured rain attenuation of 26 GHz mmW is basically consistent with the result given by the evaluation model [14], with an error at a level of approximately 1 dB/km, the influence of which can be largely ignored considering the coverage range of mmW base stations.
By substituting the frequency of 26 GHz and the thickness of the accumulated leaves of 3 m into Formula 2-6, the loss caused by leaves can be calculated to be approximately 8 dB, which is higher than the measured result. This is due to the obstruction by the selected trees, the large gaps between leaves, and the large rainfall at a depth of 3 m in the actual test. By comparing the results of the simulation model with those of the actual test, it can be seen that the existing model can mostly correctly reflect the penetration loss of mmWs in different obstructions in the NLOS path, but the specific loss needs to be tested with a combination of actual environment and obstructions.

VI. CONCLUSION
Based on the characteristics of mmW signals, which are different from medium-and low-frequency signals, the applicable scenarios of mmW base stations have been discussed, the performance of mmW base stations in different typical scenarios has been investigated, and a method of evaluating the performance of 5G mmW base stations in an existing network has been proposed. Its suitability is verified by combining simulation and field measurement. The test results have found that the EIRP of base stations deployed indoors and outdoors needs to be higher than 31 dBm and 52 dBm, respectively, to achieve good performance in terms of coverage range and throughput. The indoor scenarios are rich in refraction and scattering paths, and the additional loss is less than 3 dB when the UE is not fully obstructed. Field measurement results are of great importance to the design and actual deployment of mmW base stations.
In the future, efforts will be made to continue improving the performance evaluation of mmW base stations, including the evaluation of the delay performance of mmW base stations based on a SA network, use of UEs with different speeds to improve the handover test, and comparative tests of mmW base station performance in different seasons. The next steps of research also include the derivation and validation of empirical factors for the mmW path loss model.