Street Smart in 5G: Vehicular Applications, Communication, and Computing

Recent advances in information technology have revolutionized the automotive industry, paving the way for next-generation smart vehicular mobility. Specifically, vehicles, roadside units, and other road users can collaborate to deliver novel services and applications that leverage, for example, big vehicular data and machine learning. Relatedly, fifth-generation cellular networks (5G) are being developed and deployed for low-latency, high-reliability, and high bandwidth communications. While 5G adjacent technologies such as edge computing allow for data offloading and computation at the edge of the network thus ensuring even lower latency and context-awareness. Overall, these developments provide a rich ecosystem for the evolution of vehicular applications, communications, and computing. Therefore in this work, we aim at providing a comprehensive overview of the state of research on vehicular computing in the emerging age of 5G and big data. In particular, this paper highlights several vehicular applications, investigates their requirements, details the enabling communication technologies and computing paradigms, and studies data analytics pipelines and the integration of these enabling technologies in response to application requirements.

partitioning. Resultingly, sparse and heavy traffic frequently alternate leading to intermittent connectivity and network congestion. These dramatic changes introduce high latency variability which impacts the quality of service. These conditions complicate the deployment of vehicular applications that require real-time interactions and hinder the deployment of time-critical safety applications.
The development of connected vehicular systems paves the way for new services and business opportunities. The deployment of the technologies, infrastructure, and services relies on an interdisciplinary effort involving not only manufacturers, but also network operators, service providers, and governmental authorities. Network operators provide network access, whereas service providers provide access to specific services and bill subscribed users [3]. For example, service providers can collect real-time traffic data, detect traffic congestion, and disseminate such information to vehicles, either through RSUs or cellular communication [4]. Finally, government authorities play a critical role in aligning all actors towards providing safe, reliable, and interoperable road services. Such a collaboration opens up numerous possibilities for potential life-changing applications in the area of Intelligent Transportation Systems (ITS). These applications range from critical safety-related applications and driver assistance systems to location-based and traffic management services. The computation and communication resources involved in vehicular applications vary based on the application requirements, as shown in Figure 1. At one end of the spectrum, infotainment applications access the internet through mobile networks (LTE and 5G) thus allowing multimedia streaming or web browsing. On the other end, safety-critical applications rely primarily on vehicles, clouds, and edge servers for decision-making. These applications also tend to exploit reliable and lower-latency communication solutions, for example, cellular vehicle-to-everything (C-V2X). C-V2X enables vehicle to vehicle (V2V), vehicle to infrastructure (V2I), and other communication variants to address the real-time constraints.
Furthermore, road elements (traffic lights, lampposts, induction loop detectors, RSUs) and road users (vehicles, pedestrians' smartphones) produce traffic data resulting in significant data volume and heterogeneity due to sensor ubiquity and diversity. Thus processing heterogeneous and big data streams becomes another challenge. Additionally, as previously stated, the connection may be intermittent, resulting in massive data bursts that need to be processed in real-time. Analyzing this data and promptly extracting meaningful and useful information requires the consideration of specific big data architectures in the deployment of connected vehicular systems [10], [11].
Cloud computing (CC) enables ubiquitous, convenient, and on-demand access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) [12]. However, many applications have strict latency requirements, which makes edge computing (EC) a better candidate compared to remote centralized clouds. EC promises to deliver scalable, highly responsive cloud services for mobile computing and masks transient cloud outages. In contrast to CC, EC features proximity to the subscribing vehicles, context-awareness, dense geographical distribution, and support for mobility [13], [14]. For instance, EC-assisted traffic management enables monitoring the lane occupancy using traffic data and changing traffic signal phases accordingly.
There is a growing volume of research related to vehicular communication and computing. However, existing surveys on the topic of vehicular communication and computing lack several aspects such as the discussion of 5G integration, ML and AI-powered applications, diverse computing architectures (such as fog and cloudlet), and detailed case studies. Additionally, these surveys often have a single specific architectural focus, such as VCC [5], [6] or VEC [7], [9] rather than a more general vehicular computing viewpoint. Or they have complementary goals such as research taxonomies [8].
To help fill these gaps, we provide a systematic survey that studies the existing literature on vehicular computing, pinpoints the applications and requirements, highlights the methodologies, and determines the enabling technologies. Therefore in this work, we provide a survey that fulfills our comprehensive goals. For comparison, Table 1 summarizes the differences between related existing surveys and this work. As mentioned, the contributions of this survey are severalfold: 1) Challenges and Requirements. We characterize the vehicular environment including the challenges and requirements of real-world vehicular applications in Section III. 2) Bottom-up Overview. We provide a comprehensive study of existing communication technologies and computation paradigms in Sections IV and V. We also investigate big data analytics frameworks and ML-empowered vehicular applications in Section VI. 3) Integrated Architecture and Case Studies. We study the requirements and potential architectures of ITS systems at local, neighborhood, and city scales, and showcase real-world scenarios in Sections VII and VIII. 4) Future direction. We discuss insights and open issues, which shed light on the development of future novel vehicular applications and services in Section IX.

II. INTELLIGENT TRANSPORT SYSTEMS OVERVIEW
Intelligent Transport Systems (ITS) are traffic systems that leverage information, communication, and control technologies (ICT) to interconnect drivers, vehicles, and roads to enable smart, efficient, and safe transportation. Figure 2 illustrates the primary components that enable ITS applications to serve this purpose [15], [16]. Specifically, the data flow of a generic ITS system through varying components is as follows. Firstly vehicular or transport infrastructure sensors collect real-time spatio-temporal traffic data (e.g., vehicle speed, acceleration, direction, orientation, camera, LiDAR, location, and timestamp). These data are then processed on-board (the vehicle or infrastructure) or transmitted to computing nodes (edge or cloud) using networking protocols (Geo-Networking [17], [18], VANET, or WSMP) that leverage varying communication technologies (i.e., DSRC, ITS G5, or C-V2X). The computing nodes then aggregate, stream, map, pre-process and process the data using data analytic engines. ITS applications then use the processed data to provide services (such as intelligent traffic routing).
Given these general ITS components, we structure the remaining areas of the survey as such. We first discuss the multiple challenges that ITS applications face in Section III. Next we focus on ITS communication technologies and evaluate how these current and future technologies enable vehicular networking in Section IV. We then move on to computing architectures ranging from road to city scale in Section V. As a promising paradigm, we discuss edge computing methods including fog computing and multiaccess edge computing (MEC), a standardized ETSI architecture. Then we review different data analytics engines and algorithms to support ITS decision-making (often based on large volumes of heterogeneous data) in Section VI. Next we detail several detailed ITS application case studies in Section VIII. Finally, we conclude by discussing insights and open issues in Section IX.

III. CHALLENGES OF VEHICULAR APPLICATIONS
The deployment of vehicular applications will face multiple challenges relating to storage, computing, networking, privacy, and security.
A. DATA VOLUME, VARIETY, AND VELOCITY (3Vs) Connected vehicles contain a wide variety of sensors that continuously produce large amounts of data. RGB cameras alone generate 20 to 40 Mbps and radar sensors produce VOLUME 10, 2022 between 10 and 100 Kbps. The move from simply connected vehicles to autonomous vehicles will likely further increase sensor data volume (as autonomous vehicles will have more sensors). An autonomous vehicle may include 4-6 radar sensors generating 0.1 -15 Mbps per sensor, 1-5 lidar sensors generating 20-100 Mbps per sensor, 6-12 RGB camera sensors generating 500 -3500 Mbps per sensor, 8-16 ultrasonic sensors generating less than 0.01 Mbps per sensor, and vehicle motion, global navigation satellite system (GNSS) and inertial measurement unit (IMU) with <0.1 Mbps per sensor.  [15], [16].
The total sensor data rate is thus potentially between 3 Gbps (10.8 Tb/h) and 40 Gbps (144 Tb/h) [19]. Therefore, disseminating such massive data to remote servers or processing with near-real-time constraints are non-trivial issues.

B. LIMITED COMPUTING RESOURCES
The addition of thousands of new connected devices stresses not only the networks but also the computational resources (for example at RSU) [20]. Advanced driver-assistance systems (ADASs) and autonomous vehicles (AVs) with numerous onboard sensors will generate large amounts of data to be processed. Additionally, ensuring a holistic view of the ambient environment is beyond the capacity of any single vehicle, as it requires aggregating the points of view of multiple vehicles to recreate the entire scene [21], [22]. Therefore, off-loading computation is crucial to cope with such situations [23], [24].

C. RAPID TOPOLOGY CHANGE AND HIGH MOBILITY
The relative speed between vehicles ranges from tens of Km/h (vehicles travelling in the same direction on an urban street) to over 280 Km/h (vehicles travelling in opposite directions on a highway). Thus, vehicles may be members of a given VANET for only a very short time leading to rapid and frequent network topology changes [25], [26]. Additionally, traffic congestion can be predictable (e.g., due to rush hours) and unpredictable (e.g., due to traffic accidents) [27]. These phenomena lead to large volumes of data (due to frequent updates) in some urban areas which may run into limitations in VANET-based application scalability [28].

D. DETRIMENTAL DELAY
Propagation and queuing delay are major sources of delay in ITS. Propagation delay refers to the delay resulting from data actually traveling over the communication medium and thus depends on the medium type, such as air for V2V communication, and the physical communication distance. While queuing delay denotes the network delay of data while waiting in network queues to be initially sent by the sender, forwarded by intermediate nodes, or processed by the destination. Queuing delay depends on the number of transmitting vehicles and the volume of data sent by each vehicle (i.e., the network traffic), the available number of links between the source and destination, max queue lengths on the nodes, and the service policy (e.g., critical safety or non-safety) which determines the queuing prioritization [29]. Overall, the networking overhead and latency associated with remote cloud resources could degrade the overall performance and prove detrimental to road safety [20], [30], [31]. For example, in autonomous driving the sensors and control logic operate at the millisecond scale (compared to the human perception and reaction cycle of 1-3 seconds), thus typical cloud delays of tens or hundreds of ms can significantly impact decision making. For highway speeds of 100km/h a delay of 100ms could cause a positioning error of up to 2.7m (or more than the roughly 1.5m separation between large trucks in adjacent lanes on a highway). Moreover, traffic volume is increasing with user demand and will further heavily burden backhaul links and lead to even longer latencies [32].

E. SECURITY AND PRIVACY CONCERN
Some ITS applications and services require vehicles and RSUs to exchange messages containing potentially sensitive information such as real-time locations. This communication takes place over networks that are by design somewhat easily accessible 1 thus prompting security and privacy challenges [33]. Furthermore, the richer the data sharing the more potential exists for tactics like predatory marketing and user tracking [34]. As the number of services grow, passengers may also use services over multiple networks, thereby magnifying the potential for cyber-security problems or privacy violations. Specifically, the cyber-security issues include a diverse array of attacks such as distributed denial of service, malware for remote vehicle control, sybil attacks, replay attacks, and timing attacks among others [35]. However in comparison to many other computing contexts, in ITS these issues can have life or death consequences for both the target and bystanders.

IV. COMMUNICATION TECHNOLOGIES AND NETWORKING
Vehicles and roadside infrastructure use multiple wireless technologies to communicate. The most promising wireless communication technologies can be classified into short-range communications such as dedicated short-range communication (DSRC) [36] and ITS-G5 [37], [38], and long-range communications including long term evolution (LTE) and 5G. These technologies vary according to their range, capacity, and communication latency. Each technology is thus suitable for a specific class of applications.
A. SHORT-RANGE COMMUNICATION (DSRC AND ITS-G5) Dedicated Short Range Communication (DSRC) is a vehicular communication technology that typically operates in licensed spectrum in the 5.9 GHz band in the several countries including the united states. DSRC allows vehicles and RSUs to form vehicular ad-hoc networks through vehicle-tovehicle (V2V) and infrastructure-to-vehicle (V2I) communications. DSRC leverages the interoperability between several standards that form its protocol stack. IEEE 802.11p [39], a derivative of the IEEE 802.11 (WiFi) standard, is the native technology at the physical and medium access control layers. In data link, network, and transport layers, DSRC employs a family of IEEE 1609 standards: IEEE 1609.2 [40], 1609.3 [41], and 1609.4 [42] for security, network services (including the WAVE short message protocol -WSMP) and multi-channel operation. WSMP is a bandwidth-efficient protocol for exchanging single-hop messages and non-routed data. WSMP sends packets referred to as WAVE short messages (WSMs). IEEE 802.11p and IEEE 1609 standards allow vehicles to operate in a rapidly varying environment and exchange messages either without having to join a basic service set (BSS) or within a WAVE BSS [36], [43]. DSRC can enable vehicular collision prevention applications that depend on periodic data exchanges among vehicles and between vehicles and roadside infrastructure with strict round trip latency, broadcast frequency, and packet error rate requirements [44]. According to ETSI, cooperative collision avoidance requires a guaranteed maximum latency time of 50 ms and a minimum frequency of 10 hz to broadcast pre-crash state in cooperative awareness messages that are associated with direct V2V communication [45]. DSRC fulfills the requirements of such applications while also providing high security and low latency direct communication between entities, without involving a centralized network infrastructure [36].
ITS-G5 is an analogous European technology for vehicular communication that also uses the 5.9 GHz frequency band but is adapted to European requirements. The ITS-G5 standard is developed by the European Telecommunications Standards Institute (ETSI) to guarantee interoperability among communication devices from different manufacturers. Similar to DSRC, it carries V2V and V2I in an ad-hoc fashion. They are different only in the way they access the shared channel [37], [38]. DSRC employs an alternating access scheme with Enhanced Distributed Channel Access (EDCA) subsystems for each respective channel type, whereas ITS G5 uses a model consisting of state machines and different tunable parameters to control the medium access of all nodes. The VOLUME 10, 2022 ITS-G5 standard also adds features for decentralized congestion control methods to manage the network load [38], [46].

B. LONG-RANGE COMMUNICATION (LTE AND 5G)
The Long term evolution (LTE) standard is mobile communications standard developed by the 3rd Generation Partnership Project (3GPP 2 ). The LTE system infrastructure comprises a core network, also known as an evolved packet core (EPC), and an access network, referred to as an evolved universal terrestrial radio access network (E-UTRAN). Further details on the basics of the LTE architecture can be found from 3GPP [47].
In the vehicular context, 3GPP has developed LTE Cellular-V2X (LTE C-V2X) to operate in 5.9 GHz band (similar to DSRC) in addition to the licensed carriers via network infrastructure. This enables direct communications in the absence of cellular infrastructure in a distributed manner [48]. LTE C-V2X works in two transmission modes to support ITS services: 1) C-V2X/PC5 which supports V2X direct sidelink communications, allowing vehicles and RSUs to inter-communicate directly without the need for infrastructure, and thus providing lower delay, higher throughput, lower energy consumption, and better spectrum utilization [49], 2) C-V2X/Uu communications to connect road users (e.g., vehicles and RSUs) indirectly through LTE infrastructure. In this mode, since the V2X transmissions are scheduled, interference and collisions are lessened [48].
3GPP has been further developing LTE C-V2X to leverage the fifth-generation (5G) mobile communications standard, thus leading to New Radio (NR) C-V2X which will be compatible with the evolution of 5G. NR C-V2X is gaining global momentum with deployments in many countries [52]. In general, 5G aims to support ultra-reliable low-latency communication and ultra-high throughput (10-100× higher compared to LTE). Similar to LTE C-V2X, sidelink mode allows direct communication between vehicles, while indirect (via infrastructure) mode works inside the coverage range of a gNodeB. However, NR C-V2X supports unicast, group cast, or broadcast transmission modes while LTE C-V2X only supports broadcast transmission mode. The 5G base ensures interoperability with earlier communications systems such as LTE (i.e., non-standalone) and provides faster and more reliable telecommunications for vehicular applications. Additionally, 5G supports integration (beyond simple IP-based connectivity) with several non-3GPP communication and telecommunication systems including WiFi variants, ZigBee, and Bluetooth. This integration provides vehicular networks with more flexibility, allowing vehicles, drivers, passengers, and pedestrians to leverage the most suitable system for their selected application [53].
In addition, a variety of 5G and 5G-adjacent technologies including software-defined networks (SDN), network function virtualization (NFV), and multiaccess edge computing (MEC) are accelerating the ongoing migration of intelligence closer to the users. These paradigms are the building blocks of the network softwarization trend in mobile networks [54]. Overall, the 5G ecosystem enables vehicle manufacturers, solution integrators, network and service providers, and small and medium-sized enterprises (SMEs) to efficiently compete and cooperate. Within the 5G system, end-to-end network slicing, service-based architecture, software-defined networking (SDN), and network functions virtualisation (NFV) are the fundamental pillars to support the heterogeneous key performance indicators (KPIs) of the new use cases in a cost-efficient way. SMEs will be able to provide technological solutions which comply with the overall system design. Manufacturers and solution integrators can offer rapid deployment enabled by virtualisation and standardised interfaces to assimilate the advent level of innovation. Mobile network operators (MNOs) and infrastructure providers will create tailored slices with specific functionalities and services to address the requirements of vertical industries [55].
The international telecommunication union (ITU 3 ) envisions the capabilities of future mobile networks in the international mobile telecommunications-2020 (IMT-2020) standard. The capabilities entail flexibility, reliability and security when providing various services in three intended usage scenarios, enhanced mobile broadband (eMBB), ultrareliable and low-latency communications (uRLLC), and massive machine-type communications (mMTC). ITU sets the guidelines for 3GPP to create and maintain the technical standards for 5G technologies.
The 5G ecosystem and defined use cases (e.g., enhanced mobile broadband (eMBB) and ultra-reliable low-latency communication (uRLLC)) are promising enablers of ITS services and applications. For instance, passengers can watch an HD movie while the driver is using augmented reality applications to detect road hazards with real-time and visually interactive navigation (usage of eMBB). Figure 3 illustrates and example architecture to achieve network slicing and vehicular application use cases. While Figure 4 illustrates the importance level (Low, Medium, or High) of the key capabilities of 5G in addressing these different use cases. Figure 4a illustrates these levels for infotainment applications such as video streaming, augmented and virtual reality, and mobile cloud gaming for passengers during commuting. These applications belong to the eMBB use case, whereas critical safety and time-sensitive applications belong to the uRLLC use case, featuring stringent requirements for reliability, latency, and continuous, seamless connectivity [50]. Figure 4b illustrates the levels for critical safety applications. While Figure 4c illustrates the levels for autonomous driving (AD) requirements. AD requires ultra-high reliability, low latency, and high bandwidth, a combination of uRLLC and eMBB use cases. Finally, Figure 4d illustrates the levels for efficiency and traffic management applications [56], [57].  [50]. The car leverages two slices, a uRLLC slice to support road safety or autonomous driving, and an eMBB slice for video streaming or augmented reality. While a train leverages an mMTC slice to handle the massive smartphone traffic of the passengers [51].
These applications are more resilient and less dependent on latency and reliability compared to safety applications.

C. COMPARISONS AND COEXISTENCE
There has been extensive research into comparisons between DSRC and LTE C-V2X technologies [58]. Such comparisons have shown that the best technology depends somewhat on the deployment scenario (e.g., dense urban roads vs. highways) and the application. For example, 5G Automotive Association (5GAA) and Papathanassiou et al. [59], [60] have conducted extensive experiments to compare their suitability to deliver vehicle-to-everything broadcast safety messages. They confirm that LTE C-V2X significantly outperforms DSRC in various key areas. Therefore, LTE C-V2X seems to be a promising candidate to enable these ITS services and applications. However, DSRC has undergone several large-scale field trials and is already in production in the US, Europe, and Japan. In fact, the coexistence of both DSRC and LTE C-V2X is likely in some regions. Therefore, Ansari et. al. [58], for example, emphasize the need to enable V2X communication regardless of the underlying technology (DSRC or LTE C-V2X). As such, a hybrid V2X system is a potential comprehensive solution. The hybrid V2X scheme could apply spectrum sharing techniques such as frequency division multiplexing with a guard band. Additionally, performance and scalability issues of both IEEE 802.11p DSRC and LTE C-V2X/PC5 Mode 4 have been driving the future developments of IEEE 802.11bd DSRC and NR C-V2X [58]. The coexistence of these new future standards is an open research area.

D. SOFTWARE-DEFINED NETWORKS (SDN) DRIVEN VEHICULAR NETWORKING
Software-defined Networking (SDN) is a networking approach wherein software-based controllers direct traffic on the network (rather than dedicated hardware-based controllers). This approach enables the separation of the data plane from the control plane thus allowing an abstraction layer that standardizes the interfaces to different devices within the network. Overall, this simplifies network VOLUME 10, 2022 FIGURE 4. The importance levels (High, Medium, Low) of 5G performance indicators for different classes of applications [56], [57]. management and configuration and enables greater heterogeneity in vehicular networks. SDN also adds a network programmability feature via these external controllers. Consequently, SDN provides flexibility in developing vehicular network infrastructure and enables dynamic network resource allocation and centralized control. This allows rapid configuration management in view of the dynamic nature of vehicular networks and efficiently integrates multiple network technologies (e.g., DSRC, C-V2X) [61]. In relation to 5G, the combination of 5G and SDN enhances the 5G capabilities and supports the dynamic nature of vehicular networks by providing intelligence, resilience and flow programmability in 5G-enabled vehicular networks [62]. There have been many SDN-enabled three-tier (data, control, and application plane) frameworks. These enhance security and mobility management in 5G-enabled vehicular networks. Garg et al. [63] propose an SDN-enabled framework for 5G-enabled vehicular networks. The framework provides end-to-end security and privacy and simple network management through SDN. Specifically, through authentication and intrusion detection modules in the application plane, the framework supports mutual authentication among the peers and identifies network intrusions. SEARCH [64] is an SDNenabled three-tier architecture for better vehicle path planning. SEARCH exploits Unmanned Aerial Vehicles (UAVs), VANETs, 5G-based cellular systems, and SDN to provide better and faster communication to changing road conditions. The SDN part specifically brings flexibility, scalability, and programmability thus allowing better path planning and quicker journeys.

E. SECTION SUMMARY
While DSRC and ITS-G5 both support direct communication between vehicles (V2V) and with roadside units (V2I), NR C-V2X can provide both direct (sidelink via PC5) and indirect (Uu via 5G infrastructure) communications. NR C-V2X also supports transmitting larger data volumes directly or indirectly over short distances. Though DSRC and C-V2X could potentially co-exist through hybrid systems. In terms of additional technologies, SDN enables dynamic network resource allocation and centralized control, allowing rapid configuration management thus fitting the dynamic nature of vehicular networks. Therefore the combination of 5G and SDN provides intelligence, resilience and flow programmability to 5G-enabled vehicular networks.

V. COMPUTING PARADIGMS
External computing resources (outside the vehicle) are important for an ITS as such resources help with aggregation and fusion of heterogeneous data from multiple road users (thus providing a holistic view) [21]. They also help ensure that complex applications are accessible (through computation offloading) regardless of the capabilities of the vehicle. For example, traffic management, emergency management, fleet management, and intelligent navigation (e.g., through augmented reality overlays on the windshield) are complex applications that might require offloading of complex tasks to external compute and storage resources [57]. These resources can be remote computing resources (such as cloud servers) or intermediary nodes such as multiaccess edge computing (MEC) servers and fog computing nodes.

A. CLOUD COMPUTING
The NIST (American National Institute of Standards and Technology) defines cloud computing as a model for enabling ubiquitous, convenient, and on-demand access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services). These resources can be rapidly provisioned and released with minimal management effort or service provider interaction [12]. Cloud computing has so far been the dominant paradigm in terms of offloading intensive computation from vehicles. For instance, Toyota's connected car architecture leverages Microsoft Azure HDInsight to process millions of vehicle events a day. Furthermore, Toyota equips its vehicles with a data communication module to transmit the vehicular data to a Toyota smart center. The latter provides a mobility services platform that enables public companies to offer Toyota and Lexus vehicles various services. In other words, SMEs can provide technological solutions which will integrate with the overall vehicle system.
Beyond traditional cloud services, the huge vehicular fleets on our roadways, streets, and parking lots can be seen as massively underutilized computational resources. Given this framing, Vehicular Cloud Computing (VCC) has also emerged as a new hybrid technology that incorporates vehicular ad-hoc networks and cloud computing. In this paradigm, the underutilized vehicle's resources, computing power, internet connectivity, communication resources, and storage, are shared or rented over the Internet to various customers [65]. Through seamless and decentralized management of cyberphysical resources, VCC provides third-party or community services at low cost and enables efficient utilization of vehicle resources. Additionally, due to vehicle mobility, agility and autonomy, VCC can dynamically adapt the managed vehicular resources allocated to an application according to the dynamically changing requirements and system conditions. However, such a paradigm still faces high relative latency and high communication costs [66], [67]. In practice, stationary vehicles or mobile vehicles are controlled by cyber-physical resource management software to form VCCs. VCCs can thus be categorized into two classes: static VCCs and dynamic VCCs. These classes are suitable for different vehicular cloud services or applications [65]. VCC viability is further enhanced by 5G deployment as 5G provides capabilities, such as large bandwidth, ultra-reliability, low-latency, and V2V communication through 5G side-links, that will support even more VCC use cases.
The proven economic benefits of cloud computing make it likely to remain a permanent feature of the future computing landscape. However, the network overhead and latency of remote cloud computing cannot meet the requirements of time-critical applications and thus proves detrimental to overall network performance. Additionally, cloud computing lacks context-awareness that, for example, captures spatio-temporal traffic and driving patterns.

B. EDGE COMPUTING
Edge computing (EC) is a distributed computing paradigm that places computational resources and storage geographically close to end users (for example, vehicles and RSUs). Thus service requests typically travel a much shorter physical distance (and traverse fewer network nodes) for processing compared to requests to typical remote cloud servers. This results in significantly lower latency. Additionally, EC can complement cloud computing by masking transient cloud outages and can naturally better capture contextual and situational information due to the proximity to end users [13]. Overall, EC promises to deliver scalable, reliable, and low latency cloud services.
Edge computing encompasses three distinct frameworks in the context of vehicular networks: vehicular fog computing (VFC), multiaccess edge computing (MEC), and mobile vehicular cloudlets (MVCs). Figure 5 illustrates the architectures of these three frameworks.
Multiaccess Edge Computing (MEC) is an edge architecture standardized by ETSI 7 that brings edge computing to the mobile network context. Specifically, MEC locates computing resources at the edge of the mobile access network, typically at the first aggregation level (base stations) [82]. Being an open standard, MEC also creates a standardized and open environment that enables operators to open their radio access network (RAN) edge to authorized third-parties, to flexibly and rapidly deploy innovative applications. This new ecosystem allows different vehicles, manufacturers, and transportation agents to more seamlessly integrate their applications.  [32], [68], [70]. MEC also enables applications and services to be hosted on top of the mobile network elements [73], [83]. Different deployment scenarios address various performance, costs, scalability, and operator deployment preferences: • Deployment at the radio node (eNB or gNB). • Deployment at aggregation points (LTE EPC or 5GC). • Deployment at the edge of the Core Network (e.g. in a distributed data center, at a gateway). Figure 5 illustrates a MEC deployment where edge servers are deployed with cellular base stations including LTE evolved Node B or 5G NR gNode B.
Fog Computing (FC) in the vehicular context refers to any intermediary computation, storage, and network services between vehicles and the cloud [84]. Specifically, there are rich scenarios of connectivity and interactions in vehicular networks: vehicle to vehicle, vehicle to access points, smart traffic lights and roadside units (using Wi-Fi, DSRC), vehicle to network (using LTE, 5G), and other V2X scenarios. For instance, a smart traffic light node interacts locally with many sensors, which detect the presence of pedestrians and bikers, and measure the distance and speed of approaching vehicles. The smart traffic light in this context acts as a fog computing node (FCN). A fog computing node (FCN) can be any node with communication, computation, and storage resources. As shown in Figure 5, a FCN can be a moving or parked vehicle, also referred to as vehicular fog computing (VFC) [85], a roadside unit, or an edge device installed in a cellular base station. Vehicular Fog Computing (VFC) aggregates the abundant resources of individual and connected vehicles and exploits their available computing resources to enhance the application quality of service. VFC uses moving and parked vehicles as FCNs to offload computation tasks and provide networking services [77], [86], [87]. The fog has several characteristics which make it the ideal platform and non-trivial extension of the cloud to deliver services in infotainment, safety, traffic efficiency, and analytics. These characteristics are 1) low latency and thus real-time interaction, 2) widespread and geo-distributed deployment, 3) location, mobility, and context-awareness, and 4) interoperability, federation, and heterogeneity (deployable in various environments) [88], [89].
Cloudlet Computing represents an architecture with auxiliary proximate cloud resources for providing highly responsive services. Specifically, these cloud resources can be viewed as delegates or proxies of the real cloud and are located at the middle tier of a three-tier hierarchy, as shown in Figure 5. A cloudlet can be either a mini data center in a box [90], [91], or vehicular resources referred to as a mobile vehicular cloudlet (MVC). As an example use case, during cloud or backhaul outages, the cloudlet takes over the responsibilities and masks the outage [92]. Adjacent vehicles and roadside units can connect via DSRC communication or 5G sidelink to form MVCs. Thus MVCs harness the computational resources of the adjacent nodes in a timely and efficient manner via peer-to-peer communication. An MVC is a cluster of smart vehicles and RSUs located in a region. Such Vehicles and RSUs can share resources and information via V2V communication or indirectly via V2I communication [65], [93].
Overall, these edge computing architectures have similarities such as similar goals. However, they also have slight differences in terms of origin, the deployment location, the involved nodes, the access technologies, the geographical proximity, the level of contextual awareness, and the latency. Table 3 summarizes the differences between edge computing architectures (FC, MEC, Cloudlet) and compares them with cloud computing architectures (CC, VCC).

C. SECTION SUMMARY
The synergy of NR C-V2X and edge computing (EC) can lower the end-to-end latency significantly, allowing stakeholders to use bandwidth efficiently, and enabling time-critical applications. Edge computing provides highly responsive services for vehicular users and improves the utilization efficiency of next-generation mobile networks. VCC increases the overall utilization efficiency by leveraging the dispersed underutilized resources of vehicles. Specifically, vehicular networks can be employed to remotely offload latency-tolerant computation (into moving or parking vehicles) and storage services (into parking vehicles), or locally offload latency-sensitive computation (into moving vehicles) and caching (into moving vehicles).

VI. DATA ANALYTICS: TECHNOLOGIES AND METHODOLOGIES INTEGRATION
Road users and elements (e.g., pedestrians' smartphones, vehicles, lampposts, traffic lights, or other RSUs) generate both mobility and service-related data which is heterogeneous and large in volume. Analysing this data and extracting useful and relevant information in real-time requires an efficient data analytics architecture. This architecture must support multiple data sources, handle large data volumes, enable data streaming (to achieve low latency), and allow developers to plug-in queries and machine learning algorithms. A variety of specific technologies and frameworks can be combined to actually realize such an architecture, we briefly describe some of the most common technologies and frameworks. Additionally, we study several specific architectures from literature and two case studies.

A. DATA ANALYTICS TECHNOLOGIES
Firstly, in terms of data storage technologies, Hadoop Distributed File System (HDFS) is a distributed file system for reliably storing large amounts of unstructured, semi-structured, and structured data as files (typically on disk) [101]. HDFS was one of the first large-scale distributed file systems for big data. Several other big data storage systems actually build on top of HDFS. HBase, for example, is a key-value pair NoSQL database with master-slave replication that leverages HDFS as underlying storage [102]. Other notable systems include Cassandra, a popular keyvalue pair NoSQL database with asynchronous masterless replication [103].
In terms of data messaging, collection, and aggregation, the current dominant system is Apache Kafka [104]. Apache Kafka is a distributed event streaming system. Specifically, Kafka provides a distributed publish-subscribe messaging system that allows for decoupling of different stages of data pipelines. Kafka accommodates big heterogeneous data and Kafka event streaming includes true (event at a time) streaming with exactly-once semantics.
Finally, in terms of actual distributed computing and ML, Hadoop MapReduce is a distributed computing framework for the parallel processing of large datasets often stored on disk on HDFS (though other storage solutions are also supported) [94]. MapReduce runs on a Hadoop cluster and often leverages a cluster manager like YARN to schedule applications and services on the cluster and manage the cluster resources like memory and CPU. In comparison to the primarily disk-based MapReduce, Apache Spark is a unified big data analytics engine for distributed in-memory data processing [105]. Furthermore, Spark provides both batch and stream processing, libraries for machine learning, and an SQL-like interface. Relatedly, Apache Flink is also a big data analytics engine for distributed processing [98]. A few of the major differences between Spark and Flink are that Spark is more mature with a larger community, while Flink was designed specifically for stream processing and thus provides better support for true (event at a time) streaming. In contrast, Spark primarily supports micro-batch streaming. Kafka also provides some true (event at a time) stream processing functionality (through the Kafka streams API).

B. INTEGRATION OF TECHNOLOGIES
In vehicular environments, the big data analytics architecture relies on the integration of such technologies into a pipeline that enables computation offloading. Figure 6b illustrates a potential pipeline that integrates various technologies to process traffic data in real-time. Firstly, Apache Kafka ingests the live vehicular/traffic data and partitions the data into distinct topics which enables multiple readers and writers to operate simultaneously thus improving scalability. A separate data fusion module facilitates fusing and aggregating the topics' data for richer features and better VOLUME 10, 2022 FIGURE 6. Traffic data analytics pipeline (a) Illustrates traffic data collection, aggregation and fusion, streaming into micro-batches, and processing. (b) Online and historic traffic data collection, aggregation, and fusion (using Hadoop [94], Kafka [11], [95], or Flume [96]), micro-batch streaming then ML-assisted processing (using Flink [97], [98] or Spark Streaming and MLLib [10], [99], [100]).
context determination. Spark Streaming then either consumes data from specific Kafka topics or retrieves data from HDFS, and splits the data into micro-batches to feed into Spark MLlib which applies ML algorithms. Apache Flink could replace spark to achieve a very similar setup. The ML algorithms output meaningful information (such as predictions) which is often sent back to applications or services to assist driving or traffic control, or stored permanently in stable storage such as Hadoop HDFS or Cassandra for later use.
In terms of example integrated pipeline technologies, Amini et al. [11] employ Apache Kafka to stream traffic data in real-time and control traffic lights in a distributed manner. Carbone et al. [98] implement an Apache Flink-based platform to perform both stream and batch analytics. The platform enables pipelined fault-tolerant dataflows and supports many classes of data processing applications, including real-time analytics, continuous data pipelines, historic data processing (batch), and iterative algorithms (machine learning, graph analysis). To detect unsafe driving activities, Alhilal et al. [10] integrates Apache Kafka and Apache Spark. Specifically, Kafka collects and aggregates the vehicle data, while Spark streams and divides the data into micro-batches, then processes the batches using Spark MLlib. Anveshrithaa et al. [100] develop a real-time data stream processing model for forecasting vehicle traffic. The analytical framework integrates Spark and Kafka along with deep neural networks (i.e., Long Short-Term Memory-LSTM), where the traffic data is streamed from Kafka into the machine learning model in the Apache Spark engine. The model is intended to predict traffic flow information that assist to reduce travel time and cost. Hu et al. [97] develop a distributed dynamic pyramid map tile generation method (DPTG) based on Apache Flink. The method connect real-time data flow sources, specifically Apache Kafka. DPTG can quickly visualize real-time spatial traffic data with digital map tiles. DPTG has high efficiency and scalability in both batch processing and stream processing mode which helps to support real-time traffic monitoring data processing for timely large-scale public service in ITS.

C. ML-EMPOWERED APPLICATIONS
Traffic-related learning tasks can be primarily sorted into two main classes, basic safety and advanced efficiency. Collision warning and traffic incident detection are examples of basic safety, whereas traffic flow prediction, car-following, and driving behavior recognition are examples of advanced efficiency. To implement these tasks a machine learning algorithm is applied to traffic data in a data analytics pipeline (see Figure 6). We briefly discuss the details of these different task classes including example systems from research studies.
Traffic Flow Prediction is the real-time short-term prediction of traffic on the road network that assists in understanding the future traffic state. This prediction can leverage both longer-term historical traffic data (e.g., diurnal patterns) and up-to-date signals of traffic conditions. Such prediction plays a significant role in road network traffic planning and traffic control optimization and lays the foundation for, travel guidance, navigation, and other mobility services. In terms of existing work, studies have applied a variety of temporal ML algorithms including NN-based and more traditional algorithms to the task. The most sophisticated NN-based models including long short-term memory (LSTM), stacked LSTM, temporal LSTM, and spatial-temporal autoencoder LSTM (SpAE-LSTM) outperform the more traditional multilayer perceptron (MLP) model, decision tree model, and support vector machine (SVM) models [106], [107], [108]. In terms of details, SpAE-LSTM, for example, is a hybrid model consisting of a sparse autoencoder and an LSTM. The sparse autoencoder captures the spatial features while the LSTM captures the temporal features [108]. In fact, many of the related models leverage autoencoders as they can learn generic traffic flow features and obtain the internal relationship of traffic flow [109], [110], [111].
In addition to short-term prediction, trend-modelling of traffic can facilitate longer-term traffic forecasting. Such forecasting relies on the implicit temporal correlations among the time series observed on different days/locations due to human diurnal patterns. Specifically, the daily traffic time series at a certain location have similar M-shapes over consecutive days, in which the morning and evening rush hours correspond to the two peaks of the M-shape [112]. In terms of example studies, Li et al. [112] use principle component analysis (PCA), a well-known mathematical dimensionality reduction and feature extraction procedure, to project the traffic time series onto an n-dimensional orthogonal linear space such that the data with the k th largest variance by projection lies on the k th dimension. PCA trend can establish a link between traffic time series collected in different days/locations because the observed daily data series share the same set of latent variables. PCA also can assist to predict whether the traffic is normal or abnormal by comparing the distances between their projections in the latent space.
Traffic Incident Detection is the detection of real-world traffic incidents in some given spatiotemporal area. This detection is essentially a mining task from heterogeneous traffic data. This data can include sources such as road sensors, traffic cameras, and even social media messages from, for example, Twitter and Facebook (which are popular, real-time in nature, and may contain useful text and meta-information such as timestamp, geographic coordinates, links, hashtags, and mentions).
In terms of existing research implementations, D'Andrea et al. [113] present a real-time traffic event monitoring system that leverages Twitter stream analysis. They employ SVM to classify tweets as traffic event related with very high accuracy (>95%). Relatedly, Traffic Events Detection and Summary (TEDS) [114] uses natural language processing, spatial-temporal mining, and wavelet analysis techniques to create a traffic incident map with text summarizations (from multiple same-incident tweets) from Twitter data.
Vehicle following is the systematic control of vehicles' velocity to optimize and maintain safe, comfortable, and convenient traffic flow. To this end, car-following models set the velocity of a following vehicle in response to actions of a lead vehicle [122]. Many works have developed reinforcement learning (RL) models which control the vehicle's velocity to optimize traffic flow. Meixin et al. [120] develop a deep RL model which uses a reward function reflecting driving safety, efficiency, and comfort to fulfil the multiple objectives of the car-following model. This reward function leads to more efficient traffic flow compared to human drivers. Additionally, a collision avoidance strategy is incorporated for safety and to improve model convergence. Relatedly, Wang et al. [119] use deep RL to control lane-changing behavior for each vehicle with a reward function defined as a trade-off between the vehicle's travelling efficiency (i.e., how efficiently a vehicle maintains a target speed), traffic flow rate, and level of cooperation between the vehicles. Specifically, they utilise a deep Q-network (DQN), a type of deep neural network, as the RL model and the lane-changing of each vehicle is formulated as a Markov decision process (MDP). The MDP state is defined by the vehicle's state (at a given time t) which consists of three sequential frames of traffic snapshots and the corresponding speed difference between the actual and target speed. The action space A(t) is the corresponding driving decision such as switch to left or right lane, speed up by a fixed increment (up to a maximum speed), or maintain the current speed. Walraven et al. [123] also apply MDP, Q-learning, and neural networks to learn policies dictating the maximum allowed driving speed on highways to reduce traffic congestion. The mentioned systems are based on RL with a reward function that improves the overall traffic efficiency instead of the travel efficiency of a given individual vehicle. In short, cooperation leads to a more harmonious and efficient traffic system. Driver Behavior Recognition is the classification of a driver's behavior into classes such as normal, aggressive, or distracted driving. The classifier output is conveyed to the drivers audibly, visually, and/or haptically through an alert (typically via the infotainment system) and thus allowing them to react in time. Such feedback promotes safer driving, reduces traffic accidents, and contributes to social safety [124]. Distracted driving is an especially relevant and dangerous driving behavior. Distracted driving can be defined as any activity that diverts the driver's attention from driving including talking or texting on a mobile phone [125] or using on-board entertainment or navigation systems. In the US, 3,166 people in 2017 [126] and 3,477 people in 2015 died in motor vehicle crashes involving distracted drivers. In terms of example recognition systems, Celaya-Padilla et al. [115] leverage a ceiling-mounted wide-angle camera that feeds data to a convolutional neural network (CNN) to detect distracted drivers. The detection of distracted driving can then be conveyed to the driver audibly, visually, or haptically using, for example, the infotainment system. Relatedly, DarNet [116] is a framework utilizing CNNs and recurrent neural networks (RNNs) to process images (from a driver-facing camera) and inertial measurement unit (IMU) data (from the driver's mobile device) to detect distracted driving behavior. These data sources provide rich contextual information that allows for fine-grained validation. For instance, an image of a driver sending a text message can be cross-validated by checking the acceleration of the mobile device from the embedded accelerometer. Such multi-modal cross-validation improves the classification accuracy without the need to deploy additional sensors. More generally, the Driving behaviour Detection and iDentification system (D 3 ) [117] also detects abnormal driving behaviors using real-time smartphone sensors and an SVM-based ML algorithm. These driving behaviours include, for example, weaving, swerving, sideslipping, fast u-turn, turning with a wide radius, and sudden braking.
In addition to distracted driving, drowsy driving is another problematic behaviour that threatens road safety. Sober-Drive system [127] is a smartphone-assisted drowsy driving detection system that uses the smartphone's front camera feed and analyses the open/closed states of the driver's eyes using a NN model. Thus the system leverages drowsiness indicators such as the eyelid closure percentage, blink time, and blink rate. Furthermore, the D3-Guard system [118] detects drowsy driving using audio recording by smartphones and a long short term memory (LSTM) network. The system detects nodding, yawning, and abnormal steering in real-time by leveraging the Doppler shift of the audio signals to capture the unique patterns of these drowsy driving actions. In these systems, model training typically occurs offline whereas the application uses the real-time smartphone sensory data for inference in an online phase.

D. SECTION SUMMARY
ITS applications often require high accuracy in real-time while leveraging large and heterogeneous data. Therefore, the entire data pipeline must perform in real-time including data ingestion, streaming, processing, ML inference, and output presentation. Leveraging and combining distributed data analytics technologies such as Kafka and Spark can fulfill these requirements. Though the specific setup and placement of such pipelines will vary significantly given the diversity in requirements and scope of different ITS applications.

VII. INTEGRATED COMMUNICATION AND COMPUTING ARCHITECTURE
As mentioned some vehicular applications require offloading computation tasks to external servers. In this section, we study potential architectures that support such applications under three different scales: local scale (i.e., road, intersection, or last-mile), neighborhood scale, and city scale.

A. LOCAL SCALE
At the local level (road/intersection), the first priority is to provide basic safety functions to prevent accidents. For instance, active safety applications warn drivers of impending danger so the driver can take corrective or evasive action. Beyond basic safety, advanced efficiency functions also play a local level role. As an example, an active traffic management system (an advanced efficiency function) may adapt the local traffic control system proactively or reactively to improve local travel flow. Such a system includes consideration of peak-hour traffic, the detection of and response to incidents, and the reduction of waiting time due to congestion and incidents. Thus the system enhances the local transportation network performance in terms of safety, efficiency, reliability, scalability, and sustainability [128].

1) REQUIREMENTS
Decisions at this scale have to be made very quickly thus low overall system latency (normally less than 100ms) is important.
Basic road safety requires ultra-reliable and low latency communication. The infrastructure must have the capability to monitor the traffic situation reliably and make accurate decisions. For example, the road-side sensors must be capable of accurately recognizing and localizing various types of objects (e.g., vehicle, pedestrian, obstacle) with low latency. In advanced efficiency, the system must observe the situation and ambient environment and take quick actions on local scales to ensure smooth traffic flow. For instance, the system might change the duration of a traffic light phase based on road occupancy. Though the latency requirement of advanced efficiency system is on the order of seconds as such delays only minimally impact performance.
Importantly for both basic safety and advanced efficiency, local driving patterns vary according to the spatiotemporal context. For instance, mean driving speed varies fairly predictably due to time of day (rush hours, night hours, and so forth), day of the week (weekday or weekend), and road type (motorway or street road). Moreover, traffic flows change due to abnormal traffic events such as road incidents. Therefore, many applications on the local scale require context-awareness as well as situational-awareness to make accurate decisions [10], [92]. Additionally, the vehicles must maintain continuous, uninterrupted, and highly available communication between each other and with the RSUs. Finally, dynamic vehicle mobility leads to rapid topology changes in VANETs; while a variety of events cause transient unbalanced traffic distributions and congestion on roads and intersections. Therefore, such a category of applications should scale well with traffic flow and the incurred channel bandwidth usage [129].

2) ARCHITECTURE AND METHODOLOGY
To ensure context-awareness (in advanced efficiency) and low-latency (in basic safety), a local-scale system should offload computing tasks to nodes close to road users. The system includes these architectural considerations and meets the aforementioned local requirements. Computing nodes co-locate alongside roads (RSUs), or with cellular base stations (LTE-eNB and 5G-gNB). Relatedly, offloading computation to edge nodes provides distributed and parallel computing, allowing the system not only to scale up with load but also to ensure higher reliability by avoiding congestion in the back-haul network. Additionally, C-V2X allows vehicles to communicate with RSUs, thus allowing the collection and distribution of additional vehicular data (beyond the data from an RSUs own sensors).
The architecture contains three essential components: a data streaming module, a data management system, and a data analysis module to cope with the continuous traffic data streams. Traffic data includes but is not limited to the vehicle data, the environment data collected from the area's installed sensors, and the information obtained from the RSU. The data streaming module ingests and aggregates the traffic data streams then fuses both the aggregated data and potentially stored data from the data management system. Afterwards, the streaming module divides the fused data flow (continuous in nature) into (micro)batches for processing using the data analysis module [10], [11], [130].
Existing works [10], [130] integrate Apache Kafka and Apache Spark to detect unsafe driving activities. To provide real-time streaming and analytics of traffic data, they integrate Apache Kafka, to ingest the continuous stream of traffic data, and Spark Streaming to divide the stream into periodic micro-batches. The micro-batches then feed into a Spark-based machine learning algorithm to analyse the data. Amini et al. [11] propose a similar architecture but use a reducer-evaluator instead of Spark to analyse the data. The architecture is distributed and flexible enough to control traffic signals in real-time. It employs Apache Kafka as an intermediary module between the traffic system with its sensors (e.g., probe vehicles, loop detectors) and actuators (e.g., traffic lights, variable message sign), and the data analysis module. The arriving data are processed via user-specified reducer functions. After a time interval, a separate evaluator function is invoked to assess the results and update the settings of the traffic system accordingly.

B. NEIGHBORHOOD SCALE
Another category of (vehicular) applications involves decision-making on a larger neighborhood scale. From a transportation perspective, the neighborhood scale encompasses multiple interconnected (via road network) local-scale areas with significant inter-area traffic flow. From a mobile network perspective, it typically encompasses multiple adjacent macrocells (see Figure 7).

1) REQUIREMENTS
Traffic flow (inbound and outbound) and the underlying road network typically drive the decision-making at the neighborhood scale. Decisions are made based on traffic interactions between local areas, their cascading effects, and the acceptable latency of neighborhood scale applications. Such context-awareness allows the system to disseminate accident warnings to the relevant road users, balance road occupancy, alleviate congestion, and thus reduce travel time and CO2 emissions. Importantly, the latency requirements at this scale are typically less stringent (e.g., on the order of seconds or minutes) than the local scale. Note that such latency requirement differences can have considerable impact on the ITS system as, for example, shown in Yang et al. [131]. Specifically, Yang et al. [131] study fulfilling various latency requirements while also minimizing the required communication and computing resources in a mobility-aware task offloading scheme.

2) ARCHITECTURE AND METHODOLOGY
To ensure a holistic neighborhood view, a neighborhood scale architecture often co-locates the computational units with the aggregation points of the mobile network, known as communication-aware neighborhood scale, or the aggregation points of the RSU system, known as transport-aware neighborhood scale, as shown in Figure 7. For example, Zhou et al. [27], [132] optimize neighborhood-scale decision making through communication-aware MEC servers on two different tiers, collocated with base stations and aggregation points. In addition to the three local scale architecture components, a neighborhood architecture often contains a collaboration channel between computing nodes to exchange information (processed data), sometimes through a central computing and storage node. Specifically, some systems use inter-RSU collaboration such as CAD3 [10] and Chao et al. [131], whereas Zhou et al. [27], [132] use two-tier MEC computing as shown in Figure 7a. Relatedly, dual-mode C-V2X roadside device, supported by mobilityaware algorithms, allows the road users to communicate via NR C-V2X/PC5 direct communication channel and allows them to connect to network infrastructure via the Uu (5G) communication channel [133], [134]. This enables sharing of vehicular and traffic information within areas on a local scale, leading to coverage on a neighborhood scale (see Figure 8). Additionally, these distributed multi-tier architectures align well with certain distributed ML frameworks. For example, Zhou et al. [27], [132] apply multi-agent reinforcement learning (MARL) and federated learning (FL) to capture changing traffic patterns and maintain smooth traffic flow on a neighborhood scale.

C. CITY OR LARGER SCALE
The last category of applications require city-scale decision making such as advanced efficiency (e.g., city traffic management and planning), holiday traffic inference, and studying of large events (i.e., sporting and exhibition events) [135], [136]. These often involve the think globally, act locally concept, where data analytics of traffic data collected from road users and elements (e.g., vehicles, traffic lights, pedestrians) covers the entire city.

1) REQUIREMENTS
Many city-scale applications require urban big data that is naturally high volume, high variety, and high velocity (3Vs). This data can encompass trip and trajectory data, surveillance video, weather, social events, and a diversity of traffic data. The data typically has time and location stamps, in other words, spatio-temporal data, to enable many rich spatio-temporal applications including, for example, finding dynamic dependencies among different regions at the local and neighborhood-scale [137].
For city traffic planning and management, transport decision-making involves understanding city-scale human mobility patterns and discovering traffic problems (traffic anomaly [135] and accident detection and congestion prediction) [137]. As such this significant data scale often requires analytics engines in the cloud to handle the heavy computation. Event prediction and detection often leverages data streaming including preprocessing and feeding into a machine learning model to detect events or predict the occurrence of events [138], [139], [140]. For instance, traffic flow prediction requires city-scale traffic data analytics and urban dynamics decomposition [135] to identify the traffic flow patterns and predict the future traffic flow which then helps to ensure efficient route planning and mobility. While traffic management requires either a multi-tier communicationaware architecture [27], [132], [141], or centralized cloud architecture. This architecture allows to collect and process massive traffic data for monitoring traffic density, throughput, and events in real time. Though communication-aware systems lack spatial-temporal correlations and mobilityawareness. Efficient traffic flow requires mobility-aware traffic control so as to decrease the waiting time of vehicles traveling on signalized roads.

2) ARCHITECTURE AND METHODOLOGY
A city-level architecture primarily locates the computing nodes in the cloud given such centers' capacity to process large streams of city traffic data. In such an architecture the road elements (e.g., traffic lights, lampposts, vehicles, inductive loops) can be equipped with 5G modules to transmit the large data volumes to the nearby RSUs via NR C-V2X direct communications (PC5). The RSUs then transmit data to the cloud through the network infrastructure, i.e., I2N Uu communications. The cloud ingests the continuous stream of urban sensor data and uses data streaming and analytics engines (e.g., Apache Kafka and Spark) to process them in real-time including machine learning algorithms for decision making. Road elements receive data from the cloud to enable vehicular applications. For instance, traffic lights receive commands via downlink Uu communications, decision-makers (department of transportation) also receive information via downlink Uu communications, other road users (e.g., vehicles and pedestrians) receive information via downlink (Uu) and then I2V/PC5 communications (see Figure 8). For traffic management, distributed agents can monitor and take actions at the local scale and neighborhood scale while a city-scale central agent tunes performance parameters to optimize the overall traffic flow. However, the transmission of raw traffic data may still cause issues due to high bandwidth requirements and privacy restrictions (for example related to General Data Protection Regulation (GDPR)) [142]. A potential methodology for dealing with these issues is federated learning (FL). FL addresses privacy and bandwidth issues by training local models and uploading only the local model parameters to the cloud for aggregation into a global model (that is then redistributed to the local agents) [143] (see Figure 8).

D. SECTION SUMMARY
A robust vehicular computing architecture is crucial for satisfying the requirements of vehicular applications. These applications vary with some requiring detailed local spatio-temporal context with low latency decision making, and others requiring a city-wide holistic view for optimization beyond the local or neighborhood scale. Some applications will even require some combination of these requirements. Incorporating the V2X and 5G ecosystem, including edge computing, can help cope with such requirements. Specifically, MEC and NR C-V2X communication will likely be very important. NR C-V2X is a crucial component for reliably sharing the big traffic data. While MEC enables processing the data at the network edge thus allowing for lower potential latencies including real-time or near real-time streaming data analytics (for example with Apache Kafka or Spark).
Overall, we note the boundary between these scales is not well defined and any traffic system or application must still, in some ways, consider the holistic traffic environment on multiple physical scales. In any case, most systems and applications will actually target multiple physical scales.

VIII. REAL WORLD CASE STUDIES
In this section, we study three real-world scenarios: localscale cooperative perception, neighborhood-scale accident warning, and city-scale event detection (urban planning). These scenarios highlight the leveraged technologies, the enabling architecture, and the interactions with road users.

A. CASE STUDY: COOPERATIVE PERCEPTION
Modern vehicles have a variety of sensors to perceive the nearby environment and warn the driver of potential hazards. However, external objects (e.g., buildings, other vehicles, trees) may block the view of these sensors, causing blind VOLUME 10, 2022 FIGURE 9. Real-world case studies at local, neighborhood, and city scales. spots and raising road safety concerns. 8 In concrete terms, Figure 9 illustrates an example scenario (proposed in a public roads test project of C-V2X in Hong Kong [144]). In the example, vehicles pass through a roundabout and encounter a blind spot situation that causes a safety issue. Specifically, when a new vehicle enters the roundabout, the black vehicle's driver can not see the gray oncoming vehicle, stopped vehicles, or pedestrians crossing the road behind the stopped vehicle. The black vehicle might not have an in-vehicle communication and computation module (OBU) and thus cannot identify and relay information about the stopped vehicle and crossing pedestrian. We note that the assumption that all vehicles are equipped with OBUs does not hold in the real world.
As such, a roadside system is a potential solution to help recognize the object type (vehicle or pedestrian), position, and speed, and accordingly determine the potential danger and notify the driver. In conjunction, see-around-thecorner vehicular applications allow the RSU installed at the intersection to distribute real-time sensor data or notifications to vehicles in range using NR C-V2I communications and extend the vehicles' visibility beyond the sensors and driver's view [48], [146]. Due to the short distance and direct communication between the vehicle and RSU, the roadside system could recognise the danger and disseminate corresponding warnings well within 100 ms (maximum acceptable latency).
Additionally, when vehicles are equipped with OBUs, they can cooperatively perceive a larger area of the environment than any single vehicle alone. Using C-V2X, the adjacent vehicles can communicate directly (PC5) to share raw onboard sensor data (e.g., camera, LiDAR, radar) and processed information (e.g., information about identified objects). Thus they can obtain rich dynamic information in complex traffic environments with blocked views [147].
NR C-V2X enables sharing large volumes of sensory data with a peak data rate of 1 Gbps or more due to the wide bandwidth in the mmWave region [48]. An additional example of the benefits of OBU-enabled cooperative perception is seen in vehicle overtaking situations. Specifically, in such a situation a see-through vehicular application could allow the trailing vehicle to obtain the front camera view (using NR C-V2V communication) of the leading vehicle to help identify the non-line-of-sight (NLOS) traffic situation ahead [146].

B. CASE STUDY: ACCIDENT WARNING
Relatedly, current accident warning systems typically rely on cloud services for aggregation and then distribution to drivers' smartphones (for example Google Waze [148]). Thus the system relies on remote servers which may not always be available.
As a potential alternative, Figure 9b illustrates a detailed neighborhood-scale accident warning system that leverages the EPC of an LTE network (similar to [145]). A vehicle involved in an accident or a nearby vehicle that observed the accident sends a notification to the local evolved NodeB (eNB). The eNB delivers the notification to the serving gateway (S-GW), which, in turn, forwards the notification to the packet data network (PDN) gateway (P-GW). The P-GW provides an entry point for service providers (i.e., dedicated servers to collect information or notifications and disseminate them to the subscribed group). The broadcast multicast service center (BM-SC), part of the core network, functions as the interface between the distribution service (MBMS) and service provider (on edge servers), thus supporting evolved multimedia broadcast multicast services (eMBMS). The BM-SC transmits the notification as broadcast or multicast content through the eMBMS gateway (MBMS-GW) to the eNBs using IP multicast and then to the subscribed vehicles in each eNB cell. Additionally, it can transmit to adjacent cells using multiple eNBs, i.e., a multimedia broadcast multicast service single frequency network (MBSFN). Alternatively, the roadside system could also recognize the accident and forward warnings to the RSUs in affected local areas. Those RSUs could then disseminate the warnings to vehicles directly through C-V2X (I2V) or indirectly through LTE, depending on the vehicle communication module.

C. CASE STUDY: URBAN PLANNING
As previously mentioned, city-scale traffic planning and optimization (along with general urban planning) can benefit from detailed real-time traffic data often collected by smart devices, roadside sensors, and various kinds of road users.
Specifically, the integration of these devices can create a large-scale, cross-domain and multi-view data ecosystem. Such large-scale urban data enables, for example, detecting, analyzing and predicting large urban events (e.g., sporting and entertainment events, protests, weather, or natural phenomenon) which allows governments to take more timely actions.
Relatedly, the collection of urban spatio-temporal trajectory data in real-time can help calculate the city-wise traffic flow and identify local areas of congestion (as in [138]). The detection of congestion allows the intelligent transport system (ITS) to adjust the phases of traffic light signals dynamically and change routes on users' route recommendation applications (vehicular application in Figure 8) to alleviate the congestion. For instance, an ITS might arrange more taxis to the area near a soccer match and recommend unrelated vehicles alternative routes that bypass the area.
For illustration, Figure 9c details the logical relationship between the physical space (urban event) and the cyberspace (urban data) for the case of congestion detection. Specifically, urban events are a causal factor in urban dynamics which reflects in urban data. Likewise, urban dynamics can be inferred from spatio-temporal urban data and urban dynamics reveal the underlying urban events.

IX. CONCLUSION AND FUTURE DIRECTIONS
In this work, we surveyed the existing literature on distributed vehicular communication and computation. Specifically, we highlighted several vehicular network applications (e.g., basic safety and advanced efficiency) including their technical requirements from different viewpoints. We then detailed the enabling technologies and promising methods and architectures to support these applications. In terms of specifics, we described the available communication technologies including DSRC, ITS-G5, LTE, 5G, and C-V2X. Next, we reviewed the computation and analytics frameworks applicable to the vehicular context including general architectures such as cloud and edge computing and more specific approaches such as vehicular fog computing and mobile vehicular cloudlets. We then described the integration of communication and computation through example applications at different geographic scales (street, neighborhood, and city) including detailed case studies.
In terms of the most important lessons, we highlight the following results. NR C-V2X is a promising vehicular technology for transmitting large data volumes directly (PC5 sidelink mode) or indirectly (Uu mode) over short distances. However, long-distance communication (e.g., vehicle-to-cloud) will still incur significant latency as the data must traverse the backhaul (core) network when vehicles send sensory data to the cloud for processing. Edge computing enables data processing in geographic proximity to the vehicle, thus lowering the latency and enabling time-critical applications. Multiple edge computing variants (i.e., MEC, FC, and Cloudlet) have emerged to mask transient cloud outages and naturally capture contextual and situational information. Leveraging these communication technologies, edge computing will complement the cloud computing paradigm when combined with data analytics and streaming technologies (e.g., Apache Spark and Apache Kafka), and enable a wide range of ITS applications with diverse latency requirements, ranging from highly latency-sensitive (basic safety) VOLUME 10, 2022 to latency-tolerant (urban event prediction/detection) applications, and with varying bandwidth requirements.
Beyond the existing work in this survey, many interesting open research areas and issues remain. These include for example: (i) developing multi-tiered architectures (to handle apps with that work on multiple geographic scales); (ii) developing a traffic Metaverse application for mixedreality vehicle perception; (iii) developing programmable communication links (such as SDN) that enable on-demand bandwidth and eliminate network bottlenecks; (iv) integrating security at the node, domain, end-to-end, and service levels; and using privacy-preserving machine learning methods such as federated learning; (v) developing and applying ML and AI methods that adapt to dynamic and changing driving patterns; (vi) using augmented reality to improve the quality of experience of ITS; and (vii) assessing the interplay between vehicular communications and future 6G technologies and standards. We briefly describe each of these aforementioned open areas and issues and potential promising research directions and solutions.

A. MULTI-TIERED ARCHITECTURE FOR VEHICULAR APPLICATIONS
Some vehicular apps may consider the holistic traffic environment and act on multiple geographic scales (street, neighborhood, city). These apps necessitate a city-scale vehicular computing architecture that encompasses multitiered computing, i.e., on RSUs, regional and cloud computing. Potential multi-tiered architectures might have RSUs maintain the state of active vehicles on the road, while the regional and centralized data centers ensure the persistent state of vehicles. Such an architecture could utilize 5G New Radio (NR) Cellular Vehicle-to-Everything (C-V2X) communication technology [149], [150] to ensure ultra-reliable low-latency communications. While roadside elements and vehicles could be equipped with C-V2X device to provide both direct communication (sidelink PC5 of 5.9 GHz band) and indirect communications (Uu communication through cellular network infrastructure).

B. TRAFFIC METAVERSE FOR MIXED-REALITY VEHICLE PERCEPTION
The emergence of the metaverse could enable the creation of a city-scale traffic metaverse that leverages real-time traffic and vehicle data to create digital twins of vehicles and road conditions. Through such a metaverse, a collaborative virtual 3D map of road networks could provide road users (e.g., vehicles, traffic lights) with a pervasive view of the road conditions to increase situational awareness. Such a metaverse would probably require the non-trivial integration of many technologies including city-scale ubiquitous sensing, reliable and low-latency communications, artificial intelligence, extended reality, and a multi-tier computing architecture (see Section IX-A).

C. MULTI-CARRIER SELECTION AND AGGREGATION FOR LATENCY AND CONGESTION ISSUES
The potential for communication network congestion and the associated higher latency (due to queuing) are significant issues for latency-sensitive vehicular applications.
Possible methods to help with these issues include multicarrier network access and software-defined heterogeneous vehicular networking architecture (e.g., SDN-based Het-VNet [151]). Multicarrier network access is a network selection technology that senses and selects the best of several available networks (e.g., DSRC or 5G) given these networks' properties and current congestion levels. A study on mobile latency on multiple operators in two distinct cities [152] shows that such a carrier selection algorithm drops latencies 10 to 20% compared to single carrier operations in real-time interactive cloud-based mobile applications such as augmented reality and cloud gaming. The technology also provides the potential for aggregation of several networks to increased bandwidth.
However, such solutions might not solve the congestion issue at the next hop (edge devices) or in the core network and might even exacerbate it. An edge-assisted Vehicular Software Defined Networking (VSDN) architecture could help solve this issue by leveraging VSDN controllers on edge devices. The VSDN controllers can use continuously trained ML models to predict or detect periods of congestion and links with high utilization. They can request more bandwidth upon detection of network bottlenecks by controlling the multi-carrier algorithm which either deactivates the current carrier and activates a different carrier or activates an additional carrier and aggregates multiple carriers. The VSDN controllers could also control the selection and aggregation logic of vehicles, and also find the best route or establish multiple connections to the destination node (e.g., cloud server) in the backhaul network to satisfy the bandwidth demand. Figure 10 illustrates such an AI-assisted VSDN architecture. Vehicles and other road users may have several communication modules (DSRC, LTE-A, Wi-Fi, mmWave) and thus select the most suitable RAT or combine two or more to ensure seamless and ubiquitous connectivity.

D. PRIVACY AND SECURITY CHALLENGES
Remote vehicle diagnostics and maintenance, anomalous driving identification, and many other ITS applications are often based on learning from long-term data. Therefore, these apps often require the transmission of potentially sensitive and private data (e.g., video streams, user identifiers, locations, and insurance information) to edge or cloud servers for data aggregation and computation offloading. To deal with this privacy issue, the automotive ecosystem could adopt federated learning (FL) for use cases where privacy is a serious concern and bandwidth is limited. As mentioned, using an FL approach, users (e.g., vehicles, RSUs) share only locally trained model parameters which (compared to the raw data) are more difficult to extract insights from and are often smaller. Although FL often improves privacy and lowers bandwidth, FL raises several new challenges such as model poisoning. Specifically, an attacker may poison the model by sending parameters of an anomalous model.
Apart from FL, some vehicles (attackers) might initiate man-in-the-middle attacks to capture vehicle data (to attempt to learn about the drivers) or gain illegitimate remote control of another vehicle. Thus ITS systems need to include policies and tools to provide, confidentiality, reliability, integrity, and other security services. Vertical integration of security for data traffic from nearby vehicles and devices is critical at the node, domain, end-to-end, and service/use case levels. Besides, the integrity of transmitted and stored data is a crucial component in security provisioning. Meanwhile, a verifiable computing scheme (e.g., data attestation [153]) for vehicular users could help check the correctness of any obtained computation results from the edge servers. Further research is needed to define the authorized users, vulnerabilities, and potential threats, and to create a trusted remote computing environment.

E. MODEL LEARNING AND ADAPTATION OVER TIME
Driving patterns (along with other vehicular phenomena) are naturally dynamic and change over different timescales according to the road type, weather conditions, vehicle conditions, and driving style [154]. Thus, NN models, for example, for the prediction of driving patterns require training on significant heterogeneous historical data to account for these dynamics. However, given the possibility of rare novel events and non-stationarity, such models can benefit from continuous learning techniques (aka incremental learning). Specifically, these techniques allow learning from an online stream of incoming data (without full offline retraining) while avoiding the serious problem of forgetting previously learned data (known as catastrophic forgetting) by, for example, constraining how the network parameters can be updated during learning [155]. Continuous learning approaches have not yet reached the performance levels of offline retraining and remain a major future research area.
Beyond continual learning, specific models are also designed for dynamic phenomena, for example, Gaussianbased dynamic probabilistic clustering (GDPC) [156]. GDPC is a Gaussian mixture model-based unsupervised learning algorithm processing large amounts of data and coping with underlying dynamic phenomena (e.g., degradation). GDPC integrates three well-known algorithms: the expectation-maximization algorithm to estimate the model parameters, and the Page-Hinkley test and Chernoff bound [157]. In turn, they use multiple (heterogeneous) data sources, fuse them, and based on which they train unsupervised or reinforcement learning models. If employed, these algorithms provide the model with the capability to detect the drift in the driving patterns.

F. AUGMENTED REALITY
Augmented reality (AR) windshields is a novel research area in both academia and industry that aims to improve driving safety and experience by augmenting environmental objects (e.g., roads, vehicles, obstacles, pedestrians) by overlaying helpful information. For example, a pedestrian image could be overlaid on the windshield at the location of an out-of-view pedestrian moving quickly towards the road, thus allowing the driver to be aware of the danger. However, AR windshields present significant challenges. Specifically, high motion-to-photon latency can cause misalignment between virtual objects and the physical world [158], thus distracting the driver. Therefore methods to reduce networking and processing latency [159] including future communication technologies and computing paradigms as well as HCI methods to compensate for some degree of misalignment are important future research topics. Additionally, issues with the alignment of the driver's head with the windshield can cause other HCI issues. Finally, problems such as selecting which and how many objects to show or emphasize to the driver remain open.

G. 6G VISION
As mentioned, the widespread deployment of 5G vehicular communications will deliver, for example, higher data rates, lower latency, and more reliability to help support a variety of intelligent transportation systems. Research into 5G and 5G-advanced enabled ITS will continue to be a major research area with actual wide-scale deployment of such systems still years away. However, some future vehicular applications such as tactile internet use cases (where, for example, a remote operator would take control of an autonomous vehicle in an emergency) will require features 5G is lacking. Specifically, tactile internet requires sub 1 ms end-to-end latency and a combination of several different 5G modes including ultra-reliable low latency and enhanced mobile broadband [160]. 6G vehicular communications look to support such applications that require multiple modes and overall aim to improve on most 5G KPIs by a factor of ten. VOLUME 10, 2022 Though, 6G also presents major research challenges such as maintaining reliability even with the use of high attenuation (yet large bandwidth) THz or optical wireless communication technologies that are potential major 6G components. Thus 6G vehicular communication will emerge as a significant future research topic in the coming years.
TRISTAN BRAUD (Member, IEEE) received the degree in engineering from the Grenoble INP Phelma, France, the dual M.Sc. degree from the Politecnico di Torino, Italy, and Grenoble INP, France, and the Ph.D. degree from Université Grenoble Alpes, France, in 2016. He was a Postdoctoral Fellow at the HKUST-DT Systems and the Media Laboratory (SyMLaboratory), Computer Science Department, The Hong Kong University of Science and Technology, Hong Kong, where he is currently an Assistant Professor at the Division of Integrative Systems and Design. His major research interests include augmented and virtual reality, with interests in pervasive and cloud computing, and human centered system designs.
DONGZHE SU received the bachelor's degree from the Huazhong University of Science and Technology, and the M.Phil. degree in computer science from The Hong Kong University of Science and Technology. He is currently a Chief Engineer with the Communication Technologies Group, Hong Kong Applied Science and Technology Research Institute. He has been leading the system architecture in research and development of vehicle-to-everything (V2X) communication and application systems, connected autonomous vehicles (CAV) systems, and the Internet of Things (IoT). His role has been to define the technical scope and overall system design for ASTRI's V2X networking system. In 2021, ASTRI launched one of the world's largest C-V2X public road tests in Hong Kong, covering a 14 km route with various road environment of Hong Kong.
PAN HUI (Fellow, IEEE) received the bachelor's and M.Phil. degrees from The University of Hong Kong, and the Ph.D. degree from the Computer Laboratory, University of Cambridge. He is currently a Professor in computational media and arts and the Director of the Center for Metaverse and Computational Creativity, The Hong Kong University of Science and Technology. He is also the Nokia Chair of Data Science at the University of Helsinki. He was an Adjunct Professor in social computing and networking at Aalto University, Finland, and a Distinguished Scientist at the Deutsche Telekom Laboratories (T-laboratories), Germany. His industrial profile also includes his research at Intel Research Cambridge, U.K., and Thomson Research Paris, France. He is a World-Leading Expert in Augmented Reality and Mobile Computing, with more than 400 research papers, 30