NORA: Towards Large-Scale Vehicular Analytics for Driving Environment Monitoring/Assessment

Research in vehicular analytics has explored two different approaches to infer properties about a vehicle's surroundings: (a) using sensors on smartphone devices to infer properties about surroundings, or (b) to use in-vehicle sensors. The latter approach was less studied. In this article, we take a first step beyond research to understand, using a pilot study, how to design vehicular analytics at scale. Our pilot prototype NORA (Network Oriented Road Applications) contains novel algorithms to detect roadside phenomena (such as potholes, rough road, and slippery surfaces). These leverage multiple in-vehicle sensors to disambiguate these phenomena from other conflating factors. NORA reliably detects, on a cloud service, roadside events from multiple individual in-vehicle detections. To do this, it uses careful clustering techniques to assess the spatial scale of the event, and belief decay techniques to match event duration. It also employs aggressive fleetwide suppression of detections to minimize communication cost. Through a 50-vehicle deployment of NORA pilot over 9 months, it is shown that the results obtained from NORA in-vehicle detection methods match very well with ground truth measurements, and NORA cloud is effective at aggregating road events in an accurate and efficient fashion.


I. INTRODUCTION
In an increasingly interconnected world, it has become possible to obtain fine-grained data about various phenomena in our lives, and to use this data to make decisions.One area that has seen significant research interest is vehicular analytics, which seeks to understand the behavior of the vehicle, its driver, and properties of its surroundings.This has significant practical applications, ranging from navigation and route planning to safety warning systems, infrastructure and vehicle health maintenance.
An early line of research in this area was motivated by the increasing penetration of smartphones.This research attempted to characterize various properties of the environment (such as the existence of potholes, stop signs, speed bumps) and driver behavior (lane changes, weaving in lanes etc.) using inertial and visual sensors on smartphones.This line of research is interesting because it opens up the possibility of obtaining large-scale vehicular analytics from commodity devices.On the other hand, commercialization of this idea faces several practical hurdles, not the least of which is buy-in from the owner of the vehicle and the smartphone.
As it became clear that Internet-connected vehicles would increase year on year, another line of research considered the possibility of using in-vehicle sensors for many of the same data collection tasks.Vehicles have several hundred carefully calibrated sensors that provide input to internal control systems (Section III).Vehicle manufacturers are better positioned to commercialize this approach because: (a) they have direct access to in-vehicle sensors, (b) have sufficient market penetration to get significant spatial and temporal coverage needed for meaningful analytics, and (c) already use Internet connectivity to deliver telemetry for maintenance and roadside assistance.
In this article, we report on the design of, results from, and experience with, a pilot deployment of a crowd-sensing based vehicular analytics service called NORA (Network Oriented Road Applications), designed from the perspective of a vehicle manufacturer.Compared to previous works [1], [2], our work looks toward operationalizing detection algorithms based on streaming data analytics rather than performing analytics against a stored dataset.Scalability and latency become critical factors to system deployment feasibility in the context and we believe our work is the first to demonstrate a heterogeneous suite of insight algorithms using a scalable computing architecture.Such a pilot deployment can give insights how to architect and scale a vehicular analytics service.At a high-level, a service like NORA must address the follwing challenges: 1) Application and Sensor Data Heterogeneity: We envision a diverse set of vehicular applications that span a wide design space, creating a variety of technical requirements as the nature and composition of these applications could be different.The design space includes vehicles of different model years and build configurations, as well as sensing and data systems that vary in terms of availability and fidelity.2) Local and Dynamic Nature of Road Condition Information: Road condition information in the vehicular context tends to be spatially and temporally localized.On the one hand, a road condition event (e.g., a pothole or slippery road condition) is consumed locally by other nearby vehicles in the spatial domain; on the other hand, the information is dynamic and even interactive, changing frequently over time and thus requiring careful temporal maintenance.3) Efficient System Resource Management: For a largescale vehicular network, an efficient crowd-sensing system is required to aggregate event detection within the network in a timely manner to improve the accuracy of system output as well as maintain network bandwidth efficiency.4) System Practicality: When considering systems operating at production scale, the overall cost of operations must be kept to a minimum, while at the same time fulfilling customer needs.Therefore, sampling mechanisms must be in place to selectively report observations and maintain an accurate representation of the sensed environment.Rather than export every sensor reading, NORA's architecture (Section III) requires each vehicle to process its sensor readings to obtain a reasonably accurate detection.These succinct representations are then transmitted to a cloud service that aggregates these detections into events, and makes them available to third-party applications (e.g., navigation).In the context of this architecture, the article makes the following contributions.
First, we demonstrate how in-vehicle sensors can help detect for qualitatively different phenomena (Section IV): potholes, rough road, slippery surfaces, and severe microclimates.The challenge in designing detectors for these phenomena lies in disambiguating the phenomenon from other closely related phenomena.For instance, the behavior of the vehicle on a pothole can be conflated with normal vehicle vibrations or traversing a speedbump.Our key insight is: (a) that there exist embedded fixed sensors in parts of the vehicle that already give strong signals for the phenomenon and (b) other sensors or other control signals can be used to disambiguate and increase confidence in the detection.
Second, we demonstrate techniques by which NORA's cloud component can reliably aggregate detections into events.These techniques must surmount two challenges: (a) some phenomena like potholes are spatially localized, while others like a rough road surface might span a larger segment of a road; (b) event durations span a wide range (from hours for microclimates and slippery surfaces to weeks or months for potholes and rough road).To address these, NORA uses different spatial clustering techniques depending on the type of phenomena and also uses a unique belief decay approach to age out events for which detections have not occurred in a while.A by-product of careful event aggregation is aggressive cost minimization: NORA can suppress detections from vehicles for events for which it already has high confidence, an approach we call selective sensing.
Our third contribution is a discussion of results from an experience with a pilot deployment of NORA using 50 vehicles over a period of 9 months.During that time, NORA collected over 2 billion sensor readings from over 130,000 miles of driving data across all road types (residential, local, rural, interstate highways).When comparing to ground truth measurements, our NORA in-vehicle detections provide very high accuracy, e.g., 100% precision and 91.2% recall in the case of pothole detection, and minimal error (≤ 51.1 inch/mile) in the case of road roughness.Meanwhile, NORA cloud aggregation algorithms are shown to suppress ≥ 53% of redundant event reports while maintaining the same event detection accuracy.

II. RELATED WORK
Our work is inspired by a number of pioneering studies discussed in this section.
Vehicular context sensing: In the past decade, the research topic of detecting vehicular context has gained significant traction in the mobile sensing and computing community.There is a comprehensive set of literature on using vehicle sensors, smartphone sensors and customized cameras to detect driver behavior characteristics or road surface conditions [3], [4], [5], [6].
Automotive Programming: To enable automotive context sensing, researchers have explored different methods to synthesize various automotive sensors to detect relevant driving events or road events.Procedural abstraction for programming vehicles is explored in [7] to tune and optimize the performance of a vehicle.Declarative programming [8] is used to express the behaviors of mobile events in the context of wireless sensor networks [9].Recently, Carlog [10] has developed a programming framework to harness a large number of networked, embedded automotive sensors, and take advantage of a datalog query optimization technique to access both cloud-based virtual sensors together with vehicle sensors.
Industry Trend: Realizing the tremendous potential of applications that employ their embedded sensors, the automotive industry has begun to introduce embedded application development platforms with open APIs to access vehicle data [11], upon which second-or third-party developers can design applications to satisfy a variety of customer needs.In parallel, smartphone applications (such as OBDLink [12] or Torque [13]) are becoming increasingly popular, allowing customers to view and analyze vehicle data through self-installed OBD-II devices.
Road Event Detection: Road event detection is another area of automotive sensing that has attracted a great deal of research attention since road events are among the the most common factors that impede vehicle traffic flow and impact the driving experience.Road surface conditions have been evaluated by researchers using a variety of sensors including smartphone inertial sensors, vehicular sensors and dedicated special-purpose camera systems [14].It has been found that pothole detection is the most popular use case in the research community [15].An initial approach is to use a high precision accelerator mounted to the vehicle to detect the acceleration changes when a vehicle hits a pothole [16]; the built-in accelerometer of a smartphone is later used to develop a mobile app for empirical deployment [17].Other advanced approaches -such as vision-based pothole detection, Lidarbased environment scanning, or sound-pressure based audio processing -have been also explored [18], [19], [20], [21], [22], [23], [24], but are proven to be less feasible at this moment due to cost constraints or vulnerability to environment noise (ambient light or ambient noise).Liu et al. [25] created a mathematical model of International Roughness Index (IRI) using multiple vehicles' vibration sensors.
As emerging trend is to use smartphone sensors as a practical solution to address real-world challenges.Solutions such as Roadbump [5] and Roadbotics [6] rely on a participating user base to provide road roughness estimates or real-time road conditions.These systems typically use accelerometer and gyro sensors of a smartphone to directly measure the vertical acceleration for discrete road event detection (e.g., potholes and road bumps).
Crowd-sensing testbed: There have been a number of mobile crowd-sensing studies, which have used smartphone senors as an environment probe to enable a variety of use cases including the creation of digital maps, characterization of traffic flow and measurement of different road events.
The most relevant studies to our article are the prior works on pothole detection.The Pothole Patrol [15], [26] project leverages a high-pass filter to process vehicle information (such as speed, vertical and lateral acceleration) to identify potholes in signals [27].In a similar fashion, road bumps are detected in [5] by carefully studying the vertical acceleration and other supplementary information.Android smartphones have been explicitly utilized to analyze vertical acceleration patterns as vehicles travel through bumpy roads, providing a distributed, continuous road surface monitoring capability [28].
Interfacing with smartphones, vehicular crowd-sensing techniques have been used to support other use cases: GreenGPS [29] provides a preferred fuel-efficient route plan based on crowd-sourced vehicular measurements from a fleet of vehicles with similar source-destination patterns; GPS traces are collected and crowd-sourced at a specific set of intersections to differentiate traffic lights from stop signs [30]; SmartRoad [31] takes a smartphone crowd-sensing approach to detect various types of traffic regulations on the roadway.
Our Unique Contributions: Unlike prior smartphone-based approaches, our NORA platform directly employs highly accurate vehicle kinematic sensor measurements.Our approach not only offers high confidence in terms of road event detection results, but also employs unique sensing systems that are able to measure the physical contact between the vehicle and road surface and measure a variety of ambient conditions surrounding the vehicle.Our scalability tests demonstrate the ability to perform data stream processing at scale to enable navigation insights to vehicles in a timely manner.In addition, our system has advantages in real world applicability since active user participation is not required and vehicles automatically produce millions of sensor readings per day.Embedded technology also enables improved data fidelity and location accuracy due to fixed sensor installations and improved GPS reception due to antenna placement.To the best of our knowledge, we are not aware of other work that has attempted to quantify the efficacy and efficiency of automotive crowd-sensing systems covering a rich variety of NORA apps deployed in empirical environments.Last but not least, our naturalistic testbed differs from the earlier pioneering studies due to our goal of creating a model vehicular road event management system that could be used as a basis for eventual wide-spread commercial deployment.

III. MOTIVATION AND ARCHITECTURE
In this section, after introducing motivating NORA applications, we lay out the system architecture used in NORA pilot.
a) Applications: To motivate vehicular analytics at scale, we consider four qualitatively different phenomena (Table 1): potholes, road roughness, slippery surfaces, and severe microclimate.We use the term event to denote an instance of each phenomenon, and the term application to denote the collection of algorithms and software that together estimate events.A vehicular analytics service run multiple applications, and the generated events can (a) inform drivers about hazardous conditions, (b) augment perception and control systems to b) In-vehicle detections: In-situ vehicular sensors can detect each of these events.For example, a wheel rotation sensor can detect potholes because, as a wheel encounters a pothole, that sensor's value is different for that wheel relative to other wheels that do not encounter the pothole.Similarly, sensors in the active suspension system, whose role is to mitigate the effect of rough pavement, can characterize road roughness.We discuss detection algorithms in Section IV. c) Requirements: While a vehicle's sensors can detect each event at that vehicle, we may need to aggregate detections from multiple vehicles, for two reasons.First, while vehicular sensors are accurate, they are not perfect and can miss some detections, either because sensor is never intrinsically perfect, or because, in some cases, detecting the event repurposes existing sensors (e.g., the output of the active suspension system is used to estimate road roughness but that system can be triggered by other phenomena on the road such as railroad tracks and speed bumps).Second, events can have varying spatial and temporal extents; they may last for a short amount of time (e.g., a slippery surface or a microclimate) or may range from small spatial extents (potholes) to larger ones (rough road segments).Because vehicle manufacturers must support a large number of vehicles (some manufacturers sell millions of vehicles per year), these aggregation algorithms must scale to large fleets.
For aggregating detections, a vehicle needs to transmit its readings over the cellular network.To design a reliable analytics service, a vehicle manufacturer cannot rely on the user's mobile device to transmit detections, but must rely on an onboard communications unit (similar to GM's Onstar or Ford's SYNC).For large manufacturers, the cost of this communication can be significant, so the analytics service must minimize communication aggressively.
While we only consider four phenomena, a practical analytics service will likely estimate several other phenomena, including driver behavior and vehicle condition.An analytics service must be extensible to more phenomena.Finally, third-party services must be able to use the events detected by the service to provide value-added services to vehicle owners.For example, a navigation app can use NORA's events to improve their route guidance; service-level agreements ensure that NORA provides predictable performance to these thirdparty services.
The Architecture: To address these requirements, NORA partitions functionality between a cloud service and an invehicle component (Fig. 1).At a high-level, the in-vehicle component uses existing vehicle API interfaces (e.g., [11]) to access and collect raw sensor readings from vehicle Electronic Control Units (ECUs) over the On-Board Diagnostic Interface (OBD-II).It then runs detection algorithms for an event at each vehicle, rather than exporting the raw sensor readings.This processing can help minimize communication cost; NORA includes other mechanisms as well to minimize this cost (Section V-C).It modularizes in-vehicle processing for each event to enable extensibility.NORA's cloud service collects detections from individual vehicles and aggregates them to produce events.It uses best practices in cloud service design in order to scale to large fleets.Finally, because it stores the detected events on the cloud, NORA can easily make this data available to third-party services.

IV. IN-VEHICLE DETECTION
In this section, we first describe our universal library that is used to access vehicle Controller Area Network (CAN) communication, and then discuss synthesis logic for detecting a variety of NORA events.

A. VEHICLE DATA ACCESS LIBRARY
Vehicle CAN Communication: To support the overall operation of a vehicle, a number of Electronic Control Units (ECUs) are commonly interconnected using a serial data bus interface such as CAN, through which sensor data from one ECU is broadcast to other ECUs.An example of such ECUs is Antilock Braking System (ABS) ECU which consumes sensor inputs from individual wheel sensors to assess the braking requirements based on the current road surface conditions.The availability of such sensor data on vehicle networks enables a variety of collaborative sensing use cases that are relevant to the driving environment.
OBD-II Interfaces and Access Library: While the CAN is used for internal communication, it is possible to export CAN communication to an external computer via an On-Board Diagnostic (OBD-II) device.As a result, vehicle data can be processed by offboard algorithms to detect a road hazard and upload that data to a cloud server for additional processing and insight creation.
For this work, we developed a custom Bluetooth OBD-II device, along with a plug-in Android mini-computer module, which were installed into our data collection vehicles.Through API interfaces provided by OBD-II device, the Android mini-computer could monitor vehicle signals relevant to NORA use cases and then detect a road hazard condition based on the monitored vehicle signal values.
The OBD-II device was configured to listen to signals that are relevant to the NORA use cases.As the vehicle sensor data stream arrived at the OBD-II device, it was communicated to the Android mini-computer for local processing so that an onboard algorithm detected a road hazard condition based on the communicated signal values.
To support vehicles of different model years and build configurations, we implemented a special-purpose library in the OBD-II device that looks up the memory address of a specified vehicle signal based on its model year and configuration, and provides a universal API interface to App developers.This design hides complex vehicle implementation details and offers App developers (who are not necessarily automotive experts) a simplified programming interface.

B. DATA SYNTHESIS LOGIC
Modern vehicles delegate control to different sub-systems, each of which is responsible for one aspect of control: e.g., steering, transmission, anti-lock braking etc.Each subsystem, implemented in an ECU, uses several tens or hundreds of in-vehicle sensors that inform the control decision.ECUs communicate between themselves over a CAN bus.
Unlike prior work, NORA re-purposes in-vehicle sensors to detect the phenomena described in Table 1.This section describes these detection algorithms, which often rely on multiple sensors attached to different ECUs.These algorithms run on an in-vehicle CPU that accesses sensors over the CAN bus.The methods of accessing the sensors depend on the model and year of the vehicle; we have developed a library that abstracts these details to simplify the programming of the algorithms.
a) Pothole: NORA uses the wheel rotational acceleration sensor as the primary indicator of a pothole.This sensor, located in the each wheel, can detect slippery surface conditions (to support anti-lock braking and traction control features).We have re-purposed this sensor to detect potholes: if one vehicle's wheel encounters a pothole, its rotational acceleration will significantly differ from the other wheels that do not traverse that pothole.Pothole detection must be (a) robust to vibrations and noise caused by minor road fluctuations or turns and other vehicle dynamics, and (b) able to distinguish potholes from speed bumps.For (a), NORA uses a high-pass filter on the wheel rotational sensor, and flags a pothole if the maximum sensor reading over an interval exceeds a threshold.Moreover, it estimates the severity of the pothole from normalized change in rotational acceleration.For (b), NORA uses  the lateral acceleration sensor from the Vehicle Control ECU to distinguish a speed bump from a pothole.Speed bumps generate less lateral acceleration than a pothole since a pair of wheels encounters these simultaneously.NORA subjects the sensor to a high-pass filter with the cutoff frequency because fast turning or swirling maneuvers can also cause high lateral acceleration.The approach is shown in Fig. 2 The vehicle computes the rotational acceleration difference in real-time and outputs a value between 0 and 255.A threshold T of this value is used to detect the pothole.When the vehicle output is larger than T , the system reports a detected pothole.To determine this threshold T , we conducted data collection with a camera installed near the windshield facing forward.Multiple people watched the video independently and then they visually identified the potholes from the camera data.Those identified potholes are used as ground-truth to compare with vehicle data.In particular, we change the threshold and generate the Receiver Operating Characteristic (ROC) curve based on F 2 measure as shown in Fig. 3, in which the threshold T is different under different vehicle speeds.The final pothole severity s is a normalized value between 0 and 1.Let a z denote the vehicle sensing output, the severity is given by, s = a z −T 255−T .b) Road Roughness: The International Roughness Index (IRI [32]) is a standard measure of road roughness.IRI is the average vertical displacement of the four wheels of the car over a given stretch of road.NORA uses the shock absorber sensor in the real-time damping system to measure IRI.The damping system enhances passenger ride comfort by modulating the chassis suspension system in response to the sensed pavement conditions.The shock absorber sensor directly measures the vertical displacement on each wheel relative to its stationary state.NORA applies a high-pass filter  with a cutoff frequency threshold to remove noise in displacements caused by vehicle vibrations.We observed that vehicle velocity impacts the vertical displacement; so NORA also normalizes vertical displacement values against the velocity through a linear regression.After these steps, it averages the displacements across the four wheels to compute IRI.
Fig. 4 shows a quarter car suspension model.To determine IRI, the suspension displacement measurements are accumulated over certain distance.Let x i be the ith suspension displacement and y i be the ith sampling location on the road.Assuming that f (y i − y i−1 ) represents the travel distance between the sampling location y i and y i−1 (Note that the travel distance can also be obtained from vehicle speed information).Therefore, the IRI is given by: where 0 ≤ i ≤ N − 1, and N is the number of samples.c) Slippery Surfaces: As a vehicle drives through slippery or wet roads, several vehicle dynamic control ECUs -vehicle stability control, vehicle traction control, and Anti-lock Braking System (ABS) -monitor vehicle kinematic states and, as necessary, activate these ECUs to prevent potential accidents.NORA, rather than using sensors directly, uses these ECU determinations to assess the severity of a slippery surface.Specifically, at a given location, let a, s and t be boolean values indicating the activation of, respectively, ABS, stability control, and traction control.Then, the severity of a slippery surface is a weighted average of these three values (a, s and t), with weights estimated using an industry calibration procedure.The overall approach is shown in Fig. 5.
d) Severe Microclimate: To estimate this, NORA uses a rule-based logic synthesis function (shown in Fig. 6) that monitors wiper state, as well as ambient temperature, humidity, and barometer from the engine ECU.Several simple rules of weather events are concatenated together to determine the weather conditions; for example, if the wiper state is on and the ambient temperature is below 32F, it is highly like to be in a snowy condition.These latter sensors determine intake airflow to improve engine operation efficiency.Windshield wipers have two states: on and off.The on time of a wiper is fixed; the off time is smaller under heavier precipitation.NORA leverages this to quantify precipitation levels.NORA uses the humidity sensor and the barometer to validate the estimated precipitation, and the ambient temperature to detect snow from rain.The last step in Fig. 6 is to determine confidence.Particularly, we need to determine how confident the system can report the precipitation as rain or snow.Because both rain and snow can happen when a vehicle measures the environment temperature (close to ground) at 32F, the further the temperature is away from 32F, the better confidence the system has.Therefore, we define a threshold β temp and a linear confidence is given by |T −32| β temp , when |T − 32| < β temp .The confidence is 1 when |T − 32| ≥ β temp , where T is measured outside temperature from vehicle.

V. CLOUD AGGREGATION
NORA receives individual detections from vehicles, together with vehicle location.Using this information, it clusters detections into events, but must also determine if the event has disappeared (e.g., a repaired pothole).NORA suppresses subsequent detection transmissions when its confidence in an event is high; this also reduces communication cost (Section III).

A. SPATIAL CLUSTERING
GPS locations associated with detections can be inaccurate; NORA uses spatial clustering to identify detections of the same underlying event and to improve its estimate of the position of the event.Events in NORA can be highly localized (e.g., a pothole), or span part of a road segment (e.g., rough pavement or a slippery surface).NORA uses point-based clustering for the former and segment-based clustering for the latter.
Point-based clustering: For discrete events where multiple samples of the same event are obtained from a single location, a point-based clustering is applied.One such example is a pothole event, in which the event data from different vehicles are generated at the same location where the pothole is encountered by the vehicle.In this category, we cluster the event data nearby the location point (shown as triangles in Fig. 7) within a certain distance range into a single event.
Segment-based clustering: For spatially continuous events in which the vehicles obtain the samples of the same event at various locations, we applied a segment-based approach.Such events include severe weather, slippery road condition, and road surface roughness.In this case, we first need to map match the detected event data onto the relevant road segment, using Hidden Markov Model (HMM) model, and then we can cluster the event data into the corresponding road segments.
Once different pieces of individual event data are added into a cluster, the average event severity, and the location, of the road segment can be calculated.Meanwhile, the lifetime of the event can be managed accordingly.Algorithm 1 illustrates the pseudo code for clustering algorithm.Let C denote a set of clusters.Let |x, c| denote the distance measurement (e.g., haversine distance) between a sample point x i and the center of the cluster c i .The cluster range threshold T is different for different types of NORA events, and it is obtained by engineering calibration process used in automotive industry.
NORA calibrates the clustering thresholds using test data, in much the same way as we calibrate sensors on vehicles.In particular, the threshold of point-based clustering is determined based on GPS performance and average vehicle traveling speed or speed limit, denoted as v. Specifically, the threshold T is given by T = v • t + μ + 2 • σ where t is sampling latency and N (μ, σ 2 ) is GPS error distribution.As shown in Fig. 8.The vehicle GPS (shown as ublox) has an average error of μ = 1 meters with standard deviation σ = 0.4.Before clustering, it map-matches the reported GPS locations using a Hidden Markov model in order to improve the accuracy of the estimates.For localized phenomena, NORA computes the location of the phenomena as the centroid of  all the reported locations of the corresponding cluster.For phenomena that span road segments, it identifies the spatial extent of the cluster along the road segment.In both cases, it estimates the severity of the event by averaging the severity of the individual detections.

B. TEMPORAL MAINTENANCE
NORA needs to reliably determine when an event has disappeared.Inspired by [33], it uses Time-Decay Sequential Hypothesis Testing (TD-SHT): NORA decays its belief in an event over time, and reinforces this belief when it receives a new detection.If the belief in the event drops below a certain threshold, NORA removes the event from its database.
To be more specific, consider an event E for which NORA's cloud component receives the ith detection for this event (more precisely, this detection falls into the cluster corresponding to E ) with severity s i .It assigns an initial belief b i based on the confidence associated with that detection, which is directly obtained from automotive sensor modules.Then, to each detection, it assigns an exponentially decayed belief B i (t ) = b i e −γ (t ) .It then continuously tracks the total belief B total (t ) of all detections within the cluster as a function of time.When this total belief exceeds a threshold η h , NORA adds this event to its database.This total belief decays over time, but new detections can reinforce the total (Fig. 9).When the total belief drops below another threshold η l , NORA believes the event has been fixed and removes it from its database.
We determine the decay parameter γ from the expected lifetime of events (e.g., a few hours for slippery surfaces and severe climate, in days, weeks or months for potholes and road roughness).The thresholds determine how many detections NORA requires before it can be confident in its assessments.

C. SELECTIVE SENSING
Road events may persist for a long time before they disappear.For example, potholes may last from a few days to a few weeks before being repaired.Long-lasting road events, like potholes, could generate redundant event reports by all vehicles traversing the same pothole over a long period, unnecessarily wasting wireless bandwidth.It is not desirable to report all the detected road events; on the other hand, only relying on the first few event reports is not a good solution either, since later event reports from bypassing vehicles are used to determine if potholes still exist.
In our study, inspired by Proportional-Integral-Derivative (PID) controller logic, we propose a novel Selective Sensing algorithm, which suppresses redundant event reports if such events are already known in the cloud, but encourages newly observed events to be reported.This way, we can reduce the number of unnecessary, redundant event reports while still maintaining the confidence of event estimates in the cloud, striking a right balance between event aggregation accuracy and system resource consumption.
Selective Sensing Cloud Operation: Fig. 10 shows an overview of Selective Sensing modules.The Selective Sensing algorithms run on both the cloud side and the vehicle side.The cloud side selective sensing algorithm collects road event reports from multiple vehicles and uses the Hybrid Criteria Function to characterize the expected probability of uploading an event report.Then, the PID controller continues to calculate a re-sampling instruction, which will be sent to individual vehicles.The vehicle-side selective sensing algorithm receives the re-sampling instruction (δ) and uses it together with other local system parameters (α, β, γ ) to calculate the final probability to upload or drop the next road event report.
Criteria Function for PID: The input to the algorithm includes all the samples (i.e., event reports) uploaded in the previous time frame.A sample tuple, (type, posit ion, severity), consists of the event type (such as 'pothole'), location (such as longitude and latitude), and severity value of the detected event.Next, we define a set of three criteria to quantify the quality of these uploaded samples: r Number of uploaded samples, n; r Standard deviation of event position data from uploaded samples, s p ; r Standard deviation of event severity data from uploaded samples, s s .The number of uploaded samples n is the most important factor in determining the quality of samples, while the latter two factors (s p and s s ) compensate the first factor n since both standard deviation factors jointly offer a fine-granularity insight into error distribution functions.
We use a hybrid criteria function to characterize the expected probability of uploading a road event: where w 1 , w 2 , w 3 are weights, n 0 is the target sample size, and s p0 and s s0 are reference standard deviations for sample locations and severity.s p0 and s s0 can be determined using engineering estimates based on real-world data, such as the probability distribution later shown in Fig. 15.The target sample size n 0 can be calculated using a common method in statistics, where Z 2 a is Z-score with confidence level a, τ is the required precision, and σ is the population standard deviation, which is usually an engineering estimate.
PID Controller: Hybrid criteria function h implicitly represents the amount of redundant road events uploaded to the cloud.Due to uncertainty caused by drivers, vehicles, or environmental factors, the actual probability of individual vehicles encountering and reporting a road event often could be further complicated.For instance, once a driver is notified about a pothole, she may maneuver to avoid the pothole ahead.
It is a well-known challenge to create a comprehensive white-box model to capture all the uncertainty factors; instead, we decide to treat the model as a black box and therefore apply Proportional-Integral-Derivative (PID) controller to adapt to the change of h.The PID controller in our study is defined as: where u(t ) is the action, representing the amount of samples to be reduced.K p , K i , K d are the coefficients for proportional, integral, derivative terms.The PID controller continues to generate a re-sampling instruction δ, δ = u(t ) 1+u(t ) , and then sends it back to individual vehicles.δ represents the probability that an individual vehicle should use to either suppress or upload a report for a specific road event being detected.
NORA Vehicle Operations: Once the re-sampling instructions from the cloud are sent back to a vehicle, the vehicle-side Selective Sensing component (as shown in Fig. 1) will increase or decrease the probability of uploading road event reports based on the instructions.Fig. 10 shows more details of the vehicle-side Selective Sensing component.The Probability Function block is a utility function taking the cloud re-sampling instruction (δ), local sampling rate (α), local cost (β), and local sensor confidence (γ ) into consideration.The function combines these cloud and local parameters, calculating a final probability for event uploading.The Filter block will remove event reports based on the calculated probability derived from Probability Function block, so as to suppress the redundant events.Later in Section VI-E, we evaluate the efficacy of our proposed selective sensing algorithm.

VI. RESULTS
In this section, we first introduce the NORA pilot and its data characteristics in Section VI-A and Section VI-B, then we focus on evaluating NORA pilot to answer the following questions: r What is the end-to-end performance of NORA system?(Section VI-F)

A. NORA PILOT
NORA Pilot: Following the architecture (Fig. 1), we implemented our NORA pilot and enrolled 50 vehicles to detect and analyze road hazard events over a 9 mo period beginning in February 2017.
Emulation Platform: To test system scalability, we also developed a large-scale emulation platform that emulated thousands of virtual vehicles that seamlessly coexisted with our actual NORA vehicle fleet.To do so, we developed a ns3-based simulated environment that creates vehicle mobility traces using the Open Street Map (OSM) map [34] and SUMO Traffic Simulator [35], generating up to 10,000 virtual vehicles that move within the simulated world.In a separate module, we generated virtual road events (point-based potholes and segment-based slippery surfaces) in the simulated world, using road anomaly models trained by the empirical NORA event data.As virtual vehicles encounter a road event, they create an event report and upload it to the NORA cloud.The emulated and empirical vehicles are treated in the same fashion by our NORA cloud, enabling a seamless co-existence between empirical testbed and emulation platform.

B. DATA SET CHARACTERISTICS
For the duration of the pilot, a total of 130,000 miles of vehicle data was collected.In total, our NORA fleet detected about 110,000 instances of pothole events and 30,000 instances of slippery surface events, though the same pothole or slippery surface event could be reported multiple times by different vehicles on different days.
Vehicle usage patterns meant that the maximum vehicles online in a single day was 16 (Fig. 11(a)).On weekends, there were less vehicles online, but several vehicles continued to report data.Fig. 11(b) shows the number of miles travelled each day and the longest trip that occurred that day.Additionally, Fig. 11(c) shows that 50% of trips were longer than 50 miles during the deployment.

C. IN-VEHICLE DETECTION ALGORITHMS 1) CONTROLLED EXPERIMENTS
Controlled experiments evaluated the accuracy of NORA event detection algorithms (Section IV).During these experiments, ground truth data was collected.
Pothole Detection: We collected data along a 4-mile route at different driving speeds.The route was repeated 8 times on the same day.A dash-mounted camera captured the forward view of the road surface.Afterward, we visually inspected the videos and identified potholes.A total of 23 potholes were identified along this 4-mile route.
Results.For individual potholes, the average precision and recall over the 4 mile route were 73.05% and 71.43%, respectively.When data from 8 passes is aggregated, the results improve to 100% precision and 91.2% recall.This demonstrates that, while individual detection has reasonable detection accuracy, both false negative and false positive rates can be significantly reduced through crowd-sensed aggregation.
Road Roughness Detection: NORA estimates roughness according to the International Roughness Index (IRI) measure (a civil engineering metric).A survey vehicle was used to measure IRI ground truth and captured data across 18 miles of local roads and 46 miles of highways.The vehicle was followed by three NORA vehicles that also measured the IRI through our suspension-based solution.Over the course of two days, a total of 6 passes of NORA vehicle data over the same 64-mile route was collected.
Results: Aggregation from 6 vehicle passes resulted in a Pearson correlation of 0.86 compared to ground truth, indicating that our sensing approach using consumer vehicles could achieve similar results to specially instrumented vehicles.It achieves an overall Root Mean Square Error of 51.1 in/mile, and the half of the estimations have absolute errors less than Results: Within this area, our vehicle fleet detected 5.4 miles of slippery road, shown as red road segments in Fig. 12.An OSM query of the same area returns a total of 14.5 miles of road (red and blue).We assume all road segments were slippery since significant snowfall occurred that day (more than 6 inches) and snow had not been removed.The resulting detection precision is 1 and the recall is 0.37.The low recall value (high false negative rate) is mainly attributed to the fact that many small residential roads were not travelled by our NORA vehicles.
Severe Microclimate Detection: We selected a snowy day to drive a route that consisted of local roads and freeways (there was a significant snowstorm and various locations received 6 to 14 inches of snow on that day [36]).While vehicle data was being collected, the cloud server periodically accessed public weather station data to obtain real-time precipitation levels.
Results: Fig. 13 illustrates the snow severity level detected from wiper status compared with precipitation level.At the beginning of the trip, the dotted line shows that the precipitation level was light at around 0.  wiper signal indicates that the severity level stayed on level 2. The trend between the severity level and precipitation level achieves a good match.Spikes in wiper state were caused by human action, an occasional outlier that could be removed by cloud aggregation.

2) FIELD TESTS
We also report experiment results from testbed deployment.Due to space limitations, we only present pothole detection results.The lessons learnt here could also be applied to other NORA applications.
Characteristics of Potholes: Fig. 14 shows a summary of pothole events that were collected during a consecutive 16day period (March 5, 2017 to March 20, 2017).There were no specific routes or predetermined times for NORA vehicles to travel.During the daily drives, pothole event data was sent from NORA vehicles to the NORA cloud every minute.Fig. 14(a) illustrates the number of reports received for each individual pothole.It shows that most potholes have a severity less than 0.6, each of which has been reported 10 times or less.However, some potholes receive as many as 20 reports.Potholes with severity larger than 0.85 receive much less reports because the drivers likely took action to avoid these severe potholes.Fig. 14(a) clearly demonstrates that the vast majority of potholes are not very severe, so a real-time detection system, such as NORA, could help road network operators to timely identify and repair the non-severe potholes before they further deteriorate.
Fig. 14(b) shows the time evolution of the total number of reports across all potholes.The total reports increase rapidly during the first 6 days, with a significant spike on the 7th day.However, from the 8th day onward, the total number of reports increase slowly, because NORA cloud became more confident about detected events and thus selective sensing technique suppressed redundant reporting.different potholes.In Fig. 15(b), most reported potholes have low severity.These distribution functions obtained from NORA pilot experiments can be used to configure the models used in civil engineering for pothole monitoring purposes.
Fig. 16 shows the pothole monitoring from winter to spring.The severity is illustrated both by color of the marker, as well as the size of the marker.The more severe the pothole is, the larger the marker is.The pilot fleet vehicles commute between several cities, which monitor the local roads as well as freeways.In winter months, due to the snow removal process, the road surface frequently gets damaged.Therefore, more potholes can be detected and reported during those winter months.After the beginning of the deployment in February, the NORA started to report the potholes.The reports reach their maximum in the month of April at the end of the snow season when repeated freeze-thaw cycles create the most damage to the road infrastructure.Gradually, the road surfaces are repaired and we can see the number of potholes, particularly severe potholes, are significantly reduced in June.

D. CLOUD AGGREGATION ALGORITHMS
We executed the cloud aggregation algorithms consecutively for 16 days.At the end of each day, a snapshot of detected NORA events was taken.Fig. 17 illustrates the evolution of one such pothole event.Without losing generality, the belief value of each individual report was set to 1 for simplicity.In addition, the decay parameter was set to 0.005 for a slow decay since it may take weeks for a pothole to be fixed.
Spatial Clustering: Fig. 17(a) shows the standard deviation of the event severity value for a detected pothole over time.Due to the heterogeneous severity readings measured from different vehicles for the same pothole, an individual reading may look very different from another reading.However, by clustering those readings together based on their locations, the standard deviation of the event severity measurement reduces, obtaining a more stable consensus.From this figure, we also notice that among the 16 days of experiments, several days did not see the reported events for this pothole.Therefore, the standard deviation of the severity stays the same for those days.
Temporal Maintenance: Fig. 17(b) illustrates the evolution of total belief value for this pothole event.In the first two days, there were events reported and clustered for this pothole.However, no events were reported from the third day to the fifth day.Therefore, the figures shows an initial increase of the total belief values for first two days.After that, the total belief value kept decreasing until the new events reported on the sixth day.Similarly, for the rest of the days, the total belief value increased when there were new reports of the pothole.It started to decrease when no reports were received.Particularly, more reports were sent on the seventh and eighth day which significantly increased the total belief value.Since we set a small total belief value threshold, this pothole had been identified through the whole 16 days of aggregation, which reveals that the pothole was possibly either not reported to the road maintenance department or it takes a long time for it to be repaired.

E. SELECTIVE SENSING ALGORITHM
Experiment Setup: To study the performance of selective sensing algorithm in the large-scale emulation platform (Section VI-A), we extracted a 20 x 20 mile map (of the city where NORA pilot was deployed) from OSM and generated a 10,000-vehicle mobility trace using SUMO.Again, as an example, a total of 1,000 potholes were generated with random positions that covered all major road types, such as highways and residential roads.We assumed that the pothole event duration follows a Poisson distribution with λ = 10, which represents that a pothole exists approximately 10 days before being repaired.The experimentation duration used was 30 days.
Results: The performance metrics for selective sensing are bandwidth usage and detection accuracy.We used the number of reports and the number of detected events to compare the performance when selective sensing algorithm is enabled and disabled, respectively.Fig. 18(a) shows the number of event reports as the emulation time elapses.We find that selective sensing algorithm reduced the total number of uploaded event reports from 26,192 to 12,357 -a 53% reduction in this particular scenario.Fig. 18(b) shows that the number of detected pothole events are almost the same when selective sensing is enabled and disabled.These results verify that the selective sensing algorithm successfully suppresses unnecessary event reports, while maintaining detection accuracy.
As shown in Table 2, we discover that different road types have significantly different data reduction rates.On the one hand, roads with high traffic volumes (e.g., Motorway or Primary) have much higher reduction rates due to the high vehicle density encountering NORA events; on the other hand, roads with less traffic volumes have much lower reduction rates.This suggests that we should select different selective sensing parameters depending on the road types, achieving an even more aggressive trade-off between the confidence of event detection and communication cost.That is to say, in Section V-C, the Z-Score confidence level a can be tuned for different road types, in order to improve data reduction rates.

F. THROUGHPUT AND LATENCY
We conducted experiments using a large number of emulated vehicles to stress test the NORA system.
Scalability Test: The experiments were conducted under various scenarios (i.e., different number of emulated vehicles and different number of NORA events) to measure the system throughput, latency and system workload: r System throughput is defined as the number of events that NORA processes every 10 seconds.
r Latency is defined as the average time spent at cloud aggregation stages.
r Workload measures the average CPU utilization incurred at those cloud processing stages.To evaluate the system under different scenarios, we experimented with a different number of emulated vehicles as well as a different number of events.
In all experiment configurations, we found the CPU utilization remained at a very low level (≤ 20% utilization).The database operation also performed consistently with a throughput of up to 43,000 records every 10 seconds.On the other hand, as shown in Table 3, with an increasing number of reporting vehicles or NORA events, the amount of information being processed by the NORA cloud increases, resulting in increased throughput.However, the NORA cloud system latency was not significantly affected (in the order of several seconds, which is sufficient for navigation application) and still working efficiently under different stress-test scenarios.This set of experiments helps us conclude that our NORA cloud system design and implementation is fundamentally scalable for a city-level operation.
Performance of Integrated Service: To render an integrated, complete service of delivering timely road event notifications to drivers, we also evaluate (1) the system latency between our NORA cloud and the third-party cloud server and (2) the system latency between the third-party cloud server and the vehicle navigation system.To stress test the interaction between both cloud servers, we send a various number of events (ranging from 10 events/second to 1,000/second) from the NORA cloud to the third-party cloud.As a result, the average observed latency for the third-party cloud to receive NORA event data is 1.2 seconds, with a range between 500 milliseconds to 3.3 seconds.Meanwhile, it took an average of 78 milliseconds for the navigation system to receive and display the event once it was received from the third-party cloud server.This preliminary experiment shows that our cloud coordination architecture between multiple parties could satisfy customer navigation needs.

VII. REFLECTIONS ON LESSONS LEARNT
We derived a set of lessons or recommendations concerning the design, implementation and deployment of vehicular crowd-sensing technology in real-world environments.
Embedded Vehicle Sensing Capability: Unlike conventional road sensing research works that leverage smartphone sensors, our NORA pilot directly employs embedded automotive sensors.As such, our NORA applications either generate very accurate sensing results based on sensor quality and fixed mounting within the vehicle, or provide a satisfactory solution that smartphone sensors fail to do so (e.g., slippery surface).On the other hand, these embedded automotive sensors were originally designed for other purposes (e.g., pothole detection is derived from a system that was developed to support engine misfire detection).Since many or most vehicle sensors were not deployed with the intended use case in mind, data quality and sampling rate issues need to be overcome using various types of algorithmic techniques and data fusion approaches.To achieve desirable outcomes, we recommend that (1) application designers consult automotive domain knowledge experts to understand the intricacies of the subsystem design and (2) application designers should augment the processing of sensor data using other available vehicle context information.
Validation using Controlled Tests and External Data Sources: Another important lesson from this activity was the importance of developing approaches for measuring ground truth.It is important to spend time on developing controlled test scenarios and in accessing other information sources (e.g., weather service APIs, pavement data from government) to validate the obtained event detection results.
Iterative Software Design and Enhancement: Building a general-purpose technical framework like NORA is indeed a rewarding experience, however, it was a challenge to accommodate, deploy and operate all the applications concurrently.Software bugs and performance issues always prevented the system from functioning holistically and seamlessly.Instead, we painfully realized that using a phased deployment approach (designing and deploying a single application at a time) is a more productive and manageable approach, while carefully protecting the framework flexibility of supporting other applications or features.To support this goal, it is imperative for us to design dedicated service packages that manage the operation-plane services, such as software bug identification, function error logging, incremental over-the-air software modification, remote reflash, and software package upgrade.These operation-plane services helped our developers to improve the system design and observe any malfunctions that may have occurred.

VIII. CONCLUSION
Using embedded automotive sensors on production vehicles, our in-vehicle contextual sensing mechanism detects many aspects of the roadway and environment including potholes, road roughness, slippery road and severe weather conditions.Moreover, sensor data traces from a large number of vehicles, when crowd-sourced, provide increased spatio-temporal coverage as well as improved sensing accuracy.In this article, we demonstrate a novel vehicular crowd-sensing platform, NORA, to support a rich variety of road anomaly event applications.The results show that our NORA platform serves as an accurate, effective, and efficient mechanism to detect and communicate various types of potentially hazardous road events on the roadways.

TABLE 1 .
Examples of Crowd-Sensing NORA Applications improve their performance, and (c) inform maintenance decisions made by road network operators.

r
How accurate are event detection algorithms?(Section VI-C) r How effective are cloud aggregation algorithms?(Section VI-D) r How effective is selective sensing algorithm?(Section VI-E)

FIGURE 11 .
FIGURE 11.(a) Vehicle summaries.(b) Daily travelled distance during deployment.(c) Cumulative distribution function for single trip distance.

FIGURE 12 .
FIGURE 12. Slippery detection road segments.Red color indicates detected slippery surfaces.
01 inches per hour as captured by the weather stations.The vehicle wiper signal (solid line) indicates that the severity was also light at level 1.The snow level gradually increased to 0.08 inches per hour and a constant wiper setting was required.The wiper signal also indicates that the severity level increased to level 3.After 17:45, the precipitation level fluctuates between 0.02 inches per hour and 0.06 inches per hour.The corresponding

FIGURE 14 .
FIGURE 14. Results for daily driving experiment.

Fig. 15 (
a) shows the Cumulative Distribution Functions (CDFs) of the number of reports for each individual event in this 16-day period, indicating a significant variability among

TABLE 3 .
System Performance