An in-the-wild and synthetic mobile notification dataset evaluation

Managing the vast amounts of information being pushed at mobile users is a challenge that is becoming increasingly difficult as the number of connected devices and users continues to expand. In order to overcome this challenge, a Notification Management System (NMS), needs a number of detailed data resources in order to decide what to do with an incoming notification in-the-wild. Explicit data contained within the notification and contextual information regarding the user and immediate environment are both necessary in order for a system to accurately infer a user's preferred delivery time for a given notification. Due to the sensitive nature of notifications and contextual data, it is difficult to acquire the explicit notification datasets which sufficiently describe the incoming notifications as well as the current contextual states of the user. This poses a problem for prospective research in the domain of Notification Management as arduous and time-consuming data collection is necessary if a hypothesis depends on unique notification/user features not previously collected. Without a number of rich notification datasets, either experimentation is limited to synthetic, vague or incomplete data, or time must be invested in developing a system to capture the required features. This paper evaluates a notification dataset previously collected in-the-wild and subsequently used in an evaluation of a NMS. The necessary features of the collected dataset are outlined as well as its limitations. As a comparison, the process of creating a synthetic notification dataset derived from a mobile usage study carried out by the MIT Media lab is also evaluated. The synthetic dataset is henceforth used to optimize a previous set of knowledge base rules and membership functions used within the Fuzzy Inference System (FIS) of an NMS. The resulting optimized rules can be presented to the user as a means of throttling notifications based on their goals.