Exploring multimodal feedback for an NFC-based mobile shopping assistant

This paper explores how mobile devices can provide useful multimodal feedback during interactions with NFC-tagged objects. We exemplify our approach with a mobile shopping assistant that uses haptic-visual and audio-haptic feedback to inform diabetics about the agreeability of tagged grocery products. A preliminary laboratory study shows a preference for haptic-visual feedback which is regarded as more helpful, easier to learn and more pleasant to use. The results stress the usefulness of multimodal feedback for mobile interactions with NFC-tagged objects in this context and inform future work, e.g. about the expressiveness of different feedback modalities and patterns.

I. Introduction

Mobile devices are great means for interactions with everyday objects. They can capture and identify objects according to their location (e.g. with GPS), by taking pictures of attached visual markers (e.g. barcodes, QR codes) or by touching NFC-tags (Near Field Communication) [18]. That way, mobile devices allow users to interact with physical objects, to access associated digital information and to use the latter for different applications and services. Examples are mobile applications for the retrieval of information about groceries, medicine or other commercial products, such as availability, shelf life, origin, ingredients or nutritional value (e.g. [13], [17]). Mobile applications can use this information to compare products or to combine it with information about users, like personal preferences, allergies, diet plans or medications, in order to recommend products or to advice against them.