Pervasive Augmented Reality—Technology and Ethics

In the foreseeable future, mobile and wearable computing technology with an augmented reality (AR) interface can provide an omnipresent, environmentally adaptive, and everyday reality augmentation. This new pervasive AR technology will lead to a continuous moderation of experienced reality with the potential to support better and faster decision-making, the exploration of new information, and novel ways of communication, interaction, and collaboration. However, pervasive AR technology will also have undesired consequences, e.g., in the areas of privacy, commercial exploitation, distractions, digital inequality, and our perception of what is true and real. Little is known about how severe these effects will be when AR has become pervasive and how they can be prevented or mitigated. We draw on current developments in research and the market, sketch a near-time future of pervasive AR technology, identify ethical considerations, and discuss the development of pervasive AR systems.

T he omnipresence of today's smartphone as a ubiquitous interface to digital information might be replaced with a different interface in the near future. One of the most prominent candidates are headworn displays presenting an augmented reality (AR) interface. There are already many research prototypes and commercial products (e.g., Microsoft's Hololens and Snap Spectacles) of this technology. Future head-worn displays will integrate sensing, processing, and interaction functions within devices that are as easy to wear as regular spectacles. 1 These technologies will be aware of the user's context, including their environment, sensations, perceptions, and articulations. This will drive a continuous and omnipresent augmentation of the user's environment with digital information registered in the real world. 2 As such they have the potential to make information much more accessible in the right spatial and temporal context than AR today. They can revolutionize the way we communicate, collaborate, train, educate, entertain, rehabilitate, treat, manage, or make all kinds of decisions. They can realize true pervasive augmented reality (Pervasive AR), i.e., the continuous and ubiquitous experience of computer-mediated reality. 2 But realizing Pervasive AR will also come with unintended consequences. Pervasive AR will challenge our ability to recognize what is true and real. It will be exploited for pure commercial interest. It will further erode privacy. It will interfere with social interaction. It will distract our attention. It will empower some more than others, aggravating digital inequality. We do not know yet, how severe these effects will be once AR has become pervasive. Who is responsible for anticipating or preventing these consequences? We argue that we must recognize a shared responsibility as it cannot be placed within any one singular group. The development of technological artifacts, in particular disruptive information technology, is a complex process requiring expertise from different intertwined areas in science, research, development, innovation, commerce, social science, and policy. We, thus, argue here that the expertise to understand all aspects of Pervasive AR systems can neither sit with one individual nor can the responsibility be entirely delegated to others. In our roles as technologists, we cannot possibly delegate the judgment of ethical consequences to, say, social scientists alone. We should take responsibility for what we can influence.
We think the time is right for a discussion of the ethical issues around Pervasive AR. Conventional AR is emerging on the mainstream market, while  (2) consider how we can ethically design, develop, and integrate Pervasive AR. In this article, we will first explain the basic technical components and mechanisms of Pervasive AR, including the description of near-future Pervasive AR systems. We will then discuss the ethical challenges of the implementation, use, and integration of Pervasive AR.

TECHNICAL FOUNDATIONS FOR PERVASIVE AR
Advances in the fields of AR, ubiquitous computing, human-computer interaction, and continuous miniaturization are improving the technical components and design aspects that will soon allow for AR to become pervasive. These include displays, sensors, communication capability, the user interface, and their integration.

Display
The display is one of the key components of any visual AR system and is increasingly based on optical seethrough head-mounted displays. They use a semitransparent optical combiner that integrates a computer-generated image into the view of the physical world. In principle, this can be integrated into the frame of spectacles, but not without one or more of today's limitations: 1) low fidelity of the image, 2) a limited field of view, 3) issues with colour and brightness, 4) poor latency, and 5) accommodation-vergence conflicts. 3 However, with a number of companies and research institutions attending to these issues, we expect displays in the shape of traditional spectacles to become available soon.

Sensing and Interaction
To augment the most helpful and relevant content in any context, Pervasive AR systems need to capture the spatial and temporal context using a diverse array of sensors to determine, e.g., location, orientation, physical environment, gestures, and aspects of the user's state. We have already witnessed the proliferation of smartphones and smartwatches with high sensor integration. Similarly, there are various advances in the (AI-based) interpretation of sensor data for context-awareness. Hence, relevant context-sensing technology is maturing and will contribute to Pervasive AR.
Localization and tracking: To track their location, AR systems tend to rely on simultaneous localization and mapping techniques using cameras and Inertial Measurement Units. Future Pervasive AR systems will use those not just for building 3D representations of selected indoor environments but for reconstructing the entire physical world around the user while at the same time using those representations to track the position of the user. With an increasing number of wearable devices utilizing 3D representations, we will also see cooperative approaches emerging that share those reconstructions across devices and users.
Sensing the user state: Over the last decade, we have seen a rapid increase in body-worn sensors to capture physiological measurements. Initial prototypes have used dedicated sensors that are now being integrated into the latest AR solutions. 2,4 Eye-tracking, for example, has been integrated into the most recent commercial AR displays, not just to track the gaze but also to capture the user's state. 4 Furthermore, researchers started to capture and integrate human facial expressions, brain waves, heart rate, skin temperature, with the first commercial applications now entering the market. With continued research and development, AR devices will be increasingly adept to capture the user's physiological and even emotional states.
User input: Increasingly, AR systems leverage cameras and other sensors to capture direct user input. Some systems utilize microphones and the use of voice. Some allow for input through hand gestures, typically directly in front of the user by detecting depth. In the near future, with a wider array of sensors, we expect AR devices to allow for more input modalities, including hand and arm gestures around the body, facial expressions, and eye movements. The increased diversity of user input means that the AR systems can be controlled in almost any situation.

Communication
One key component of current and future mobile and wearable technologies is the ability to seamlessly connect to the internet using different technologies, including mobile data networks and WiFi. Future Pervasive AR technologies are likely to be released with 6G capable network technologies, with predicted speeds of up to 95 Gb/s. This would allow for a continuous exchange of data, including 3D environment data needed for tracking and information transmission on services and points of interest in the user's proximity.

User Interface
We assume that Pervasive AR interfaces will mainly rely on visual display modalities ( Figure 1), possibly with auditory input, cues, or audio augmentations. With its ability to sense the user's context, the user interface can adapt to and blend into the broader context. This fits in with a broader trend toward calm computing, where technology stays in the background to silently serve the user. With this disappearing interface and advances in fundamental technology developments, what is real and what is virtual will fuse to a seamless mixed-reality experience. It will be increasingly difficult to distinguish between the two.

Integrated Pervasive AR System
Combining all available technologies and integrating them into one coherent system to be worn continuously has the soon-to-be-realized potential to deliver a seamless Pervasive AR experience. Akin to and extending the concept of smart eyeglasses, 1 cameras, microphones, and other sensors capture the environment in tandem with physical and physiological input streams such as gaze, heart rate, galvanic skin response, and temperature, while working with high resolution, high fidelity, calibrated visual, auditory, and haptic displays to allow for context-tailored Pervasive AR use. In sum, many required technical components and aspects are already available. Their improvement and integration are underway with research in academia and in industry. Thus, we can assume that it is just a matter of time until we will have access to such integrated technology allowing for a continuous Pervasive AR experience. In a not too distant future, users will potentially wear their Pervasive AR glasses in the same way as they would wear traditional glasses. Similar to smartphones or recent smartwatches, they can become our constant companion being worn all day. Pervasive AR glasses can continuously, and without any further direct user interaction, adapt to the current spatial and temporal context of the wearer and continuously present overlaid information appropriate to that spatial and temporal context. Figure 2 illustrates how a user would continuously wear the glasses in different situations during one day.

PERVASIVE AR AND ETHICS
Ethics in computing is a widely discussed topic and in particular, privacy has received much attention. In the following, we will extend the discussion on ethics to Pervasive AR using a broader view than privacy alone, consisting of four dimensions of interest.

Ethical Dimensions of Interest
Moral philosophy would normally take one of the following three positions on ethics 5 : 1) rule-based ethics (e.g., Kantian duty ethics); 2) utilitarian ethics (as, for instance, exemplified by the trolley problem); and 3) virtue ethics (cf. Aristotle's Nicomachean Ethics).
We do not want to favor one view here, rather we would like to take into consideration aspects of each of those positions, e.g., on the importance of codes of conduct, regulatory mechanisms, and the individual's moral action. Because Pervasive AR is not a reality yet, we cannot reflect on its observed ethical implications, rather we have to rely on our expert-driven judgment.
When identifying our main ethical dimensions of interest for AR, we examined related technologies,  such as virtual reality with identified issues arising from, e.g., "superrealism". 6 We also informed our work by earlier discussions of ethics in AR by McEvoy, 7 Brey 8 highlighting misrepresentation and biased representation as the main ethical shortcomings of AR, and Madary and Metzingers 9 who in particular emphasized the subjects' well-being when using an AR system. This includes motion sickness, information overload, and reentry into the real world. We further adapt selected guiding motifs of the data ethics commission , a in particular the principles of ensuring a human-centered and valueoriented design of technology, fostering responsible data utilization, and the introduction of effective oversight of algorithmic systems. Considering the abovementioned guiding principles and applying them to a value-driven approach to guidelines 10 , we identified four guiding dimensions of ethical interest for AR and Pervasive AR: data and privacy, health and safety, illusion and belief, and rights and access. Those dimensions will be discussed in the following sections.

Data and Privacy
How to handle user data and privacy is a question now relevant to almost all disciplines within computing. However, maintaining privacy is also a controversial topic in other disciplines such as in journalism and street photography with respect to the agreed-upon procedures and processes around data capture, processing, identification, and storage and analysis.
AR: The introduction of Google Glass in 2013, and its adoption in public spaces sparked great controversy and demonstrated that AR raised new ethical issues. While one could argue that Google Glass is not an AR device according to commonly used definitions, the issue that, similar to actual AR glasses, it integrates a camera that is able to capture the environment was identified as being problematic. Since then AR has been implemented mainly on mobile phones, for playing mobile games (e.g., Pok emon Go), and for marketing and shopping guidance purposes. Consequently, the privacy discussion came often from a marketing point of view and the issue of unwanted exposure, bias perception of reality, and contextualized marketing. 11 In fact, several of the authors bring up the notion of privacy as a cost that is traded against perceived gains. [11][12][13] Pervasive AR: Privacy issues of Pervasive AR applications go beyond those of traditional AR (e.g., on mobile phones). While traditional AR applications tend to be used for short periods of time (e.g., to inform about certain products that are virtually integrated into the user's view), Pervasive AR can be always on. Similarly, Pervasive AR benefits from the use of "intelligent user interfaces," which adapt to the current context of use by incorporating a comprehensive user model, environmental awareness, and continuous sensing and tracking. Integrated camera(s) capture the environment and everything within view (people and objects) over potentially extended periods of time. Similarly, Pervasive AR devices can long-term capture detailed information (e.g., locations/movement patterns, physiological data) about the user. Hence, if exploited, the data captured in Pervasive AR opens up pathways to build models not only about the user but also everything else in the user's view and environment. The potentially "always on" and "disappearing interface" aspects of Pervasive AR exacerbate the issue and make it less noticeable. This is not only an issue for private use but equally problematic once Pervasive AR is used professionally at the workplace.
Discussion: Current industrial guidelines to mitigate negative consequences for privacy consider rather simple mechanisms like Facebook's (now Meta) Let that capture LED light shine and Power off in private spaces ("Ray-ban Stories") that would only work for noncontinuous use of isolated applications. Pervasive AR faces the challenge that there is no simple opt-out mechanism as data is collected as part of its functionality. This is even more true for people affected by using Pervasive AR once it becomes omnipresent. We cannot assume all people in the environment to indicate (e.g., via a beacon) their consent. While conceptually possible to process all captured data only internally but never expose it to third parties (or even the user) once processed (e.g., do not keep a history of events), experiences from past events tell us that such mechanisms can always be undermined in practice and are hard to monitor for compliance. We also do not support the idea of trading away privacy for potential tangible and intangible user benefits; while this might be applicable to certain contexts where users can truly make well-informed decisions, the intentional unobtrusiveness of Pervasive AR would make explicit, frequent "privacy-trading" impractical.

Health and Safety
The more technology is part of our daily lives, the more we need to consider aspects of health and safety as part of any ethical consideration to guarantee physical and psychological well-being.
AR: When considering traditional AR applications, the main health and safety concerns are similar to a htt_ p://www.datenethikkommission.de those considered for VR, namely effects resulting from simulator sickness (e.g., the work of Madary and Metzinger 9 ) and health risks from not being aware of one's surroundings because the world is only seen through the AR display. In addition, in particular with commonly used mobile AR applications, the phenomenon of "looking down" can lead to serious health and safety issues.
Pervasive AR: We see the following three areas of particular relevance when discussing health and safety concerns: 1) perceptual distraction and blind spots; 2) ergonomics; and 3) long-term effects of AR exposure.
While already present in traditional AR, these risks will be more prevalent in Pervasive AR through its extended usage patterns. For example, with respect to (1), there is only limited research and understanding of how information is integrated into the users view to limit mental workload and reduce distraction. 4 Similarly, with respect to (2) and (3), we know from other emerging technologies that ergonomics paired with long-term usage can create unexpected adverse health effects (e.g., touchscreen-induced pain). While Pervasive AR requires less user input, gaze and eyebased interfaces are likely to play a more important role with potential unknown long-term effects. While there is now a better understanding of the risks of near-eye displays (e.g., the Peli's work 14 ), the knowledge is inferred from short-time experiments not reflecting the possible long-term exposure of Pervasive AR technology.
Discussion: Pervasive AR is intended to be "always on." While this is one of its main characteristics and underpins its benefits, it is also its main risk. From a health and safety point of view, akin to the "looking down" problem with mobile phones, the continuous augmentation of information into the field of view of a user undoubtedly will lead to issues with attention, cognitive workload, and distraction, if not addressed properly. For instance, visual occlusions and clutter, interface elements seeking users' attention, and conflicting reality-overlay cues can lead to serious issues. One way to address those issues is to carefully modulate what is seen by the user as, e.g., demonstrated by Sutton et al.'s work. 15

Illusion and Belief
With the rise of computing technology and computergenerated realities came a fear of how these technologies will challenge our beliefs and understanding of physical reality. These considerations have a long history in computer games and the movie industry and it is reasonable to assume that these issues apply to AR and even more so to Pervasive AR.
AR: AR, as a computer-mediated reality technology, aims to provide illusions, i.e., a perception of reality which is actually not present. AR has the potential to turn those illusions into beliefs. 6 In fact, similar to VR, AR is considered as a promising technology for rehabilitation by utilizing illusions to treat for example phobias or stroke. 16 This capability also triggered a discussion on the re-entry into the real world. 9,17 Issues of changing beliefs have been discussed from a marketing perspective 9 (e.g., showing virtual products) or in discussions arising from "superrealism" in XR such as when VR and AR merge together. 6 Pervasive AR: For VR, one needs a great deal of suspension of disbelief 18 since users are very aware of the act of entering a virtual environment, in large part due to the fact that they have to put on a headworn device (the "device gap" 6 ). Pervasive AR, however, is bridging this device gap in a seamless way since the device would be always worn and there is no explicit start or stop of an application. Instead, the system responds to changes in the environment to change the augmentation and its configuration. Hence, Pervasive AR makes the interface virtually disappear 19 with consequences for our judgment of reality. For example, when filters, messages, or other perceptual artifacts become more pervasive and highly relevant to current situations and contexts, they become more potent in shaping people's knowledge, beliefs, and attitudes. While this effect is clearly desired in some scenarios, e.g., in therapy and rehabilitation, education, learning, and entertainment, it can lead to deception, manipulation, and reinforcement of incorrect or even dangerous beliefs. For instance, the effects of psychological ownership of ("holographic") AR objects could already be shown in experiments. 20 Besides the continuous experience utilizing the concept of the disappearing interface, future Pervasive AR devices will likely also offer a higher visual fidelity 3 further diminishing a clear separation between real and virtual. In fact, early perception studies indicate that high visual fidelity is often not even needed to change the human perception of what is real and even simple video filters already contribute to distorted perceptions. b b htt_ ps://bit.ly/3a1suDR Discussion: While it seems that we are still quite far away from having systems, which make it impossible to distinguish what is real from what is not (illusion and belief), in certain situations, occurring during the continuous use, ambiguities will happen even with early Pervasive AR systems. In particular, when snap decisions by users are required in potentially high workload situations. One way to mitigate this is to reintroduce the disbelief similarly to the "device gap." There is already research on different styles how to finish an immersive experience and effects for breaks in illusion 17 and we would advocate for more research into how to leave and re-enter computer-generated realities. Similarly, we should give the users control over the level of deception, e.g., by placing "halos" around virtual objects, or a level 0-10 control of the degree of visibility of virtuality in the experienced scene (cf 6 ). Overall, we strongly argue for a "Breaking the Magician's Code" action; by revealing the following: a) the visible illusion; b) other sensory illusions (e.g., acoustic, haptic); c) the logical relationship of actions and simulations in the environment; d) references to sources of information used to produce the illusion.
A starting point could be the utilization of a "Corsican Twin" technique 21 to reveal and overlay the underlying digital information.

Rights and Access
Recently, technology has not only challenged our definition and rights for privacy but also brought up issues about ownership. In particular ownership of virtual replications of physical objects or properties and rights for universal access have been discussed.
AR: AR systems have already created some controversy with respect to rights and access. For example, Pok emon Go has a well-documented history of challenging the rights of property owners and in several countries, discussions are ongoing on how to control the placement of virtual objects on physical property. For instance, similar to Google Streetview, property owners can soon demand removal of a place of interest within Pokemon go (e.g., Poke stops). While this issue was observable with early mobile AR applications on AR graffiti, it will probably intensify with Pervasive AR.
Pervasive AR: Pervasive AR has the potential to further introduce changes to the notion of private and public property ownership and use. It similarly might redefine who will have access to what quality and quantity of information-and hence widening or narrowing the digital divide across education, gender, ability, etc., Once usage of AR is commonplace it will change the perception of our surrounding environment. Places and spaces currently defined by, e.g., streets, buildings, people, and their appearance will suddenly also be affected by the presence and appearance of their virtual counterparts. Equally important, it will also raise the social issue of who can access and see these virtual objects. In the following, we describe some of the particular issues we see with respect to ownership rights over virtual space, ownership rights over personal data, and universal access to Pervasive AR.
Property owners have legal rights over their property but this does not explicitly extend to the virtual space, which can lead to various inappropriate uses of AR. Commercial or political entities could put virtual billboards on private property. Conflicts could lead to disparaging messages or virtual vandalism. This raises the discussion for potentially adapting laws to give property owners also rights over the virtual space of their property. However, inappropriate augmentation can also occur in public spaces, as it already has. It may be necessary, for example, for city councils to restrict virtual overlays, which are allowed on public places similar to geo-fencing for drones. Regulations around augmentations in public places may be necessary to reduce potential virtual graffiti and ensure the public is protected from those who may use augmentations inappropriately.
Omnipresent cameras in the surrounding environment and in users' Pervasive AR devices will increase the data obtainable about individuals, including facial expressions, location, and activities. Recent usage of photographic material and AI technology to produce deepfakes illustrates some of the issues likely to arise. Unfortunately, negative usage of "digital user twins" and their potential for exploitation and manipulation does not stop here, raising voices that demand that data ownership laws should address this.
Finally, like other digital technologies, Pervasive AR can increase inequities if access to it is not universal. One impediment to universal access is intentional disruption by governments, which has already drawn condemnation from the United Nations Human Rights Council in 2016. c Like the case with the internet more broadly, access to virtual space with Pervasive AR can be manipulated or restricted in ways that limit c htt_ ps://digitallibrary.un.org/record/1639840?ln=en freedom of expression and the ability to learn about diverse perspectives. More (empirical) research is needed to ascertain potential inequalities and access issues before and when Pervasive AR technology becomes more widespread in use.
Discussion: In summary, the creation of digital twins of real places and potentially real people that can be created and assessed with the help of Pervasive AR technology highlights challenges in the traditional understanding of rights and in enabling access. In particular, (1) who has the right to provide augmented information, who owns the virtual real estate, or does one own its digital twin including the right to not have a digital twin? but also (2) which users get the right of access not only to the enabling technology (glasses, operating infrastructure) but also to the information to be displayed continuously? Many of these decisions should not be left to the developers, rather to be developed social norms and legal frameworks should also extend to an augmented space.

CONCLUSION
In this work, we have outlined future Pervasive AR technologies and their technical foundations. We identified four main ethical dimensions for discussing ethical issues for continuous AR interfaces as envisioned in the concept of Pervasive AR: Data and privacy centering around issues in the application of consent and data protection. Health and safety issues arise from the always-on and always in the field of view (and other senses) with the potential to obstruct and distract. Illusion and belief outlining factors that might make it very difficult for users to establish what is plausible when continuously altering perceptions with the potential for biased beliefs. Finally, rights and access issues address the way we deal with (virtual) ownership rights and how we give access to information via an AR interface.
It is our belief that much of the current work on technology-related ethics is addressing issues retrospectively; ethical issues are discussed when they are already present. With our work, we try to look ahead into a future of a continuous augmentation of our environment with computer-generated information.
Our discussion of ethical issues shows that the current focus is insufficient when being applied to a future of Pervasive AR. While there are shared issues with AR, in particular around privacy, the continuous nature combined with a context-aware interface amplifies the ethical issues, calling for a more in-depth and coordinated approach in research and innovation. What is needed is a more comprehensive and holistic approach-the "blinking LED" can only be a starting point, but not the actual solution.
To constructively progress with the design and development of Pervasive AR, further exploration of computer-mediated realities with intelligent, and adaptive AR interfaces (including human-AI integration) is needed. A future research agenda should include ethical considerations in designing artifacts and studies. We need an "ethics-by-design" approach when designing Pervasive AR systems: for the products, the services, the infrastructure, and the usage. For empirical research with Pervasive AR systems, ascertaining end users' and clients' views on ethics is paramount and needs to be reported on; preferably in every single (technological) publication.
Technologists have to be an integral part or even the driver for ethical considerations. In essence, they are the only ones who can appropriately judge the technical factors of Pervasive AR. They should work in partnership with a range of stakeholders and interested parties. We are strongly arguing for a technoempirical approach to Pervasive AR ethics, i.e., tangible, testable artifacts should be built and empirically tested for ethics in the lab and in the field. Not only because of the inherent complexity of Pervasive AR systems but also because of the projected continuous, long-term use of it.
Our ethical discussions are intended to be used as a proposal for wider consideration, in particular among us technologists and between us and social scientists and other stakeholders in the development and integration of future Pervasive AR technology.