Better Anticipating Unintended Consequences

If people want the benefits of innovations, must they simply accept the unintended adverse consequences? Versions of this question haunt many who care about the social implications of technology. Technological design processes could include impact assessment steps, but not all do. Adoption in the marketplace may ignore spillover effects. Jurisprudence is often reactive and focused on remediating obvious wrongs. Public policy also often requires evidence of harm before legislators or administrators are willing to act. The failure to anticipate adverse consequences is sometimes framed as a moral lapse, but it could equally be about competence or incentives. This paper considers the relative merits of methodology (analogizing, interpolating, projecting,) and procedure (reflecting, reasoning, discourse) as systematic approaches to anticipating unintended consequences of innovation. It weighs the efficacy of such approaches against current reactive remedies, highlighting the importance of tailoring approach to context, and building in early learning opportunities (observing and testing). Several examples suggest that society is often playing catch-up and trying to avoid adverse consequences before the innovation is widely deployed rather than before it is initially introduced.


I. INTRODUCTION
A SURPRISE is an unexpected event or thing.The nature of surprises has changed since Shakespeare [1] embodied them in his mischievous character Puck.Adverse historical surprises were mostly natural disasters, plagues, wars, and plots, whereas today many are technological.We invent new things, deploy them at scale, apply them in new ways, and discover surprising outcomes, some adverse.
Unexpected and unintended outcomes are distinct effects.Unexpected benefits, unexpected drawbacks, and perverse results of purposeful social action are all unintended [2].However, not all unintended consequences are unanticipated.For example, when dispensing a prescription for a painreducing drug that also induces drowsiness, a doctor will advise the patient to avoid driving to reduce the odds of an unintended but expected adverse outcome.Approaches for reducing unintended consequences therefore include efforts to reduce surprise or the unexpected, and to reduce the odds of adverse consequences.
This paper argues that the ability to manage unintended adverse consequences of technological innovations depends on discussing them more thoroughly.However, some modes of discussion are more effective than others.Effective discussion has both substantive and procedural dimensions that this paper unpacks.The substantive challenge is to contribute insights based on current knowledge to inform scenarios for a future in which the technology is deployed at scale.The procedural challenge is to get the right people discussing the technology and its potential consequences at times and places where they can exert leverage on the technology's trajectory.This paper draws on a literature in communicative rationality, technology policy, and interpretation of illustrative examples to suggest what the desired "more thorough discussion" might mean in practice.
This paper first discusses its methods, and then the institutions societies use to help respond to adverse consequences, and the risk-benefit tradeoff calculus underlying efforts to prevent such consequences.The institutional description focuses on the U.S. context to illustrate more general considerations, while acknowledging that national frameworks vary.The paper then examines the substantive and procedural dimensions of discussions seeking to anticipate unintended adverse consequences of technological innovations, considering the promise of specific strategies.It highlights the value of quickly learning from experience early in the technology deployment process.The paper applies its communicative analysis framework in an illustrative manner to four widely different historical examples including chlorofluorocarbons, drones, Thalidomide, and Facebook's Newsfeed.It draws tentative conclusions regarding the contributions of specific substantive and procedural strategies and recommends next steps.

II. METHODS
This paper employs qualitative research methods that focus on how people interpret interactions among evolving science, technology, and policy narratives.Following [3], it investigates public policy development as a process of developing a narrative: "policy narratives are stories (scenarios and arguments) which underwrite and stabilize the assumptions for policymaking in situations that persist with many unknowns, a high degree of interdependence, and little, if any, agreement."This methodology involves reading and interpreting texts to extract their meanings (and not hubristic and implausible social 'laws') [4], and which "privilege meanings as ways to grasp actions" [5] while acknowledging the contingency of such context-specific insights.The paper employs a conceptual framework originating in theories of communicative action [6] and critical policy analysis [7].The short case examples interpret the historical literature to extract suggestive rather than conclusive insights.c 2024 The Authors.This work is licensed under a Creative Commons Attribution 4.0 License.
For more information, see https://creativecommons.org/licenses/by/4.0/Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
III. RESPONDING TO ADVERSE CONSEQUENCES Social theorists such as Marx, Weber, Merton and Sartre have speculated that most human interactions yield unintended consequences [8].Many have catalogued unintended adverse outcomes from the accumulated impacts of many small events, the system features that emerge from individual interactions, changes in contextual factors, spillover effects such as network externalities, and dual-use applications in fields of innovation such as energy, environment, medicine, and urban planning (e.g., [9], [10], [11], [12]).Society's institutions regularly respond.
Humans respond to problems both individually in personal decisions, and collectively through participation in systems of economic and political decision making.
As individuals, humans operate in a world that they do not fully understand by relying on habits, heuristics, and biases that work sometimes but not all the time [13].Their mental models are incomplete and inconsistent [14].Neuroscientists studying individual humans identify "puzzlement" surprise, measured by how unexpected an event is, that may trigger changes to habitual (model-free) behavior.This differs from "enlightenment" surprise, measured by "how much an event changes our model of the world and, as a consequence, our expectations," that may trigger changes to reasoned behavior and encourage pursuit of better information [15].People carry their individual capacities and biases to social roles as consumers and citizens.
Markets are relatively efficient, voluntary and adaptive mechanisms for allocating societal resources, and many innovations diffuse through marketplace choices.Innovations that exhibit relative advantage, compatibility with existing assets, low complexity, trialability, and observability are likely to diffuse quickly [16].Each of these characteristics represents a means by which a potential adopter can assess benefits and costs of the innovation.Many bad ideas fail the market test.
Political institutions set the rules for marketplace and societal actors.Most modern governments have a constitutional basis that assigns roles to each branch.For example, in the U.S. context, legislatures promulgate laws, the executive branch implements laws, and the courts adjudicate the constitutionality of laws as implemented.When innovations that succeed in the marketplace turn out to produce negative spillovers or unintended adverse consequences, aggrieved parties can seek governmental redress.
Often, a first point of entry is the court system, where one party can seek relief from harm caused by another party.A successful lawsuit under tort liability (a widely used construct with Roman origins) typically requires evidence of harm, hence it is a reactive mode of redress that attempts to cure a problem after the fact [17].In U.S. parlance, the most relevant torts for the purposes of this paper are negligence (breach of a duty of care) and nuisance (activity that harms others).
The unhappy reality is that legislative and executive branch solutions are also often reactive.Issues usually must climb systemic agendas (by garnering newspaper headlines, airtime, and organized advocates) before ever getting onto institutional agendas (legislative hearings, legislation, executive actions) [18].Modern social media technologies simultaneously raise the possibility of more democratic, bottom-up [19] and more top-down, algorithmic forms of agenda-building [20].
Designers' good judgement, organizational concerns about reputation and liability, market forces and governmental interventions have reduced some of the adverse consequences experienced by society from technological innovation.But new problems continually emerge.Society's institutions therefore scan for and assess emerging risks.
Risk assessment is the applied science of anticipating the likelihood of adverse consequences, and risk management is the practice of reducing them to the extent that society supports [21].The utilitarian, consequence-focused framing of this practice has been criticized for hubris (its claims of objectivity, precision and comprehensiveness) and for embedding unreasonable decision rules (such as net social benefit, with winners and losers) in rational methodology [22], [23].Nonetheless, risk-based policies are widely adopted by governments and enterprises.The challenge has been to build the political will to adopt timely policies.
Precautionary approaches change the basis for policymaking to emphasize risk avoidance over risk management.They have been widely adopted in Europe but not North America.Precaution depends on similar risk assessment analytics for anticipating adverse consequences, but uses the knowledge gained to determine whether to allow an innovation to deploy rather than whether to regulate the extent and circumstances of that deployment [24].Some critics see this as a distinction without a difference because the assessments and management activities are so similar [25].Others are concerned that excessive precaution (often by government) stifles innovation (by market actors with an appetite for risk) and hence call for innovation systems that "counterbalance the generation of novelty and the risk of negative unintended consequences" [26].
The human ability to anticipate adverse consequences remains limited enough that both precaution and risk management are necessary in practice.The Amish are well known for deliberating in a precautionary way before allowing members of their community to adopt new technologies, yet they also manage other risks-such as weather-related crop failure-in a very consequentialist way [27].At a larger scale, foresightoriented activities such as technology roadmaps lay out an organization's or sector's technology adoption strategy, and deliver insights that benefit both risk management and precautionary management approaches.Research and development investments in particular capture the goal of being anticipatory rather than reactive or precautionary.
The standard tools for managing unintended consequences thus are largely reactive.Such tools are practical because they can be evidence-based and are less likely to waste society's resources by responding to insignificant concerns.However, they are morally and intellectually unsatisfying.Is it possible to do better by attempting to anticipate adverse effects?
A search for better ways to anticipate unintended adverse consequences is also a quest for more rational thinking about the future.This paper follows [28] in distinguishing Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
between substantive and procedural dimensions of rationality."Behavior is substantively rational when it is appropriate to the achievement of given goals within the limits imposed by given conditions and constraints" [p.66] and "behavior is procedurally rational when it is the outcome of appropriate deliberation" [p.67].Because deliberation often involves multiple parties, many call for a communicative rationality that seeks for every deliberation a social context in which participants can debate the "truth, rightness, and sincerity" of claims [29].These concepts have entered practice in the form of varied creative efforts to marry substance and process including analytic deliberation [30], participatory technology assessment [31], and joint fact-finding [23].

IV. SUBSTANTIVE APPROACHES FOR ANTICIPATING UNINTENDED ADVERSE CONSEQUENCES
Predicting the future is something people do regularly, but they are better at predicting some events than others.After decades of investment in data collection networks and modeling systems, people now have become pretty good at predicting tomorrow's weather, but not next month's weather.Although people have invested vast effort in data and models to predict tomorrow's stock market prices, they often miss wildly.
Uncertainty bedevils predictions.Relevant types include statistical variation, subjective judgment, linguistic imprecision, variability, inherent randomness, disagreement, and approximation [30].Some uncertainties are irreducible, such as the inherent unknowability of the future or the actions other entities might take in response to current conditions, but people can address some types of uncertainty with improved data, tools, relationships, and wisdom.
In situations where uncertainties are rife and predictions are often inaccurate, the focus shifts toward management of the associated impacts.Anticipating surprise is a fundamental survival skill requiring adaptability and foresight.Adaptability, the quality of being able to adjust to new conditions, improves with access to diverse resources, internal capacity, and a flexible mindset [33], [34].Foresight, the ability to consider the future in a systematic way, improves when actors identify salient issues or concerns, track their development over time, and anticipate changes in their trajectories [35].Futurists warn that "predictable surprises" happen when we fail to recognize salient factors, track trends, and commit resources in time [36].Accurate assessments of what matters (saliency) are difficult because people may fail to focus on important goals due to the many distractions they experience [37].Additionally, they may encounter unpredictable surprises.
Prediction seeks to identify a specific, most likely future, and forecasting attaches a date when that future will occur.Foresight instead seeks to help people understand the ramifications of several plausible futures so that they can manage the consequences [38].
Anticipating consequences requires a logical basis for projecting what might happen, given what has already happened and what people know about how the world works.It requires argumentation about cause and effect, and it requires a system conception capable of delivering a "then" when given an "if."A simple physical system may have more data available and more tractable modeling tools to characterize it than would a complex sociotechnical system.Thus, system representations vary in formality and usefulness for both prediction and foresight.
Disciplines integrate causality, probability, and mechanism in different ways that make multidisciplinary communication both valuable and challenging [39].Futurists distinguish among anticipated-intended consequences, anticipatedunintended consequences, and unanticipated-unintended consequences, and their methods seek to turn unanticipatedunintended into anticipated-unintended consequences [12].Such methods identify relevant events, trends, and disruptions using means that vary from informal to formal, depending on data availability, analytical resources, time horizon, technological status, and other factors.
It is useful to array methods for characterizing systems on a spectrum, from analogizing, to interpolation, to projection.Each method has strengths and weaknesses, and each exhibits better fitness for some applications than for others, as discussed below and summarized in Table I.These methods illustrate but do not exhaustively cover the spectrum of possibilities.

A. Analogize
Are there relevant historical analogies from which we can learn?What are the limits of analogizing?Examples include: (1) "the old network was like a narrow country road, but the new one will be like a superhighway" [40]; and (2) the "paperworld" permitted by the commodification of books due to invention of the printing press presages problems in today's cyberspace [44].
Writers distinguish rhetorical gradations from simile (this is like that), to analogue (this is a stand-in for that), to metaphor (this is that).Metaphor is usually poetic, but analogy is often literal and quite useful for thinking about 'what-if' scenarios.Speculative fiction relies heavily on analogy [41].
Analogical reasoning is a cognitive heuristic-a shortcutthat is widely used to make sense of the world [42].Perceived similarities between cases become the basis for claiming that further similarities may exist.For example, scientists regularly perform laboratory experiments on mice to predict likely human responses to biomedical interventions.Good biomedical researchers also understand the limits to analogy and thus conduct follow-up studies with human subjects.
Logicians note that the strength of analogical reasoning depends more on the content of the argument than on its form, that is, the extent to which similarities between cases are present and relevant, and not offset by inevitable dissimilarities [43].If the measure of an analogy is its empirical strength, then one necessarily needs to evaluate it in a specific context.Historians and other humanists specialize in contextualized interpretation when crafting and contesting analogies [44].
There is an ethical dimension, because while "analogical imagination" has great value in exploring possible futures for emerging technologies, users may wield specific analogies Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

TABLE I ANTICIPATORY TOOLS
rhetorically to advance strategic arguments, leading to the need to develop a critical "analogical sensibility" [45].The practical implications are to appreciate similarities, dissimilarities, and limits to generalizability when deploying any analogy, and to encourage stakeholders to offer multiple analogies as they deliberate.

B. Interpolate
How does a fine-grained appreciation of the dynamics of social practices guide us to unbundle social meanings and human competencies from old technologies and successfully re-bundle them with new technologies?An example is: "Patient mortality increased significantly. . .following introduction of a new computerized physician order entry system aimed at reducing medical error. . .[because the system] did not perform as anticipated in real-world situations" [46, pp. 1 and 2].
Social and behavioral considerations at the level of individual technology adoption decisions can scale up to be just as important as those affecting systemwide technology diffusion.Anthropologists and sociologists have documented strong determining roles for everyday social practices that cause naturalistic adoption behaviors to differ from those observed in controlled experimental settings [47].One useful framework defines a social practice as bundle incorporating a material element (a technology), a procedural element (competencies and social rules needed to use the technology), and a meaning element (symbols, ideas, and aspirations people attach to the technology) [48].Changing a social practice involves unbundling all three elements and then re-bundling a new set of material, procedural, and meaning elements into a new social practice.Thus, an apparently simple technology swap, say, adopting an electric car in place of one with an internal combustion engine, is also likely to require the user to develop new procedures (such as learning to use regenerative braking and managing limited driving range) and meanings (such as re-imaging the prestige of a luxury brand or the ruggedness of a pickup truck with an 8-cylinder engine).
A common refrain from social practice researchers is to expect new technology adoptions to progress more slowly than a simple technology substitution analysis might predict, because people only slowly develop new procedures and assign new meanings.Technology developers therefore ought to ask how close a substitute to what it replaces is the new technology, how user friendly is it, and how do its social meanings diverge from the status quo?If technology adoption is in fact a highly social process, then it makes sense to use social knowledge and tools, from consumer education to peer pressure, to encourage adoption.This is also an opportunity to identify unintended consequences.

C. Project or Extrapolate
How do our understandings of physical, chemical, biological, economic, social, and political system dynamics help us predict the systemic effects of widespread deployment of an innovation?An example is: "We estimate robust negative effects of robots on employment and wages" [49, p. 1].
Researchers have long been interested in the dynamics of natural-science, economic, social, and political systems, and in recent decades the toolkit has improved dramatically.The systems approach was embraced by military technology planners in the 1950s and 1960s, reviled due to misconceived efforts to apply these tools to civil problems in the 1970s and 1980s, and rehabilitated as computing and visualization technologies opened new possibilities in the 1990s and 2000s [50].In economics, anecdotes and extrapolations from historical narratives to support conceptual models (such as Marx's political economy) have been joined by mathematical modeling of simplified systems (such as Walras' general equilibrium model of the macroeconomy), quantitative documentation of observed economic system dynamics (such as Schumpeter's business cycles and Kondratiev's long waves) and structures (such as Leontief's sectoral demand model), and formal modeling of dis-equilibrium conditions (such as Schelling's agent-based models).High interest from governments and businesses, and an abundance of detailed time series data have fueled these advances in economic modeling and projections.
Social and political system dynamics are less well understood and there is less of a tradition of predictive modeling.This is due to a lack of consensus on appropriate conceptual models of these systems, a lack of adequate time series data, the contingent nature of social knowledge, and the high importance of contextual factors [7], [51].
Integrated modeling to predict the impacts of technological innovations on society nonetheless has deep roots.The intellectual framework of public choice theory advocated by Black, Arrow, Downs, Olsen, Buchanan, Tulloch, and others portrays many social, political, and economic phenomena as the outcome of interactions among individual, essentially microeconomic actors [52].Contrasting theories that attribute causality to structural factors in society, whether Marx's class system or Carmichael's institutional racism, can be folded into an integrated assessment using the logic of structuration [53].Structuration theory asserts that agency and structure are interdependent because structure constrains agency, even as agents must actively decide to reproduce structure.A multilevel perspective has gained currency in recent years to acknowledge multiple levels of resolution in both agency and structure [54].Agent-based modeling has been useful for formalizing these concepts and exploring observed implications such as tipping points [55], [57].
Integrated assessment is widely used for public policy analysis, and it is most advanced in applications to climate change policy [57].These models marry bottom-up technology characterizations, such as types of electric power generation plants, with technology substitution models that predict their market share over time, with top-down macroeconomic models that account for changing prices and quantities, as mediated by policies and fundamental resource constraints, and the resulting system simulations report salient metrics such as net carbon emissions and changes in Gross Domestic Product.The virtue of an integrated assessment is its internal consistency and comprehensive scope.An unwanted byproduct is the extreme level of uncertainty associated with any prediction because so many factors are involved.Thus, integrated assessments are most useful for "what-if" scenario analysis to explore a decision space.
Many technological innovators lack the resources to conduct formal, integrated modeling of social, economic, and political system dynamics as part of the design process.Instead, they use approximations and rules of thumb.For example, if introducing a more efficient technology reduces the price to consume a unit of output, then consumers might use the money saved to consume even more of the product, causing a rebound effect that takes back some of the net resource savings [58].If the efficiency improvement is dramatic enough, when it scales up it can change the entire economic structure, as was the case with 18 th -century steam engines and 20 th -century integrated circuits.Relevant metrics in these cases are the price and the price sensitivity of demand for the product.Each rule of thumb analyzes only a small part of the integrated system to assess impact.
Whether the sociotechnical system model is integrated or piecemeal, open or closed, explicit or reduced form, it can only rarely claim to be comprehensive across all relevant phenomena.Exogenous factors affect nearly all system models.Thus, there are inevitable limits to the value of these models in projecting future outcomes.Communicating these limits to users and asking whether the model is fit for its purpose can improve the odds that it makes a substantive contribution to the investigation of unintended consequences.

V. ADDING A PROCEDURAL DIMENSION Systematic thinking is only part of what society needs to anticipate adverse consequences of technological innovations.
There is also a procedural dimension that assigns responsibilities to certain people and organizations.One can distinguish between "technologies of hubris" (predictive methods) and complementary "technologies of humility" that identify unforeseen consequences, admit normative biases, and promote pluralistic discussion [59].Such a communicative rationality requires a marriage of substance and process.Lively literatures on anticipatory governance (e.g., [60]) and responsible innovation (e.g., [61]) attempt to establish procedural norms within innovation ecosystems [62].Procedures comprise a spectrum spanning individual reflection, moral reasoning, and discussion.Each has strengths, weaknesses, and appropriate applications, as discussed below, and summarized in Table I.Again, the list is not exhaustive.

A. Reflect
What can we learn from engaging in reflective professional practice?How can we remember to look back and assess our own work periodically?An example is: "The [nuclear] accident at Three Mile Island was caused by a combination of hard-to handle machinery, poorly trained or incompetent operators, and a regulatory process that lulled management into neglecting its own responsibilities.Congress, the public, the regulators, and the nuclear power industry reacted.Some overreacted. . ." [63].
The everyday professional activities of engineers and designers are social practices with high relevance to new technology development.By reflecting on designers' practical actions, learning may occur, and outcomes may improve [64].Observers note that experience alone does not guarantee better performance, because it also requires deliberate reflection on the experience [65].The act of reflecting can be unsettling because it raises the possibility that one has some personal responsibility for what one's designs do to the world, that the social dimensions of professional work-from employee diversity to managerial style-might affect outcomes, and that living with uncertainty and doubt are intrinsic features of professional practice [66].
Design work involves making tradeoffs, and each tradeoff decision brings the potential for unintended consequences.Reflective practice provides opportunities to revisit those tradeoff decisions and play out what-if scenarios.Whether institutionalized as an "after-action debriefing" or undertaken Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
as a personal journey, reflection enhances understanding.It pursues answers to three inter-related questions: "What?" "So what?" and "Now what?" that detect unintended consequences [67].

B. Reason
Are there usable frameworks for moral reasoning about professional practice?An example is: Principles that should govern age-appropriate digital services include "presenting information in an age-appropriate way, upholding children's rights, offering fair terms for children, recognizing childhood, and putting children ahead of commercial interests and ahead of platform status" [68].
Moral reasoning by individual professionals can help them anticipate unintended consequences.The personal guidance of an ethical framework, whether motivated by concern over potential consequences or a sense of mutual obligation, can encourage timely questioning of motives and consideration of outcomes.This personal ethical stance might be guided unconsciously by deeply held assumptions, informally by social norms, or formally by regulative constraints.
Culture, defined as the shared attitudes, beliefs, practices, and values that characterize a group [69], instills within each person a set of unwritten framing assumptions that influence their behavior.Each person also has cognitive biases that limit their ability to think logically, comprehensively, and quickly.
Social norms are patterns of behavior that are selfreinforcing within a group and may infer an evolutionary advantage on members of the group [70].Informal but explicit social norms may powerfully constrain individual behavior.Many professions and organizations write ethical codes to guide member behavior.Some elements merely protect the group from ethical lapses by participating individuals, but most also advance some vision of "the public good."When the consequences of behavioral lapses have the potential to become significant, informal normative constraints give way to formal legal and policy constraints [71].
Moral reasoning involves asking good questions.Thus, one professional code of ethics asks members how they "hold paramount the safety, health, and welfare of the public" and "improve the understanding by individuals and society of the capabilities and societal implications of conventional and emerging technologies" [72].Such questions can help a designer to weigh tradeoffs and reduce adverse consequences.This specificity might be as or even more useful in a practical context than the overarching injunction to avoid doing harm [73].

C. Discourse
Can talking about planned new technologies help designers avoid adverse outcomes?An example is: "Deliberation enabled the participants to develop a shared understanding of New Jersey's energy future" [and] "forged a consensus on targets for three key energy indicators" (greenhouse gas emissions, percent of income spent on energy, local selfsufficiency) while recognizing "that the future will be created by active and concerned citizens, consumers, businesses, workers, and investors" [74, pp. 18 and 19].
Opening an inquiry up to broader participation can introduce more varied points of view and bring more types of knowledge to bear on a problem, provided participants communicate successfully [75].Communication can fail when one or more participants decline to participate voluntarily in the discussion, or when distortions hinder effective communication [76].In complex technical domains, just talking might not deliver enough rigor, hence it may become necessary to marry the instrumental (technical analysis) and communicative (open discussion) dimensions of rationality by using careful processes of co-design [77], joint fact-finding [23], or analytic deliberation [74].
Open discussion of the social implications of new technologies is such a well-established value in social studies of science and technology that people sometimes forget how difficult it is to do well [78].Each emerging technology presents specific challenges to effective discourse.For example, the opacity of current AI has limited "deep" discussion of its merits and demerits [79].Yet the desire to communicate, the norms of ethical inquiry, and the value of discourse persist even as the deliberations take new forms and move online [80].

VI. LEARNING BY DOING
There are limits to the human ability to anticipate unintended outcomes in the real world.An important role for learning quickly remains.The slow, reactive form of social learning embedded in governance processes such as lawsuits, legislation, and regulation may not be timely.Observation and testing are tools that speed up the learning process.

A. Observe
If adverse consequences are suspected, can they be detected promptly even as the innovation deploys at scale?An example is: "Although the need for early drug safety monitoring is widely recognized, a harmonized approach at a global level is still lacking" [despite] "the globalization of the clinical landscape over recent decades" [81].
Science advances through an iterative process of observing phenomena, making empirical generalizations, formulating theories, proposing testable hypotheses, and carrying out those tests through observation [82].Repeated, time-series observations are valuable for detecting change in dynamic systems and allowing the "wheel of science" to spin rapidly and advance knowledge.Thus, if adverse consequences of deploying an innovation are suspected, then it makes sense to set up a surveillance process capable of detecting such consequences.
Surveillance processes are longstanding in the fields of public health and environmental management.One of the most challenging aspects is to figure out what to observe.In public health, it is customary to do surveillance planning to identify hazards warranting attention [83] and in environmental management it is common to screen candidate contaminants of emerging concern using existing knowledge of hazard characteristics and exposure potential [84].Both fields receive regular governmental support for surveillance activities although they are chronically under-funded relative to their scopes of concern [8].
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

B. Test
Is it possible to perform low-risk tests to identify potential adverse consequences of innovations before they are deployed at scale?An example is: "Online education is rapidly expanding in response to rising demand for higher and continuing education, but many online students struggle to achieve their educational goals. . .Scaling behavioral science interventions across various online learning contexts can reduce their average effectiveness by an order-of-magnitude.However, iterative scientific investigations can uncover what works where for whom" [86].
Designers typically test many alternatives before settling on a preferred solution, so experimentation is already a core element of innovation.Unlike the role of experimentation in science, the focus of a design experiment is "not to definitely accept or reject a hypothesis, but rather to iteratively refine the intervention" [87, pp. 163 and 164].Many of these tests will be low risk, performed using sketch models or small prototypes.However, these tests are often evaluated on metrics of interest to funders who are uninterested in unintended adverse consequences to society.Some sectors systematically build lower-risk trials into the innovation process [88].The pharmaceutical industry, which is heavily regulated by government, follows a strict sequence in testing novel drug formulations, first in laboratory conditions on animals, then on a small number of human volunteers, then on a much larger number and variety of human volunteers, and then it performs follow-up studies on results within the broad human population.The electric power industry, which is heavily regulated by government and in some jurisdictions publicly owned, tests innovations on a small scale such as at an individual power plant or substation, before allowing systemwide deployments.For example, "in Denmark's early phase of developing offshore wind power, utilities were encouraged to experiment with offshore wind, and were allowed to pass on costs to consumers" [89, p. 28].
Other sectors, especially those driven by network effects and that lack tight governmental oversight, such as social media platforms, perform localized trials only to confirm technical performance.New products and services then enter the marketplace at scale.Adverse society-wide impacts sometimes result [90].
The marketplace itself can identify innovations that have unintended adverse consequences, especially if there is a clear way to assign liability for damage caused.The nuclear power industry experienced such disciplining following accidents at Chernobyl and Fukushima, and construction cost overruns at many sites.However, in cases where internalizing the externality proves difficult, governments and social institutions may need to step in to correct failures of market decision making.There is even room for small-scale experimentation in the design of governmental innovation policies [91].

VII. SUGGESTIVE EXAMPLES
This section briefly considers four well-known technological disasters to ask whether the suggested approaches for anticipating adverse unintended consequences would have been feasible at the time of these innovations' deployments.Is there room for "cautious optimism" [92]?Each case has received prior scholarly attention, so this paper synthesizes previous works rather than interpreting primary sources.As such, reasonable people may differ in their interpretations of the evidence.

A. CFC Refrigerants
The invention of inert, non-flammable chlorofluorocarbons in 1930 by GM's Frigidaire division allowed refrigeration to safely scale up to create cold chains for food and medicine, comfortable buildings located in any climate, and many other beneficial products [93].Unfortunately, dissipative uses of these same miracle chemicals contributed to global warming and created the Antarctic ozone hole that exposed plants and animals living in the Southern Hemisphere to increased levels of harmful ultraviolet (UV-B) radiation [94].A global treaty enacted in 1987 phased out the most damaging formulations [95], but a somewhat-diminished ozone hole persists [96].
The following timeline draws on [97].The industrial R&D to develop CFC refrigerants began in 1928 and resulted in patents by 1930, with industrial-scale production accelerating from the late 1930s and CFCs being measured everywhere in the ambient environment by 1971.The chemical's ubiquity prompted reflection on the part of the chemical industry which formed a nineteen-company study panel on the ecology of fluorocarbons in 1972.Scientists confirmed existence of a stratospheric ozone layer in 1931 and its role in protecting life on Earth from harmful ultraviolet radiation in 1934, but the link between CFCs and destruction of the stratospheric ozone layer was not identified until 1974.This unintended consequence was confirmed in 1975 using observational data collected from 1957 onward as part of an International Geophysical Year effort.Projections using known chemistry provided the basis in 1974 for Molina and Rowland's hypothesis, and observations allowed a test of that hypothesis, but only 40 years after widespread deployment of CFCs.Projections, observations, and much discussion facilitated the international agreement in 1987 that phased out these problematic chemicals.

B. Drones
Unmanned aerial vehicles (UAVs) for military purposes date back to French hot-air balloons in 1783, but modern drone technology dates to 1935 when De Havilland deployed a radio-controlled aircraft allowing more realistic target practice training for British military pilots.Development of modern UAVs accelerated with the invention of radio, microprocessors, and geo-spatial positioning systems [98].UAVs played important roles in Middle Eastern and other conflicts from 1982 onward [99].In 2006, non-military UAVs were first allowed in U.S. airspace to help respond to the devastation of Hurricane Katrina, and since then a vibrant civilian drone industry has developed.Mass production, standardization, lower costs, and clear regulatory frameworks have encouraged many new applications of drones such as wedding photography, real Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
estate sales, wildlife tracking, and farming [100].Access to inexpensive versions of this dual-use technology in turn have dramatically empowered those engaged in asymmetrical warfare, as shown in both terrorist attacks and the first year of the underdog Ukrainian response to its invasion by Russia in 2022 [101].
Early thinking about drones relied on analogies to manned aircraft and applications based on remote sensing like what hot air balloons (and later, satellites) could deliver.Even after commercial drones emerged in 2006, many military strategists persisted in viewing military and civilian drones as different and evaluating capabilities according to traditional aircraft-oriented metrics such as payload and speed [102].However, some engaged in reflection and reasoning to ask whether inexpensive drones would lower the threshold for war [103].The recent successful use of commercial drones for military purposes by Ukraine eventually prompted Russian military planners to interpolate and adapt [101].Projections of emerging AI capabilities and potential swarm deployments are leading strategists to anticipate new and highly destructive functionalities for drone fleets fielded by the great powers [104].

C. Thalidomide
Chemical Industry Basel (CIBA) in 1952 synthesized the drug Thalidomide, and in 1957 it was acquired by Chemie Grunenthal and marketed as a sedative having fewer sideeffects than others available at the time.It was widely used in Japan, Australia, and Europe, but not in the United States because it did not receive FDA approval [105].Unfortunately, when taken by pregnant women to combat morning sickness, it caused high rates of congenital malformations in their offspring.Thalidomide affected thousands of children.By 1961, it was removed from the market in most countries.Since then, drug companies have increased the rigor with which they test drugs before bringing them to market, including tests for developmental toxicity [106].Thalidomide has subsequently found use for treating leprosy and a type of bone marrow cancer [107].
The following interprets the rich reporting of [105].Projection failed because this drug was invented by combining "two rather innocuous compounds. . .both very safe compounds.When you put them together, we find a very powerful birth-defect causing agent" [105, p. 8].Analogy also failed, because trials in animal models showed none of the adverse effects that emerged in humans.Procedural tools likely would have helped if they had been implemented.Instead of reflection, the inventors hurried the drug to market and ignored early adverse reports from local physicians.Moral reasoning was tainted by the immediately prior corruption of medical ethics by the Nazi regime.Individual victims and their physicians were not aware that others were also seeing similar symptoms because there were few forums for sharing such information.Instead of conducting experiments with a small number of human subjects to limit risk, the inventors sold their product over the counter to the masses because regulators in Germany did not prevent it [106].The inventors did not log observations of adverse reports; this was eventually organized by others after the fact.
In the U.S. context, a more consultative procedure was in place in which "safety evaluation involved review of submitted data by a chemist, a pharmacologist, and a medical officer," none of whom were satisfied with the data received, which delayed approval until the application was withdrawn [108, p. 221].The medical officer, who had acted while still in her first year on the job, was subsequently "honored with the President's Award for Distinguished Federal Civilian Service from President John F. Kennedy in 1962" [109, p. 292].

D. Facebook's Newsfeed
Facebook incorporated in 2004 pursuing a mission to "give people the power to share and make the world more open and connected" within ever-expanding online social networks [110].Advertising provided a path to profitability.Its Newsfeed feature was launched in 2006 to deliver algorithmically selected content to users.It was spectacularly successful in driving growth in the user base and revenues.With growth came a variety of unintended consequences, including a proliferation of pornography, hoaxes, scams, and hate speech in the Newsfeed.Algorithmic updates addressed these concerns on a piecemeal basis, seeking to balance the potential social harms against the need for revenue.Facebook's content-serving algorithms and those of other social media platforms are now understood to have spurred political hyper-partisanship, misinformation, and violence at an unprecedented scale in many countries [111].In response, the company changed its name to Meta and its mission statement to "give people the power to build community and bring the world closer together" [112].Its founder articulated a new imperative to develop "the social infrastructure for community -for supporting us, for keeping us safe, for informing us, for civic engagement, and for inclusion of all" [113].It remains to be seen whether this positive outcome emerges.
Algorithmically selected content provided by Facebook and other social media companies has become the major source of news for half of U.S. adults [114].Since 2005, Americans of all ages have greatly increased their use of social media (from 12% to 84% for ages 18-29, from 2% to 45% for ages 65+) [115], [116]."Despite a string of controversies and the public's relatively negative sentiments about aspects of social media, roughly seven-in-ten Americans say they ever use any kind of social media site -a share that has remained relatively stable over the past five years" [117].The tension in this case is that social media algorithms are achieving an intended outcome of holding users' attention without effectively reducing their unintended consequences because both rely on network effects.
Early analogizing noted that social media, like the printing press, could be expected to induce both positive and negative changes [118].In 2004, industry actors equated connectedness with social good.Clear evidence that it could also spread social ills came later [119], [120].Projections rooted in behavioral science were at first inconsistent but have more clearly identified modest negative effects in recent years [121].Social media Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
companies collect observational data and conduct experiments on an ongoing basis but rarely share the results.Discussion takes place but it is mostly adversarial rather than solutions oriented.Reflection occurs but often those who engage in it must leave their firms.Moral reasoning is becoming enshrined in standards (e.g., [122]), although adoption of such standards by the industry is just beginning.
Analogy had value for anticipating consequences of drones and Newsfeed but misled about Thalidomide.Interpolation showed value in the drones case.Projection had value in all cases but was rarely timely.Reflection had value in the CFCs, drones, and Newsfeed cases, and unfortunately seemed not to have been attempted by the manufacturer in the Thalidomide case.Reasoning played a role in the drones and Newsfeed cases.Discussion emerged as a key element in the Thalidomide case.Observation helped in the CFCs and Newsfeeds cases.Testing did not seem to help anticipate unintended consequences in any of cases, although it helped refine products.
The apparent lack of testing for adverse societal consequences in the four brief technological case studies is a noteworthy finding that warrants further investigation.Is it infeasible or just a low priority?The examples suggest that procedural tools may be somewhat more effective than analytical methods in identifying unintended adverse consequences of innovation.This suggestive finding reflects the difficulty of imagining-whether through analogy, projection, or interpolation-surprising outcomes that lie beyond the reach of current scientific understanding.Open discussion seems to help.Very often, it is a game of catching up and remedying adverse outcomes before the innovation becomes widely deployed.

VIII. CONCLUSION
Technological innovations sometimes surprise people with unintended adverse consequences when deployed at scale, and current economic and political institutions mostly respond reactively.There are many tools available to aid people in better anticipating these consequences, and they should learn their appropriate applications and use them more widely.However, people are unlikely to avoid adverse consequences all the time, so people will also continue to need to advance their capabilities for timely and effective reaction.
Better anticipation of adverse consequences has utilitarian benefits, but it is also an obligation that innovators owe the rest of society.Individual innovators, organizations that employ them, and nations that host them each can and should perform anticipatory activities.Market forces and political mobilization sometimes help to minimize undesirable outcomes, but they are intrinsically reactive, and interest focused.Anticipation is a difficult task that requires much knowledge and a rich model of cause and effect.But even a simple requirement that invention disclosures include a paragraph of reflection on possible unintended consequences could advance the discussion.Collaborative deliberations can strengthen both the substance of an effort and its impact.Approaches for anticipating adverse consequences are teachable and deserve wider use.