Doing Remote Controlled Studies with Humans: Tales from the COVID Trenches

How should empirical researchers conduct controlled, remote “lab” studies in the uncontrolled, noisy conditions of each participant's own home? Volatility in participant home environments, hardware, internet connection, and surrounding distractions takes the “controlled” out of controlled studies. This paper recounts our in-the-trenches mitigations for designing and conducting two complex controlled studies under COVID, in which participants, from home, interactively localized faults in an AI system. The studies with our COVID-era mitigations in 5 categories-Privacy/Security, Data Collection, Control, Technology Issues, Payment-ultimately produced crisp results beyond what we thought possible under such uncontrolled circumstances.