Carry-Over Effects Ruin Your (Cybersickness) Experiments and Balancing Conditions Is Not a Solution

This position paper discusses the complexities and challenges related to order and carryover effects in within-subjects designs. Through a detailed examination of case studies involving the measurement of cybersickness, the paper highlights the importance of careful experiment design, including counterbalancing condition sequences and its limitations when mitigating potential biases. It also emphasizes the need for larger sample sizes and the adoption of strict protocols and common reporting standards to enhance the reliability of research findings. The paper ultimately serves as a guide for researchers in HCI and related fields, offering insights and practical solutions for navigating the intricacies of experimental design and data analysis especially when facing carry-over effects with largely unknown characteristics.