Reflections on ‘What do we think we are doing’: 20-Year most influential paper award talk

My 1996 paper [1] challenged the VL community to ask What do we think we are doing? It might now be called a Systematic Literature Review, although formal procedures for SLR were not developed until later [5]. It made a textual analysis of publications in which authors described a cognitive rationale for VL research, observing that many relied on insights from folk psychology, from introspection, or speculative computer analogies to the brain. This was a study of metacognition - beliefs about one's cognitive ability that shape the mental strategies we choose. In the case of programming language designers, the choices being shaped were not their own problem-solving strategies (something we all do), but the design rationale for new languages (which will affect others). The paper was written in the first year of my PhD, which should be encouraging to students wondering what will happen to their own work in 20 years time. It was inspired by research I had been doing for Hitachi, in a team designing a new VL. Sadly `AppGallery' was not a commercial success. More interestingly, my own scientific assumptions turned out to be wrong - as reported in later VL papers [2], in my PhD dissertation, and in a ToCHI publication where I tried to reconstruct why we had believed it was a good idea in the first place [3]. At VL'96, scientific attention to human factors was a minority interest, and few user studies were reported. I remember that Judy Gurka seemed to ask the right questions about testing effectiveness of algorithm animation [4]. And the two keynote speakers - Thomas Green and Ben Shneiderman - demonstrated growing engagement of VL with HCI. However, within a few years, the whole VL community became convinced that these were the central questions in our work - first with a radical change of name to Human-Centric Computing, and then to the integration of VL and HCC One might assume it is easy to recognize influential papers. But this paper was controversial. It didn't include any technical work, let alone report user studies. It was a textual critique, but written by an engineer who simply needed better information about users, and had set out to ask what guidance was available for the design of VLs. Yet although the motivation came from engineering, the paper didn't make an engineering contribution, so people wondered whether some other publication venue would be more appropriate. When aiming to demonstrate conference quality through low acceptance ratios, it is often easier to reject a discussion paper like this one in favour of research that offers simpler conclusions, hard evidence and scientific consensus. This question of whether papers at a conference like VL should only present scientific or engineering results continues to be active 20 years later. In 1996, VL had been proceeding on the assumption that the primary focus of the conference should be on engineering work. The fact that engineering work relied on hidden psychological assumptions had not been noticed. However, bringing in more insights from human-facing fields such as HCI, and broadening the base of disciplinary enquiry, did help to overturn fallacies that had been acquired by holding to a strictly engineering mindset. We continue to face similar problems today. HCI has changed greatly over the past 20 years. In particular, HCI researchers now understand that hypothesis-testing experiments are not sufficient for interaction design: different kinds of empirical data must be acquired and integrated into explanatory and predictive theories of human behavior. But there are still advocates for approaches to programming language research that reject theories of design, relying on common sense and experimental measurements. It is hard for young researchers to argue for better theory in the face of demands for engineering evidence, so I hope that this award is a reminder that we still need to have informed debate about what we think we are doing.