Mixed method approach to identify analytic questions to be visualized for military cyber incident handlers

Our multi-disciplinary team developed and applied a 6-step mixed method approach to efficiently identify the cognitive work of early stage military cyber incident handlers, extract a subset of that work that could benefit from visualization, and specify the information needs as Analytic Questions (AQs) posed by operators that the visualizations would have to support. The methodology included a survey of subject matter experts to validate that the major findings of prior research on the cognitive work of cyber defenders, conducted over a decade ago, are still valid today. It also utilized a Goal Directed Task Analysis (GDTA) structure to represent the major task, goals, decisions, AQs and data source requirements of early stage cyber incident handlers. This yielded 40 AQs which are reported in this paper. Knowledge Elicitation (KE) interviews of domain practitioners were used to select the AQs with greatest potential for incorporation into a follow-on project to measure the effects of visualization on early stage incident handler performance. The AQs represent measurable units of cognitive work which must be performed using available data in a severely time-constrained work environment. Thus, they can serve as indicators of operator performance to be used in experiments on the effectiveness of visualization for event detection and preliminary analysis. They can also provide requirements for visualization designers and security products.