Concentus: Applying Stream Processing to Online Collective Interaction

The collective experience is the experience of unity, belonging, and purpose that occurs when large numbers of people come together and perceive themselves and others as part of a single social entity, and interact with each another accordingly. We are exploring how the collective experience can be supported in a fully computer-mediated environment through activities where a virtual crowd performs synchronous collective action over a shared focal state (e.g. collectively controlling a character in a game, pulsing text-based messages in time to form collective chants). Supporting collective interaction requires a system architecture that is able to process large numbers of input actions into an aggregated collective representation at low latency. We have created a scalable distributed system called Concentus that applies approaches found in distributed stream processing to online collective interaction. Concentus allows for different implementations of aggregation engine, the primary component of the system, to be measured in-situ with other core components (e.g. client connection handlers). We have evaluated the performance of two aggregation approaches: one based on Spark Streaming, a general purpose distributed stream processing engine, and another that performs aggregation on a single thread on one machine, and have measured their performance against the key metric of interaction latency (time from input submission to perceiving the effect on the shared state) as the crowd size scales. The evaluation revealed that both approaches are capable of supporting 50,000 participants with latencies under 1 second, with the single threaded approach performing better on smaller data sizes, and Spark Streaming on larger data sets. We discuss the implications on collective application design.