Towards Hybrid Crowd-AI Centered Systems: Developing an Integrated Framework from an Empirical Perspective

Crowdsourcing has shown to be a valuable problem-solving approach to handle the increasing complexity and scale of tasks for which the current AI algorithms are still struggling. Crowd intelligence can be particularly useful to train and supervise AI systems in a symbiotic, co-evolutionary relationship that raises long-term research challenges to the hybrid, crowd-computing design space. With the increase in the scale of mixed-initiative approaches, we need to gain a better understanding of the implications of crowd-powered systems as a scaffold for AI through the study of massive crowd-machine interactions. In this paper, we identify some open challenges and design implications for future crowd-AI hybrid systems. A framework is also proposed based on the practical challenges of addressing human-centered AI methods and processes.