Design Patterns for Hybrid Algorithmic-Crowdsourcing Workflows

Crowd sourcing has shown to be a powerful technique for over-coming many challenges in data and information processing where current state-of-the-art algorithms are still struggling. This is especially true for workflows that transparently combine algorithmic heuristics and dynamically crowd sourced tasks that are performed by human workers, and which promise to solve even more complex tasks effectively and efficiently. But still, such hybrid crowd sourcing workflows can be difficult to approach, and they are often designed in an ad-hoc fashion. Therefore, in this paper, we extensively investigate such crowd sourcing workflows as described in the literature, and abstract generic design patterns, which codify commonly recurring challenges and their best-practice solutions. Each design pattern is described and discussed with a special focus on its requirements, constraints, and effects on the overall workflow. We illustrate the practicality of these patterns by providing real-world application examples where such patterns can or have been applied. Furthermore, we show-case how the individual design patterns can be extended and combined to support more complex workflows. Our design patterns provide an extensive overview of the hybrid crowd sourcing workflows' design space, and allow for a more efficient modeling, analysis, and documentation of such workflows.