Evaluation And Workflow Design And Quality Assessment

1919 Words Jun 22nd, 2016 8 Pages
Although crowdsourcing has been successfully applied in many fields in the past decades, challenges still exist especially in task/workflow design and quality assessment. We take a deeper look at crowdsourcing classification tasks, and explore how task and workflow design can impact the answer quality. Our research is intended to use large knowledge base and citizen science projects as examples and investigate the workflow design considerations and its impact on worker performance as well as overall quality outcome based on statistical, probabilistic, or machine learning models for quality answer prediction, such that optimal workflow design principles can be recommended and applied in other citizen science projects or other human-computer hybrid systems to improve overall quality.

section{Introduction} label{sec:introduction} subsection{Motivation}
In recent years, we have witnessed a trend of leveraging the wisdom of the crowd (cite{Howe2006,kittur2008crowdsourcing}) to help tackle the problems and tasks that are either extremely time-consuming and expensive for domain experts to complete, or difficult for machine to handle automatically. Crowdsourcing, by outsourcing the work to the crowd, has been widely applied and studied in various areas, covering ontological engineering (cite{Markotschi2010a,eckert2010crowdsourcing,Noy2013a,sarasua2012crowdmap,mortensen2013developing}), Linked Data management (cite{simperl2011crowdsourcing}) and quality assessment…
Open Document