Microtask Crowdsourcing Has Been Applied In Many Fields

1273 WordsMay 12, 20176 Pages
Microtask crowdsourcing has been applied in many fields in the past decades, however there are still important challenges not fully addressed, especially in task/workflow de- sign and quality assessment. We take a deeper look at crowdsourcing classification tasks and explore how task and workflow design can impact the quality of the classification result. Our research will use large online knowledge base and citizen science projects as examples and investigate the workflow design considerations and its impact on worker performance as well as overall quality outcome based on statistical, probabilistic, or machine learning models for true label prediction, such that design principles can be recommended and applied in other citizen science…show more content…
While the wisdom of crowd (Howe, 2006) has shown great potentials in these areas, it also exhibits many challenges (Kittur et al., 2013; Buettner and Buettner, 2016) no matter it is via volunteered activities (Lease, 2011; Newman et al., 2012) or paid microtask crowdsourcing (Kittur et al., 2013; Demartini, 2015; Bernaschina et al., 2015). In a typical microtask crowdsourcing scenario, from the requester’s point of view, the most important challenges include: 1, task and workflow design to make sure that it is easy to be carried out by non-expert workers, and at the same time risk of spammers or low-quality workers can be mitigated to achieve good quality of result; 2, quality assessment approaches to evaluate the work from crowd workers. Task and workflow design (Little et al., 2010b; Kittur et al., 2011; Demartini et al., 2012) are crucial in ensuring the task to be outsourced to the crowd is properly understood, mitigating the chance of spam and keeping users engaged. They are essential to obtain high quality and quantity of input from crowd workers. Quality assessment techniques can be either on the fly (Ipeirotis et al., 2014) during the task running that can be used to optimize task assignment hence reduce cost, or post aggregation (Whitehill et al., 2009; Ipeirotis et al., 2010; Bachrach et al., 2012; Difallah et al., 2015) which involves various algorithms and strategies that evaluate how
Open Document