Please use this identifier to cite or link to this item:
Title: An online cost sensitive decision-making method in crowdsourcing systems
Authors: Gao, J.
Liu, X.
Ooi, B.C. 
Wang, H.
Chen, G.
Keywords: Crowdsourcing
Issue Date: 2013
Citation: Gao, J.,Liu, X.,Ooi, B.C.,Wang, H.,Chen, G. (2013). An online cost sensitive decision-making method in crowdsourcing systems. Proceedings of the ACM SIGMOD International Conference on Management of Data : 217-228. ScholarBank@NUS Repository.
Abstract: Crowdsourcing has created a variety of opportunities for many challenging problems by leveraging human intelligence. For example, applications such as image tagging, natural language processing, and semantic-based information retrieval can exploit crowd-based human computation to supplement existing computational algorithms. Naturally, human workers in crowdsourcing solve problems based on their knowledge, experience, and perception. It is therefore not clear which problems can be better solved by crowdsourcing than solving solely using traditional machine-based methods. Therefore, a cost sensitive quantitative analysis method is needed. In this paper, we design and implement a cost sensitive method for crowdsourcing. We online estimate the profit of the crowd-sourcing job so that those questions with no future profit from crowdsourcing can be terminated. Two models are proposed to estimate the profit of crowdsourcing job, namely the linear value model and the generalized non-linear model. Using these models, the expected profit of obtaining new answers for a specific question is computed based on the answers already received. A question is terminated in real time if the marginal expected profit of obtaining more answers is not positive. We extends the method to publish a batch of questions in a HIT. We evaluate the effectiveness of our proposed method using two real world jobs on AMT. The experimental results show that our proposed method outperforms all the state-of-art methods. Copyright © 2013 ACM.
Source Title: Proceedings of the ACM SIGMOD International Conference on Management of Data
ISBN: 9781450320375
ISSN: 07308078
DOI: 10.1145/2463676.2465307
Appears in Collections:Staff Publications

Show full item record
Files in This Item:
There are no files associated with this item.

Google ScholarTM



Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.