Wsdm

Crowdsourcing 101: Putting the WSDM of of Crowds to Work for You
Omar Alonso
Microsoft

Matthew Lease
University of Texas at Austin

9 February 2011

1

Tutorial objectives
• • • • What is crowdsourcing? How and when to use crowdsourcing? How to use Mechanical Turk Experimental setup and design guidelines for working with the crowd • Quality control: issues, measuring, and improving• Research landscape and open challenges
Crowdsourcing 101: Putting the WSDM of Crowds to Work for You.

2

Tutorial Outline
I. Introduction to crowdsourcing II. Amazon Mechanical Turk (and CrowdFlower) III. Design of experiments

Crowdsourcing 101: Putting the WSDM of Crowds to Work for You.

3

INTRODUCTION TO CROWDSOURCING

Crowdsourcing 101: Putting the WSDM of Crowds to Work forYou.

4

Crowdsourcing
• Take a job traditionally performed by a known agent (often an employee) • Outsource it to an undefined, generally large group of people via an open call • New application of principles from open source movement
Crowdsourcing 101: Putting the WSDM of Crowds to Work for You.

5

Examples

Crowdsourcing 101: Putting the WSDM of Crowds to Work for You.

6Less Serious Examples

Crowdsourcing 101: Putting the WSDM of Crowds to Work for You.

7

Wisdom of Crowds (WoC)
Requires • Diversity • Independence • Decentralization • Aggregation Input: large, diverse sample (to increase likelihood of overall pool quality) Output: consensus or selection (aggregation)
Crowdsourcing 101: Putting the WSDM of Crowds to Work for You.

8

WoC vs.Ensemble Learning
• Combine multiple models to improve performance over any constituent model
– Can use many weak learners to make a strong one – Compensate for poor models with extra computation

• Tend to work better when significant diversity
– Using less diverse strong learners better than dumbingdown models to promote diversity (Gashler et al.’08)

• cf. NIPS’10 Workshop
– ComputationalSocial Science & the Wisdom of Crowds
Crowdsourcing 101: Putting the WSDM of Crowds to Work for You.

9

Human Computation
• Use humans as processors in a distributed system
– Humans do tasks computers cannot (do well) – System makes opaque “external call” to the “HPU” (“AAI”)
• HPU has describable functional capabilities (J. Davis et al., ACVHL’10)

• Ex. 1: Detect CPU (Captcha – “reverseTuring test”) • Ex. 2: HPU computation (e.g. labeled data)
– linear, with minimal post-processing of HPU output – E.g. ReCaptcha, ESP game, most Mechanical Turk work

• Ex. 3: Integrate CPU + HPU computation
– HPU part of core system architecture, many invocations – E.g. CrowdSearch, Soylent, Monolingual Translation
L. von Ahn has pioneered the field. See bibliography for examples of hiswork.
Crowdsourcing 101: Putting the WSDM of Crowds to Work for You.

10

A New Class of Applications
CPU + HPU hybrid applications blend automation with human computation to achieve new capabilities exceeding components
– CrowdSearch (T. Yan et al., MobiSys 2010) – Soylent: A Word Processor with a Crowd Inside. M. Bernstein et al. UIST 2010.

– Translation by Iteractive Collaborationbetween Monolingual Users, B. Bederson et al. GI 2010
Crowdsourcing 101: Putting the WSDM of Crowds to Work for You.

11

Human Computation
• Not a new idea • Computers before computers

Crowdsourcing 101: Putting the WSDM of Crowds to Work for You.

12

Pay-based Marketplaces / Vendors
• • • • • • • • • • • Mechanical Turk (since 2005, www.mturk.com) Crowdflower (since 2007,www.crowdflower.com) CloudCrowd (cf. DoMyStuff Livework Clickworker SmartSheet uTest Elance oDesk vWorker (was rent-a-coder)
Crowdsourcing 101: Putting the WSDM of Crowds to Work for You.

13

Amazon Mechanical Turk (AMT, Mturk)
• • • • Crowdsourcing platform On-demand workforce Went online in 2005 “Artificial artificial intelligence” (AAI) • Named after fake chess playing machine by Wolfgang von...