Design of AI Products and Services: An Annotated Syllabus

Original article was published by Qian Yang on Artificial Intelligence on Medium

Part 2: Crowd as AI Proxy

In week 4–6, students will begin to consider how their data-driven designs can motivate users to provide the desired data in order to feed the underlying AI system.

By studying crowdsourcing systems, students will learn to investigate and envision the when, where, and why of user producing data. AI and crowdsourcing are a perfect match: AI empowers crowds and enhances the value of their production, while corporates and organizations access the crowd not only for co-creation of products or their ingenuity but rather as trainers for AI systems.

Students will read about two typical paths towards using the crowd as a proxy for AI. They will engage both paths as starting places for their project:

  1. Designers want to deploy an AI system to address a particular user need; however, there is no data set available to train the system. Or, the problem could be that statistical approaches are not very good at fulfilling the specific need. (VizWiz, a system we will read about, offers one example of this innovation path.)
  2. Designers recognize that people are engaged in an activity that could be recaptured to build data for training an AI system. A crowd plays the role of an AI system through the work they are already doing, and this produces valuable training data to develop such a system. (Luis von Ahn’s ESP Game and reCAPTCHA both provide an example of this innovation path.)


Students will begin by learning about crowd-sourcing and human-computation.

  • VizWiz: nearly real-time answers to visual questions (UIST’10)
    by Jeffrey P. Bigham et al.
    Link to article
  • Human computation: a survey and taxonomy of a growing field
    by Alexander J. Quinn and Benjamin B. Bederson
    Link to article

Next, about motivating crowd in the context of designing social network platforms:

  • Chapter 2: Encouraging contribution to online communities
    Building successful online communities: Evidence-based social design (2011): page 21–76.
    by Robert E. Kraut and Paul Resnick
    This is a very long chapter. Please skim and focus on where they discuss interaction design techniques to increase people’s willingness to make contributions.
    Link to article
  • Online forums supporting grassroots participation in emergency preparedness and response (Communications of the ACM, 2007)
    by Leysia Palen, Starr Roxanne Hiltz, and Sophia B. Liu
    Link to article

Finally, students will read about human-centered machine learning, where motivating crowds converged with enabling machine learning systems.

  • Power to the people: The role of humans in interactive machine learning (AI Magazine, 2014)
    by Saleema Amershi et al.
    Link to article

Project 2: Crowd as AI Proxy

Through this project, students will learn:

  • To investigate user motivation using value flow models
  • To gain a felt understanding of what people can do, what AI can do, and what AI needs data to do
  • To use the crowd as AI proxy, designing the when, where, and why of user producing data

Students will work on teams to design a system that uses a crowd of people as a proxy for an AI computing system. The project has three distinct stages:

  1. Explore design space: Teams will consider many possible opportunities for services that could benefit from a crowd as an AI proxy.
  2. Model preferred future: Teams develop and refine a value flow model that describes how all of the stakeholders within their ecology gain value. At this stage, it is critical to understand how the crowd is motivated to produce the required data in a time-sensitive way.
  3. Refine interaction: Teams will develop a set of wireframes that show the transactional flows for crowd participants who generate the data. This should include scenarios of use, that describe a typical interaction with the system.

Design Sprint: Modeling Value Flow

Design Sprint: Framing a Good AI Problem

Unlike the popular portrayals of AI in the media, what AI can do even provided a sufficient amount of data can often be frustratingly limited and brittle. We will provide some rules of thumb to help students check the technical feasibility of their designs.