CS 303: Designing Computer Science Experiments

Spring Quarter, 2011
Friday 2:15-5:05 pm
Wallenberg 124

Instructor and office hours:
Scott Klemmer
    Wednesdays 3:15-4:30pm, Gates 384

Philip Levis
    Mondays 11:30-12:30pm, Gates 358

Course assistant office hours:
Katherine Breeden
    Thursdays 4:30-6pm, Gates 381    

CS 303 is a graduate course that examines experimental design in computer science research. Papers often succeed or fail based on their evaluation section. The goal of CS 303 is to help you improve how you design experiments to evaluate your research. It will do so by teaching you how to

  • Reason through the strengths and limitations of an experimental design
  • Design new experiments to unambiguously measure something
  • Decide which experiments to include, given limited space

The class will also teach basic uses of R for data analysis. Note to participants: please take a few minutes prior to the first class to download R.  

The course will have two major parts. The first is a series of experimental case studies from human-computer interaction, natural language processing, and computer systems. These case studies will include examples of exemplary depth, standard practices, innovative designs, and unforeseen flaws. The second major part of the course is a project, where students design and execute experiments for either their own research or prior work. Members of the class will constructively critique and discuss each other's designs. Coursework for the class involves problem sets in R that recreate experimental results in papers as well as a final project.

The course is paper-centric and has no textbook. Participants will be expected to keep a "lab notebook" in the form of a blog (please see Blogger). Version control will be handled using Git.


Instructor notes

Date Topics Reading Assignment Due

April 1

  1. In-class Experiment: Comparing Pointing Devices
    Consent Form · FittsStudy Software
  2. Using R to analyze Fitts' Data
  3. Introduction to Experimental Design
  4. Lab Notebooks and Brainstorming Experimental Questions

Fitts' Law: The Information Capacity of the Human Motor System in Controlling the Amplitude of Movement.

Card et al.: Evaluation of Mouse, Rare-Controlled Isometric Joystick, Step Keys, andText Keys for Text Selection on a CRT.
If you have one, please bring your laptop to class with R installed.

April 8

  1. Wireless Networks
  2. Introduction to significance testing
  3. "Round Robin" of Experimental Questions
Read this first: Background materials.

Aguayo et al.: Link-level Measurements from an 802.11b Mesh Network.
Assignment 1

April 15

  1. Parallel vs. Serial Prototyping
  2. Google Analytics
  3. Preparing your "Methods" section
Dow et al.: Parallel Prototyping Leads to Better Design Results, More Divergence, and Increased Self-Efficacy.
Assignment 2

April 22

  1. Clickthrough Experiments
  2. Non-parametric statistics
  3. Checkpoint: Pilot experiments
Read this first: Background materials.

Joachims et al.: Accurately Interpreting Clickthrough Data as Implicit Feedback
[longer journal version]
Assignment 3

April 29

  1. Stanford power consumption project [slides]
  2. Exploratory data analysis,               power testing, and effect size.
  3. Presentations of Pilot Experiments
"Green" Computing for Enterprise Systems
Assignment 4

May 6

  1. Social Software: Guest speaker Eric Gilbert
  2. Making graphs
  3. Data visualizations using existing datasets
Chapter 2 from The Elements of Graphing Data by William Cleveland.
Assignment 5

May 13

  1. Mechanical Turk
  2. Checkpoint: Project updates with feedback from buddies & course staff.
Snow et al.: Cheap and Fast -- But is it Good?

Abelson: Statistics as Principled Argument, Ch. 1
(available from course staff.)
Assignment 6

May 20

  1. Mobile Applications paper
  2. Methodology Matters
  3. Results and graphs.
Huang et al.: Anatomizing Application Performance Differences on Smartphones Assignment 7

May 27

  1. Ethnography of Mobile Devices
  2. Results and discussion.
Bell et al.: The Age of the Thumb: A Cultural Reading of Mobile Technologies from Asia

Additional reading:
Falaki et al: Diversity in Smartphone Usage
Assignment 8

June 3

Final presentations begin at 3pm sharp. (Location as usual: Wallenberg 124) 

(you may be wish to view the cs376 presentations from 12:45 - 2:45pm.)

Pre-presentation snacks will be served during the break 2:45 - 3pm!

Final Project Guidelines

June 6  (Monday)

Due date for final project write-ups (see Final Project Guidelines for details.)