CS 303: Designing Computer Science Experiments

CS 303 is a graduate course that examines experimental design in computer science research. Papers often succeed or fail based on their evaluation section. The goal of CS 303 is to help you improve how you design experiments to evaluate your research. It will do so by teaching you how to

• Reason through the strengths and limitations of an experimental design
• Design new experiments to unambiguously measure something
• Decide which experiments to include, given limited space

The course will have two major parts. The first is a series of experimental case studies from human-computer interaction, natural language processing, and computer systems. These case studies will include examples of exemplary depth, standard practices, innovative designs, and unforeseen flaws. The second major part of the course is a project, where students design and execute experiments for either their own research or prior work. Members of the class will constructively critique and discuss each other's designs. Coursework for the class involves problem sets in R that recreate experimental results in papers as well as a final project.

The course is paper-centric and has no textbook. Participants will be expected to keep a "lab notebook" in the form of a blog (please see Blogger). Version control will be handled using Git.

Syllabus

	Date	Topics	Reading	Assignment Due
	April 1	In-class Experiment: Comparing Pointing Devices Consent Form · FittsStudy Software Using R to analyze Fitts' Data Introduction to Experimental Design Lab Notebooks and Brainstorming Experimental Questions Lecture Slides	Fitts' Law: The Information Capacity of the Human Motor System in Controlling the Amplitude of Movement. Card et al.: Evaluation of Mouse, Rare-Controlled Isometric Joystick, Step Keys, andText Keys for Text Selection on a CRT.	If you have one, please bring your laptop to class with R installed.
	April 8	Wireless Networks Introduction to significance testing "Round Robin" of Experimental Questions	Read this first: Background materials. Aguayo et al.: Link-level Measurements from an 802.11b Mesh Network.	Assignment 1
	April 15	Parallel vs. Serial Prototyping Google Analytics Preparing your "Methods" section	Dow et al.: Parallel Prototyping Leads to Better Design Results, More Divergence, and Increased Self-Efficacy.	Assignment 2
	April 22	Clickthrough Experiments Non-parametric statistics Checkpoint: Pilot experiments	Read this first: Background materials. Joachims et al.: Accurately Interpreting Clickthrough Data as Implicit Feedback [longer journal version]	Assignment 3
	April 29	Stanford power consumption project [slides] Exploratory data analysis,               power testing, and effect size. Presentations of Pilot Experiments	"Green" Computing for Enterprise Systems	Assignment 4
	May 6	Social Software: Guest speaker Eric Gilbert Making graphs Data visualizations using existing datasets List of Class Blogs	Chapter 2 from The Elements of Graphing Data by William Cleveland.	Assignment 5 Class visualizations
	May 13	Mechanical Turk Checkpoint: Project updates with feedback from buddies & course staff.	Snow et al.: Cheap and Fast -- But is it Good? Abelson: Statistics as Principled Argument, Ch. 1 (available from course staff.)	Assignment 6
	May 20	Mobile Applications paper Methodology Matters Results and graphs.	Huang et al.: Anatomizing Application Performance Differences on Smartphones	Assignment 7
	May 27	Ethnography of Mobile Devices Results and discussion.	Bell et al.: The Age of the Thumb: A Cultural Reading of Mobile Technologies from Asia Additional reading: Falaki et al: Diversity in Smartphone Usage	Assignment 8
	June 3	Final presentations begin at 3pm sharp. (Location as usual: Wallenberg 124)  NOTE NEW CLASS TIME! (you may be wish to view the cs376 presentations from 12:45 - 2:45pm.) Pre-presentation snacks will be served during the break 2:45 - 3pm!		Final Project Guidelines
	June 6  (Monday)	Due date for final project write-ups (see Final Project Guidelines for details.)