Unit 15: Experimental Design Revisited

Home | Contact us   
  Main Concepts | Demonstration | Teaching Tips | Data Analysis & Activity | Practice Questions | Connections | Fathom Tutorial | Milestone 


 Teaching Tips

• Just do it!!! Students will not learn about the design of experiments by reading the text. They need to get dirty with the data: planning, collecting, organizing, summarizing, analyzing, communicating, and revising the design. Or, as Dan Teague once put it (in a slightly different context): "Students won't be good at it until they're bad at it first." So give them plenty of opportunities to make mistakes and improve.

• The bulls-eye diagram on BIAS and VARIATION in Yates Moore & Starnes is helpful and so is the histogram question in the Practice Problems.

• Some students will find it helpful to think of data as a combination of a "signal" with "noise." Any factor that adds to the "noise", thereby making it harder to detect the signal, is increasing the variability. Anything that distorts the signal -- makes it "tune" to the wrong station -- is adding bias.

• If you have students in journalism, they can improve the quality of their newspaper surveys by having you and your students edit the questions, distribute the surveys (usually in clusters by randomly selected classrooms at a certain time) and then you can use these data for discussion, review, or exam questions.

• Students can use what they learn in statistics to improve their projects/reports in other courses, such as Biology, Physics, Econ or Government, by adding a statistical component to their work. Better yet, statistics students can teach tools like chi-squared to other biology students and regression to fellow econ students.

• When assessing students' work, make sure they are not just mindlessly parroting the key terms (such as "control", "blocking", "bias"). Make sure the students understand why these concepts are important within the context of the design of the experiment.

• If doing projects earlier in the course, a useful exercise for students might be to ask them how they would change their (earlier) experimental design, now that they’ve learned all of this useful information about experimental design.

Student Misconceptions and Confusions

• The Gettysburg address activity (see the Activity page) introduces the concept of BIAS, so that students can distinguish BIAS and VARIATION in sampling. Students tend to overuse the word "bias" as a critique of experiments. They'll say a study is "biased" by some factor when, in fact, that factor merely introduces extra variability. However, "bias" occurs only when one treatment is favored over another by the experimental design.

• One of the most common misunderstandings students have is that they confuse treatment groups with blocks. Subjects are assigned to blocks BEFORE a treatment is performed, and are assigned to blocks based on their similarities. Subjects are then assigned to treatment groups WITHIN a block, so that all treatments are represented within each block.

• Students often confuse randomizing treatments in an experimental design with choosing a random sample for a survey. Selecting a simple random sample from a population is done so that we can make inferences to the larger population. However, often in experimental design we're more interested in establishing whether one factor CAUSES another. And the purpose of randomizing subjects to treatment groups is to help us establish a causal relationship EVEN IF IT IS JUST BETWEEN THESE SUBJECTS. In other words, often in an experiment it is not possible to make a statistical inference to a larger population, and sometimes this is not desirable.

• Students are easily confused by confounding variables. A confounding variable is not one that "stumps you". Confounding variables are variables that have a strong association with at least one of the treatment variables and the response variable. We might want to know, for example, if there is an association between race (treatment variable) and chances of admission to UCLA (response). But race is confounded with socio-economic status, which in turn influences which schools children attend which influences whether or not they apply to UCLA and also affects their chances for admission. Randomization across treatment groups minimizes the effects of confounding variables. (Of course, it is not possible to randomly assign someone a race.)


Historical Footnotes on randomization: for those of us who believe a little history makes things more interesting.

• The NCSSM Leadership Institute of 2001 had some useful notes on experimental design (pdf format).

• You might just find the entire NCSSM leadership institute website to be useful.

• George Cobb's books