Week 5 Lab Activity: Experimental Design

Note: this activity is due by the end of your Lab A (Monday or Tuesday) this week!

Suppose I want to know how well a new chickenpox vaccine works. I plan to take a random sample of 10 people and give half the vaccine. In eight weeks, I will check to see how many people in each group contracted chickenpox. In order to see whether the vaccine works, I will test whether more people in the vaccine group or in the control group got the chickenpox.

There are many issues with this set up. First, there are only ten participants, so the power of the study is likely to be very low. There is no indication that the control group was given a placebo, so the placebo effect is a concern. There is also research suggesting that medical professionals impact patient outcomes (without meaning to) based on how much they believe a particular treatment will work. If the people administering the vaccine (or not administering the vaccine) know which treatment they are giving, that may also impact the results of this study. The proposed timeline for this research is also only eight weeks! This may not be a long enough timeframe. A good chickenpox vaccine should work for much longer than eight weeks. Finally, there are so many other possible factors here and none of them are being taken into consideration. Typically, medical research includes taken a lot of additional information in order to examine how additional factors - sex, weight, current prescriptions, etc. - may impact the effectiveness of a treatment.

Did you think of other issues? You’re probably right! This is a bad study design.

Learning the red flags of study design is a great thing to keep in mind while reading the news. Being inquisitive and questioning the details will often lead you to a more nuanced conclusion (usually the actual scientific paper is much better than the news reporting!), but it may also help steer you away from misinformation of all kinds!

Now it’s your turn - take out a piece of paper or open up a new Word document. This is an exercise in thinking about experimental design, so you do not need to identify absolutely every possible problem. As long as you are thinking crticially about the study and give several (3-4) reasons why it is problematic, there are no wrong answers! Discuss some of the issues with the following study design:

Suppose I want to know whether higher intelligence causes people to be more anxious. I bring 11 people into the lab and have each of them take a survey. Each one is asked (1) “On a day-to-day basis, how anxious are you?” and (2) “What was your SAT score?”. We will compare the level of anxiety from question (1) to the SAT score reported in (2) to see if higher intelligence is related to higher levels of anxiety.