Research Design
Your research question determines your design — the design is your plan for getting reliable answers
📌 Before You Start
Prerequisites: Modules 1 and 2. You should already have a research question in mind.
Estimated time: ~45 minutes including the exercise.
What you need: The research question you developed in Module 2.
By the end of this module you will be able to distinguish experimental from non-experimental designs, explain internal and external validity, and choose an appropriate design for a given research question.
💡 The Big Idea
Your research question determines your design. The design is your plan for getting reliable answers. No single design is best for every question — choosing the right one requires understanding what each design can and cannot tell you.
🔍 Deep Dive
Experimental Designs
Experimental designs are the gold standard for establishing causation — whether X actually causes Y. They require a researcher to intervene (manipulate the IV) and observe the effect on the DV.
True Experiment
What it has: Random assignment to groups, a control group, and a treatment group.
Random assignment means each participant has an equal chance of being in either group, which distributes confounders evenly.
Example: Randomly assign 60 students to either a tutoring program (treatment) or no program (control). Compare final exam scores.
Strength: Strongest evidence for causation.
Quasi-Experiment
What it lacks: Random assignment. Groups are pre-existing.
Example: Compare exam scores between two classes — one taught with flipped classroom, one with lecture — where students were not randomly assigned to classes.
Strength: More feasible in real settings than true experiments.
Weakness: Groups may differ in ways other than the intervention.
Pre-Experimental
What it lacks: Random assignment AND a control group.
Example: Give a survey to one class before and after a mental health workshop. No comparison group.
Strength: Easy to conduct. Good for pilot studies.
Weakness: Cannot rule out alternative explanations. Low internal validity.
Non-Experimental Designs
Non-experimental designs do not manipulate variables. They observe, describe, or explore. They are essential for ethical, practical, or exploratory research.
| Design | What it does | Best for |
|---|---|---|
| Survey / Questionnaire | Collects self-reported data from a large sample using structured questions | Measuring attitudes, behaviors, prevalence of experiences |
| Case Study | Deep, detailed investigation of one person, group, or event | Rare or unique phenomena; exploratory research; clinical contexts |
| Ethnography / Observation | Researcher observes participants in their natural setting, often over an extended period | Understanding culture, daily practices, group dynamics |
| Content Analysis | Systematically analyzes existing texts, images, or media | Studying communication, media representation, historical documents |
| Meta-Analysis | Statistically combines results from multiple existing studies | Synthesizing evidence across many studies to find overall effect sizes |
Cross-Sectional vs. Longitudinal
These terms describe the timing of your data collection — they apply to both experimental and non-experimental designs.
| Timing | What it means | Example |
|---|---|---|
| Cross-Sectional | Data collected at one point in time. Like a snapshot. | Survey 500 college students about their sleep and GPA this semester. |
| Longitudinal | Data collected from the same participants over time. Like a film. | Follow 200 first-year students across 4 years, tracking sleep and GPA each semester. |
Internal Validity: Did It Actually Work?
Internal validity is the confidence that your independent variable (and nothing else) caused the change in your dependent variable. In other words: Did the intervention really work, or could something else explain your results?
Common threats to internal validity:
- Selection bias: The groups differ before the study starts. (Students who volunteered for tutoring were already more motivated.)
- Attrition (dropout): Participants leave the study unevenly. (Only struggling students drop out, making the treatment look more effective.)
- History effects: Something happens during the study that affects results. (A major exam occurs during your stress intervention.)
- Placebo effect: Participants improve simply because they believe they are receiving treatment.
- Maturation: Participants change naturally over time, regardless of the intervention.
External Validity: Does It Apply to the Real World?
External validity is the extent to which your findings can be generalized beyond your specific sample and setting.
- A study of 30 college students at one university may not generalize to all adults.
- Lab conditions may not reflect real-world behavior.
- The time period of the study may limit generalizability.
Internal Validity
"Did X cause Y in this study?"
Increased by: random assignment, control groups, blinding, controlling confounders.
External Validity
"Can we apply this finding more broadly?"
Increased by: larger samples, random sampling, diverse participants, real-world settings.
📋 Real Example: Coffee, Cancer, and Observational Data
For decades, studies suggested that drinking coffee was associated with higher rates of certain cancers. Governments warned against it. Coffee lovers worried.
The problem? Nearly all of these studies were observational: researchers noted who drank coffee and who got cancer, but they did not control for confounders. One massive confounder was smoking: in the mid-20th century, heavy coffee drinkers were also much more likely to smoke. Smoking causes cancer. When researchers began controlling for smoking in their analyses, the coffee-cancer link largely disappeared — and some studies now find that coffee may actually be protective against certain cancers.
What this teaches us:
- Observational data can tell you two things are associated, but not that one caused the other.
- Confounding variables are the most common reason headlines mislead us.
- The phrase "linked to" or "associated with" in a headline does NOT mean "causes."
🖐️ Your Turn
What you need: Pen and paper or a Google Doc. About 15 minutes.
For each of the three research questions below, choose the best research design and explain why. Consider: Can you manipulate the IV? Are there ethical concerns? What is most feasible?
- "Do students who sleep more get better grades?"
What design would you choose? Why? What is the main threat to validity in your design? - "How do first-generation college students experience imposter syndrome?"
What design would you choose? Why? Is this question better suited to quantitative or qualitative methods? - "Does tutoring improve exam scores in introductory statistics?"
What design would you choose? Could you run a true experiment here? What ethical issues might arise?
There is often more than one defensible answer. The goal is to justify your choice with the concepts from this module.
🧠 Brain Break — 2 Minutes
Think of a headline you have seen recently about a health finding.
Ask yourself: What design do you think that study used? Was it observational or experimental? What confounders might have been lurking? Would you change your behavior based on that study alone?
Good researchers ask these questions automatically every time they read a finding.
✅ Key Takeaways
- True experiments (random assignment + control group) are the best design for establishing causation, but they are not always ethical or feasible.
- Non-experimental designs (surveys, case studies, ethnography, content analysis) are essential for questions that cannot be manipulated or that need depth and context.
- Internal validity asks: did the intervention cause the outcome? External validity asks: does the finding generalize?
- Common threats: selection bias, attrition, history effects, and the placebo effect.
- "Associated with" means correlation, not causation. Confounders are the reason observational studies so often mislead headlines.
🎯 Module 3 Complete!
You can now plan a study. In Module 4, you will learn to find and evaluate the sources you need to ground your research question in existing knowledge.
Continue to Module 4: Finding & Evaluating Sources →