Planning a statistical investigation and critiquing statistical claims
Using the Problem-Plan-Data-Analysis-Conclusion cycle to run a fair investigation, and spotting the flaws in misleading survey reports and graphs
About three to four lessons of 45 to 60 minutes
"9 out of 10 people agree": agree with what, asked how, and out of how many?
A headline claims a new study proves a surprising result. Before believing it, three questions matter more than the result itself: who exactly was studied, how were they chosen, and does the conclusion actually follow from the data? A badly planned investigation, or a report that skips these questions, can make almost any claim sound convincing.
This unit has two connected halves. First, planning your OWN fair investigation using the Problem-Plan-Data-Analysis-Conclusion (PPDAC) cycle, so every step, from the question you ask to the conclusion you draw, holds up to scrutiny. Second, turning that same scrutiny outward: reading someone else's survey, graph or news report and working out exactly what it does, and does not, actually prove.
- "New study finds coffee drinkers live longer"does the study compare against non-coffee-drinkers, or is there no comparison group at all?
- A bar graph with a y-axis that does not start at zeroa small real difference can be stretched to look dramatic
- A poll of an online forum's most active usersa large sample can still be a biased one, if it is not chosen fairly
- Ice cream sales and drowning incidents rising togetherboth are driven by hot weather, one does not cause the other
What students will be able to do
Students will plan a statistical investigation using the Problem-Plan-Data-Analysis-Conclusion cycle, explain why a larger, fairly chosen sample gives a more reliable estimate than a small or biased one, and critique a statistical claim, survey report or graph by identifying missing comparison groups, biased sampling, misleading displays, or confusion between correlation and causation.
- I can name and describe the five stages of the PPDAC cycle: Problem, Plan, Data, Analysis, Conclusion.
- I can explain why a larger random sample tends to give a more reliable estimate of a population than a small one.
- I can identify when a survey sample is likely to be biased (not representative of the population it claims to describe).
- I can explain why a graph with a non-zero or inconsistent axis can be misleading, even when the numbers on it are accurate.
- I can explain why two variables changing together (correlation) does not prove one causes the other (causation), and suggest a possible lurking variable.
Standards this unit teaches
- AC9M9ST05Australian Curriculum v9 (ACARA)Statistical investigations
Plan and conduct investigations that involve the collection and analysis of different kinds of data; report findings and discuss the strength of evidence to support any conclusions.
- AC9M9ST01Australian Curriculum v9 (ACARA)Survey reports in the media
Analyse reports of surveys in digital media and elsewhere for information on how data was obtained to estimate population means and medians.
- AC9M10ST01Australian Curriculum v9 (ACARA)Critique statistical reports
Analyse claims, inferences and conclusions of statistical reports in the media, including ethical considerations concerning the use of statistics and the identification of bias.
- AC9M7ST03Australian Curriculum v9 (ACARA)Statistical investigations (Year 7 foundation)
Plan and conduct statistical investigations involving data for discrete and continuous numerical variables; analyse and interpret distributions of data and report findings in terms of shape and summary statistics.
Prior knowledge
This unit builds on skills students should already have met. Revisit any that are shaky first.
- Grade 7 sampling and making inferences about a population teaching unitrandom versus biased sampling, the foundation this unit's critique half builds on
- Year 7 statistics: mean, median, mode and rangepopulation 'means and medians' are exactly what a survey report tries to estimate
- Year 10 boxplots and the five-number summarycomparing the spread of two data sets is one of the analysis tools an investigation can use
Words to teach and display
- PPDAC cycle
- the five stages of a statistical investigation: Problem, Plan, Data, Analysis, Conclusion
- Population
- the entire group an investigation is trying to learn about, not just the people or items actually surveyed
- Sample
- the smaller group actually surveyed or measured, used to estimate something about the whole population
- Bias (sampling bias)
- a systematic tendency for a sample to over- or under-represent parts of the population, making it unrepresentative
- Correlation
- a relationship where two variables tend to change together, without necessarily meaning one causes the other
- Lurking variable
- a hidden third factor that influences two other variables, making them appear correlated even though neither causes the other
Teach it: concrete, pictorial, abstract
The lesson moves from things students can hold, to pictures and diagrams, to the written maths. The diagrams below are drawn from data, so they are accurate and print cleanly. Teach straight from them.
1. Planning a fair investigation: the PPDAC cycle
ConcreteA statistical investigation follows five stages, in order: pose the Problem (a clear question), make a Plan (how and from whom you will collect data), collect the Data, carry out the Analysis (organise, display and summarise it), and draw a Conclusion that answers the original question.
Consider the question: 'Do students who walk to school get more daily exercise than students who are driven?' Problem: state the question precisely, including how 'exercise' will be measured (e.g. daily step count). Plan: decide on a fair sample, for example randomly selecting an equal number of walkers and non-walkers across every year level, not just one friendship group. Data: collect step counts consistently, on comparable days. Analysis: compare the two groups' typical step counts (e.g. medians) and their spread. Conclusion: state clearly whether the data supports the original question, and acknowledge any limitations, such as the sample only covering one school.
A student wants to investigate: 'Does the amount of sleep Year 9 students get affect their test scores?' Outline the PPDAC stages for this investigation.
- Problem: define the question precisely, e.g. 'Is there a difference in test scores between students who sleep 8+ hours and those who sleep less, on average?'
- Plan: decide how sleep will be measured (a self-reported average over a week) and how students will be chosen (a random sample across several classes, not just one).
- Data: collect both the sleep data and the matching test scores for the same students.
- Analysis: compare the median (or mean) test score of the two sleep groups, and look at the spread within each group.
- Conclusion: state what the data shows, being careful to note this only shows an association, not proof that sleep directly causes better scores (other factors could differ between the groups too).
Answer: A five-stage plan: define the sleep/scores question precisely, plan a fair random sample and consistent measurement, collect matching data, analyse by comparing typical scores and spread between groups, then conclude cautiously, noting the limits of what the data can prove.
- Why must the Problem stage state exactly what will be measured, not just a vague question?
- What could go wrong in the Analysis stage if only the mean is reported, with no mention of spread?
2. Why sample size and fairness both matter
PictorialImagine the TRUE proportion of a population that prefers something is exactly 50%. Take many small random samples and many large random samples from that same population: the small samples will vary a lot from 50% just by chance, while the large samples cluster much more tightly around it.
Six different random samples of 10 people each, all drawn from a population that is truly 50% in favour, gave these percentages in favour: 40%, 60%, 50%, 70%, 30%, 60%, ranging all the way from 30% to 70%. Six different random samples of 100 people each, from the exact same population, gave: 47%, 52%, 49%, 51%, 48%, 53%, all within 3 percentage points of the true 50%. Neither set of samples is 'wrong', both are genuinely random, but the LARGER samples are far more reliable because random sampling variation shrinks as sample size grows.
- If both sets of samples above are genuinely random, why do the smaller samples vary so much more?
- A friend says 'my sample of 10 people is fine because I asked them randomly'. What would you tell them about relying on that single result?
3. Spotting a biased sample or a misleading display
PictorialA sample can be large and still be badly biased, if the way it was chosen systematically favours some people over others. A display can show completely accurate numbers and still be misleading, if its scale exaggerates or hides the real pattern.
A phone app asks users to rate a restaurant, and reports '95% satisfied' from 2,000 responses. That sounds like strong evidence, but only people motivated enough to open the app and leave a review did so, a SELF-SELECTED sample. Diners having an average, unremarkable experience are far less likely to bother rating it than diners having an unusually great or unusually terrible one, so the 2,000 responses likely over-represent strong opinions in both directions, not the typical customer. Sample size alone (2,000 is a lot of people) does not fix a biased selection method.
Separately, a bar graph comparing two products' sales, $48,000 and $52,000, can be drawn with a y-axis starting at $47,000 instead of $0. The second bar then looks roughly FOUR TIMES as tall as the first, even though the real difference is under 10%. The numbers on the graph are not wrong, but the visual impression is.
A news article says 'Ice cream sales and drowning incidents both rise every summer, so eating ice cream increases your risk of drowning.' Identify the flaw in this conclusion.
- Both variables (ice cream sales and drowning incidents) do genuinely rise together, so there is a real correlation.
- But a correlation between two variables does not prove either one causes the other.
- Look for a lurking variable that could explain both: hot weather increases ice cream sales AND increases the number of people swimming, which increases drowning incidents.
- The honest conclusion is that hot weather (a lurking variable) drives both, not that ice cream causes drowning.
Answer: The flaw is treating a correlation as proof of causation. Hot weather is the lurking variable that genuinely explains both ice cream sales and drowning incidents rising together.
- Why does a large sample size (2,000 reviews) NOT automatically mean a sample is unbiased?
- Besides temperature, can you think of another example where two things correlate because of a shared, hidden cause?
Common misconceptions and how to address them
MisconceptionA bigger sample automatically means a fairer, less biased sample.
Why it happens: Students conflate the two separate ideas of sample SIZE (reducing random variation) and sample SELECTION METHOD (avoiding systematic bias).
How to address it: Size and fairness are different problems needing different fixes. A huge but self-selected or non-random sample (like online reviews) can still be badly biased; a small but genuinely random sample is unbiased, just imprecise. The best studies need both: large AND randomly chosen.
MisconceptionIf a graph's numbers are accurate, the graph cannot be misleading.
Why it happens: Students check only whether the plotted values are correct, not how the axis scale shapes the visual impression.
How to address it: A graph can be numerically accurate and still visually misleading, most often through a non-zero y-axis, inconsistent scale spacing, or 3D effects that distort area. Always check where the axis starts and whether its intervals are even before trusting a visual comparison.
MisconceptionIf two things happen together consistently, one must cause the other.
Why it happens: Correlation is the pattern that is easiest to notice, and 'X causes Y' is a simpler story than 'a hidden third factor affects both'.
How to address it: Before accepting a causal claim, actively search for a lurking variable that could explain both trends, and ask whether a controlled experiment (not just an observed pattern) has actually tested cause and effect.
MisconceptionA conclusion can claim anything as long as it sounds like it follows from the data discussed.
Why it happens: Once an investigation has produced results, there is a pull to state a strong, exciting claim rather than a cautious one.
How to address it: A conclusion should refer directly back to the original Problem, state only what the Analysis actually supports, and explicitly note the investigation's limitations (sample size, possible bias, or what was NOT tested).
MisconceptionSurveying an entire class or school is the same as surveying the whole population you care about.
Why it happens: Students confuse a CENSUS of one convenient, accessible group with a representative sample of the wider population the question is really about.
How to address it: Ask explicitly: what is the population this conclusion is meant to apply to? A result from one school's students only safely describes that school, not 'all teenagers', unless the sample was deliberately chosen to represent the wider group.
Guided practice (with answers)
1. Which PPDAC stage involves deciding HOW data will be collected and from whom, before any data is gathered?
Answer: The Plan stage, which sets out the method and sample before collection begins.
2. A researcher takes one random sample of 15 people and another random sample of 150 people, both from the same population. Which is likely to give a more reliable estimate, and why?
Answer: The sample of 150, because larger random samples show less variation due to chance than smaller ones, so their results cluster more tightly around the true population value.
3. A weight-loss product's website only publishes testimonials from customers who lost weight. What is the sampling problem?
Answer: It is a self-selected, biased sample: customers who did not lose weight (or had a bad experience) are excluded, so the testimonials overstate how well the product typically works.
4. A graph shows a company's profit rising from $9.8 million to $10.2 million, but the bar for $10.2 million looks twice as tall. What should you check?
Answer: Whether the y-axis starts at zero and uses even spacing; a non-zero axis can make a small real change (about 4%) look like a huge one.
5. A study finds that towns with more fire stations have more fires. Does having more fire stations cause more fires?
Answer: No; a lurking variable, town size (or population), likely explains both: bigger towns need more fire stations AND naturally have more fires.
6. Why should a statistical conclusion mention the investigation's limitations, rather than just stating the result?
Answer: Because a limitation (such as a small sample, one school only, or a possible source of bias) tells the reader how much to trust the conclusion and whether it can be generalised beyond the group actually studied.
Independent practice worksheets
Practise the investigation cycle, sampling reasoning, and critiquing statistical claims with computed, never-wrong answer keys.
Differentiation
- Give students a PPDAC checklist card (five labelled boxes) to fill in for a simple, familiar investigation question before attempting an unfamiliar one.
- For sample-size reasoning, physically demonstrate with a bag of coloured counters: draw small samples versus large samples and record how much the proportion varies each time.
- Provide a short, explicit list of things to check in any claim: Who was sampled? How many? Is there a comparison group? Does the graph's axis start at zero?
- Work through the ice cream/drowning example as a whole class before asking students to find a lurking variable in a new example independently.
- Have students design and actually run a small investigation of their own on a real school question, following all five PPDAC stages and presenting the conclusion with its limitations.
- Find (or construct) a real misleading graph and redraw it fairly, comparing the visual impression of both versions side by side.
- Research a real correlation-versus-causation news story and identify the most likely lurking variable, explaining what evidence would be needed to test a genuine causal claim.
- Debate: should online review platforms be required to disclose what percentage of customers actually left a review, to help readers judge sampling bias?
Assessment: exit ticket
A three-question exit ticket sampling the PPDAC cycle, sample size and fairness, and critiquing a claim.
1. Name the five stages of the PPDAC cycle in order.
Answer: Problem, Plan, Data, Analysis, Conclusion.
2. A pollster surveys 40 people outside a gym at 6am about exercise habits, and concludes '90% of adults exercise regularly'. Identify the sampling problem.
Answer: The sample is biased, not random: people at a gym at 6am are far more likely to exercise regularly than the general adult population, so the sample does not represent all adults.
3. A report shows students' screen time and grades both changing over the school year, and claims screen time directly lowers grades. What should a careful reader ask for before accepting this as proven?
Answer: Evidence that rules out lurking variables (such as workload, stress, or time of year affecting both) and, ideally, a controlled study rather than just an observed correlation, before accepting a direct causal claim.
Teacher notes and timings
- Rough timing across three to four lessons: Lesson 1 the PPDAC cycle (section 1), Lesson 2 sample size and variation (section 2), Lesson 3-4 spotting bias and misleading displays plus the exit ticket (section 3 and assessment).
- This unit assumes comfort with random versus biased sampling (the Grade 7 sampling and inference unit) and with mean/median as measures of a typical value (the Year 7 statistics unit). Revisit either first if the vocabulary is shaky.
- Language to keep repeating: PPDAC in order (Problem, Plan, Data, Analysis, Conclusion); a bigger sample reduces random VARIATION but does not fix a biased SELECTION method; correlation is not causation, always ask about a lurking variable.
- The two dot-plot figures in section 2 use the exact same underlying idea (six random samples from a population truly 50% in favour) at two different sample sizes, so the spread difference is the only thing changing, making the 'bigger samples vary less' point as clean as possible.
- Curriculum note: this unit deliberately spans four ACARA v9 codes across Years 7, 9 and 10 (AC9M7ST03, AC9M9ST01, AC9M9ST05, AC9M10ST01), since planning a fair investigation and critiquing someone else's are two sides of the same statistical-literacy skill, taught here together rather than as separate, disconnected worksheets.
- Present mode and print both work: use the Print button for a clean handout, or project the ice cream/drowning scatter plot and have the class suggest the lurking variable before revealing it.