If every student in the class completed the SCP, the response rate would be 100% and indicative of the true collective experience of the entire class. In reality, however, a response rate of 100% is extremely rare so we rely on an estimate of the true experience of the collective and estimates always include some degree of measurement error.
Don’t assume that the average score represents the collective experience of the entire class. Exercise caution, especially when there is a low response rate. Scores associated with higher response rates are a better reflection of the collective experience.
Don’t make assumptions about the collective class experience if the response rate is low. The size of the course determines what should be considered to be an adequate response rate (see Table 2). Consult Table 2 to determine how much confidence you can have in thescores based on the level of “precision” for the response rate in a particular course.
Class size and response rate
Generally speaking, smaller classes require a higher response rate to achieve the same confidence that scores reflect the collective experience. Larger courses result in more data than smaller courses, so even with similar response rates, a smaller class is more likely to achieve a “precise” estimate of the overall experience. There is a high degree of inaccuracy in mean scores calculated in samples with fewer than 20
respondents. In a smaller class, one unhappy student can seriously undermine the overall average. The outlier cases can have a significant impact on the overall score. With smaller classes, it is extremely important to review the distribution of scores.
The following table outlines the response rate necessary to estimate the collective experience of students based on class size. Adapted from University of Toronto data.
| Interval around mean | Quality of mean estimate | Course size | ||||
|---|---|---|---|---|---|---|
| 1-25 | 26-50 | 51-100 | 101-200 | 200+ | ||
| <±0.1 | Very reflective of the collective experience | >90% | >80% | >80% | >60% | >50% |
| <±0.2 | Reflective of the collective experience | >80% | >70% | >70% | >50% | >40% |
| <±0.5 | Somewhat reflective of the collective experience | >70% | >50% | >40% | >20% | >10% |
| <±1.0 | Generally not reflective of the collective experience | >60% | >20% | >10% | >10% | >10% |
| >1.0+ | Not at all reflective of the collective experience | <30% | <10% | <5% | <3% | <1% |