Grant recipient:
Christine Logel
Social Development Studies
(Project timeline: September 2014-January 2015)
Description
Too many students reach university without learning skills that are essential for success in higher education. In Arts lecture courses, instructors note weaknesses in two key areas: study skills (specifically learning textbook material in-depth rather than memorizing it) and exam-writing skills (specifically giving focused, complete, clear answers to short answer format questions). These weaknesses can impact students’ academic achievement as the former impairs students’ independent learning, and the latter prevents any learning from translating into higher grades. Poor grades may further impair deep learning when stress compromises concentration and working memory capacity.
Most universities have entire departments dedicated to teaching students these skills. uWaterloo’s Student Success Office (SSO), for example, employs four full-time success coaches and offers skill-specific tutorials. But do these tutorials actually improve students’ academic outcomes? No one knows. The majority of student support initiatives are never evaluated for effectiveness. Conversations with student success staff across North America, related to another project, reveal that staff are deeply committed to delivering effective programs, and motivated to measure their success and seek improvement. But they lack the mandate and research-specific expertise to design rigorous tests. This project provided such a rigorous test.
Participants were 121 students in three sections of an introductory psychology course. The majority were women entering first year in an interdisciplinary social science program.
In the department in which this class was run, students are block-enrolled into one of three sections, approximating random assignment to section. Sections were then randomly assigned to one of three conditions:
- Study Skills Tutorial: A success coach from the SSO presented a standard study skills tutorial.
- Exam Short Answer Tutorial: A success coach from the SSO developed a tutorial on writing clear, concise, and complete short answers on exams.
- Control: A manager of Student Life developed a presentation on the importance of managing finances throughout university.
The tutorials took place during an 80-minute class period between Test 1 and Test 2, such that Test 1 provided a baseline. As an incentive to attend the tutorials, 5% of students’ Test 2 grade was awarded for simply showing up. This 5% was removed from attending students’ test scores before data were analyzed.
Instructors were kept unaware of each section’s condition assignment. They encouraged students to attend the sessions. Few students asked questions about the purpose of the tutorials or appeared to notice that sections received different versions. These (rare) questions or comments were addressed truthfully as a chance to evaluate whether any of the tutorials were helpful for students.
Teaching assistants, who were unaware of students’ sections or condition assignments, graded tests with some help from the instructors. The instructors recorded students’ experimental participation marks from a list sent by the coordinator of the experimental pool.
Questions Investigated
- Empirically test the degree to which student success tutorials on “Writing Short Answers” and “Study Skills” improve students’ exam grades
- Test whether tutorials improve the clarity of students’ short answer responses
- Test whether grade effects are strongest among those who scored poorly on the first midterm
- If successful, implement such skills training annually for all incoming SDS students
Findings/Insights
Summary of Results
On the subsequent three in-class exams, study skills participants outperformed controls (p =.017) and nonattendees (p = .033), and improved their test grades by 6.5% on average. Effects were strongest for students who performed poorly on the premeasure midterm. These results suggest that study skills training is valuable for first year students.
This study provided a rigorous test of the effect of student success tutorials on student test performance. A study skills tutorial improved test scores among first-semester Arts students by over 6%. A short answer tutorial showed effects in a similar direction that did not reach conventional levels of statistical significance, perhaps because only a small portion of each students’ test grades were based on their short answer scores. The benefit was primarily experienced by lower-performing students, who would be expected to score around 66% on post-tutorial tests if they took one of the tutorials, compared to 58% if they did not.
If a teaching technique could improve students’ test scores by over 6%, faculty members would be sharing it on social media and discussing it at department meetings. If a pill could improve students’ test scores by over 6%, students would be clamoring for it at the pharmacy and selling it to each other in the library. But, as demonstrated in the present study, it is a standard study skills tutorial, conducted during class time, that has this test-boosting effect. This result suggests that study skills tutorials should be offered, in class, to every first-year student in Arts programs.
Did the Tutorials Improve Students’ Scores?
The effect of tutorial condition on students’ average scores across Tests 2, 3, and 4, controlling for Test 1 as a baseline, were examined using ANCOVA.
Results revealed a main effect of tutorial condition, F(3,120)=2.74, p=.047, as illustrated in Figure 1 (PDF). Post-hoc analyses using LSD indicated that students in the Study Skills condition outperformed controls (p =.017) and the nonattenees (p = .033). No other comparisons reached statistical significance.
How many points did each tutorial add to students’ test scores? Table 1 (PDF) displays the difference between students’ pre-tutorial test scores (i.e. Test 1) and post-tutorial test scores (i.e., Tests 2, 3, and 4).
Coders rated the degree to which the short answer portion of students’ exams was clear and complete. There was no effect of tutorial on clarity and completeness of short answers.
Did the Tutorials Improve Scores Most for the Lowest-Performing Students?
Given that no significant differences emerged between control participants and nonattendees, or between study skills and short answer participants, these groups, respectively, were combined to simplify the analysis. Regression analyses thus assessed the degree to which the effect of attending tutorials, compared to no tutorials, depended on students’ pre-tutorial test performance. Test 1 score was centred and tutorial condition was effect-coded (1- attended a tutorial; -1- did not attend a tutorial) and entered at Step 1; the interaction between them was entered at Step 2.
Results revealed main effects of tutorial condition, β=.16, t(117)=2.69, p=.008, and pre-tutorial test performance, β=.75, t(117)=12.79, p<.000001 which="" were="" qualified="" by="" an="" interaction="" t="" p="047." as="" illustrated="" in="" href="https://uwaterloo.ca/centre-for-teaching-excellence/sites/ca.centre-for-teaching-excellence/files/uploads/files/figure_2.pdf" title="Figure 2">Figure 2 (PDF), the tutorial increased test scores for the lower-performing students (-1SD), β=.14, t(116)=2.27, p=.025, but not the highest-performing students.
Dissemination and Impact
- At the individual level: Results shared with collaborators on other projects at Waterloo, and with struggling students in one-on-one meetings, encouraging them to take study skills courses at the student success offices. Report was read by a faculty member at St. Jerome’s who has instigated a collaboration to test its effectiveness among only those students who volunteer to take it when offered.
- At the Department/School and/or Faculty/Unit levels: Report shared with the Student Success Office, where it was read by the Director and core staff. Results shared with the Department of Social Development Studies. The study skills tutorial will now become a standard part of the course curriculum, beginning immediately.
References
Project reference list (PDF)