TAP Update: Fall 2022

Learning about changes to Waterloo’s teaching assessment processes for the first time? Read “The path to here: 2014-2022” 

The Teaching Assessment Processes (TAP) office continues to consult, innovate, and support the campus community through the administrative changes and cultural shift required to adopt a holistic approach to teaching assessment. This update offers a summary of current activities, including TAP's work supporting the University’s work towards equity, supporting Faculties, and conducting ongoing research and development activities, in addition to a note about the University of Waterloo in relation to its peers.

Supporting the University’s work towards equity

Collaborating with the Office of Equity, Diversity, Inclusion & Anti-racism and the Office of Indigenous Relations to support marginalized instructors. These connections are of primary importance for the TAP office because equity is a driving force behind Waterloo’s commitment to a holistic model of assessment. 

Analyzing SCP data through the lenses of gender and race. With the winter 2022 launch of the new Student Course Perception (SCP) survey, the TAP office has been tasked with regularly reviewing results to determine how well it performs, and in particular to consider whether there are important differences in results depending on instructor characteristics irrelevant to teaching performance. Analysis of data from winter 2022 SCP data looked at mean differences in ratings assigned by students to male and female instructors and found that differences in scores (with a very small number of exceptions) were not statistically significant (p>.05). Moreover, the difference was close to 0 in almost all cases explored. These results suggest that gender bias is endemic to the instrument itself. While this is good news, this of course does not mean that sexism does not exist, nor does it mean that it does not affect SCP scores—especially as this is just a one-term study. As noted, the TAP office will continue to examine SCP results in future terms. A research report presenting the winter 2022 results will be posted on the TAP office website soon. Note: gender-related data are gathered through Human Resources.

The TAP office remains committed to analyzing and monitoring SCP results with respect to race, but a low response rate for Waterloo’s Equity Survey (together with the already small number of faculty members in some racialized groups) means that we will need more than one term’s worth of data to gather meaningful results. This analysis will be repeated as needed, in addition to the ongoing collection of data that will eventually illustrate multi-year trends. Processes and survey items will be adjusted where necessary. 

Preparing to conduct a qualitative analysis of responses to open-ended questions. In many cases, our most vulnerable instructors are victimized by abusive comments. This research will determine the extent to which marginalized instructors are exposed to inappropriate comments from students. The goal of this analysis is primarily to inform decisions with respect to: (1) How to best screen comments to protect instructors, and (2) development of instructional material for students about what sorts of comments are helpful and appropriate. 

Supporting Faculties

Extending support to Faculties updating documentation for Annual and Biennial Performance Reviews. Administrative documents will need to be revised to match the new Student Course Perception survey. If you would like support for this adjustment, please reach out to Kathy Becker at kabecker@uwaterloo.ca. 

Consulting with Faculties to design Faculty-level SCP questions grounded in Faculty-specific teaching & learning priorities. Once these questions have been crafted, piloted, and finalized, similar work will be undertaken at the department level. Only core institution-level questions will be included in summative assessments. 

Working with Faculties to establish consistent Peer Review of Teaching processes. These processes, based on recommendations from the Complementary Teaching Assessment Project Team and endorsed by Deans and Senators, will take into account factors like timing and frequency of peer reviews. 

Ongoing research and development

Launching (1) a working group to establish guidelines for the use of data from SCPs and other TAP methods for summative purposes and (2) an advisory committee that will guide the ongoing work of the TAP office. The groups’ Terms of Reference have been drafted. Next steps include identifying subject matter experts who have capacity to support this work. 

Considering the work of the Task Force on Graduate Supervision and how the quality of student supervision might be assessed. These guidelines will create a space for Faculty who regularly supervise Waterloo students to have their contribution to student development assessed and recognized. 

Continuing to revise the Student Course Perception survey user guides based on input from both administrators using them for assessment and instructors bringing concerns to the TAP office. Please continue to provide feedback. Enter instructor feedback. Enter administrator feedback. 

A note about the University of Waterloo in relation to other institutions 

Many institutions use survey questions that are not based on best practices, inspiring a number of publications with inflammatory headlines about gender- and race-based bias within student surveys. 

The good news is that most of these articles recommend mitigating the impacts of bias with SCPs by incorporating additional mechanisms for assessing teaching – such as peer review and teaching dossiers, both of which are being implemented at Waterloo. 

We – the TAP office and the Waterloo instructors who take the time to provide constructive feedback supporting this massive culture shift – are making great strides to change the culture around teaching assessment, with the ultimate goal of creating a fairer and more equitable system of teaching assessment. Waterloo’s work and progress toward this goal is recognized by our colleagues at other U15 institutions who look to Waterloo’s work and the institutional culture shift that is underway as both guidance and inspiration for their own work in this field.