Frequently asked questions: Student Course Perceptions

Student Course Perceptions

About the survey

Concerns

General

What is the Student Course Perceptions survey?

The SCP survey has been designed to replace the course evaluations that have been in use at Waterloo for decades. We are referring to it as a “student course perceptions” survey to emphasize that it is not, by itself, a means to “evaluate” an instructor’s teaching, a student’s learning, or the quality of a course. The SCP survey gives students a voice to express their perception of their learning experience (see “Why are we revamping the course evaluation process at Waterloo?”) and will be used with complementary methods such as teaching dossiers and peer review, in addition to specific strategies for assessing graduate supervision.

Why are we revamping the course evaluation process at Waterloo? What are the benefits of the new SCP process?

The decision to come up with a new process emerged from a few directions, including the following concerns:

  • As the single source of data about course instruction, Waterloo’s course evaluation surveys were limited and problematic. We needed to find a way to prioritize the student voice while paying close attention to concerns that teaching assessments should not rely solely on student input. In addition to updating the student survey questions, we are in the process of developing additional assessment tools designed to complement data collected by the SCPs.
  • Waterloo’s course evaluations have historically been decentralized: each faculty administered its own questions, which were used by instructors (as formative input) and academic administrators (for summative assessments). The new SCP survey includes a set of core questions designed to provide insight into teaching and learning priorities that matter institutionally at the University of Waterloo. Each faculty will also have an opportunity to develop its own additional questions as needed.
  • The previous series of faculty surveys included problematic questions, including: An emphasis on outdated questions applicable mainly to a lecture format; Questions students are not in a position to answer (e.g., “Is the instructor knowledgeable?”)Gendered language that risks generating biased responses from students based on their preconceived expectations of how instructors should act based on gendered stereotypes (e.g., “Is the instructor “caring,” “understanding,” “available”?) The new core survey questions were designed to be applicable for all types of delivery methods (e.g., online/in-person) and teaching modalities (e.g., lecture, seminar, active learning), emphasize questions students are well-positioned to answer, and tries to avoid gendered language. The questions were also designed to address concerns about conflating scores for course “design” and “implementation” by distinguishing course design principles from implementation (delivery) principles, as supported by the pilot test factor analysis.

Additionally, the core items on the new instrument provide the opportunity to examine trends over time in the performance of the survey instrument itself, as well as with respect to how scores are associated with other variables that do not impact teaching (class size, instructor gender, instructor racialization, etc.).

What are the new survey questions?

See "the survey questions"

How did you come up with the core survey questions?

The core survey questions were developed after a thorough review of the research literature (scholarship of teaching and learning, best practices for use of course evaluations, the teaching and learning priorities that matter at the University of Waterloo, and consultations with other U15 institutions engaged in this type of work.

Members of the Course Evaluation Project Team visited each Faculty multiple times throughout their development, and instructors at Waterloo had an opportunity to view and critique the questions via an open survey that included open-ended questions. Focus groups with students in each Faculty confirmed that the core questions were aligned with student expectations and needs.

Can the survey questions be changed?

They might, depending on input that the TAP team receives as well as data trends as the university starts using the surveys. Administering the same survey across the entire campus will allow the university to track trends centrally, which should bring gaps and flaws to the surface so that they can be addressed. If the data highlight problems with any of the questions, the TAP team will review it thoroughly and make changes as needed.

How does the University of Waterloo define effective teaching?

In designing any “course evaluation” survey, careful consideration must be given to the institutional context in which the instrument will be used (Ory & Ryan, 2001; Gravestock & Gregor-Greenleaf, 2008; Theall & Franklin, 2001; Willits & Brennan, 2017). The validity and utility of any evaluation tool depends heavily on the development of questions that reflect institutional goals and practices of teaching (Gravestock & Gregor-Greenleaf, 2008; Ory, 2001; Theall & Franklin, 2001).

As part of the University of Waterloo’s initiative to develop a fairer, more equitable, and transparent system for the evaluation of teaching, the Teaching Assessment Processes team is collaborating with key stakeholders, including instructors and students, to develop a holistic characterization of effective  informed by the University of Waterloo's institutional priorities, the research literature, and extensive consultations with campus stakeholders.

See the teaching effectiveness framework, which informs our assessments of teaching quality at Waterloo. Work on this framework continues, but the consilience of these sources of information to date shows that the SCP core questions are directed at features essentially everyone recognizes as important aspects of effective teaching.

Why do we need an institutional characterization of teaching effectiveness?

“Good teaching” can mean many different things. We therefore prefer not to talk about defining effective teaching, as this can be a source of endless and fruitless dispute. By gaining institutional endorsement for a framework that reflects agreed upon values for the local context--that is, a framework that includes items that we all recognize as aspects of the kind of teaching we want to see at Waterlloo--we can incentivize that kind of teaching.

Buy-in for assessment practices is more likely when campus stakeholders can see how those practices provide real evidence about how well important aspects of good teaching are being carried out, and this can be made clear if we have an institutionally endorsed framework.

Measures of “teaching effectiveness” are grounded in a concrete framework that is built on answers to the following questions:

  • What matters for instructors at Waterloo with respect to teaching?
  • What matters for students at Waterloo with respect to teaching?
  • What does the literature say matters with respect to effective teaching?
  • What does Waterloo need to strive for to be the leading institution for teaching effectiveness in Canada?

Does the SCP survey measure student learning?

No. This is deliberate. Measuring student learning is what assignments and exams are for.

The SCP is designed to measure things students are able to reliably assess within their course experience including their perceptions of their learning experience, specifically with respect to Waterloo’s institutional values, expectations, and priorities for teaching. Students are not well positioned to assess their depth or degree of learning in a given course. Often this type of insight comes later, as students realize the impact of their formal education. As Berk (2006) explains, “isolating students’ course achievement at one point in time or their gains over time that are attributable directly to teaching is nearly impossible” (p. 32).

Some researchers have demonstrated a positive correlation between student ratings and their grades (or expected grades) (Abrami, 2001; Abrami, Dickens, Perry, & Leventhal, 1980; Benton & Li, 2015; Eiszler, 2002; Feldman, 1976; Greenwald & Gillmore, 1997; Stumpf & Freedman,1979), but that does not suggest a causal relationship. As Linse (2017) suggests, ““Student ratings have never been intended to serve as a proxy for Learning"

The SCP questions don’t seem to differentiate between the instructor and the course. What if an instructor taught a course but didn’t design it (or vice versa)?

The literature suggests that those types of distinctions are not overly clear for students when filling out a course survey: “this course” and “the instructor for this course” are somewhat conflated in their minds. The new SCP instrument deliberately circumvents this issue by focusing instead on things that are most likely to be distinguished in a student's mind, such as the “design” of a course versus its “delivery”: each of the core SCP questions belongs in one or the other category. That students do distinguish these things was supported both in the focus groups conducted before the 2018 pilot test of the new instrument and in the factor analysis carried out on the results of the pilot test.

Why doesn’t the SCP include any overall ratings (e.g., “What do you think of this instructor overall?”)?

The literature shows that students are more likely to draw on implicit or explicit biases (instead of their specific experiences taking a course) when asked to respond to an overall question (Gravestock, & Gregor-Greenleaf, 2008). The methodological literature shows that overall questions are prone to measurement error and are not as reliable as specific measures. (Ref)

Our pilot test gave some reason for concern about the two “overall” items originally proposed in designing draft questions (“Overall, I learned a great deal from this instructor”, and “Overall, the quality of my learning experience in this course was excellent”). These items were found to exhibit the highest potential for generating biased scores for female instructors (b=0.09/p=0.05). Additionally, student focus groups carried out in the Spring 2018 revealed that these global items failed to connect with the themes identified by students that influenced their perception of effective instruction.

Concerns

Question x is biased.

Although our team has made efforts to design set of questions that is as unbiased as possible, we recognize that the SCP survey is prone to bias just like any other survey instrument (Berk, 2013). The pilot test gave us some preliminary indication of the extent to which scores on each of the SCP questions are associated with a range of other variables. We plan to continue to monitor survey results over time to observe trends (including potential bias) and will update the questions as needed. It would be helpful if you could share your specific observations about particular questions, so that we can add them to our notes about survey design.

I’m worried that my scores will go down with the new instrument.

The University of Waterloo’s Statistical Consulting and Collaborative Research Unit compared pilot test results to regular end-of-term course evaluation results for the fall 2018 term. While the surveys are different, and so the actual numbers a given instructor receives may change, that is not what matters for assessment of teaching. The comparative analysis showed that you should not expect dramatic differences in how your scores compared to those of other instructors. If your scores were high with the old instrument, they are likely to remain so using the new instrument You are welcome to view the comparative analysis.

I am opposed to collecting student course perceptions at the University of Waterloo.

We understand that student course perception surveys are a “percolating cauldron of controversy”[1] and that not everyone supports this initiative. The director of Teaching Assessment Processes is charged with responsibility for ensuring that the SCP survey (and complementary processes) are effective and equitable, measure priorities that matter for teaching and learning at UW, address institutional policies related to teaching and learning, and that they meet the University of Waterloo’s needs to understand what is happening at an institutional level. In the meantime, we are happy to hear your reasons for opposing SCPs, so that we can consider them as a part of our due diligence while we continue to monitor results, review the literature, and consult with other Canadian institutes of higher learning.

Course evaluations are flawed and can’t be fixed.

Of course, there is no perfect instrument for assessment of teaching. We anticipate that by continuously monitoring trends, we can continue to monitor issues of bias and other associations of scores with variables beyond an instructor’s control. SCP surveys  provide an important avenue for the student voice. Having participated in a course for up to 12 weeks in a term, students are well-positioned to share their learning experience. The student voice provides an important piece of evidence to understand teaching at our institution, and it is the only feasible source of information about some important aspects of effective teaching. It is also important to note that SCP scores have not generally been, and in the future will not be, the only source of evidence used in assessment of teaching. In addition to making assessment of teaching better, having multiple sources of evidence will help to identify teaching problems, aid in the design of improvement strategies, and determine follow-up steps to improve teaching (Berk, 2006; p. 29).

We recognize that revamping this system (including peer review and dossiers) will require an immense culture shift at the University of Waterloo, which will take time. We invite you to be a part of that culture shift.

I have concerns about this project.

Please feel free to contact Sonya Buffone, TAP Director, with your questions—or invite her to speak to your group.

General queries

What is the “instrument”? Are we getting rid of Evaluate?

We will continue to use Evaluate to administer the new SCP instrument, and are in the process of rebranding the software to reflect the name of the new SCP instrument. As is common in the social sciences, we sometimes refer to the new set of survey questions as  the new “tool” or “instrument.”Since it has been drawn to our attention that this has been a source of confusion, we are trying to avoid this terminology when we can.

I’m conducting my own research on student ratings at Waterloo.

We’re so glad you’re so invested in this work! We strongly believe that the best research will be focused on our local context. As the Teaching Assessment Processes team currently includes only one full time staff member, it can be challenging to cover all the bases—as you can imagine. We hope that you will share your results with our office so that we can add them to the growing body of Waterloo-specific research on this subject.

[1] See Berk (2013) Top 10 Flashpoints in Student Ratings and the Evaluation of Teaching.