Consultations Phase 2: Summary of Findings

Introduction

The literature review and environmental scan conducted by CTAPT both indicate that the University of Waterloo should adopt the use of Teaching Dossiers (TD) and Peer Review of Teaching (PRT) across campus as complementary methods in addition to student course perception surveys (SCPSs) to assist in the formative and summative assessment of teaching.
The objective of Phase 2 of CTAPT consultations was to consult with faculty members about the tools, mechanisms and conditions needed in order to implement and support ongoing use PRT and TD.
CTAPT will use the findings from consultations to help inform CTAPT recommendations to Senate.

Who participated in the consultations?

169 participants in total, which is approximately a 13% faculty response rate.
72 faculty members participated in the in-person sessions and 97 completed the online survey.
High participation rates (survey and in person combined) from the Faculties of Mathematics (32% of participants), Science (20%) and Engineering (20%).
Survey only: The majority of survey respondents (69%) are Tenured / Continuing Faculty; 13% of survey respondents are Chairs/Directors.

What did participants say?

Findings from Phase 2 of consultations (in person and online survey) fall into three main thematic categories:

Value and Impact – Usefulness of complementary methods
Procedures and Guidelines – Clarity on using complementary methods
Resources – Needs for and feasibility of implementing complementary methods

The themes that emerged from consultations are consistent with the literature and provide information on needs and concerns from faculty members’ perspective. The frequency of mentions are provided in brackets.
Survey results are described below in the left column. We have included CTAPT responses to some of the items in the right column.
Thank you for your time and input to work towards improving teaching and the assessment of teaching at Waterloo!

1. Value and impact: Usefulness of complementary methods

Survey results	CTAPT response
More agreement than disagreement with this direction (32), as well as support for establishing use of these methods for formative purposes or “when needed” (20).	n/a
Others are concerned with push back or apathy from those who might not care or see the value in these methods (20).	n/a
Need to outline and realize the benefits and value of using PRT and TD (28) – will using these methods improve teaching?	The literature shows that these methods do help improve teaching. CTAPT completely agrees that whatever is implemented must be useful to improve teaching.
Concern with biases and/or subjectivity (including possible increased vulnerability of marginalized groups); need to mitigate or minimize this (38).	We recognize that this is one of the potential barriers to implementation that needs to be addressed.
Need “formal” recognition and incentives for the added work that peer reviewers will undertake (40), such as service or teaching credit or establish as a service role.	We agree.

2. Processes and guidelines: Clarity on using complementary methods

Survey results	CTAPT response
Need University commitment and leadership (20) from the top down to ensure that these methods are taken seriously, and to ensure supports for ongoing professional development are in place.	We agree.
Need clear and transparent procedures (37), with safeguards to ensure equity.	We agree.
Clarification on frequency and timing of PRT and TD (31) - how often and when?	Implementation details will be addressed in our recommendations to Senate.
There were also concerns about how to give feedback to peers (22).	Peer reviewers will need careful and thoughtful training, which will help with this.
Purpose/use (21): clarify how these methods will be used – for formative, summative, or both? Participants also asked why we are doing this – if we are doing this to improve teaching then time to gain skills is needed.	The intention is that complementary methods be useful for both formative and summative feedback.
Participants overwhelmingly stated that they need Guidelines (45) for developing a TD and conducting PRT, as well as guidelines on using the evidence for assessment (either formative or summative).	We agree.
Criteria (23): Need to know what could and should be assessed, noting that there should be recognition of different teaching contexts and disciplinary approaches. - Outline the weight of each method, and how to turn evaluation of PRT and TD into a number (9).	We agree. The question about how to translate information gained through PRT, TD and SCPSs into Waterloo’s summative assessment process is an important one that will be addressed. CTAPT does not anticipate assigning a “weight” to each method, but anticipates using our definition of teaching effectiveness as a rubric to help guide this translation.

3. Resources – Needs and feasibility of implementing complementary methods

Survey results	CTAPT response
Participants made it very clear that training (52) would be a necessity - training on how to conduct peer observations and use tools, how to give feedback, how to evaluate a TD or use PRT evidence.	We agree.
Participants suggested similar types of tools or mechanisms (58) that would need to be in place in order to implement these methods, in particular rubrics, PRT observation tools, and TD templates and examples.	CTAPT will recommend the development of tools based on the needs identified in consultations.
Tools should be consistent or standardized (17) but also flexible (19) to accommodate different course context and teaching approaches; tools should also be simple (13) and easy to use.	We agree.
The University will need to commit to investing in people resources (48) for PRT to be feasible in both large and smaller units.	We agree.
Participants had some questions or suggestions about who should be doing PRT: • Participants said reviewers should be ‘qualified’ expert or trained, motivated, experienced and knowledgeable of teaching practices. • Many suggested having 2 reviewers (content expert and teaching expert) would be beneficial, whether they be internal or external (e.g. CTE or cross-discipline), while others specified reviewers should be external to their unit.	CTAPT will recommend training be in place.
Time constraints (36) represents another potential barrier to implementation, in particular the time needed to do PRT properly. - Many stated that they either “just don’t have time” or “would need to have time allocated” to use these methods. - Others were concerned about the increased workload (19). - Some suggested hiring support staff (11) to help coordinate processes, scheduling, flow of documents, and support the Chair.	We agree that resourcing will be important and will be addressing this in our recommendations to Senate.

Next steps

Based on the findings from the literature review, environmental scan, and two rounds of consultations with key stakeholders, CTAPT is preparing a report for Senate, which we hope to submit by early 2020. We will continue to share updates and news through appropriate university communication channels and our webpage: Complementary Teaching Assessment Project Team.

Methodology and method

CTAPT produced a backgrounder document, which contains a summary of key research findings on assessing teaching, PRT and TD “fact sheets,” and a table mapping the three assessment methods to the definition of teaching effectiveness developed by CTAPT. This information was posted on our webpage and provided to participants for review in advance of the consultation.
CTAPT provided two ways to participate in consultations: 1) Complete an online survey available from October 7, 2019 to October 20, 2019, or 2) Attend an in-person session – six Faculty-specific sessions and one session for the Affiliated and Federated Institutions of Waterloo took place between October 7, 2019 and October 15, 2019.
Discussions from the in-person sessions were anonymized. Each session was read as a complete document with attention to comments for each method; the researcher conducted preliminary analysis using an inductive approach to code comments in addition to coding to the literature to create a preliminary list of topics.
Survey responses (Qualtrics) and in-person consultations were imported to NVivo and analyzed in aggregate. The researcher reviewed all responses by method and question, and coded comments based on the preliminary list of topics; new thematic nodes were added as they emerged.
The researcher analyzed data through a process of examining individual thematic nodes and using queries to identify prevalence and links, then reorganized and grouped nodes into high level themes. Synthesized findings were presented to the CTAPT committee in a report for review and discussion.