CTAPT Teaching Effectiveness Survey Results

Introduction

CTAPT spent several months developing a literature- and evidence-based definition of teaching effectiveness framed by four dimensions – Design, Execution, Experience, Development (DEED).
Two documents posted on the CTAPT webpage provide details of this definition and outline key findings from our research:
- Backgrounder: Defining Teaching Effectiveness.
- Dimensions of Teaching Effectiveness: Links to Literature.
Following a review by Institutional Analysis and Planning (IAP), CTAPT invited the University of Waterloo community to provide feedback on the proposed definition of teaching effectiveness through a survey designed on Qualtrics.
In accordance with CTAPT’s mandate to consult widely, we sent all campus stakeholders (Faculty, Staff, Graduate, and Undergraduate Students) a call to participate in this consultation survey from April 17 – May 10, 2019. In addition, seven (n=7) Faculty members attended a 90 minute, in-person consultation session held on May 15th, 2019.

Methodology

The survey described each of the four dimensions of teaching effectiveness on a separate page (questions one to four) and asked participants: “Based on your teaching and/or learning experience, do you believe anything is MISSING from [this] dimension of teaching effectiveness described below? If so, please specify and explain why in the space provided.”
Question five (Q5) was an open-ended question: “Is there anything else you would like to add?”
Responses from the anonymous survey were imported into NVivo for coding and analysis.
Discussion from the in-person session was anonymized, imported into NVivo, and aggregated with the survey responses.i
The researcher conducted preliminary coding and analysis, which involved an initial reading of all responses for each question (Q1 - Q5), and coding the responses based on the type of feedback and content received.ii The researcher then analyzed each response type using queries to identify initial themes. Using an inductive thematic analysis approach, the researcher continued coding and analysis through a process of querying, synthesizing, and grouping data sets to identify and verify the prevalent themes in the data.
While some of the thematic nodes naturally corresponded with the sub-dimensions of teaching effectiveness, the researcher also identified new themes and additions.

Who responded to the survey?

In total, 526 people completed the CTAPT Teaching Effectiveness Survey. 23% of those respondents are faculty members, 43% are undergraduate students (see Table 1).
The three largest faculties at the University of Waterloo – Arts, Engineering, and Mathematics – have the greatest number of respondents (see Table 2).
Based on the University of Waterloo 2018 Count, the response rate for Faculty members is approximately 10%. The breakdown of Faculty member participation by faculty affiliations is more or less representative of UWaterloo demographics for faculty members (see Table 3).

Table 1: Percentage and number of respondents by role

Respondents	%	Count
Faculty	23%	122
Staff	16%	82
Graduate student	17%	89
Undergraduate student	43%	228
Other	1%	5
Total	100%	526

Table 2: Percentage and number of respondents by faculty affiliation (all roles)

Faculty	%	Count
Applied Health Sciences	10%	51
Arts	22%	116
Engineering	22%	114
Environment	7%	39
Mathematics	17%	90
Science	15%	79
Other	2%	12
No faculty	5%	25
Total	100%	526

Table 3: Faculty participation numbers and percentage based on University of Waterloo count, 2018

Faculty	University of Waterloo count	University of Waterloo %	Survey count	Survey %
Applied Health Sciences	79	6%	11	9%
Arts	336	26%	36	30%
Science	204	16%	15	12%
Mathematics	268	20%	30	25%
Environment	89	7%	7	6%
Engineering	335	26%	21	17%
Other	n/a	n/a	2	2%
All Faculties	1311	100%	122	100%

Survey results

Overview of findings

We received 607 responses to questions 1 to 4 (Q1 – Q4) and 167 responses to question number five (Q5), for a total of 774 comments ranging in length from a few sentences to 1-2 paragraphs.
Overall, responses to the question “what is missing?” suggest additions that are present to varying degrees in the literature. Some responses relate to the University of Waterloo context; in particular, comments in reference to the Development dimension, and the themes of “accessibility,” “diversity,” and “implementation.”
Comments from the in-person session echoed survey results, with some additional suggestions for the Development dimension and comments related to implementation and standards.
Approximately 2/3 of respondents said they had nothing to add to the Design, Execution, and Experience dimensions (Q1 – Q3); 75% of respondents said they have nothing to add to the Development dimension (Q4).
There were also 67 mentions of thanks or agreement with the dimensions and/or the overall scheme of teaching effectiveness proposed.
Based on an analysis of 774 responses, the proposed definition of teaching effectiveness appears to be on track. The central themes identified from the survey are:

Design (n=161)

Four prevalent themes emerged:

Reference to ‘real world’ application of content
Aligning a course with sequential courses
Planning for accessibility
Course / time management

Execution (n=192)

Main suggestions for additions or modifications:

Adding a bullet about communicating objectives
Add to “Variety of Elements” variety of assessment and content forms
Clarify that feedback should useful and actionable.
Some respondents expressed concern with the phrasing “delivering content” and with the emphasis on the use of technology

Experience (n=158)

Main concerns and suggestions:

“Environment” sub dimension is too vague
Concern with potential interpretations of “approachable” and “appropriate”
Many also said the “diversity” bullet does not go far enough.

Development (n=96)

Additions and clarification suggested:

Include an awareness of emerging and evidence-based practices
What about curriculum and committee work, conferences, grants, and awards related to teaching?
Many stated that “making changes” to practices and courses should be purposeful and based on reflection and/or feedback
Expectations for SoTL?

Is there anything you would like to add? (n=167)

Responses are included in the findings outlined above. Apart from that, 65 comments, the majority from Faculty, relate to questions about how this conception of teaching effectiveness will be implemented. Main themes:

The need for an institutional culture that values and rewards teaching.
Who is qualified to assess these dimensions and how? How does this relate to annual reviews and Promotion and Tenure procedures?
Will there be support, training, and resources if needed? Will you provide examples?
How does this relate to the student perception surveys?

Response from CTAPT

CTAPT thanks everyone who participated in the Teaching Effectiveness Survey and in-person session for their time and suggestions.
We heard your feedback! CTAPT reviewed results from the survey and in-person session and used the findings to help refine the proposed definition of teaching effectiveness. The finalized definition of teaching effectiveness is presented in Table 4A,B and C (below).
The first column describes the original definition proposed in the survey; the second column outlines the finalized definition of teaching effectiveness based on comments from the campus consultation, which are summarized in the third column. The last column provides information on how CTAPT responded to comments and feedback. (Please note, the use of “or” in the bullet lists is inclusive and means “and/or”).
As noted in the Backgrounder, this is a broad definition of teaching effectiveness that cuts across disciplinary boundaries and modes of instruction; however, teaching effectiveness is also “context bound.” Therefore, context is an important factor for consideration, as not every item will necessarily apply to every context (e.g. co-instructors may have little input in “Design”).
To clarify, “scholarly approach to teaching” refers to the process of consulting the literature and peers on teaching methods, applying this knowledge and disciplinary approaches to guide teaching learning, and then reflecting on your teaching practices and outcomes. In comparison, the “Scholarship of Teaching and Learning” involves a systematic examination and analysis of research questions about student learning and teaching activities that are shared publicly with the aim of improving learning, strengthening teaching, and advancing the field of teaching and learning.

What’s Next?

In the fall of 2019, CTAPT will be moving forward with aligning the revised definition of teaching effectiveness with complementary methods for the assessment of teaching. At that time, we will be asking for feedback from key stakeholders at the University of Waterloo through another consultation process (Phase 2).
Updates and further communications will be posted on the CTAPT webpage. Stay tuned!

Table 4A: Refining and finalizing the definition of teaching effectiveness - DESIGN

Original description	Finalized description	Comments from campus	Response from CTAPT
Planning	Planning	n/a	n/a
Builds course around evident design framework	Builds course around a design framework from the literature, from the field of study, or created by the course author	Clarify “evident design framework”	Clarified
Clearly defines course-level and activity-level learning objectives	Clearly defines course-level and activity-level learning objectives	n/a	n/a
Includes learning material that is relevant and/or current scholarship from the field	Includes learning material that reflects current scholarship from the field or that is clearly relevant	n/a	n/a
Structures material in a logical and coherent order	Structures material in a logical and coherent order	n/a	n/a
Sets workload and performance standards appropriate for the course level and topic	Sets pacing, workload and performance standards appropriate for the course level and topic	Include something about time management	Included
Includes experiential and/or professional connections, when possible	Includes experiential components, professional connections, or practical applications, when possible.	Include “real world” applications and skills	Included “practical applications”
Plans a variety of teaching/learning strategies to promote student engagement and deep learning	Plans a variety of teaching/learning strategies to promote student engagement and deep approaches to learning	Clarify “deep learning”	“Deep learning” is defined in the teaching and learning literature, but we have used a variation here for clarity
n/a	Incorporates a diversity of experiences, viewpoints, and backgrounds in course materials	n/a	NEW bullet: Diversity is important in design as well as in execution (from comments on EXPERIENCE)
n/a	Adheres learning materials, activities, and assessments to University accessibility policies	Include something about accessibility	Included

Table 4B: Refining and finalizing the definition of teaching effectivEness - EXECUTION

Original description	Finalized description	Comments from campus	Response from CTAPT
Communication	Communication	n/a	n/a
	Communicates course-level and activity-level objectives as well as teaching/learning approach and rationale to students	Include something about communicating objectives and approach	Included
Describes and explains material clearly using an appropriate pace	Describes and explains material clearly using a pace appropriate to the context	Clarify “appropriate”	Clarified
Demonstrates enthusiasm for the subject	Demonstrates enthusiasm for the subject	Include “subject knowledge”	After considerable discussion, CTAPT decided to leave “subject knowledge” out. We feel that the focus of this definition is on teaching specifically; subject expertise is considered at hiring and/or teaching assignments.
Uses technology and/or media effectively	Uses technology, media or other teaching tools effectively	No or low technology might be best.	CTAPT agrees that this might be the case in some situations, and intends that this be included in “uses…effectively”.
Student engagement	Student engagement	n/a	n/a
Promotes student participation, peer interactions, and engagement with course content	Promotes student participation, peer interactions, or other active engagement with course content	n/a	n/a
Uses teaching/learning strategies that encourage student engagement and deep learning	Uses teaching/learning strategies that encourage student engagement and deep approaches to learning	n/a	"Deep learning" clarified as above
Variety of elements	Variety of elements	n/a	n/a
Adapts to evolving classroom contexts	Adapts to evolving classroom contexts	n/a	n/a
Adopts a variety of teaching/learning strategies, methods, and/or modalities that recognize diverse talents and ways of learning	Adopts a variety of instructional practices, content types, and assessments that recognize diversity of learners Assessments	Include varied forms of content and assessments for different ways of learning.	Included above in DESIGN.
Assessments and feedback	Assessments and feedback	n/a	n/a
Prepares students for assessments by delivering content	Enables students to prepare for assessments through instructional practices	There are other ways to prepare students besides “delivering content”	Adapted to correct this and to capture students’ responsibility in this
Communicates clear expectations and instructions for assessments	Communicates clear expectations and instructions for assessments	Include “fairness” of assessments	See DESIGN above.
Provides performance feedback in a timely manner	Provides performance feedback in a timely manner	n/a	n/a
Provides directions for student improvement individually or collectively	Provides directions for student improvement individually or collectively	Include that feedback needs to be actionable	CTAPT believes “Provides direction for student improvement” captures this

Table 4C: Refining and finalizing the definition of teaching effectiveness - STUDENT EXPERIENCE

Original description	Finalized description	Comments from campus	Response from CTAPT
Environment	Rapport	Whose “experience”? What does “Environment” mean?	Clarified
n/a	Fosters a supportive learning environment	n/a	n/a
n/a	Establishes a climate of intellectual openness	n/a	n/a
Is approachable	Shows concern for students' success and wellbeing	What about respect? Shows concern or care? What about gender and racialized biases?	CTAPT has found what it believes to be the best middle ground here.
Builds appropriate rapport with students	Interacts professionally and respectfully with students	What is appropriate? Maintains professional boundaries, respectful, courteous? What about student responsibilities?	CTAPT has found what it believes to be the best middle ground here. Captured in Assessments and Feedback
Responsiveness	Responsiveness	n/a	n/a
Provides sufficient opportunities for student contact inside and outside of class	Provides sufficient opportunities for student contact inside and outside of class	n/a	n/a
Responds to student inquiries and questions in an appropriate timeframe	Responds to student inquiries and questions in an appropriate timeframe	How one responds equally important	Addressed above with “Interacts professionally and respectfully”
Diversity	n/a	n/a	n/a
Promotes inclusivity and diversity by acknowledging variety of experiences, viewpoints, and backgrounds	Promotes inclusivity and diversity by acknowledging variety of experiences, viewpoints, and backgrounds	There could be more added to this, e.g. inclusive scholarly contributions in course design	See NEW point about diversity in DESIGN
Engagement and learning	Engagement and learning	n/a	n/a
Generates and maintains student interest	Generates and maintains student interest	n/a	n/a
Fosters intrinsic motivation	Fosters students' intrinsic motivation and responsibility for their own learning	n/a	n/a
Seeks student input on course learning experience	Seeks student input on course learning experience	Include “responds” to student input	This is captured under DEVELOPMENT
Provides evidence of student learning	Provides evidence of student learning	What evidence and to whom?	This may be easier to provide in some cases than others (e.g. clear evidence from success in follow-up course); not all items will apply in all circumstances

Table 4D: Professional development

Original description	Finalized description	Comments from campus	Response from CTAPT
Reflection	Reflection	n/a	n/a
Reflects on and assesses teaching and learning practices	Reflects on and assesses teaching and learning practices	n/a	n/a
Engages in a scholarly approach to teaching through determining and implementing best practices	Engages in a scholarly approach to teaching through determining and implementing best practices	Include something about awareness of emerging best practices, staying up to date, and using evidence based practices	This is included by implication in a “scholarly approach”
Takes an iterative approach to continuous improvement and growth.	n/a	n/a	Removed for overlap below
Growth	Growth and continuous improvement	n/a	n/a
Participates in professional development activities	Participates in professional development activities	What about curriculum work, course design, conferences, grants, awards, etc.?	Included by implication in “professional development activities"
Makes changes to practices and/or develops innovations	Makes thoughtful and deliberate changes to practices or develops innovations in response to new information about best practices or to other opportunities as they arise	"Makes changes" for change’s sake is not a positive in itself	Changed to reflect thoughtfulness required
Regularly revises or updates course content, assignments, format, or teaching strategies	Regularly revises or updates course content, assignments, format, or teaching strategies in response to feedback and reflection	Changes should be based on feedback or reflection	Included
Collaboration and mentorship	Collaboration, mentorship, and leadership	n/a	n/a
Leads activities to promote teaching and learning	Demonstrates leadership related to teaching and learning within the University and in the broader educational community	Demonstrates leadership related to teaching and learning within the University and in the broader educational community	Adjusted
Interacts and works with colleagues around teaching	Interacts and works with colleagues around teaching and learning	Include “teaching assistants”	Included in next bullet
Provides and receives mentorship related to teaching	Provides and receives mentorship related to teaching, including with teaching assistants	n/a	Adjusted
Contributes to the Scholarship of Teaching and

ⁱ Note: Comments from the in person session were coded by theme and included in number of thematic mentions, but not included in overall number of responses by dimension (i.e. all comments about the Design dimension were coded as one “Design” response).

ⁱⁱ Preliminary coding and analysis resulted in a compilation of coded data (or nodes) to be included and excluded from analysis, based on content type. Inclusions include all data related to suggestions for additions, modifications, concerns, implementation concerns, and agreement and thanks. Data reiterating existing bullets, personal experiences and opinions, comments outside of the scope of this project, and comments providing context specific descriptions were not included in number of mentions. Note: context specific examples may prove useful in the next stage of consultations.