Backgrounder: Defining Teaching Effectiveness

Introduction

The Complementary Teaching Assessment Project Team (CTAPT) was formed in Winter 2018 to research and develop methods of assessing teaching and learning and to provide recommendations that are useful for both formative and summative assessment, based on empirical evidence and feedback consultations with the University of Waterloo community. The focus of CTAPT is to recommend additional methods other than student course perception surveys that should be used campus-wide. This backgrounder summarizes key findings on one of CTAPT’s research questions – “What is effective teaching?” While defining and evaluating teaching effectiveness (TE) are interconnected, this backgrounder focuses on defining TE. In the coming months, CTAPT will produce more information about tools. 
 
Grounded in an extensive literature review and environmental scan of current U15 teaching evaluation practices, the project team developed a broad definition of teaching effectiveness framed by four dimensions – Design, Execution, Experience, Development – each consisting of subdimensions describing evidence-based principles of effective teaching. These dimensions and principles have wide application across faculties and disciplines and build on existing University of Waterloo conceptions of effective teaching outlined in Policy 77 and proposed in the Undergraduate Learning: Issue Paper (May 2018). (For more information on how these dimensions connect to the literature, please see the Dimensions of Teaching Effectiveness: Links to Literature document). 
 

.

The consultation survey

CTAPT is consulting with the University of Waterloo community through the CTAPT Teaching Effectiveness Survey to gather feedback on this definition. This is the first phase of consultations. The next phase will focus on key findings on current and best practices of complementary methods of teaching assessment. Phase two of the consultation process will also illustrate how the proposed definition of TE may be incorporated into evaluation tools and instruments. 

Key findings

Shared understanding of teaching effectiveness

Recent initiatives aimed at fostering and promoting teaching quality in Higher Education have stimulated a renewed interest among scholars in reexamining how to define and measure teaching effectiveness for a variety of purposes such as annual reviews, promotion and tenure, and the improvement of teaching and learning.1 One of the essential components of a complementary or multi-faceted approach to the assessment of teaching is a clear and shared understanding of teaching effectiveness.2 One goal of CTAPT, therefore, is to recommend a definition of teaching effectiveness (TE) that would be useful for complementary methods of assessing teaching and learning. As a starting point, Policy 77 (University of Waterloo) includes a statement of effective teaching that forms the basis for a shared understanding:  “The purpose of teaching is to facilitate learning. Thus, effective teaching draws the strands of a field together in a way that provides coherence and meaning, places what is known in context, lays the groundwork for future learning, and opens the way for connections between the known and the unknown.” It further states that faculty members are expected to evaluate student work fairly, provide constructive feedback, be reasonably available for consultations, and respect the integrity, diversity, and confidentiality of students.   

Broad definition of teaching effectiveness and central features

The task of determining how to define TE has been tackled by identifying broad dimensions and sub-dimensions using a variety of theoretical perspectives and quantitative and qualitative approaches from disciplinary, teacher, and student perspectives.3 While no standardized definition of TE exists, our review of the literature confirms that there is consensus on essential characteristics. Decades of research has resulted in the identification of several central features that cut across disciplines and modes of instruction.4 Scholars also agree effective teaching is focused on student learning, which requires “alignment” facilitated by teaching skills and practices that promote “higher order learning processes” or “deep learning.”5 In a similar vein, the Undergraduate Learning: Issue Paper proposes a “broad and conceptually simple” definition from the literature of effective teaching as being “oriented to and focused on students and learning,” where learning is “a process that leads to change, which occurs as a result of experience and increases the potential for improved performance and future learning” (p2). It identifies five evidence-based principles of effective teaching relevant to the Waterloo context: “use of alignment in course and curriculum design, fosters motivation, embodies inclusivity, encourages deep learning, and enables lifelong learning” (ibid).  

Informed by this research and existing University of Waterloo policies, CTAPT developed a broad definition of teaching effectiveness framed by four central dimensions: Design, Execution, Experience, Development. Each dimension consists of sub-dimensions describing evidence-based principles of effective teaching adapted from Undergraduate Learning Issue Paper (2018), Allen et al. (2009), Bain (2004), Chickering and Gamson (1987), Hativa et al. 2001, and Ramsden (2000, 2003) that are relevant and applicable to the University of Waterloo context (see Table below): 

  1. DESIGN: Planning and Alignment  
  2. EXECUTION: Communication, Student Engagement, Variety of Elements, Assessments and Feedback  
  3. EXPERIENCE: Environment, Responsiveness, Diversity, Engagement and Learning
  4. DEVELOPMENT: Reflection, Growth, Collaboration and Mentorship  

Central features are consistently identified by different stakeholders

Studies investigating teacher and student perspectives on TE provide insight on where subdimensions are consistent or divergent; variances tend to be a matter of emphasis or a result of context. For example, studies examining teachers’ perspectives show that descriptors of “good teaching” are consistent across disciplines.6 Findings on student perspective are similar, although students place greater emphasis on the importance of the “Experience” dimension, particularly student-teacher interrelationships such as approachability, rapport, caring about students, respect, patience, trust, and enthusiasm.7 Another study reveals students’ perception of good teaching varied based on their expectation of approaches normally used in their discipline, demonstrating the importance of “context.”8 

Rationale for a broad definition of TE - consideration of context  

Conceptions of teaching effectiveness are multi-faceted and “context-bound.”9 Following Allen et al. (2009) and Devlin and Samaracwickrema (2010: 118), factors such as course parameters (size, level, etc.), mode of teaching, student ability, and disciplinary pedagogical approaches to teaching and learning are viewed as “contexts” that influence teaching effectiveness rather than a determining factor of TE in itself.10 Defining TE in a way that accounts for all these factors would be a difficult task. There are also university and department contexts, as well as wider social, political, economic, technological and demographic change forces that influence conceptions of TE (ibid). The University of Waterloo already acknowledges this rationale; Policy 77 outlines a broad conception of TE that recognizes the complexity and diversity of teaching at the University of Waterloo by stating: “because these standards are intended to apply university-wide to faculty members engaged in complex intellectual endeavors, they cannot be expressed in absolute quantitative terms.” A broad conceptualization of TE, therefore, acknowledges the need for a university-wide shared understanding of TE as one aspect of an effective and transparent evaluation framework while recognizing the complexity and diversity of teaching and learning contexts. Because some sub-dimensions would be more or less relevant depending on context, additional or more detailed sub-dimensions descriptors and weights should be refined at the faculty and/or department level by considering context, faculty values, and discipline specific practices.11  

Design  Execution Experience Development

Planning 

- Builds course around evident design framework

- Clearly defines course level and activity-level learning objectives

- Includes learning material that is relevant and/or current scholarship from the field

- Structures material in a logical and coherent order

- Includes experiential and/or professional connections, when possible

- Sets workload and performance standards appropriate for the course level and topic

- Plans a variety of teaching/learning strategies to promote student engagement and deep learning 

Alignment

- Aligns program expectations, course objectives and learning outcomes with course content, delivery of course and learning activities, and assessments

- Develops appropriate and sufficient assessment methods that align with course objectives and outcomes 

Communication 

- Describes and explains material clearly using an appropriate pace

- Demonstrates enthusiasm for the subject

- Uses technology and/or media effectively

Student Engagement 

- Promotes student participation, peer interactions, and engagement with course content 

- Uses teaching/learning strategies that encourage student engagement and deep learning 

Variety of Elements 

- Adapts to evolving classroom contexts 

- Adopts a range of teaching strategies, methods, and/or modalities that recognize diverse talents and ways of learning 

Assessments and feedback 

- Prepares students for assessments by delivering content

- Communicates clear expectations and instructions for assessments

- Provides performance feedback in a timely manner

- Provides directions for student improvement individually or collectively

Environment 

- Is approachable

- Builds appropriate rapport with students 

Responsiveness 

- Provides opportunities for student contact inside and outside of class

- Responds to student inquiries and questions in an appropriate timeframe 

Diversity 

- Promotes inclusivity and diversity by acknowledging variety of experiences, viewpoints, and backgrounds 

Engagement and Learning 

- Generates and maintains student interest

- Fosters intrinsic motivation 

- Seeks student input on course learning experience

- Provides evidence of student learning 

Reflection

- Reflects on and assesses teaching and learning practices 

- Engages in a scholarly approach to teaching and learning through determining and implementing best practices 

- Takes an iterative approach to continuous improvement and growth. 

Growth 

- Participates in professional development activities

- Makes changes to practices and/or develops innovations

- Regularly revises or updates course content, assessments, format, or teaching strategies 

Collaboration and Mentorship

- Leads activities to promote teaching and learning

- Interacts and works with colleagues around teaching

- Provides and receives mentorship related to teaching 

- Contributes to the Scholarship of Teaching and Learning (SoLT) 

Bibliography

Allan, Jo, Karen Clarke, and Michael Jopling. 2009. “Effective Teaching in Higher Education: Perceptions of First Year Undergraduate Students.” International Journal of Teaching and Learning in Higher Education 21 (3): 362–72.

Arreola, Raoul A. 1995. Developing a Comprehensive Faculty Evaluation System: A Handbook for College Faculty and Administrators on Designing and Operating a Comprehensive Faculty Evaluation System. Anker Pub. Co.

Asarta, Carlos J, Regina Bento, Charles J Fornaciari, Kathy Lund Dean, Ben Arbaugh, Alvin Hwang, and Kathy Lund Dean. 2018. “The Scholarship of Teaching and Learning: Changing the Dominant Narrative About (and In) Research Institutions.” Journal of Management Education 42 (6): 731– 48.

Bahia, Sara, Isabel P Freire, Maria Teresa Estrela, Anabela Amaral, José António, and Espírito Santo. 2017. “The Bologna Process and the Search for Excellence: Between Rhetoric and Reality, the Emotional Reactions of Teachers.” Teaching in Higher Education 22 (4): 467–82.

Bartram, Brendan, Tanya Hathaway, and Namrata Rao. 2018. “‘Teaching Excellence’ in Higher Education: A Comparative Study of English and Australian Academics’ Perspectives.” Journal of Further and Higher Education, July, 1–15.

Berk, Ronald A. 2005. “Survey of 12 Strategies to Measure Teaching Effectiveness.” International Journal of Teaching and Learning in Higher Education 17 (1): 48–62.

Carnell, Eileen. 2007. “Conceptions of Effective Teaching in Higher Education: Extending the Boundaries.” Teaching in Higher Education 12 (1): 25–40.

Cassuto, Leonard. 2014. “Can We Create a Culture That Values Good Teaching? We like to Talk about the Value of Pedagogy, but We Never Seem to Get around to Rewarding It.” The Chronicle of Higher Education 60 (37).

Chickering, Arthur W., and Zelda Gamson. 1988. Principles of Good Practice in Undergraduate Education.

Chism, Nancy. 1999. Peer Review of Teaching: a Sourcebook. Anker Publishing Co.

Crookes, Patrick A., Steve Outram, and Fabienne C. Else. 2018. “Developing a Collaborative Pilot Programme to Promote Innovation in Higher Education Focusing on the Reward and Recognition of Teaching.” Journal of Further and Higher Education 42 (8): 1156–66.

Devlin, Marcia, and Gayani Samarawickrema. 2010. “The Criteria of Effective Teaching in a Changing Higher Education Context.” Higher Education Research & Development 29 (2): 111–24.

Díaz‐Méndez, Montserrat, and Evert Gummesson. 2012. “Value Co‐creation and University Teaching Quality.” Edited by Evert Gummesson. Journal of Service Management 23 (4): 571–92.

Gan Joo Seng, Mark, and Johan Geertsema. 2018. “Sharing Practices, but What Is the Story? Exploring Award-Winning Teachers’ Conceptions of Teaching.” Higher Education Research & Development 37 (2): 254–69.

Gourlay, Lesley, and Jacqueline Stevenson. 2017. “Teaching Excellence in Higher Education: Critical Perspectives.” Teaching in Higher Education 22 (4): 391–95.

Hativa, N., R. Barak, and E. Simhi. 2001. “Exemplary university teachers: Knowledge and beliefs regarding effective teaching dimensions and strategies.” Journal of Higher Education, 72 (6), 699–729.

Hill, Lilian H. 2014. “Graduate Students’ Perspectives on Effective Teaching.” Adult Learning 25 (2): 57– 65.

Knol, Mariska H, Conor V Dolan, Gideon J Mellenbergh, and Han L J van der Maas. 2016. “Measuring the Quality of University Lectures: Development and Validation of the Instructional Skills Questionnaire (ISQ).” PLOS ONE 11 (2):

Magin, D J. 1998. “Rewarding Good Teaching: A Matter of Demonstrated Proficiency or Documented Achievement? The International Journal for Academic Development 3 (2): 124–35.

Nabaho, Lazarus, Joseph Oonyu, and Jessica Norah Aguti. 2017. “Good Teaching: Aligning Student and Administrator Perceptions and Expectations.” Higher Learning Research Communications 7 (1): 16 pp.

Oravec, Jo Ann. 2017. “The Manipulation of Scholarly Rating and Measurement Systems: Constructing Excellence in an Era of Academic Stardom.”
Parpala, Anna, Sari Lindblom‐Ylänne, and Henna Rytkönen. 2011. “Students’ Conceptions of Good Teaching in Three Different Disciplines.” Assessment & Evaluation in Higher Education 36 (5): 549–63.

Ramsden, Paul. 2003. Learning to Teach in Higher Education. RoutledgeFalmer.

Samples, Jerry W, and Susan E Copeland. 2013. “The Universality of Good Teaching: A Study of Descriptors Across Disciplines.” International Journal of Teaching and Learning in Higher Education 25 (2): 176–88.

Saroyan, Alenoush, and Keith Trigwell. 2015. “Higher Education Teachers’ Professional Learning: Process and Outcome.” Studies in Educational Evaluation 46: 92–101.

Saunders, Daniel B, and Gerardo Blanco Ramírez. 2017. “Against ‘Teaching Excellence’: Ideology, Commodification, and Enabling the Neoliberalization of Postsecondary Education.” Teaching in Higher Education 22 (4): 369–407.

Seijts, Gerard, Lynn Taylor, and Gary Latham. 1998. “Enhancing Teaching Performance through Goal Setting, Implementation and Seeking Feedback.” The International Journal for Academic Development 3 (2): 156–68.

Sherman, Thomas M., L. P. Armistead, Forest Fowler, M. A. Barksdale, and Glenn Reif. 1987. “The Quest for Excellence in University Teaching.” The Journal of Higher Education 58 (1): 66.

Skourdoumbis, Andrew. 2017. “Assessing the Productivity of Schools through Two ‘What Works’ Inputs, Teacher Quality and Teacher Effectiveness.” Educational Research for Policy and Practice 16: 205–17.

Smith, Shelley L. 2014. “Peer Collaboration: Improving Teaching through Comprehensive Peer Review.” To Improve the Academy 33 (1): 94–112.

Trigwell, Keith, Michael Prosser, Michael and Fiona Waterhouse. 1999. “Relations between teachers' approaches to teaching and students' approaches to learning.” Higher Education 37 (1): 57-70.

Wieman, Carl. 2015. “A Better Way to Evaluate Undergraduate Teaching, Change: The Magazine of Higher Learning” 47 (1): 6–15.

Wieman, Carl, and Sarah Gilbert. 2014. “The Teaching Practices Inventory: A New Tool for Characterizing College and University Teaching in Mathematics and Science.” Edited7 by Erin Dolan. CBE—Life Sciences Education 13 (3): 552–69.

Wright, Alan, Beverley Hamilton, Joy Mighty, Jill Scott, and Bill Muirhead. 2014. “The Ontario Universities’ Teaching Evaluation Toolkit: Feasibility Study.” Centre for Teaching and Learning Reports, June. https://scholar.uwindsor.ca/ctlreports/4.

Wood, Margaret, and Feng Su. 2017. “What Makes an Excellent Lecturer? Academics’ Perspectives on the Discourse of ‘teaching Excellence’ in Higher Education.” Teaching in Higher Education 22 (4): 451–66.



Endnotes

1 See Donna Ellis https://cte-blog.uwaterloo.ca/?p=4166 and University of Waterloo Strategic Plan in Action 2013-2018, and Bridge to 2020 Issue Papers. Internationally, the United Kingdom (UK) and Australia have led the way in terms of conducting studies to gather evidence (e.g. Bahia 2017; Crookes et al. 2018; Dias-Mendez et al. 2012; Gourlay and Stevenson 2017; Wood and Su 2017). Debates about standardizing definitions of TE is felt strongly in UK because of 2015 Green Paper and 2016 white paper, which in introduced assessment of teaching at the HEI level through the Teaching Excellence Framework that would allow institutions to increase fees depending on results (Gourlay and Stevenson 2017). Critiques of UK Model include criticism of the notion of ‘excellence’ itself and neoliberal approach, which views students as customers and uses a marketing model for HEI (see Bahia 2017; Bartram et al. 2018; Saunders et al. 2017). On the other hand, a climate of accountability and focus on quality enhancement in the UK also led to the establishment of 74 Centres for Teaching and Learning, with the purpose of recognizing and rewarding excellence in HE and promote development (Allan et al. 2009: 362).

2 Arreola 1995; Chism 1999; Smith 2014; Wright et al. 2014.

3 Devlin and Samarawickrema 2010: 112.

4 See for example Allan et al. 2009: 364; Bain 2004; Bartram et al. 2018: 2, 5; Hill 2014; see also Harris 1998.

5 For example Biggs 2003; Bain 2004; Devlin and Samarawickrema 2010: 112; Parpala et al .2011; see also Ramsden 2003; Trigwell, Prosser, and Waterhouse 1999.

6 See studies by Samples and Copeland 2013, and Bartram et al. 2018, for example. Samples and Copeland’s large mixed method study comparing perspectives of Engineering academics with a wider university sample across faculties showed that essential descriptors defining good teaching where consistent across both samples (2013: 181), echoing findings from other studies (e.g. Bain 2004 and Lowman 1995). Bartram et al.’s (2018) mixed methods study compared English and Australian academic views of teaching excellence and found views to be consistent with both samples.  

7 Allan et al. 2009: 366, 368 and Hill 2014: 58.

8 Parpala et al. 2011: 560-1.

9 Bartram et al. 2018; Biggs 2003; Parpala et al. 2011: 3.

10 For example, characteristics such as “includes interactions” or “includes group work” would depend on course context, and thus it would be problematic to standardize such a characteristic across disciplines (Parpala et al. 2011: 557-8).

11 Arreola 1995: 1; Berk 2005; Wright et al. 2014: 14.