Grant recipients:
Judene Pretti, Waterloo Centre for the Advancement of Co-operative Education (left)
Bob Sproule, School of Accounting and Finance (centre)
David Drewery, Waterloo Centre for the Advancement of Co-operative Education (right)
(Project timeline: May 2018-April 2019)
Description
Many programs in Canada aim to develop students into “lifelong learners”. Lifelong learners are those who engage in learning opportunities throughout their lives. Assessing lifelong learners’ characteristics is important to implementing activities that promote lifelong learning. Previous assessments of lifelong learning rely on students’ self-reports. The goal of this study is to advance a rubric that could be applied to students’ written reflections to assess lifelong learning. Reflections provide a “window” into students’ learning experiences and so present opportunity to assess lifelong learning. Advancing this rubric provides an innovative approach to the assessment of lifelong learning characteristics which have been previously tied to students’ deep learning.
Research Questions
The purpose of this study was to develop a rubric for the assessment of students’ lifelong learning characteristics via students’ written reflections, particularly those following a work-integrated learning (WIL) experience. We sought to address the following questions:
(1) what are the characteristics of a lifelong learner?
(2) how might those characteristics be assessed using a rubric that is applied to students’ post-work term reflections?
(3) is such a rubric valid and reliable for use by assessors, such as course instructors?
These questions guided our study which ultimately informs our understanding and assessment of lifelong learning – an increasingly important learning outcome in post-secondary education settings. In particular, the study advances the ways in which we might develop students within the School of Accounting and Finance at the University of Waterloo into lifelong learners.
Findings/Insights
Our research has identified five characteristics of lifelong learners: (1) curiosity, (2) initiative, (3) transfer, (4) resilience, and (5) reflection. These five dimensions were identified using three complementary methods. First, we derived an initial set of characteristics from a review of the extant literature on the topic of lifelong learning. Second, we conducted a focus group with exemplary students, ones whom others might define as lifelong learners. We asked these students to define and describe lifelong learning and the characteristics of lifelong learners. Third, we surveyed a select group of course instructors on the topic of lifelong learning. These instructors were thought to have a useful perspective on the meaning of lifelong learning and on the characteristics that make students lifelong learners. Instructors were asked to identify the key characteristics of lifelong learners. All three methods produced a similar set of descriptions of lifelong learners which are encapsulated in the five aforementioned characteristics.
We incorporated these five characteristics into a rubric in order to better understand whether such a rubric could provide a useful approach to assessing students’ lifelong learning characteristics. The rubric (Appendix A, attached) consists of rows representing each of the five characteristics of lifelong learners and columns denoting the degree to which students’ written reflections contain evidence of each characteristic, on a scale from Level 1 to Level 4. This framework was adapted from the Association of American Colleges and Universities VALUE Rubric pertaining to Lifelong Learning. Our goal was to determine whether the rubric was valid and reliable for use. Reliability was assessed in terms of inter-rater reliability. Inter-rater reliability refers to the degree to which multiple raters would assign consistent scores to students using the rubric. Establishing inter-rater reliability is important because it suggests that the rubric provides unambiguous guidance to assessors regardless of who that assessor is. Four raters applied the rubric to a sample of students’ reflections. Results suggested that inter-rater reliability across the assessors was adequate for all five dimensions of the rubric and for an aggregate score. This suggests that the rubric has acceptable inter-rater reliability.
Validity of the rubric was assessed in two ways. First, we assessed face validity of the rubric by presenting it to SAF students and course instructors and asking those individuals to comment on where the rubric adequately represents their conceptualization of a lifelong learner. Participants reviewed the rubric, commented on the appropriateness of the proposed dimensions, and commented on whether any dimensions were missing. Responses indicated that the rubric was intuitive and inclusive of the general notion of lifelong learning held by participants. This provided initial evidence of the face validity of the rubric.
Second, the validity of the rubric was tested in terms of criterion validity. Criterion validity refers to the degree to which scores obtained with the rubric are positively associated with others scores that would be expected to correlate with the rubric. Our focus has been on two sources of such scores: (1) students’ self-reported lifelong learning characteristics, and (2) supervisors’ assessments of students’ performance during a WIL experience. Preliminary results with a small sample are inconclusive. Results suggest that the rubric is positively associated with other scores, as expected. Yet, the sample size is too small to determine whether this association is statistically significant. We will be examining these relationships with a larger sample in the coming months.
A secondary focus of our research was on the ways in which a lifelong learning mindset may be associated with success at work. Workplace success refers to both objective (e.g., pay and promotions) and subjective (e.g., job satisfaction) aspects of thriving at work (Ng, Eby, Sorensen, & Feldman, 2005). In addition to the data sources described above, data about lifelong learning and career success were also collected from SAF alumni. We examined whether self-reported lifelong learning mindset would be associated with work success not just for co-op students but also for alumni. We found evidence that developing a lifelong learning mindset is positively associated with work success in both samples, but particularly for alumni. The direction of the association cannot be determined in our study (i.e., whether more successful people adopt a lifelong learning mindset or those with a lifelong learning mindset become more successful). However, our research provides some important initial evidence that work success and lifelong learning attitudes and beliefs are linked at both early and alter stages of one’s career.
Dissemination
A tangible outcome of the project at the student-level has been the dissemination of the Lifelong Learning rubric across SAF. Students in SAF now have the opportunity to review the rubric as they complete various assignments related to their co-operative education work terms. Prior to our project, no such rubric was available. We believe that the rubric acts as an additional prompt or guide for students as they reflect on their experiences and report on those experiences. In this way, the rubric directs attention to the components of a lifelong learning mindset as students review their experiences and development.
Data from the study has been shared within the Experiential learning group of the SAF. The template used for the end of work term reflection of SAF students has been modified to focus on lifelong learning development and assessment. The rubric used for the assessment of this reflection has been modified to almost exclusively assess lifelong learning.
Our work was shared with the University audience during the 2018 University of Waterloo Teaching and Learning conference. The title of our presentation was “Assessing lifelong learning characteristics using students’ written reflections.”
Insights from our research have been shared through multiple venues beyond the University. We gave a presentation on our work, entitled “Assessing lifelong learning using a novel rubric in a work-integrated learning program,” at the 2018 Ontario Universities Council on Quality Assurance (Toronto, ON). Our work was also shared at the 2018 Society for Teaching and Learning in Higher Education (STLHE) conference, in Sherbrooke, QC. The title of the presentation was “Development of a rubric to assess lifelong learning in work-integrated learning reflection assignments.” A manuscript by that same title that more fully describes our work to date has been accepted to and is to appear during 2019 in Collected Essays on Teaching and Learning (CELT). CELT is the annual publication of work presented at STLHE conferences.