Design and development

Outcomes are statements that describe what students will be able to know (cognitive), do (psychomotor), and feel/model (affective) by the end of the program or course of study. Outcomes describe measurable behaviours. Most outcomes have three parts: an action, a topic, and a criterion or context. Note that terminology is used interchangeably; you may find "outcomes," "objectives," and "goals" used to mean the same thing in different settings. The semantics are less critical than the essential notion: statements describing what students will have learned, articulated in a way that allows that learning to be measured.

In general, we recommend developing eight to twelve outcomes for the program. In practice, many graduate programs have fewer, perhaps six to eight outcomes.

The ideal graduate

To generate program outcomes, we encourage departments to begin with their ideal graduate. What knowledge, skills, and values would students in your department develop if there were unlimited resources?  If you had the freedom to focus on any area in your discipline, what would you want your students to learn? Ultimately, this list of attributes, which is often developed through a brainstorm at a program retreat or a department meeting, reflects who the department wants its students to be.

From the ideal graduate to program outcomes

Realistically, we cannot focus only on the ideal. There are many contextual factors that influence the design of our program, including the availability of resources, the expectations of key stakeholders such as students and the institution, accreditation requirements, etc. Based on these ideal attributes, we develop program outcomes that account for the relevant contextual factors.

Program outcomes

• Writing intended learning outcomes – Although many of the examples on this Centre for Teaching Excellence tip sheet relate to outcomes at the course level, the same principles apply at the program level, although the context and scope might be broader.
• Program outcomes database – The University of Ottawa’s Centre for University Teaching developed the Directory of University Program Learning Outcomes. It is an extensive database of outcomes from programs around the world, with a focus on Canadian and American institutions. Outcome samples are available at the undergraduate, masters and PhD level.
• Information Literacy Competency Standards - These competencies describe a variety of knowledge and skills related to information literacy and were developed by the Association of College & Research Libraries
• Research Skill Development Framework for curriculum design and assessment - The University of Adelaide, Australia created this framework, which articulates outcomes related to various facets of research and students' level of autonomy in the research process.

These statements are used throughout the curriculum design process as well as the evaluations of the program. A balance must be strict; few outcomes can mean they are too broad to assess while too many can make design restrictive.

This first draft of the program outcomes reflects the interests and requirements of the department. The next step is to gather input from other key stakeholders, which will help the department hone these outcomes. Key stakeholders might include students, staff, alumni, and employers. Sample data sources include:

• exit surveys completed by graduating students;
• focus groups with current students;
• alumni surveys; and
• interviews with current or potential employers (particularly for co-op programs).

Other key aspects to consider are external requirements, such as requisite components for accreditation, or the undergraduate or graduate degree-level expectations.

Degree-level expectations

All undergraduate and graduate programs in Ontario must meet the requirements outlined in the university undergraduate degree-level expectations (UDLEs) (DOC) and graduate degree-level expectations (GDLEs) (DOC) respectively. The DLEs represent a set of minimum requirements all students must attain to earn the relevant degree.

The UDLEs were developed by the Ontario Council of Academic Vice-Presidents and institutions could choose to adopt them outright, develop their own, equivalent expectations, or add to the existing ones. At Waterloo, we adopted the original six and added two more to reflect the overarching goals of our institution. Additional information regarding the history of the degree-level expectations and their role in program evaluation is available in our program review and accreditation section.

It is beneficial to compare your program outcomes to the DLEs. First, this comparison can be used in program reviews to demonstrate the fulfillment of these requirements. Second, although they are generic requirements of any program, they might highlight an area of your program that requires greater emphasis or is not articulated fully in your own program outcomes. The following templates, provided in MS Excel format, can be used to document the relationship between your program outcomes and Waterloo’s degree-level expectations.

• Undergraduate template (XLSX)
• Masters template (XLSX)
• PhD template (XLSX)

Professional associations and accreditation bodies

In designing the program, it is beneficial to seek guidance from groups outside Waterloo. For accredited programs, it is critical that the new or modified program meets the requirements of the accreditation body. Other groups can also provide guidance on the attributes to consider developing in our students. For example, the Association of College & Research Libraries has developed Information literacy competency standards for higher education. These standards effectively demonstrate how measurable outcomes can be developed related to research tasks such as the selection and critical evaluation of sources, the application of information to a specific product, and the legal, ethical and social issues associated with the use of information.

Another input to consider when reviewing your program outcomes relates to what your students do upon graduation. Do they pursue professional degrees, such as law or medicine? Do they continue their education in the same field or a related field? Do graduates enter a specific sector? A better understanding of the academic and career paths of our students provides us with additional input to consider in our program’s design.

It is not our task to prepare our students for each of these unique settings; that simply is not feasible. There are, however, benefits to understanding the direction our students will take when they complete our program. As well, there are other elements in the program’s design to consider. Perhaps it makes sense to build some flexibility into the program’s design when students pursue a plethora of options. If a significant number of students are pursuing a similar path, it might be worth investing in creating an option or minor in that area.

Association of Accrediting Agencies of Canada  – The members list provides an extensive list of contact information for various accrediting bodies in Canada, such as Engineers Canada, the Canadian Medical Association, TESL Canada, and the Canadian Psychological Association.

While we encourage developing eight to 12 program outcomes, it might be necessary to define each outcome at a deeper level. Consider the following example:

By providing additional information, the outcome is clearer and more easily measured. Using the Research Skills Development Framework, the outcome was modified as follows.

Integrate evidence-based practice to execute a student-initiated research project.

• Articulate suitable research questions within structured guidelines to address the student-generated thesis statement
• Critically evaluate information and its sources to review the literature related to the research questions
• Execute an appropriate research methodology to answer the research questions
• Present results, discussion and conclusions using the language of the discipline through appropriate media for the selected audience

Feedback from faculty, staff, students, and other stakeholders can be used to develop this next layer of the outcomes.

As described in the previous sections, program-level learning outcomes are statements that describe what students should be able to achieve by the end of the program. Learning Outcomes reflect the disciplinary context in which the program is situated and serve as criteria against which to evaluate the program. 

Design

1. Do the Learning Outcomes specify what all students who graduate from the degree program should be able to accomplish? The emphasis on all is a reminder that Learning Outcomes reflect the degree requirements of the program regardless of the specializations, options, minors, electives, or experiential opportunities a student completes. In this sense they help you to set the bar for minimum expectations to leave with your stamp of approval.
2. Is each Learning Outcome stated from the perspective of the learner? Although we often use the terms goals, objectives, and outcomes interchangeably, Learning Outcomes reflect a student’s knowledge, skills, and abilities whereas program goals shape the program’s design. For example, a goal of the program might be to draw on a variety of disciplinary perspectives. As a result, a relevant Learning Outcome would be “By the end of the program, the student should be able to integrate tools, techniques and perspectives from multiple disciplines to solve complex problems”.
3. Are the Learning Outcomes comprehensive, representative of both the depth and breadth of the program? One of the challenges when constructing these statements is to capture the right level of specificity. Consider the Learning Outcome, “By the end of the program, a student should be able to write a persuasive essay”. While that might be the case, is that the only form of written communication expected of program graduates? That outcome might be more appropriate at the course level whereas at the program level, written communication might reflect a broader scope of audience, format, and purpose, such as “By the end of the program, the student should be able to communicate (findings/ information/ arguments/ analyses) accurately in written, oral, and multimedia formats to their colleagues and a general audience.”

Function

4. Is each Learning Outcome measurable? By designing specific, measurable statements, Learning Outcomes serve as criteria against which to measure the program. Consider the example, “By the end of the program, a student should be able to integrate evidence-based practice to design and implement a research project.” To complement this outcome, indicators can be specified either as a sequenced collection of assessments throughout the curriculum or in terms of an integrative final artefact (a project or thesis) to clarify the scope and context of the outcome (Is this a collaborative project? What is the level of autonomy expected of the student when designing and implementing the project?). This idea is further explored on the CTE web page Refining the outcomes.
5. Do the Learning Outcomes sufficiently distinguish between programs offered within the department?Many departments at Waterloo offer multiple degrees at the same level (e.g., undergraduate programs with both co-op and regular streams, masters programs in the same discipline offered as either a professional degree or a thesis-based degree, similar degrees offered at the undergraduate level with key differences (e.g., the “X and Business” programs). If two students receive a different degree, the distinctiveness of the degree should be reflected in the Learning Outcomes.

Discipline

6. Do the Learning Outcomes integrate the language and frameworks of the discipline? As an example, consider the concept of design. What does design mean in the discipline? What is a student designing (e.g., a solution to a problem, a process or procedure, art, music, a written composition, something tangible such as a tool, model, or building or perhaps something intangible such as a computer algorithm)? What does the program emphasize: the end result, the process, models, techniques? How is design assessed? Learning Outcomes need to reflect the program’s disciplinary context, not only the generic skills developed in any degree.
7. Do the Learning Outcomes include elements that demonstrate the uniqueness of the program and distinguish it from similar programs at other institutions? For some programs, this might not be a priority, but it is worth reflecting on whether it is important to the department. For programs that have counterparts at several institutions, capturing this uniqueness helps articulate the value of studying this discipline at Waterloo.

Curriculum mapping is an invaluable tool to gain a snap shot of the program. Mapping can help you identify the flow of the outcomes through the program and detect gaps and redundancies in the program. More complex maps also identify the relevant teaching methods and assessments used to measure students' progress. The maps can be created at facilitated department retreats or data might be gathered from faculty members with the map generated by a curriculum committee.

When mapping the program, it is important to identify not only the core courses but to also include key educational experiences. These might include experiential opportunities, such as service-learning or co-operative education, or research requirements, including all phases of a dissertation. Although you might not list all electives, if there are breadth requirements or a suite of options (e.g., students must complete two technical electives from the following list), these requirements should be presented in the map, although you can choose whether to map the associated courses.

At the development phase, there are a variety of ways to map the curriculum. If you are modifying an existing program, you might want to map the program in its current format (core courses, elective courses, milestones, key learning experience) to the new program outcomes. This map provides foundational information about the current program and can help the curriculum committee identify areas for development in the program to achieve the new program outcomes. This format captures at what level the element contributes to the fulfillment of the outcome.

For undergraduate maps, we often map three levels: introduction; reinforcement; and completion. For graduate programs we suggest showing whether the degree requirement has a major or minor relationship to the outcome. The sample map below shows a snapshot of a program’s map in which courses were mapped to the UDLEs. As you can see in the map, there is not always a linear progression from introduction to completion. For this program, the map helped identify that completion was not met for all outcomes.

• Sample Map (undergraduate) (XLSX)

Another option at the development phase is to create a less detailed program map, focusing on components, rather than courses, and their relationship to the outcomes. Rather than identifying the level to which the element contributes to the outcome, we simply identify whether it makes a major or minor contribution toward fulfilling the outcome. This style is particularly useful for new programs at their early development stages. A more detailed map can be created as the program takes shape.

• Sample map (detailed) (XLSX)

The final option in mapping the curriculum is to use a visual model rather than a curriculum map. Consider the following example from the Master of Peace and Conflict Studies (MPACS).

• MPACS Visual Model - Located at the bottom of the page

This visual representation of their curriculum provides an overview of the MPACS program, its components, and expresses the core values of the program. Thank you to Lowell Ewert, director of peace and conflict studies, for sharing this example.

We offer several services to support your activities during the design and development phase.

• Consultations:  We provide consultation services with individual members of the department and with curriculum committee members.
• Retreats:  We facilitate retreats that vary from one to two hours in length to full-day sessions. Topics include all aspects of design and development.
• Data collection and instrument design: We can assist you with your design of various data collection activities. For example, we can design, organize, and facilitate student focus groups, or help you develop a survey for alumni.

For more information on these services, please contact Veronica Brown, Senior Instructional Developer, Curriculum and Quality Enhancement.

Wolf, P. (2007). A Model for facilitating curriculum development in higher education: A Faculty-driven, data-informed, and educational developer–supported approach. New Directions for Teaching and Learning, 112, 15-20.