Evaluation Guide

Evidence: Is the policy effective for the populations it’s designed to serve?

• What works to improve outcomes — overall, and for various groups? Consider dimensions of equality identified at the consultation stage—e.g. gender, race, disability,

• What works to reduce disparities? Compare average outcomes across all groups, and differences between groups, to see whether they improved over time.

• What works under what conditions? Compare how service delivery, program resources, and quality may vary across groups.

Once you’ve developed your evaluation plan, share it with the equity lead in your department or unit, then with the EDI-R office and the Office of Indigenous Relations. Experts on equity, diversity, anti-racism, and decolonization can help you make sure your evaluation plan anticipates and includes, as far as possible, all the groups whose equity interests might be affected by your policy and its evaluation.

This guide makes recommendations about (1) how to develop an evaluation plan, and (2) what questions your policy evaluation should address.

1. Identify the outcomes the policy is designed to achieve, and how and when you will know whether it’s achieving them. What would success look like?

• Identify any benchmarks against which you might measure changes after your policy is implemented. What was the status quo before the new/changed policy, and how did those measures change after the policy was implemented?
• Set a schedule for when the policy evaluation might take place. When are results expected to be measurable? 12 months? 24 months?

2. Practice data equity: consult affected groups about which data to collect, and how to analyze and interpret them. Consider the principles of data equity in developing your plans for data collection and analysis. Consult the groups whose data is to be collected to ensure that your evaluation process does not inadvertently put their interests or well-being at risk.

Many racialized and other groups have been subjected to exploitative and harmful data collection and research. As Mathematica points out, data can be used in equitable or harmful ways, depending on which data are collected, which analytic questions are addressed, and how those findings are presented. For example, Mathematica observes, racially disaggregated data have been used to expose structural inequity in the underfunding of majority-minority schools and secure equitable and much-needed reforms. At the same, time, disagreggated data on disparate academic outcomes “have been used to argue the inferiority of specific racial groups, primarily Black and Indigenous people.”

In the same vein, data on nonstraight or noncisgender identity risks “outing” the people whose data is recorded, especially if identifiable data falls into the hands of law enforcement, university administrators, or government agencies who want to discriminate (or might even be required by law to discriminate). Consider ensuring that data are collected in ways that do not identifiably record names, social insurance numbers, or exact dates of birth so that your raw data cannot be traced to individual respondents.

Indigenous medical researcher Dr. Nadine Caron warns against “what we call 'helicopter research' or 'vampire research,' where researchers come into a community with a question that isn't even a priority for the communities. [They] come in, take what they need and leave and don't even share the results with the community — let alone the potential benefits." To avoid doing this, make sure affected communities participate in creating your plan for data collection and evaluation.

3. Identify, collect, and review any existing data on the effects of similar policies. Conduct a literature review, review any data currently held by this university or other institutions, and consult with affected groups to identify gaps in prior data collection and analysis (e.g., lack of racial data, inability to cross-tabulate identity groups). Ensure that these gaps are addressed by your data collection and analysis plan.

4. Identify the people who will make up your evaluation team. As with policy development, evaluation should involve participation by all stakeholders identified in the consultation plan (including implementers, intended beneficiaries, and other affected groups), as well as additional stakeholders, if effects on additional groups were observed or reported during policy implementation. Consider the following:

• Lived experience
• Subject-matter expertise
• Data analysis
• Community connections and trust (for qualitative feedback)
• Stakeholder representation
• Commitment to equity

5. Establish mechanisms to record, store, protect, and analyze quantitative data about the effects of the policy. Ideally before implementation begins, ensure that the data needed to answer your questions (including data about process and outcome measures) will be available for evaluation in a form that allows for disaggregating outcome measures by, e.g., race, gender, disability, gender identity and sexual orientation, or any other relevant dimensions of equity.

6. Establish mechanisms to receive and record feedback from people affected by the policy.
Make a plan to solicit feedback from implementers, intended beneficiaries, and anyone else who you anticipate may be affected by the policy.

• Ensure that all feedback received on policy implementation is recorded in a systematic way, so it will be accessible when you conduct your evaluation.
• Take steps to ensure that participants--especially those with less institutional power (e.g. students; non-unionized staffers; adjunct faculty; racialized, disabled, and 2SLGBTQIA+ participants, etc.)--will feel free to disclose criticisms or negative feedback.
• Be open to receiving unsolicited feedback, whether from known stakeholders or others who might be affected.

7. Qualitative analysis. Involve relevant stakeholders to assess how the policy is working. Consider questions like these:

 a. Assess the policy as written:

• Is the policy written in clear language? Does the policy allow a lay reader to easily know who is required to do what?
• Do the words and substance of the policy take equity into account? How?
• What outcomes does the policy aim to achieve? Whom does it aim to benefit, and how?
• What steps does the policy require, and who is supposed to take those steps?
  • Implementation: What detailed recommendations, procedures, and action steps are mandated or allowed?
  • Accountability: What does the policy say should happen if the policy isn’t implemented as described? Who decides whether the steps have been taken?
  • Funding: How are the actions required by the policy funded? If the action requirements are unfunded, how are the steps required by the policy going to be resourced?
• Does the policy set explicit goals to address racial, gendered, disability, or sexual disparities?
• Does the policy require collection and accessibility of data to measure whether the policy is achieving its objectives?
  • Does it require collection and accessibility of disaggregated data by factors such as gender, age, race, disability, 2SLGBTQIA+ identity, or other relevant factors?
• Does the policy establish a timeline and deliverables?
• Does the policy require that the results of the evaluation be made public?

b. Ask stakeholders about the effects of the policy

• If the policy or program contains eligibility requirements, how do these requirements affect groups such as racialized, disabled, 2SLGBTQIA+ and other students who may be at risk of exclusion?
• Was the policy implemented as planned? (Ask both implementers and intended beneficiaries, as their perceptions may differ.)
• Did the groups who were intended to benefit from the policy know about it?
• Did the groups who were intended to benefit from the policy apply for it (if applicable)? Did they receive the intended benefits?
• Did respondents benefit equitably across gender, race, disability, 2SLGBTQIA+ identity, and other potential forms on inequity? Consider intersections between these groups.
• Has the policy been working for the people whom it was intended to benefit?
• Did the intended beneficiaries feel they were treated fairly? Did they feel they benefited from the policy? Did members of equity-seeking groups feel they benefited fairly compared to other groups?

8. Quantitative analysis: Collect and analyze data to see how your policy is working:

• Did the policy achieve its intended outcomes for the overall population of intended beneficiaries?
• Did it benefit all groups of intended beneficiaries equitably?
  • Analyze policy effects by gender, race, disability, 2SLGBTQIA+ identity, and other relevant groupings as identified at the consultation stage (as well as groups whose relevance became apparent during the implementation or evaluation process)
  • Ensure that your analysis disaggregates outcomes intersectionally, e.g. by race + gender identity, race + disability, etc.
• Analyze whether:
  • a proportionate share of people from equity-seeking communities benefited from the policy, and
  • members of these groups benefited as much as similarly situated white, non-disabled, straight etc. comparators.
• Consider adverse effects: Has the implementation of the policy resulted in worse outcomes for any groups of people? (e.g., increased disparities or harmful effects)
• Don’t let the perfect be the enemy of the good. Rigorous causal analysis may not be possible using administrative data about a new policy or one that affects small numbers of people. Consider alternate ways of knowing whether (beneficial or adverse) changes in measured outcomes were related to the policy, or not. These might include descriptive statistics as well as interpretations informed by feedback from implementers, intended beneficiaries, and others affected by the policy.

9. Reflect on findings, identify directions for change. What changes to the policy or its implementation could sustain the beneficial effects, reduce disparities, and mitigate adverse effects?

• Ensure that stakeholders from all affected groups are represented in determining what policy, program, or practice changes might improve beneficial effects and prevent adverse ones.