COMPASS Technical Report Series, Volume 4, Issue 3, October 2017
Development of a mental health module for the COMPASS system: Part 2 (PDF)
Table of Contents
Acknowledgements
Introduction
Method
Results
Pilot Test
Mental Health Module Tool Revisions
Focus Groups Results
MHPP Pilot Test Results
Revisions
References
Acknowledgements
The COMPASS Mental Health team would like to thank Dr. Patrick Carney and Jeffrey Cole of Simcoe Muskoka Catholic District School Board, and Becky Mann of Simcoe Muskoka District Health.
Authors
Karen A. Patte, PhD (Department of Health Sciences, Brock University, St.Catherines, ON)
Chad Bredin, BA (School of Public Health and Health Systems, University of Waterloo, Waterloo, ON)
Joanna Henderson, PhD CPsych (Centre for Addiction and Mental Health (CAMH), Toronto, ON; University of Toronto, Toronto, ON)
Tara Elton-Marshall (CAMH, London, ON)
Guy Faulkner, PhD (University of British Columbia, Vancouver, BC)
Catherine Sabiston, PhD (University of Toronto, Toronto, ON)
Mark Ferro, PhD (School of Public Health and Health Systems, University of Waterloo, Waterloo, ON)
Kate Battista, MMath (School of Public Health and Health Systems, University of Waterloo, Waterloo, ON)
Scott T. Leatherdale, PhD (School of Public Health and Health Systems, University of Waterloo, Waterloo, ON)
Report funded by:
The development and testing of the COMPASS Mental Health Module was supported by a bridge grant from Canadian Institutes of Health Research (CIHR) (PJT-149092; grant awarded to Patte and Leatherdale). The COMPASS study (2012-2015) was supported by a bridge grant from the CIHR Institute of Nutrition, Metabolism and Diabetes (INMD) through the “Obesity – Interventions to Prevent or Treat” priority funding awards (OOP-110788; grant awarded to Leatherdale) and an operating grant from the CIHR Institute of Population and Public Health (IPPH) (MOP-114875; grant awarded to Leatherdale). The COMPASS study extension (2016-2021) was supported by a CIHR Project Grant (PJT-148562; grant awarded to Leatherdale). The COMPASS expansion to additional jurisdictions was funded by a Health Canada grant through the Substance Use and Addictions Program (SUAP).
Suggested citation:
Patte KA, Bredin C, Henderson J, Elton-Marshall T, Faulkner G, Sabiston C, Battista K, Ferro M, Cole A, Doggett A, Godin K, Aleyan S, Butler A, Leatherdale ST. Development of a mental health module for the COMPASS system: Improving youth mental health trajectories. Part 2: Pilot test and focus group results. 2017; 4, 3. Waterloo, Ontario: University of Waterloo. Available at: www.compass.uwaterloo.ca.
Contact:
COMPASS Research Team
University of Waterloo
200 University Ave West, BMH 1038
Waterloo, ON Canada N2L 3G1
compass.uwaterloo.ca
Introduction
COMPASS is a 9-year longitudinal study (starting in 2012-13) designed to follow a prospective cohort of grade 9 to 12 students attending a convenience sample of Canadian secondary schools over several years to understand how changes in school environment characteristics (policies, programs, built environment) and provincial, territorial, and national policies are associated with changes in youth health behaviours1. COMPASS originated to provide school stakeholders with the evidence to guide and evaluate school-based interventions related to obesity, healthy eating, tobacco use, alcohol and marijuana use, physical activity, sedentary behaviour, school connectedness, bullying, and academic achievement. COMPASS has been designed to facilitate multiple large-scale school-based data collections and uses in-class whole-school sampling data collection methods consistent with previous research2-5. COMPASS also facilitates knowledge transfer and exchange by annually providing each participating school with a school-specific feedback report that highlights the school-specific prevalence for each outcome, comparisons to provincial/territorial and national norms or guidelines, and provides evidence-based suggestions for school-based interventions (programs and/or policies) designed to address the outcomes covered in the feedback report. (refer to: www.compass.uwaterloo.ca).
Purpose
In spring 2016, our team received CIHR Bridge funding (PJT-149092) to develop and test a robust tool to determine the real-world effectiveness of school-based mental health interventions – the COMPASS mental health module (MH-M). The MH-M was designed to build capacity and provide infrastructure to continually evaluate and improve school-based programs, policies, and resources for the advancement of youth mental health.1 With a longitudinal, hierarchical, and quasi-experimental design, this module will allow us to evaluate the hundreds of mental health programs being implemented in schools across the country (the majority of which remain untested),2 and to facilitate knowledge exchange in order to continually improve youth mental health and schools' ability to support their students.
The MH-M was designed for seamless incorporation into the COMPASS system.3 In line with the
COMPASS design,3 the MH-M has three components:
1. Student mental health questionnaire (MHq)
2. School-level mental health program and policy scan tool (MHpp)
3. School-specific mental health knowledge exchange tool (MHkte)
After the development of the draft MH-M tools (refer to Technical Report Part 11 for details on tool development), the module was tested using both quantitative (pilot test) and qualitative (focus group and interviews) methods. In follow-up to the MH-M development technical report,1 the current report provides details on the testing of the draft tools, including the procedures and results of the pilot test and focus groups. The findings outlined in this report informed revisions to the tools. The finalized MH-M will be incorporated into the COMPASS system.
Method
Development and testing of the MH-M took place over a 1-year period, using a design and protocol similar to that used for the development of the original COMPASS tools, and included both quantitative (pilot testing) and qualitative (focus groups) methods. Initial conceptualization of the MH-M tools was informed by past and ongoing school-based youth and mental health research, national and provincial strategic priorities, school programming, and consultations with researchers, clinicians, school stakeholders, and public health partners, in order to reflect both science- and practice-based concerns. The draft MH-M was pilot tested in a sample of purposefully selected schools participating in COMPASS. Focus groups with students were conducted in one of these schools to ensure appropriateness, comprehension, and comprehensiveness of the MHq. Following necessary revisions, the tools were automated and incorporated into the existing ongoing COMPASS system infrastructure, available for future implementation.
Pilot Testing
Pilot Sample Recruitment
The initial plan for the pilot test sample was to include three Ontario secondary schools. The resultant sample reflects the high level of school interest in the project and need for support for student mental health. Upon hearing about the development of the mental health module – a key priority identified by the COMPASS schools in past years – representatives of a central east Ontario school board inquired about the project. The board expressed interest in having the MH-M pilot testing take place in all 9 secondary schools included in their board. The schools within the board were invited to participate in the MH-M pilot testing via a school information letter which outlined the project. In addition, we invited 5 B.C. secondary schools (including schools in Vancouver, Chilliwack, and Coquitlam, and two schools in Victoria) for the value of testing the MH-M in another region and samples naïve to the host COMPASS system.
Pilot Test Data Collection Procedures
As pilot testing occurred at the same time as the COMPASS host study data collection, the same procedures were followed,4 with the addition of the MHq and MHpp.
The school administrator at each participating school selected a school contact to assist project staff in implementing the project and to set a data collection date. The school contact was mailed a school welcome folder that provides a step-by-step guide and information documents to assist the school contact with implementing the project in their school.
Consistent with the COMPASS study, the MHq was designed for use with active-information passive-permission protocols. The school contact was provided permission materials to be distributed to parents in the method(s) preferred by schools (e.g., by mail, social media, email, taken home by students). Once adequate time was given for parents to review the information (and refuse permission for their child if necessary), questionnaires were prepared for shipment to the school.
Within one week of the scheduled data collection date, the recruitment coordinator mailed out classroom questionnaire bundles to the school contact. The school contact received a School Contact Questionnaire Distribution Instruction Letter along with bundles of questionnaires (including the MHq insert) for each participating classroom. The letter detailed the steps required for the school contact to effectively distribute the student questionnaires to the classroom teachers.
Each classroom questionnaire bundle included a Teacher Instruction Questionnaire Letter, Classroom Envelope Form, Classroom Instructions and FAQs, No Permission List, and enough student questionnaires and MHq inserts for all students with permission to participate. Teachers used the Teacher Instruction Questionnaire Letter included in their package to guide them in implementing the questionnaire to their class. Teachers were asked to read the Classroom Instructions aloud. These instructions emphasized the anonymity of questionnaire responses, the need for honest, accurate reporting, and information on the addition of the MHq. They also indicated that the students could end their participation at any time without penalty or recourse.
Classes had a set of instructions and a No Permission List listing students whose parents did not provide permission. The students on this list were to not receive a questionnaire to complete. If parental refusal was obtained after the questionnaire bundles are mailed to the school, an updated No Permission List was distributed to the classroom teachers just prior to the data collection testing period.
Teachers were to remain at the head of the classroom until the Cq and MHq were completed by all students. While students were completing the questionnaire, teachers completed the Classroom Envelope Form indicating total classroom enrolment, time required to complete the survey, and the number of questionnaires completed. To ensure student confidentiality, students placed their completed questionnaires and MHq inserts in an envelope and sealed it. A student was then assigned by the teacher to collect the sealed envelopes and seal them in a larger classroom envelope. The assigned student then delivered the classroom envelope containing the questionnaires and inserts to project staff waiting in the school's main office.
Administrators most knowledgeable in student health programming, policies, and resources at the participating schools were asked to complete the SPP and MHpp items at the time of student-data collection or prior. As the COMPASS SPP is now online, an email was sent to the school contact with a link to the tool. The draft MHpp items were included within the SPP.
A data collector attended each school data collection. Their role was to: problem solve any distribution and questionnaire completion issues; assist the school contact and teachers with the survey implementation; ensure all eligible classrooms returned questionnaires by checking off each returned classroom package of completed questionnaires on school class lists; locate any missing classroom packages before leaving the school (if applicable); confirm the completion of the SPP and MHpp questionnaires; and verbally thank the school/school contact for participating.
After the school data collection, the data collector returned data collection materials to UW for processing. Upon receipt of a data collection shipment at UW, the boxes were checked for completeness, unpacked, and processed.
Post pilot testing, student-level data were used to create an individualized MHkte for each of the participating schools. The MHkte was provided to schools at the same time and as part of the COMPASS study School Health Profile feedback report. The school’s Knowledge Broker followed up with school contacts to arrange a time to discuss the MHkte and School Health Profile feedback report, identify priorities, appropriate action, and connect and mobilize necessary resources. When following up, the Knowledge Broker requested feedback from school contacts on the utility and comprehension of the MHkte.
MHq Student Focus Groups
We convened three student focus groups sampled from one of the Ontario schools that participated in the pilot testing. The objective of the focus groups was to obtain student feedback of the draft MHq. More specifically, the goal was to ensure appropriateness, comprehensiveness, and student comprehension of the tool.
Focus Group Recruitment
One of the schools participating in the Ontario pilot test was approached via email and sent an information letter about also participating in the student focus groups. The student focus groups required active permission protocols. With the assistance of the school contact, information letters and active permission forms were sent home with interested students at least 2 weeks prior to the focus groups. The school contact was instructed to choose one class of each grade (grade 9, 10, 11, and 12) to send letters home with, in an effort to ensure representation of different grades. The parents or guardians of eligible students were asked to (a) sign and return the form with their child to the school primary contact should they and their child consent to participating in the study, and/or (b) call/email the recruitment coordinator using the 1-800 phone number or email address provided in the information letter should they have any questions. The primary contact collected the signed permission forms and created a list of the students’ initials, grade, and gender of those with permission. This information was used to gauge adequate recruitment and varied grade and gender representation.
Focus Group Procedures
The focus groups took place over the lunch break on the day of the school’s COMPASS study data collection, which occurred in the morning. Students were divided into three groups by grade and groups took place in separate rooms within the school. A healthy lunch was served. Students were required to sign assent forms to participate. At any time during the process, students could decline to participate or withdraw from the study. The discussion was approximately 40-45 minutes long. Each group had a trained moderator and note taker. Each moderator followed a script with an introduction of the purpose and format, and questions and probes questions. Questions were intended to assess student perceptions and comprehension of the MHq that they completed in the pilot test in the school class period prior to the lunch break. Also, students were asked if any key areas were missing and whether they perceived the items as important and relevant to individuals their age. Groups were audio recorded to ensure all student comments are adequately captured. Note that no qualitative analytic procedures were performed.
Remuneration
As part of the host COMPASS study, a thank-you package including an honorarium was mailed to all schools after their school data collection day. Schools participating in the COMPASS host study receive an honorarium of $250. An additional $50.00 was paid to the schools participating in pilot testing of the MH-M, due to the extra time required for staff to complete the MHpp items. The school participating in the student focus groups received an additional honorarium of $100.00 for the added burden of sending home and collecting active-permission forms, and arranging space during school hours. Students in the focus groups were provided with a free lunch as an acknowledgement of their participation and time over the lunch period.
Results
Pilot Test
Pilot Test Sample
In total, 8,344 grade 9 to 12 students and the 14 secondary schools they attend participated in the pilot test. See Table 1 for sample demographic frequencies.
The 9 secondary schools that comprise a central Ontario Public Catholic school board (including schools in Barrie, Bracebridge, Collingwood, Midland, Orillia, and Tottenham) participated in the pilot test, with a total enrollment of 5921 eligible students in grades 9 to 12. Overall, only 14 students were withdrawn from the study by their parent(s)/guardian(s). The MHq was completed by 80% (n = 4727) of eligible students. See table 1 for demographics of the final Ontario pilot sample.
In the 5 B.C. schools, 3617 of the 4948 eligible students completed the MHq (72% participation rate). Three students did not participate due to parental refusal, and the remaining missing students were due to absenteeism or scheduled study periods at the time of the data collection. The timing of the data collection near the end of the school year accounted for a relatively lower participation rate. In particular, one school scheduled the data collection on June 21, 2017 and due to an error by the school contact, many students were not present (355 students participated of 1085 enrolled students).
Table 1. Ontario and B.C. COMPASS Mental Health-Module pilot sample demographics
| Ontario Sample n (%) (n = 4727) |
B.C. Sample n (%) (n=3617) |
Total Pilot Sample n (%) (n = 8,344) |
|
|---|---|---|---|
| Grade | |||
| 9 | 1459 (31.9%) | 895 (24.7%) | 2354 (28.2%) |
| 10 | 1329 (28.1%) | 1115 (30.8%) | 2444 (29.3%) |
| 11 | 1088 (23.0%) | 925 (25.6%) | 2013 (24.1%) |
| 12 | 833 (17.6%) | 664 (18.4) | 1497 (17.9%) |
| Missing | 18 (0.4%) | 18 (0.5%) | 35 (0.4%) |
| Age | |||
| 13 | 15 (0.3%) | 17 (0.5%) | 32 (0.4%) |
| 14 | 928 (19.6%) | 498 (13.8%) | 1426 (17.1%) |
| 15 | 1307 (27.7%) | 1021 (28.2%) | 2328 (27.9%) |
| 16 | 1194 (25.3%) | 939 (26.0%) | 2133 (25.6%) |
| 17 | 907 (19.2%) | 813 (22.5%) | 1720 (20.6%) |
| 18 | 356 (7.5%) | 312 (8.6%) | 668 (8.0%) |
| Missing | 60 (1.3%) | 17 (0.5%) | 37 (0.4%) |
| Gender | |||
| Male | 2318 (49.04%) | 1824 (50.4%) | 4142 (49.6%) |
| Female | 2349 (49.69%) | 1733 (47.9%) | 4082 (48.9%) |
| Missing | 60 (1.3%) | 60 (1.7%) | 120 (1.4%) |
MHq Pilot Test Results
1. Self-rated mental or emotional health
Overall, 4.4% (n=369) of participating students did not respond. “Fair” or “poor” emotional or mental health was reported by 42.9% (n=3,425) of students who responded. As found in previous studies,4 females were more likely to report lower scores than males (p 4 In the 2015 OSDUHS, “fair” or “poor” mental health was reported by 14.2% of grade 9 students, 18.8% of grade 10 students, 23.2% of grade 11 students, and 18.9% of grade 12 students.4
Figure 1. Self-rated emotional or mental health by gender among secondary school students in the COMPASS study Mental Health-Module pilot sample
Figure 2. Self-rated emotional or mental health by grade among secondary school students in the COMPASS study Mental Health-Module pilot sample
2. Family and friends
a. “I have a happy home life”
Excluding students with missing responses (4.4% [n=369]), 76.3% (n=6131) of students agreed or strongly agreed that they have a happy home life. Males were more likely to agree/strongly agree than female students (71.0% vs. 82.1%; p
Figure 3. Responses to the statement “I have a happy home life” by gender among secondary school students in the COMPASS study Mental Health-Module pilot sample
Figure 4. Responses to the statement “I have a happy home life” by grade among secondary school students in the COMPASS study Mental Health-Module pilot sample
Responses to the statement “I have a happy home life” by grade among secondary school students in the COMPASS study Mental Health-Module pilot sample
b. “My parents/guardians expect too much of me”
Excluding students with missing responses (2.7% [n=111]) females and 6.1% [n=247] males), 35.3% of grade 9 to 12 students (35.5% of females and 34.9% of males; p 5
Figure 5. Responses to the statement “my parents/guardians expect too much of me” by gender among secondary school students in the COMPASS study Mental Health-Module pilot sample
Figure 6. Responses to the statement “my parents/guardians expect too much of me” by grade among secondary school students in the COMPASS study Mental Health-Module pilot sample
c. “I can talk about my problems with my family”
Over half (56.1%; n=2818) of students agreed or strongly agreed that they could talk about their problems with their family (51.6% of females and 60.9% of males; p 5
Figure 7. Responses to the statement “I can talk about my problems with my family” by gender among secondary school students in the COMPASS study Mental Health-Module pilot sample
d. “I can talk about my problems with my friends”
Four-percent of students did not respond to the item “I can talk about my problems with my friends”. Among students who responded, 72.6% reported they could talk about their problems with their friends (72.8% of females and 72.5% of males; see Figure 8). Results resemble responses to similar items on the Canadian HBSC. For instance, 77-78% of grade 9 and 10 girls and 67-70% of grade 9 and 10 boys felt they could count on their friends when things go wrong.5 Likewise, 81% of females and 74% of males reported it was easy or very easy” to talk to their same-sex friends about things that really bothered them, and 58% and 63% of grade 9 and 10 females, and 64% and 68% of grade 9 and 10 males, respectively, reported it was easy or very easy to talk to their opposite-sex friends about things that really bothered them.
Figure 8. Responses to the statement “I can talk about my problems with my friends” by gender among secondary school students in the COMPASS study Mental Health-Module Pilot Sample
Figure 9. Responses to the statement “I can talk about my problems with my friends” by grade among secondary school students in the COMPASS study Mental Health-Module Pilot Sample
3. Psychosocial functioning or well-being (Flourishing Scale)
A sum score was taken, scoring the eight items one through five for strongly disagree to strongly agree, for a possible range from 8 (strong disagreement with all items) to 40 (strong agreement with all items). Higher scores are said to indicate “flourishing” and low scores signify “languishing”.6 According to Diener et al.6 “flourishing” indicates the individual has many psychological resources and strengths. That is, high scores mean they view themselves strongly in many areas of functioning, including positive relationships, feelings of competence, and meaning and purpose in life. Rates of missing data were consistent and acceptable at 4.5 to 5.2% across the eight items comprising the Flourishing Scale (see Table 2), with 92.6% (n=7724; 7.4% [n=620] of total scores missing) of students completing all eight items. The scale had a high internal consistency with an alpha coefficient of 0.91. A factor analysis supported the use of a single sum score, with one factor explaining 62.5% of the variance. Total scores ranged from 8 to 40, with an overall mean of 31.4 (SD=5.91). The distribution of scores approximated normal (Skewness = -0.777; Kurtosis = 1.036). Flourishing scores were higher among males (33.4; SD=4.01) compared to female students (32.6; SD=3.99; p<.0001 and="" in="" grade="" relative="" to="" grades="" figure="" see="" for="" means="" plot="" by="" gender.="">
Table 2. Flourishing Scale item response frequency among secondary school students in the COMPASS study Mental Health-Module Pilot Sample
| How much do you agree with the following statements? | Agree/Strongly agree | Neither agree or disagree | Disagree/Strongly disagree | Missing |
|---|---|---|---|---|
| a. I lead a purposeful and meaningful life | 5272 (63.2%) | 1828 (21.9%) | 867 (10.9%) | 377 (4.5%) |
| b. My social relationships are supportive and rewarding | 5785 (69.3%) | 1570 (18.8%) | 596 (7.5%) | 393 (4.7%) |
| c. I am engaged and interested in my daily activities | 5458 (65.4%) | 1637 (19.6%) | 855 (10.8%) | 394 (4.7%) |
| d. I actively contribute to the happiness and well-being of others | 6320 (75.7%) | 1300 (15.6%) | 307 (3.9%) | 417 (5.0%) |
| e. I am competent and capable in the activities that are important to me | 6679 (80.0%) | 945 (11.3%) | 314 (4.0%) | 406 (4.9%) |
| f. I am a good person and live a good life | 5977 (71.6%) | 1444 (17.3%) | 530 (6.7%) | 393 (4.7%) |
| g. I am optimistic about my future | 5529 (66.3%) | 1541 (18.5%) | 839 (10.6%) | 435 (5.2%) |
| h. People respect me | 5478 (65.7%) | 1799 (21.6%) | 670 (8.4%) | 397 (4.8%) |
Figure 10. Flourishing Scale mean score by grade and gender among secondary school students in the COMPASS study Mental Health-Module Pilot Sample
4. Self-concept
Students were instructed to choose the answer that best describes how they feel from the options: “true”, “mostly true”, “sometimes true, sometimes false”, “mostly false”, and “false”. The proportion of missing data remained consistent across the five self-concept items at about 5% (see Table 3). Use of a single total score appears appropriate, with one factor explaining over 93.5% of the total variance and item factor loadings of 0.933 and above. Internal consistency of the 5 items was 0.983. Total self-concept scores were higher among male students relative to females, and in grade 9 students when compared to their peers in higher grades. Gender and grade trends were consistent across all items.
Table 3. Self-concept item response frequencies among secondary school students in the COMPASS study Mental Health-Module Pilot Sample
| Choose the answer that best describes how you feel. | True/ Mostly true | Sometimes true, sometimes false | False/ Mostly False | Missing |
|---|---|---|---|---|
| a. “In general, I like the way I am” | 5129 (61.5%) | 1811 (21.7%) | 1014 (12.2%) | 390 (4.7%) |
| b. “Overall, I have a lot to be proud of” | 5258 (63.0%) | 1693 (20.3%) | 999 (12.0%) | 323 (3.9%) |
| c. “A lot of things about me are good” | 5390 (64.6%) | 1667 (20.0%) | 855 (10.3%) | 432 (5.2%) |
| d. “When I do something, I do it well” | 5188 (62.2%) | 2107 (25.3%) | 626 (7.5%) | 423 (5.1%) |
| e. “I like the way I look” | 4212 (50.5%) | 2086 (25.0%) | 1633 (19.6%) | 413 (5.0%) |
5. Barriers to help seeking at school for hypothetical mental health concerns
Students were asked “if you had concerns regarding your mental health, are there any reasons why you wouldn’t talk to an adult at school (e.g., a school social worker, child and youth worker, counsellor, psychologist, nurse, teacher, or other staff person)?” and instructed to “mark all that apply” from the provided options. Thirty-six percent of students (36.2%; n=3022) reported that they would have no problem talking to an adult at school about their mental health. Reports of having no concerns seeking help at school were more likely among male students (43.9% in males vs. 28.8% in females; p
Table 4. Reasons secondary school students in the COMPASS study Mental Health – Module Pilot Sample would not seek help for mental health concerns at school
| Deterrents to seeking help from an adult at school if students had mental health concerns: |
All students (n=8344) N (%) |
Excluding students who reported they would have no concerns seeking help (n=5322) N (%) |
|---|---|---|
| Prefer to handle problems myself | 2956 (35.4%) | 2839 (53.3%) |
| Lack of trust in these people – word would get out | 1959 (23.5%) | 1893 (35.6%) |
| Worried about what others would think of me (e.g., I’d be too embarrassed) | 1783 (21.4%) | 1705 (32.0%) |
| Don’t think these people would be able to help | 1723 (20.6%) | 1676 (31.5%) |
| Wouldn’t know who to approach | 1303 (15.6%) | 1255 (23.6%) |
| There is no one to talk to | 458 (5.5%) | 432 (8.1%) |
In response to a similar question on 2014 School Mental Health Survey,7 55.9% of Ontario students aged 12 to 16 reported they would not seek help at school. The relatively lower willingness to seek help indicated on the SMHS survey may reflect differences in question wording or participant ages. Also, with the expansion of awareness campaigns regarding mental health and reductions in stigma, more students may have been willing to talk about concerns in 2017 than in the 2014/2015 school year when the SMHS was conducted. As in our findings, the most common reason reported on the 2014 SMHS for why students would not seek help was that they preferred to handle problems themselves (71.5%).7 This response was followed by perceptions that adults at school would not be able to help (47.8%), not knowing who to approach (45.8%), and a lack of trust and concern that word would get out (44.1%).7
6. Generalized anxiety
Missing data among the 7 GAD-7 scale items ranged from 7.8 to 8.8%. The scale had an alpha coefficient of 0.988. Based on a factor analysis, over 93% of the variance was explained by one factor.
Based on the accepted cut-offs, 28.1% of students had GAD-7 scores indicative of moderate to severe anxiety, with 14.1% in the severe range. Likewise, on the 2015 OSDUHS, 34% of grade 7 to 12 students had moderate to severe levels of psychological distress in the past month, and 14% had scores at the severe level.4
Consistent with the literature, females were more likely to report moderate-to-severe anxiety than males (43.3% vs. 18.8%; see Figure 11), and females (M=9.26, SD=6.21) had a higher average score than males (M=5.24, SD=5.35; p<.0001 the="" overall="" mean="" was="" students="" in="" higher="" grades="" were="" more="" likely="" to="" report="" anxiety="" levels="" than="" their="" peers="" lower="" figure="">
Figure 11. Levels of generalized anxiety by gender based on GAD-7 score cut-offs among secondary school students in the COMPASS Mental Health – Module pilot sample
Figure 12. Levels of generalized anxiety by grade based on GAD-7 score cut-offs among secondary school students in the COMPASS Mental Health – Module pilot sample
7. Emotional skills
One item from each of the 6 DERS subscales was included. The rate of missing data was fairly consistent across the 6 items, from 8.3% to 9.7%, similar to the scale listed directly above on the MH-M. While each item is from 6 separate DERS subscales, analyses indicate the use of sum score is appropriate. Factor analysis results support the extraction of one factor, explaining over 93% of the total variance, and with factor loadings of 0.836 and above. The alpha coefficient the 6 items was 0.986. Males reported greater emotional regulations skills on all items, with the exception of emotional awareness.
a. Emotional clarity
The item “I have difficulty making sense out of my own feelings” is from the “lack of emotional clarity” DERS subscale. Overall, 17.9% of students reported they had difficulty making sense out of their feelings “most of the time” or “almost always”. Males were less likely to report difficulties with emotional clarity than females (23.6% of females and 11.5% of males responded “most of the time” or “almost always”; p
b. Emotional awareness
The item “I pay attention to how I feel” is from the DERS subscale labelled “lack of emotional awareness”. Overall, 39.0% reported they “most of the time” or “almost always” pay attention to how they feel. Males were more likely to report “almost never” paying attention to how they feel than females (19.6% vs. 9.4%; p
c. Engaging in goal-directed behaviour
The item “When I’m upset I have difficulty concentrating” is on the “difficulties engaging in goal-directed behaviour” DERS subscale. Forty-percent (40.1%) of students reported they “most of the time” or “almost always” have difficulties concentrating when upset. Males reported “almost never” more often than females (25.7% vs. 8.6%; p
d. Access to emotional regulation strategies
The item “When I’m upset, I believe there is nothing I can do to make myself feel better” is from the “limited access to emotional regulation strategies” DERS subscale. Overall, 19.0% of students reported they “most of the time” or “almost always” believe they cannot make themselves feel better. Males had higher reports of “almost never” than female students (50.8% vs. 27.7%; p
e. Impulse control
The item “When I’m upset, I lose control over my behaviour” is from the “impulsive control difficulties” DERS subscale. About three-quarters (74.2%) of students reported they “almost never” or “sometimes” lose control over their behaviour when upset, and 15.8% do so “most of the time” or “almost always”. Males reported they “almost never” lose control over their behaviour when upset more often than female students (48.5% vs. 42.4%; p
f. Acceptance of emotional responses
The item “When I’m upset, I feel ashamed for feeling that way” is from the nonacceptance of emotional responses” subscale. Overall, 20.2% of students indicated they “most of the time” or “almost always” feel ashamed about their feelings when upset, while 41.4% reported they “almost never” feel that way. Males reported not feeling ashamed about their feelings more often than females (50.6% vs. 33.1%; p<.0001>
8. Depressive symptoms
The CESD-R-10 scale had more missing data relative to the other MH-M components, with seventeen percent (17.7%) of students missing one or more item on the scale. Missing response frequency on the 10 individual items comprising the CESD-R-10 ranged from 8.7% to 10.4%. Response fatigue or timing constraints may account for the differences, as the CESD-R-10 was listed last on the MH-M insert. In addition, the negatively phrased items and content of the scale may have contributed to missing items. Internal consistency of the scale was high (α = 0.984). The use of a total score is supported based on factor analysis. Extraction of a single factor best fit the data, with factor loadings of 0.811 and higher, and one factor explaining 87.1% of the total variance.
Total scores were calculated for the 82.3% (n=6871) of participants that responded to all CESD-R-10 items. Among these students, 41.6% (n=2860) scored 10 or higher on the CES-D 10-item version of the scale, the cut-off used to indicate clinically significant depression symptoms.
As expected, there were gender and grade differences. Over half (51.8%) of females reported significant depressive symptoms over the past week, compared to 30.6% of male students. Depressive symptoms were generally more common with increasing grade (see Figure 13).
For comparison, on the Canadian HBSC, 41-43% of grade 9 and 10 girls and 20-22% of grade 9 and 10 boys felt so sad or hopeless almost every day for 2 weeks or more in a row that they stopped doing some usual activities in the past year.5 The relatively higher prevalence in the current sample may reflect differences in assessment or the higher age in the current sample, with mental health problems and depression rates known to increase over adolescence.
Figure 13. Depression (CESD-R-10) scores by grade among secondary school students in the COMPASS Mental Health – Module pilot sample
Relationships among the MH-M scales
Bivariate correlations were conducted for the four scale total scores: the CES-D, GAD-7, Flourishing Scale, and the DERS (see table 10). Higher scores on CES-D and GAD-7 represent higher depression and anxiety symptoms, respectively, and on the DERS, poorer emotional regulation skills. In contrast, higher scores on the Flourishing Scale are said to indicate better psychosocial functioning or wellbeing. As expected, the DERS, CES-D, and GAD-7 all demonstrated moderate positive associations, and had negative correlations with the Flourishing Scale (Table 10).
Table 10. Correlation matrix of MHq scales among secondary students in the COMPASS Mental HealthModule pilot sample
| Flourishing Scale | GAD-7 | CESD-10-R | |
|---|---|---|---|
|
GAD-7 |
-0.493* | -- | -- |
| CESD-10-R | -0.613* | 0.772* | -- |
| DERS | -0.522* | 0.704* | 0.735* |
* P
In further support of the scales’ construct validity, students with better self-rated emotional or mental health scored lower on the depression (CESD-R-10) and anxiety scales (GAD-7), and had better emotional regulation scores (DERS), and higher Flourishing Scale scores (see Figure 14).
Figure 14. MH-M scale means by self-rated mental or emotional health and gender among secondary students in the COMPASS Mental Health – Module Pilot Sample
Note: Higher Flourishing Scale scores indicate better psychosocial functioning, and higher GAD-7, CESD10-R, and DERS scores indicate greater anxiety, depressive symptoms, and difficulty regulating emotions, respectively.
Students with higher anxiety and depression scale scores were less likely to report having no concerns seeking help from an adult at school.
Alarmingly, students with higher depressive or anxiety symptoms were less likely to indicate they could talk about their problems with their family or with their friends, and had much lower likelihoods of reporting they would have no problem talking to an adult at school if they had mental health concerns. Over half (50.9%) of students without significant depressive symptoms (CESD-R-10
Comparison between the Ontario and B.C. Pilot Test Samples
Comparing the Ontario and B.C. pilot test results, no difference was found in the Flourishing Scale, DERS, and CESD-R-10 total scores. However, the GAD-7 scores were higher among students in the Ontario pilot sample relative to the B.C. pilot sample. See table 11. Note that pilot samples did not aim to be representative.
Table 11. Ontario and B.C. pilot test results for the COMPASS Mental Health-Module
| B.C. Pilot Sample M (SD) | Ontario Pilot Sample M (SD) | p-value | |
|---|---|---|---|
| GAD-7 | 7.10 (5.98) | 7.50 (6.29) | 0.005 |
| CESD-10-R | 9.56 (6.26) | 9.63 (6.58) | 0.679 |
| DERS | 14.62 (4.90) | 14.85 (5.18) | 0.056 |
| Flourishing Scale | 31.37 (5.81) | 31.35 (5.98) | 0.874 |
MHq Pilot Test Summary
Overall, the MHq tested well, with high participation and minimal missing data. Rates of missing data were consistent across the items on the first page of the MH-M, and then increased and remained consistent across the second page. This pattern suggests students skipped or missed the second side of the insert, potentially due to response fatigue, time constraints, or not realizing the insert was doublesided. The second page also included the negatively-phrased items whereas the items on the first page are mostly positively framed.
The scales appear reliable and valid for the intended purpose. All scales had high internal consistency, and the expected relationships were found amongst them. With the exception of the self-rated emotional or mental health item, results generally resemble nationally and/or provincially representative samples when available. Differences by gender and grade are as expected based on the research literature.
The proportion of students with CESD-10-R and GAD-7 scores above accepted cut-offs indicating clinically significant depression and anxiety symptoms was higher than the expected levels of major depressive disorder (MDD) or GAD in the general population; however, while these self-report screening scales are found to be valid and reliable tools, they are also shown to overreport9 and should be taken as risk of MDD or GAD, or as clinically significant distress, rather than diagnosable mental disorders. The relatively higher rates to similar studies may reflect the older sample (given that the prevalence of mental health problems or disorders increase with increasing age through adolescence) or the more recent data collection (given that rates of distress or perceptions of internalizing symptoms have shown increases over recent years5). Also, students may have felt more comfortable being honest in reporting anxiety and depression symptoms given that they are not required to provide their name with the passive permission protocols used in COMPASS. Similarly, high-risk students are more likely to participate in studies when these procedures are used, and their inclusion may have contributed to higher rates of mental health problems in the pilot sample relative to studies requiring active permission. Lastly, it should be noted that he COMPASS study and the MH-M pilot sample are not meant to be representative. These factors may have also contributed to the high proportion of students rating their emotional or mental health as “fair” or “poor”.
Focus Groups Results
Focus Group Sample
Parental permission was obtained from 30 students, and 23 of these students were present on the day of the focus groups, signed assent forms, and participated. Participants were divided into three groups:
1. Grade 9 and 10 students (n = 7, including 4 grade 9 students [3 females; 1 male] and 3 grade 10 students [3 females])
2. Grade 11 students (n = 6; 4 females; 2 males)
3. Grade 12 students (n = 10; 9 females; 1 males)
Focus Group Results
The focus group results are summarized below. Note that no formal qualitative analysis was performed.
1. Comprehension
Students agreed the MHq was worded clearly and items were easy to understand.
“Easy to pick and answer them. They’re easy to understand. Things people can relate to, and know about themselves” (female, grade 12)
When asked about the Flourishing Scale, some students considered some items difficult to answer due to being too “vague”, “broad”, or “general”.
“‘I am competent and capable in the activities that are important to me’… you could be talking about school or you could be talking about sports activities … kind of general. Kind of hard to answer that question… Question ‘c’ is also vague – ‘I am engaged and interested in my daily activities’ that is also kind of vague. Are you talking about school or personal?” (male, grade 12)
“What do you mean by daily activities? School or extracurricular?” (female, grade 11)
“Very broad. Engaged in brushing teeth or in a sport?” (female, grade 11)
“And ‘a’ (‘I lead a purposeful and meaningful life’) could be about a few things … Could be talking about an activity or could be talking about school” (female, grade 12)
Students wanted more specific questions or more description/definitions of terms (e.g., definitions of a ‘purposeful life’, ‘depression’, whether items are referring to school or to other aspects of life, etc.).
Regarding the self-rated emotional or mental health item (item 67):
“for me, I think it’s really hard to choose an option for mental health because they might be all over the place. Don’t know if you guys want to put in a description. Maybe want to describe your options a bit more. Maybe to know difference between agree and disagree.” (female, grade 12)
Regarding the Flourishing Scale (item 68):
“Question ‘a’ ‘I lead a purposeful and meaningful life’ is there a certain type of life that you guys would see as purposeful compared to what someone else would see as purposeful? Your opinion of what a good life is.” (female, grade 11)
Regarding the Depression scale (CESD-R-10):
“A lot of people have different definitions of the word ‘depressed’” (female, grade 12) “Yeah, some people would think they have a bad day, that they were depressed that day, so they’ll put 1 to 2 days” (female, grade 12)
Two students reported difficulty answering items that were meant to tap their perceptions of how others view them (Flourishing Scale item 68h).
“How do you know if other people respect you… Don’t know unless you ask them. Can assume. Really hard to accurately judge what others think of you.” (female, grade 11)
“How do you know that other people respect you? They could be different behind your back and act differently upfront.” (female, grade 9)
However, when asked to interpret items in their own words, students were consistently able to explain the meaning correctly. For example:
Flourishing Scale item: “I actively contribute to the happiness and well-being of others”:
“Are you polite, supportive, nice to others.” (female, grade 11)
“Means you make people happy, you’re supportive of people.” (female, grade 10)
“You’re a good person and friendly.” (female, grade 9)
“You care about their health, check in on people, make sure they’re alright.” (female, grade 9)
Flourishing Scale item: “I am engaged and interested in my daily activities”:
“Means you’re involved.” (female, grade 10) “Means you’re doing it because you like it, not because you have to.” (female, grade 9)
Regarding CESD-R-10 item: “I could not get ‘going’”:
“Maybe it means you didn’t want to do it, due to low motivation and energy” (female, grade 9) “Other people might want to do it but you can’t do it” (female, grade 10)
“Did not feel motivated. Could not get motivated. Did not want to do anything” (male, grade 11)
2. Format:
a. Spacing and order within the Cq booklet:
Grade 12 students suggested spacing out or splitting up items more because they found the mental health questions required more time to answer relative to other Cq domains. Some students suggested to spread the MHq items throughout the Cq booklet, while others preferred the MHq as separate because they perceived the section to be important. Instead, they advised moving the MHq earlier in the Cq booklet, as students may be tired and trying to rush by the time they got to the MHq. Overall, student felt the MHq items required greater reflection and were important to answer correctly.
“I agree with *** they should be spaced out. I mean you’re answering a full booklet of questions, and by the time you get to a mental health section. I don’t want to say you’re over it, but you kind of have other goals to complete this, you’re not really concentrated on, and they’re important questions. Or in-depth questions. I agree they should be spaced out. Whether you should remove some questions, or make more specific questions.” (female, grade 12)
“Maybe instead of putting it all in one section, maybe spreading some of the questions out throughout” (female, grade 12)
“I think it should be its own separate thing. Because mental health is important. Maybe word everything different so kind of flows” (female, grade 12)
b. Order of the MHq items:
The grade 12 students suggested to move more general questions first as a lead in, followed by questions they considered more difficult to answer. The grade 12 students suggested moving item 66 (self-rated mental health) to the end of the MHq.
“The first question – just like bang, how do you rate your mental and emotional health…yeah, you’re just answering how much do you weigh and then how do you rate your mental health… You’re not really in that mindset. Maybe ask a couple more questions before that. Or maybe put that a little bit down, so that you’re thinking about it, and then you’re like ok… it’s kind of coming…” (male, grade 12)
“Yeah, I agree with ***, yeah maybe (item 66 ‘how do you rate your emotional and mental health’) should be at the bottom” (female, grade 12)
“Yeah like start with more general questions that don’t require too much thought, and then as you go more and more, you’re having more and more thought as you’re going. Because you need to lead a student into more thought. Yeah, you can just be like ‘am I depressed?’ Maybe, I don’t know, and then more general questions next” (female, grade 12)
3. Comfort in completing the MHq honestly
When grade 9 and 10 students were asked if they would answer the questions honestly, two females (grade 9 and 10) responded positively and two students responded negatively (a grade 10 male and a grade 9 female). Other students wanted to complete the questions correctly because they perceived the MHq and student mental health as important.
“I felt comfortable doing it because you know people don’t talk about it a lot, and this is the first time introducing it, so you wanted to fill it out correctly. You’ll probably use this to help people, so may as well answer correctly…” (female, grade 12)
Reasons for concerns about answering honestly and correctly included seeing family matters are private or denial of personal issues.
“Some people have private things they do with their family, don’t want to share this information (regarding item: “I have a happy home life”) (female, grade 9)
“Hard to talk about appearances with some people” (regarding self-concept item: “I like the way I look”) (female, grade 9) “Some people don’t like to think about these things until they have to” (female, grade 10)
“Some people might not want to say to themselves…like admit to themselves, like 68 ‘a’ or something that they strongly disagree that they have a meaningful life. People don’t usually say that on a daily basis. Nobody wants to admit that to themselves” (female, grade 12)
A couple of students suggested that some individuals may have difficulty answering the questions if they had not given thought to them before.
“These are deep questions. These aren’t something you think about on a daily basis. So when you get to it, it really makes you think and makes you think how should I answer.” (female, grade 12)
“About 73 (Depression scale [CES-D]), a person needs to be thinking about, they need to reflect on their past days, so they need to have some idea of how they feel. But like before if they’re not thinking about it before they can’t answer that question. Like even if they are depressed, if they’re not thinking about how depressed they are. If it’s just normal to them, you can’t really ask about how depressed they are.” (male, grade 12)
Some students suggested formatting changes (e.g., colour, title font size, placement within the Cq) to increase comfort when responding in the classroom environment and with time constraints.
“Have you guys ever thought about doing this online, so some people can do it at home maybe. Because some people in the environment they don’t want to... They can think about it to themselves at home versus when they’re surrounded by people its kind of rushed…” (female, grade 12)
“It just lets a person answer the questions more discretely. Having its own page. Especially when starts at 66, people are starting to rush through the book a bit more. These questions require more thought and you won’t be truthful…” (male, grade 12)
“I know it’s not the point of the questions but usually when I do them I’m in a group environment. Like today I was sitting around a table, like a round table, you could see what everyone was doing. Maybe it’s because this part was loose that it was more obvious to see. It was kind of hard to answer questions truthfully when other people are... Maybe you wouldn’t write mental health in big letters and put it in different colours, which you obviously wouldn’t do when you put it into the thing, but like stuff like that I’d take out so kids can answer it more discretely in an environment when there is a lot of people around. Like when one of your friends is saying stuff like that … a lot of guys at least will kind of joke around.” (male, grade 12)
4. Kids Help Phone
Students had positive impressions of the inclusion of the Kids Help Phone information and logo at the end of the MHq. The Kids Help Phone information was one of the sections identified by grade 9 and 10 females when asked what they considered to be the most important items on the MHq.
“Kids Help Phone, if they don’t like their answers and they think there’s something wrong with them and if they said they didn’t want to talk to someone they can use this.” (female, grade 10)
“Kids Help Phone, if by the end of the survey they realize they have a problem, they can take that away as well since it’s at the end of the survey.” (female, grade 9)
“Seeing it at the end can help you remember you have it.“(female, grade 10)
5. Perceived importance and relevance to secondary school students
Several students expressed seeing the MHq and the content as important to the mental health of students. When asked about the most important questions, items 70 (deterrents to seeking help in school settings) was consistently rated as the most important. Other items and scales identified as particularly important included the Kids Help Phone information and logo (as discussed above), the family and friends related items (particularly 67cd [able to talk about problems with friends and with family]), the Flourishing Scale, and the depression scale (CESD-R-10).
“In my opinion it’s question 70, because I know a couple of people who wouldn’t want to talk to someone because of these reasons. And I just think it is something to draw attention to. Because usually everyone asks ‘oh are you depressed’ and that is the other questions, but it’s like, why wouldn’t you go to someone?” (female, grade 12)
When asked about the least important items, students agreed on item 66 (‘how do you rate your emotional or mental health’) because they felt other areas covered the same content and regarded the item as too general.
“66… yeah, you basically get an idea through the rest of the questions.” (male, grade 12)
“I agree about 66, because the other questions answer that in pieces. Kind of broad to ask someone that question.” (female, grade 12)
6. Repetition
Items comprising the anxiety scale (GAD-7) were identified as repetitive.
“Question 71 (Anxiety scale [GAD-7]). All the options. You’re basically asking the same thing over and over again. Like about nervous, anxious, trouble relaxing, and everything like that. Just seemed really repetitive. Seemed like I was answering the same thing… could probably be asked in two questions instead of seven” (female, grade 12)
“yeah, some are (different). Some could be combined together. like what *** said. Like trouble relaxing or being nervous or anxious… being restless is same as trouble relaxing” (female, grade 12)
7. Additional content suggestions
a. Sources of stress and context:
“Sources of stress and level of stress” (female, grade 11)
“It’s talking a lot about how do you feel about yourself. But not a lot of questions about the surrounding, and community you live in. So like the life at home, what is it like? What you’re like with your friends? Is there a good community of friends you have at your school… yeah at school, family, community, extracurricular activities, coaches… they just talk about yourself. So like makes you feel bad, but if you think about the community you live in, you can think about oh yeah, maybe this is the reason you feel like this. Like the workload you have at school and how much you work, maybe that is having an effect on how much you stress out.” (female, grade 12)
b. Several students suggested asking school specific questions – stress of workload and how handle stress (re: tests and exams).
“Doesn’t touch on school life, how we would handle stressful situations like tests presentations exams” (female, grade 9)
c. Pressures and sources of high expectations (e.g., teachers and peers, in addition to parents):
“Add teacher’s expectations. Not just parents. Expectations for workload in relation to everything else.” (female, grade 11) “Expectations for homework” (grade 11s)
“Peers expect too much. Peer pressure” (female, grade 11)
“A lot of people other than parents have high expectations.” (female, grade 11)
“Ask to write down what you feel pressured to do. Get good grades, get a job. Figure out what to do when older. Hold up an image of self. To act happy. Act like you’re OK.” (female, grade 11)
d. Sexual orientation and gender identity
“Add LGBTQ. Issues not feeling accepted. Not feeling adequate” (female, grade 11)
“Add question do you identify as LBGTQ” (female, grade 11)
e. Family structure and loss (e.g., parental divorce, loss of parent)
“A lot of kids have divorced parents. (item 67) Refers to parents. Add separate question asking if parents separated. Divorce (is) important to mental health.” (female, grade 11)
“Any kind of loss (is) important to mental health.” (female, grade 11)
f. Connection between sleep and mental health
“Connect sleep with mental health” (female, grade 11)
“Maybe a question about sleep as an important part of mental health” (female, grade 10)
“Reasons for not sleeping. On phone. Stress. Living situations – too cold, baby siblings. Homework.” (female, grade 11)
g. Additional mental health problems or disorders. Three grade 12 female students specifically referred to eating disorders.
“a lot of them are anxiety and depression related. There is another full spectrum of mental health problems … They’re all aimed at stress, anxiety, depression, and that… there is a whole spectrum. Add more questions that have to do with, broader spectrum” (female, grade 12)
“Eating disorders for sure” (female, grade 12)
h. More on self-esteem
“Should be more on self-esteem, they touch on it in some parts but don’t cover it fully” (female, grade 9) “Maybe a question describing how you view yourself” (female, grade 9)
6. Additional comments by specific MHq items:
a. Self-rated mental or emotional health (Item 66)
As discussed, self-related emotional or mental health was chosen as the least important item and the one to remove when asked to choose because students believed the other more specific items already give an idea of students’ emotional or mental health. Other concerns included the order (considered “abrupt” as the first item), generality (“too broad”), and perceptions of emotional and mental health as different concepts. Student suggestions for revisions included: moving the item to the end of the MHq, separating mental and emotional health, defining each term or response option for students (i.e., what “excellent” or “very good” mental health looks like), and adding an “I don’t know” option.
b. Family and friends (Items 67a-d)
As mentioned, the friends and family related items (Items 67a-d) were identified as one of the most important content areas overall, particularly the items regarding students’ perceived ability to talk about their problems with their friends and family (67cd).
Students suggested revisions to item 67b (“My parents/guardians expect too much of me”), such as:
- Add parentheses and ‘other’ (i.e., parent(s)/ guardian(s)/ other)
- Add a question about whether parents are divorced
- Students had concerns about how to answer if parents differed in expectations (i.e., one parent may have high expectations and another lower expectations)
- There was confusion about whether expectations were positive or negative – one student suggested rephrasing the item to “unrealistic expectations”
c. Flourishing Scale (Items 68a-h)
Despite comments that some items were “vague” or broad, students were consistently able to interpret the Flourishing Scale items correctly when asked and perceived the content to be important. Students suggested spacing out items more or breaking them up.
Some student concerns about the Flourishing Scale included:
- Items required a lot of self-reflection (i.e., required more time and energy than other COMPASS domains, and may not have thought about questions before)
- Some items were considered too broad, could be interpreted different ways, and/or subjective (e.g., questions regarding whether “activities” means school, extracurricular activities, sports, etc.; definition of “purposeful life,” a “good person” and “good life”, a “rewarding” relationship).
- Regarding item 68h “People respect me”, groups reported difficulty understanding how individuals could know if others truly respect them and uncertainty regarding the context (e.g., friends, in school, etc.)
- One student commented that some items were too mature for their age group (e.g., if living a “purposeful” or “a good life”) but considered them to be important:
“Too young to understand if fully live a purposeful and meaningful life. Good question though… Mature wording – ‘live a good life’.” (female, grade 11)
d. Perceived deterrents to help seeking at school (item 70)
All students perceived item 70 as the most important item and found the wording and format clear. Several additional response options were suggested as reasons that students may not seek help:
- Family/ “someone at home” might not want you to talk about problems or student may prefer to keep family or home-related concerns private
- May already have support (e.g., already have talked to friends, family, mental health professionals outside of school)
- May not consider staff to be qualified (i.e., in terms of certifications/training) or able to understand and help (i.e., “haven’t been through it”)
- A number of students expressed concerns about school staff/teachers becoming involved in their lives (e.g., “overhelp”, bring more attention to problem, go to friends or parents)
- Problem might be stemming from school (therefore they may not want to address it at school)
- Several students discussed the type of relationship they had with school staff/teachers a barrier (e.g., if counsellor is known to student outside of counselling role as a teacher or otherwise; concern might use material against student; all groups saw certain teachers or teacher-student relationships as “too joking” and not serious; teacher in authority role)
- Concerns that they would not be taken seriously or believed because of their age
- Felt “intimidated” or “threatened” - “Not just too embarrassed but too uncomfortable”
- May not know how to approach someone (in addition to who), both in terms of how to express their feelings and how to approach/access school’s mental health worker
- May not recognize they have a mental health problem
- May not know how to express their feelings or may not understand their own feelings
Student suggestions for revisions to item 70:
- Examples of staff (e.g., social worker, child and youth worker, etc) seen as unnecessary by two students (one student replied they didn’t read this) and able to remove
- Remove “word would get out” because could other reasons for not trusting staff
- Suggested to remove “I’d be too embarrassed”
- Add “other” option and space to fill in
- Change to two questions – school specific and general
- Add question about school support/resources capable to deal with it (not just specific staff)
e. Generalized Anxiety (Item 71a-g [GAD-7])
Students consistently perceived the content as important and clearly worded but reported there was a lot of repetition within the scale and overlap with other scales. Also, one group commented on the time frame (2 weeks). Students advised using a longer period (i.e., students commented the question does not acknowledge if always feel anxious or specific to last 2 weeks when there could have been a reason [e.g., test])
f. Emotional skills (Item 72a-f [Difficulty with Emotional Regulation Scale [DERS])
A student suggested changing the order of items (advised to move [b] “I pay attention to how I feel” before [a] “I have difficulty making sense out of my feelings”).
Item 72d (“When I’m upset, I believe there is nothing I can do to make myself feel better”) was identified as a question people may feel uncomfortable answering. A student advised changing the item to: “I have trouble making myself feel better”.
g. Depressive symptoms (Item 73a-j [CES-D])
Identified as among the most important and clearly worded items.
Some concerns that were mentioned:
- Regarding item 73c, students argued “depression” could be defined differently by different people (i.e., ‘depressed’ as in a sad mood or a chronic depressive disorder)
- The time frame of seven days was considered too short because it may have been a bad week; students suggested changing it to a month and/or rewording to specify “in a typical week”
- Regarding item 73j (“I could not get ‘going’”), all three groups reported concerns about the meaning and wording of this item, and preferred the word “motivated,” but students consistently interpreted the item correctly when asked
Focus Group Summary
Overall, comprehension, comprehensiveness, and perceived importance and relevance appear favourable. Some students have difficulty with the lack of specificity or abstract nature of questions aiming to assess their global mental health or psychological and social functioning; however, they were consistently able to correctly interpret items and perceived them to be important. Formatting changes (e.g., spacing, order, colour) may assist in more careful and honest responses when in classroom environments, and with fatigue and time constraints. Additional content areas and response options (e.g., regarding barriers to help seeking) will be considered in light of space constraints. Potential revisions to the wording, time frame, or repetitiveness of items within scales are restricted by the need to be consistent with validated measures and to allow comparison to nationally and provincially representative youth samples.
MHPP Pilot Test Results
1. Primary mental health concerns
School representatives at each of the 14 schools were asked to “Please rank the following areas of primary concern related to your students’ mental health” and provided the instructions “Rank items from 1 to 8 where 1 =highest priority, 10 = lowest priority.” Eight mental health-related concerns were listed. The question was completed correctly by all 14 school representatives. The leading concerns included anxiety symptoms, self-harm and/or suicidality, and depressed mood (see Table 12). Depression and anxiety are addressed in the student-level measure (MHq), while we are unable to assess self-harm and/or suicidality due to the passive permission procedures used by COMPASS. Note that substance use is already addressed in COMPASS. In response to the follow-up question, “Are there any other mental health-related issues that are important to your school that are not listed in Question 51 above?” Eleven schools reported “No”, two schools did not respond, and one school reported “Yes” but did not provide further details on the mental health issue missing from the response options.
Table 12. Primary concerns related to student mental health as reported by school representatives on the MHpp in the COMPASS Mental Health – Module pilot study
| Mental health concern | Number 1 rankings by schools (n = 14) | Top 3 rankings by schools (n = 14) |
|---|---|---|
| Anxiety symptoms | 5 | 10 |
| Self-harm and/or suicidality | 5 | 10 |
| Depressed mood | 1 | 9 |
| Disruptive behavioural issues | 2 | 5 |
| Trauma | 1 | 2 |
| Substance use | 0 | 4 |
| Disordered eating | 0 | 1 |
| Attentional problems | 0 | 0 |
2. School Staff Training
School respondents were asked “during the past 12 months, how many staff have received the following training related to mental health?” The item was completed by all 14 school respondents. According to some recommendations (e.g., School Mental Health ASSIST), all school staff should be trained at the level of mental health awareness, and some staff should have additional training to provide support. Seven schools reported “all or most” staff had received mental health awareness or mental health literacy (e.g., basic information, key warning signs) training in the last 12 months, while seven schools reported “some (e.g., 1-5)” staff had received this level of training. One school reported none of their staff had received training in providing mental health support, and two schools reported no staff had been trained in suicide prevention in the last year. For other training types, one school reported “all or most” staff had been trained in “mindfulness, brain-based research and behaviour”. Training was most often delivered in the form of workshops provided on professional development days, followed by presentations by community organizations, conferences, teacher-initiated on the internet, and in-service training (e.g., by Public/Regional health) (see table 14).
Table 13. School staff mental health-related training in the past 12 months reported by school representatives on the MHpp in the COMPASS Mental Health – Module pilot study
| During the past 12 months, how many staff have received the following training related to mental health? | All or most | Some (e.g., 1-5) | None |
|---|---|---|---|
| Mental health awareness/literacy (e.g., basic information, key warning signs) | 7 | 7 | 0 |
| Providing mental health support (e.g., mental health first aid, Supporting Minds, etc.) | 2 | 11 | 1 |
| Suicide prevention | 4 | 8 | 2 |
| Other | 1 | 0 | 3 |
Table 14. Delivery form of school staff mental health-related training in the past 12 months reported by school representatives in the COMPASS Mental Health – Module pilot study
| During the past 12 months, have school staff received the following forms of training for mental health? | Schools (n = 14) |
|---|---|
| In-service training (e.g., by Public/Regional Health) | 5 |
| Conferences | 6 |
| Workshops on professional development days | 11 |
| Presentations by community organizations | 7 |
| Teacher initiated self-training on the internet | 6 |
| Faculty of Education courses | 1 |
| Other | 8 |
3. Mental Health Specialist Staff
The 14 schools respondents listed any mental health professionals available at their school (e.g. Social Worker, Child and Youth Worker, Psychologist, etc.) and indicated their availability (on-call, on-site fulltime, or regularly scheduled for an indicated number of hours per month). Two schools filled in two specialist staff and 12 schools filled in three different specialist staff that were available. The most common types of specialist staff indicated included: psychologists, social workers, child and youth workers, mental health nurses, and counsellors. Three of mental health specialist staff were on-site full time, 12 were on-call only, and 26 were regularly scheduled for a set number of hours or days per week or month (e.g., one morning, one day, or 12 hours per week; 15 to 40 hours per month). One school reported being unsure how often a specialist staff was available.
4. Mental health services available on-site
See Table 15 for school administrator responses to the MHpp item “Are any of the following mental health services available on-site at your school? (Check all that apply).” One school respondent did not report any of the mental health services listed were available on-site, and one school reported all 10 of services were available on-site at the school. Two schools had all services except group counselling/therapy. The most common services available were referrals to specialized programs or services, crisis intervention, individual counselling/therapy, substance abuse counselling.
Table 15. Mental health services available on-site at the school reported by school representatives on the MHpp in the COMPASS Mental Health – Module pilot study
| Are any of the following mental health services available on-site at your school? (Check all that apply) | Schools (n = 14) |
|---|---|
| Assessment for emotional or behavioural problems (including behavioural observation, psychosocial assessment and observation checklists) | 10 |
| Diagnostic assessment (comprehensive psychological evaluation) | 5 |
| Behavioural management consultation with teachers, students, or families | 8 |
| Case management, including monitoring and coordination of services | 9 |
| Referral to specialized programs or services for emotional or behavioural problems or disorders | 11 |
| Crisis intervention (e.g., response to traumatic events, including disasters, serious injury/death of a member of the school community) | 11 |
| Individual counselling/therapy | 11 |
| Group counselling/therapy | 2 |
| Substance abuse counselling | 11 |
| Family support services in school setting (e.g., child/family advocacy, counselling) | 5 |
5. Referral procedures
All 14 school administrators responded to question, “what are your general practices for routine referral to and coordination with community-based mental health organizations or providers? (Check all that apply)” (See table 16). None of the schools reported that they did not make any referrals. Four schools reported all five options provided (passive and active referrals, following up with student/family and with the provider, and hosting/attending team meetings with community providers). Three of the schools made passive but not active referrals, with two of these also hosting or attending team meetings with providers and the other school reporting no additional procedures.
Table 16. Referral procedures and coordination with community mental health organizations and providers reported by school representatives on the MHpp in the COMPASS Mental Health – Module pilot study
|
What are your general practices for routine referral to and coordination with communitybased mental health organizations or providers? (Check all that apply) |
Schools (n = 14) |
|---|---|
| Staff make passive referrals (e.g., give brochures, lists and contact information of providers or organizations) | 9 |
| Staff make active referrals (e.g., staff complete form with family, make calls or appointments, assist with transportation) | 11 |
| Staff follow-up with student/family (e.g., calls to ensure appointment kept, assess satisfaction with referral, need for follow-up) | 6 |
| Staff follow-up with provider (via phone, e-mail, mail) | 9 |
| Staff host or attend team meetings with community providers | 10 |
| Staff do not make referrals | 0 |
6. Role of the local Public Health Unit
Greater collaboration with public health units is encouraged and desired by both schools and public health; however, both sides have reported a disconnect. Consistent with the other domains of COMPASS, the MHq item “During the past 12 months, what role does your local Public Health Unit when working with your school on improving mental health for students (Check all that apply)” was added. Twelve of the 14 school administrators responded (see Table 17). Five schools reported they had no contact with their local Public Health Unit in the past year in regard to the mental health of their students. Six schools reported the Public Health Unit had provided resources, information, or programs, but no schools had developed or implemented programs together.
Table 17. Role of the local Public Health Unit in improving student mental health reported by school representatives on the MHpp in the COMPASS Mental Health – Module pilot study
| During the past 12 months, what role does your local Public Health Unit when working with your school on improving mental health for students (Check all that apply) | Schools (n = 14) |
|---|---|
| No contact with local Public Health Unit | 5 |
| Provided information/resources/programs (e.g., posters, toolkits) | 6 |
| Solved problems jointly | 1 |
| Developed/implemented program activities jointly | 0 |
7. Mental Health Programs
In response to the question “other than classes/curriculum, does your school provide any programs to promote mental health?”, school respondents were asked to indicate programs run by the school, by their local Public Health Unit, or by an external organization. Six programs were run by schools, 6 by their local Public Health Unit, and 4 programs were run by external organizations. When asked for further details, one school reported a school-wide presentation/assembly, a girls and a boys group for anxiety, and peer support programs. A second school also reported peer support programs, which were run by both the school and external organizations. A third school reported that information and activities were provided to students in teacher advisory classes. Four schools reported providing programs but did not provide additional details.
MH-M Tool Revisions
MHq Revisions
Based on the pilot study and focus group results, the following revisions to the MHq are planned:
1. Format:
The MHq will be printed in the same colour and format as the larger student questionnaire COMPASS booklet to allow students to complete the questions more discretely. Students in the focus groups reported that the separate insert in bright orange made them less comfortable filling out the mental health items in a crowded classroom.
2. Order within the COMPASS student questionnaire (Cq):
The MHq items will be moved forward to be completed earlier in the questionnaire. This change was made based on student feedback that they were tired or rushing for time by the time they got to the MHq insert at the end.
3. Order of the Mhq items:
Self-rated mental health will be moved to the end of the MHq questions, instead of the beginning. Students in the focus groups considered self-rated mental health too abrupt as the first item and suggested they would be better prepared to answer once they had reflected on the other MH items.
4. Rewording:
The self-rated mental health item will be reworded to “In general, how would you rate your mental health” from “How would you rate your mental or emotional health”. Students reported difficulty answering the item as they considered emotional and mental health to be two different concepts that they would not necessary rate the same. Also, as students in the focus groups were unclear whether questions were referring to the school, social, family, or other contexts and activities, “in general” was added to help portray that the question is intended to be a global measure. For the item assessing potential deterrents to students asking for help from an adult at school if they had mental health concerns, the response option “there is no one to talk to” will be changed to “there is no one I feel comfortable talking to”. This option was the least frequently chosen response in the pilot test. When students in the focus groups were asked if any potential barriers to seeking help were not listed, several students indicated the type of relationship they had with teachers was a concern.
5. Additional item:
The item “I generally recover from set-backs quickly” will be added to assess student resiliency. More additions were not due to time constraints. Classrooms completed the MHq and the Cq within one classroom period, but some students in the focus groups reported feeling rushed near the end.
MHpp Revisions
1. Mental Health-Related Programs:
Given that four schools reported running programs but did not provide further detail, revisions will be made to better assess the mental health-related programs running in schools. Also, a question will added to assess when the programs were implemented, in order to allow evaluation of their effect on students over time. The question will be revised to the following:
2. Student Lead Mental Health Initiatives:
The MHpp draft pilot test version only assessed mental health-related programs and other initiatives being run by the school, outside organizations, or the local Public Health Unit. A question will be added to the revised MHpp that will assess the presence of student lead mental health initiatives, given that several of these programs, campaigns, and related activities continue to emerge in schools.
3. Mental Health Professionals:
The MHpp pilot test draft version asked school representatives to fill in the type of mental health professionals available at their school. Given that the same types of mental health professionals were consistently mentioned by schools, and that a few schools reported specific names or agencies, the question will be modified to list the most common types of professionals that were listed in the pilot test. Not having to type/write in this information will ease burden on school respondents, and in later data entry. An “other” option will be provided to capture any mental health professionals that are not listed.
The revised question format will be as follows:
References
-
Patte KA, Henderson J, Elton-Marshall T, Faulkner G, Sabiston C, Bredin C, Battista K, Leatherdale ST. Development of a mental health module for the COMPASS system: Improving youth mental health trajectories. Part 1: Draft Development and Design. 2017; 4, 2. Waterloo, Ontario: University of Waterloo. Available at: www.compass.uwaterloo.ca.
-
Mental Health Commission of Canada. School-based Mental Health in Canada: A Final Report. School-Based Mental Health and Substance Abuse (SBMHSA) Consortium. Calgary, AB: Mental Health Commission of Canada, 2013.
-
Leatherdale ST, Brown KS, Carson V, Childs RA, Dubin JA, Elliott SJ, et al. The COMPASS study: a longitudinal hierarchical research platform for evaluating natural experiments related to changes in school-level programs, policies and built environment resources. BMC Public Health. 2014;14:331.
-
Thompson-Haile A, Leatherdale ST. Student-level Data Collection Procedures. COMPASS Technical Report Series. 2013; 1(5). Waterloo, Ontario: University of Waterloo. Available at: www.compass.uwaterloo.ca
-
Boak A, Hamilton HA, Adlaf EM, Henderson JL, Mann RE. The mental health and well-being of Ontario students, 1977-2015: Detailed OSDUHS findings (CAMH Research Document Series No. 43). Toronto, ON: Centre for Addiction and Mental Health, 2016.
-
Freeman JG, King M, Pickett W, (Eds). Health Behaviour in School-aged Children (HBSC) in Canada: Focus on Relationships. Her Majesty the Queen in Right of Canada, 2016. http://healthycanadians.gc.ca/publications/science-research-sciences-recherches/health-behaviour-children-canada-2015-comportements-sante-jeunes/index-eng.php
-
Diener E, Wirtz D, Tov W, Kim-Prieto C, Choi D, Oishi S, Biswas-Diener R. New measures of well-being: Flourishing and Positive and Negative Feelings. E. Diener (ed.), Assessing Well-Being: The Collected Works of Ed Diener, Social Indicators Research Series 2009;39, DOI 10.1007/978-90-481-2354-4 12, Springer Science+Business Media B.V. 2009.
-
Offord Centre for Child Studies. Student mental health in Ontario: the role of schools. Selected findings from the School Mental Health Surveys. Presented by K. Georgiandes at the Children’s Mental Health Ontario (CMHO) Annual Conference 2015, November 24, 2015. Toronto, ON.
-
Stockings E, Degenhardt L, Lee YY, Mihalopoulos C, Liu A, et al. Symptom screening scales for detecting major depressive disorder in children and adolescents: a systematic review and meta-analysis of the reliability, validity and diagnostic utility. J Affective Disorders 2015; 174: 447-463.