COMPASS protocol changes and recruitment for online survey implementation during the COVID-19 pandemic

COMPASS Technical Report Series, Volume 7, Issue 2, December 2020

COMPASS protocol changes and recruitment for online survey implementation during the COVID-19 pandemic (pdf)

Table of Contents

Acknowledgements
Introduction
Process to switch from paper-based to online survey administration
Revised protocols
Supporting documents
Knowledge brokering process
Recruitment
Administering the online student survey
Participation rates 
References

Acknowledgements

Authors

Breanne Reel, MPH1
Kate Battista, PhD(c)1
Scott T. Leatherdale, PhD1

1- School of Public Health and Health Systems, University of Waterloo, Waterloo, ON Canada.

Report funded by:

The COMPASS study has been supported by a bridge grant from the CIHR Institute of Nutrition, Metabolism and Diabetes (INMD) through the “Obesity – Interventions to Prevent or Treat” priority funding awards (OOP-110788; awarded to SL), an operating grant from the CIHR Institute of Population and Public Health (IPPH) (MOP-114875; awarded to SL), a CIHR project grant (PJT-148562; awarded to SL), a CIHR bridge grant (PJT-149092; awarded to KP/SL), a CIHR project grant (PJT-159693; awarded to KP), and by a research funding arrangement with Health Canada (#1617-HQ-000012; contract awarded to SL). The COMPASS-Quebec project additionally benefits from funding from the Ministère de la Santé et des Services sociaux of the province of Québec, and the Direction régionale de santé publique du CIUSSS de la Capitale-Nationale.

Suggested citation:

Reel B, Battista K, Leatherdale ST. COMPASS Protocol Changes and Recruitment for Online Survey Implementation During the Covid-19 Pandemic: Technical Report Series. (2020); 7(2): Waterloo, Ontario: University of Waterloo. 

Contact:

COMPASS Research Team
University of Waterloo
200 University Ave West, TJB 2319
Waterloo, ON Canada   N2L 3G1
compass@uwaterloo.ca 

Introduction

COMPASS is a 9-year longitudinal study (starting in 2012-13) designed to follow a prospective cohort of grade 9 to 12 students attending a convenience sample of Canadian secondary schools over several years to understand how changes in school environment characteristics (policies, programs, built environment) and provincial, territorial, and national policies are associated with changes in youth health behaviours [1]. COMPASS originated to provide school stakeholders with the evidence to guide and evaluate school-based interventions related to obesity, healthy eating, tobacco use, alcohol and marijuana use, physical activity, sedentary behaviour, school connectedness, bullying, and academic achievement. Based on feedback from participating schools as well as emerging issues in youth health, COMPASS expanded its topic areas to include mental health, prescription drug use, and gambling. COMPASS has been designed to facilitate multiple large-scale school-based data collections and uses in-class whole-school sampling data collection methods consistent with previous research [2-5]. COMPASS also facilitates knowledge transfer and exchange by annually providing each participating school with a school-specific feedback report that highlights the school-specific prevalence for each outcome, comparisons to provincial/territorial and national norms or guidelines, and provides evidence-based suggestions for school-based interventions (programs and/or policies) designed to address the outcomes covered in the feedback report (refer to: COMPASS system).

In March 2020, the World Health Organization characterized the coronavirus outbreak as a pandemic [6]. In response to the growing concern of this novel virus, provinces and territories in Canada swiftly suspended all in-class instruction as a means to limit its spread, with no certainty of when in-class instruction would resume [7,8]. At the time of these in-class closures, there were many COMPASS data collections scheduled for the remainder of the school year. With in-class instruction suspended, the COMPASS team had to cancel the scheduled in-class data collections and consider alternative modes of administering its annual student survey. In order to continue to collect valuable data, the COMPASS team decided to administer an online version of the survey for students to complete at home (in alignment with the new mode of delivery for the school curriculum). In addition to the annual core survey questions, the COMPASS team also decided to add pandemic-specific questions. The purpose of this report is to (a) outline the process taken to switch from paper-based to online survey administration; (b) highlight the recruitment process and participation numbers; and (c) summarize the process for administering the online COMPASS survey.

Process to switch from paper-based to online survey administration

The online version of the COMPASS survey had previously only been piloted in the classroom among students with special needs at a handful of schools upon request. This mode of delivery had yet to be piloted in a whole-school sample, nor outside of the classroom. As such, new procedures, which are outlined below, were required. Details on the original procedures can be found in this technical report [9].

Revised protocols

  1. Qualtrics was secured as the platform to house the online student survey. Considerations for this platform included data security, an easy to use survey development platform, an already existing license agreement with the University of Waterloo, and that the data collected through this program are housed in Canada. Additionally, using Qualtrics allowed the COMPASS team to create a unique survey link for each school and therefore collect school-specific data.
  2. It was decided to close the surveys two weeks after the scheduled data collection day in Ontario and British Columbia to allow students ample time to complete the survey at their convenience. In Quebec, it was decided to close the surveys four weeks after the scheduled data collection day. This decision was made by the Quebec leadership team after the first online survey opened in Quebec and was an attempt to improve participation after seeing some low initial rates.
  3. Under the circumstances where in-person data collection was not allowed, no school built environment data (Co-Sea) were collected for these data collections.
  4. Since wrap-up packages would not be mailed to schools until COMPASS staff were allowed back in the office, it was decided that the Recruitment Coordinator would email the permission sharing agreement form to the school contacts along with the PDF version of the School Health Profile (SHP).
  5. School contacts were made aware that their honorarium cheque would not be mailed until the COMPASS staff were allowed back in the office and that this date was undetermined.

Supporting documents

  1. The previous parent information script was modified to inform parents of the online survey. This new script included an email address for parents to use to withdraw their child(ren) from the study, whereas previous scripts included a phone number. Schools no longer had the option to send a paper copy of this script to parents. Instead, a digital copy of this script was sent to parents via the preferred method of the school – either email, voicemail, school newsletter, or social media.
  2. New student communications were drafted by the COMPASS team. These communications included two email templates that were sent to students by school staff. The first email was an invitation email that included the survey link as well as pertinent information regarding the survey. The second communication was a reminder email. The purpose of this reminder email was to increase participation rates, as previous research has shown that reminder emails improve participation rates [10].

Knowledge brokering process

  1. The SHP was edited to reflect the questionnaire changes. These changes included the following:
    1. Removing pages from the SHP that no longer had measures to report as a result of questionnaire changes and/or small sample sizes (e.g., obesity and bullying outcomes and the gender breakdown);
    2. Consolidating or reformatting health behaviour pages that had fewer measures to report as a result of questionnaire changes and/or small sample sizes (e.g., physical activity, healthy eating, tobacco use, and mental health outcomes);
    3. Removing the prior year comparison page due to comparability issues related to sample differences;
    4. Adding a covid-19 outcomes page; and
    5. Adding a participation rate and text at the beginning of the SHP to explain that due to smaller sample size and other sample differences, that this year’s data may not be entirely comparable to previous years.
  2. It was decided not to share a School Policies and Practices (SPP) summary report with schools that participated online as this report primarily included data from questions that were removed from the student survey.
  3. Previously, both a PDF and hard copy version of the SHP was shared with the school. However, without being allowed on campus and to limit the number of packages delivered during this unprecedented time, it was decided to send solely a PDF version of the SHP.

Once the process was finalized and supporting documents were drafted, the COMPASS team sought ethics clearance from the University of Waterloo (UW) as well as their associated institutions (i.e., Laval University, University of British Columbia, and University of Alberta). Protocols along with supporting documents were submitted to and received clearance from UW ethics in April 2020. Ethics approvals from the associated institutions were received in April and May 2020. Once ethics clearance was obtained, COMPASS staff reached out and obtained approval from school boards regarding the new protocols and additional pandemic-related content.

Recruitment

At the same time the revised protocols were being organized, province-specific Research Coordinators reached out to the schools that previously had a data collection scheduled to first, cancel their in-school data collection and, second, to gauge interest in and ability to administer the student survey online. To administer the student survey online, schools needed to have direct communication access to both parents and students. This ability to access students directly (i.e., through student email addresses instead of parent/guardian emails) was essential due to the confidential nature of the survey.

The feedback from these schools was positive with many schools across BC, ON, and QC expressing interest in and capability to administer the student survey online. As such, it was decided to also invite the remaining schools that had yet to schedule a data collection for the 2019-20 data collection cycle (Year 8) to participate in the online student survey.

The following describes the recruitment attempts and decisions made in each province and territory:

  1. Schools in Alberta were unwilling to participate after the schools were moved to online format. Therefore, no Alberta schools participated in the online COMPASS survey during the 2019-20 online data collection cycle.
  2. Due to the various difficulties with administering an online option in Nunavut – including limited access to internet and electronic devices – schools in this territory did not proceed with the online version of the COMPASS survey.
  3. In Ontario, there was a varied response from schools. Some schools were keen to participate, citing the importance of collecting health data during this time, while others opted out, citing they were overwhelmed with the transition to online teaching and did not have the capacity for additional non-essential projects. Schools that agreed to participate were asked to choose a preferred data collection date. At the end of the Year 8 data collection cycle, 20 Ontario schools participated in the online student survey.
  4. Similar to Ontario, there was a varied response from schools in Quebec. Many schools agreed to participate, while others opted out, citing the transition to online teaching was already challenging enough. The difference in recruitment in Quebec compared to Ontario was that the COMPASS-Quebec team recruited all schools to participate on the same date – May 12th, 2020. A small minority of schools were not able to participate on this date and opted for another date of their choice. The decision to have schools all participate on the same date was to facilitate the data collection planning. At the end of the Year 8 data collection cycle, 29 Quebec schools participated in the online COMPASS survey. Most schools participated on May 12th, 2020, with 5 schools participating on a different day.
  5. In British Columbia, necessary school board approval for the modified survey was not obtained from two boards. Therefore, four schools within these boards were not recruited and did not participate. Two additional schools, where board approval was obtained, decided not to participate online. The remaining two B.C. schools were successfully recruited and participated in the online COMPASS survey.

As of the end of the Year 8 data collection cycle, a total of 51 schools participated in the online student survey and 49 schools (22 ON, 3 AB, 18 QC, and 6 BC) chose not to participate online.

Administering the online student survey

Once the schools agreed to participate in the online COMPASS survey, the following process for administering the student survey started immediately:

  1. Recruitment Coordinators contacted schools to confirm a data collection start date. In Ontario and B.C., the data collection date was required to be set no sooner than two weeks from the date of contact in order to allow time to send the parent information materials. The data collection date was also required to be set before June 15th, the year-end deadline set by the COMPASS team to allow for a two-week participation window before school finished for the year. In Quebec, the data collection date chosen by COMPASS-Quebec staff was relayed to school contacts. A data collection time was no longer needed when setting up a data collection, only a data collection start date.
  2. Once the data collection start date was confirmed, the Recruitment Coordinator sent the parent information script to the school contact. This script included information regarding the COMPASS survey including but not limited to the main themes of the survey, the data collection date, and how to withdraw their child(ren) from participation.
  3. The Recruitment Coordinators contacted the COMPASS Program Manager, who created a school-specific survey link in Qualtrics. At this time, the link was activated, but was not shared with school contacts.
  4. Two weeks before the data collection date, the parent information script was sent by the school contact to parents via the preferred mode of communication typically used by the school (usually email or voicemail). It was sent to parents at least two weeks before the data collection start date to allow parents time to withdraw their child(ren) from the survey if desired.
  5. Two days before data collection start date, the Recruitment Coordinator emailed the school contact a list of any students who did not have permission to complete the survey (if applicable).
  6. In addition to the no-permission list, the Recruitment Coordinator also emailed the survey invitation email template to the school contact two days before the data collection. This email template included the survey link as well as pertinent information regarding the survey. It was asked that the school send this invitation email to students on the day of the data collection.
  7. The school contact was again notified the day before the data collection of any additional students added to the no-permission list.
  8. On the start day of the data collection, the school emailed the invitation template to all students except those on the no-permission list. The invitation email was sent on the data collection date at the convenience of each school administrator – usually in the morning by 9AM.
  9. On the morning of the data collection start date, the Recruitment Coordinator emailed the school to confirm that the invitation email was sent to all students, except those included on the no-permission list.
  10. Once the school contact confirmed that the invitation email was sent to students, the Recruitment Coordinator emailed the school contact a link to the School Policies and Practices (SPP) questionnaire, which is completed by school personnel. The purpose of emailing the SPP this link after the student invitation email was to limit the number of survey links sent at the same time and to minimize confusion.
  11. Four business days after the data collection start date, the Recruitment Coordinator emailed the school contact with the second student email template. This second communication was a reminder email with the purpose of increasing participation rates. Schools were asked to send this reminder email to all students except those with no permission one week after their data collection start date.
  12. One week after the data collection start date, the school emailed the reminder email to all students except for those on the no-permission list.
  13. One week after the data collection start date, the Recruitment Coordinator emailed the school contact to confirm that the invitation email was sent to all students, expect those included on the no-permission list.
  14. In Ontario and B.C., the survey was scheduled to close two weeks after the data collection start date and students were no longer able to access it. In Quebec, the survey was scheduled to close four weeks after the start date. These scheduled closing dates were not strictly adhered to and there were some exceptions.
  15. Once the survey was closed, the Recruitment Coordinator reached out to the school contact notifying them that the survey was closed, thanking them for their participation, and letting them know that their data would be sent to them shortly.
  16. As per original protocol, if the school-level data was not completed at time the student survey was closed, the Recruitment Coordinator also followed-up with the school contact asking them to complete it.
  17. As per original protocol, if the school-level data was completed at the time the student survey was closed but had missing responses, the Recruitment Coordinator also followed-up with the school contact regarding those specific questions.
  18. Once the data were processed, the SHP was created, and the SPP questionnaire was completed, the Recruitment Coordinator emailed a PDF copy of their SHP to the school contact and connected them with their Knowledge Broker. This first part is standard protocol. As a result of the revised protocol, the means by which the permission sharing agreement form is shared with the school changed. That is, the permission sharing agreement form was also sent in this email along with the SHP and the school contact was asked to complete it at their earliest convenience. The purpose of the permission sharing agreement form is to let the UW COMPASS team know whom they can share the school’s results with – one, neither, or both their school board and Public Health Unit.
  19. As per original protocol, the Knowledge Broker emailed the school contact one week after the school received their Profile – to allow the school some time to review their data. The purpose of the Knowledge Broker is to answer any questions the school may have regarding their data and to assist the school with any further data requests or general requests such as help with grants.

The above protocols regarding student communication were the minimum protocol standards that the UW COMPASS team and participating schools adhered to. Some schools decided to go above and beyond these protocols to engage with their students and those parents more frequently. Best practices for various engagement strategies will be discussed in a future report.

Participation rates

Encouraging student participation using an at-home, online questionnaire format during a time of limited student-school engagement proved to be challenging. Typical participation rates for in-class, paper-based administration have averaged 80%, with only slight variation by school or province. A post-hoc analysis of participation rates for the online survey showed an average participation rate of 29% in the overall sample, with provincial averages of 22% in Ontario, 37% in Quebec and 19% in B.C.

In addition to provincial variations, participation rates also varied dramatically by school, with participation rates ranging from as low as 5% to as high as 63%. In general, schools in Quebec and private schools had higher participation rates. A detailed investigation into the impact of various administration protocols and school engagement attempts on participation rates will be completed in a future report.

There was also a noted decline in the number of students fully completing the online questionnaire, as compared with the traditional in-class, paper-based administration mode. Overall, approximately 82% of participating students completed to the end of the survey. Considerations for the lower completion rates include technical limitations and a lack of scheduled, dedicated time for participation.

In addition to lower overall participation rates, the demographic distribution of online survey respondents was found to differ from previous years. Meaningful within-school sample differences in terms of age and sex were noted. To adjust for these sample differences and to increase comparability across years, within- and between-school sample weighting will be used. Weighting procedures will be detailed in a future report.

References

  1. Leatherdale ST, Brown KS, Carson, V, et al: The COMPASS study: a longitudinal hierarchical research platform for evaluating natural experiments related to changes in school-level programs, policies and built environment resources. BMC Public Health. 2014,14,33 doi:10.1186/1471-2458-14-331
  2. Leatherdale ST, Burkhalter R: The substance use profile of Canadian youth: exploring the prevalence of alcohol, drug and tobacco use by gender and grade. Addict Behav 2012, 37:318-32
  3. Leatherdale ST, Manske S, Faulkner G, Arbour K, Bredin C: A multi-level examination of school programs, policies and resources associated with physical activity among elementary school youth in the PLAY-ON study. Int J Behav Nutr Phys Act 2010, 25;6. doi: 10.1186/1479 -5868-7-6.
  4. Leatherdale ST, McDonald PW, Cameron R, Brown KS: A multi-level analysis examining the relationship between social influences for smoking and smoking onset. Am J Health Behav 2005, 29:520-530.
  5. Leatherdale ST, Papadakis S: A multi-level examination of the association between older social models in the school environment and overweight and obesity among younger students. J Youth Adolesc 2011, 40:361 - 372.
  6. World Health Organization, 2020. Timeline of WHO’s response to COVID-19. Accessed on July 14, 2020 from: https://www.who.int/news-room/detail/29-06-2020-covidtimeline
  7. The Star, 2020. B.C. Becomes latest province to close schools indefinitely amid novel coronavirus concerns. Accessed on July 14, 2020 from: https://www.thestar.com/news/canada/2020/03/17/bc-becomes-latest-province-to-close-schools-indefinitely-amid-novel-coronavirus-concerns.html
  8. Macleans, 2020. Could the school year be over because of coronavirus? Access on July 14, 2020 from: https://www.macleans.ca/education/school-closures-due-to-coronavirus-may-last-longer-than-you-think/
  9. Thompson-Haile A, Leatherdale ST. Student-level Data Collection Procedures. COMPASS Technical Report Series. 2013; 1(5). Waterloo, Ontario: University of Waterloo. Available at: www.compass.uwaterloo.ca.
  10. Monroe, M. C., & Adams, D. C. (2012). Increasing response rates to web-based surveys. Journal of Extension, 50(6), 6-7.