1.1 Aims

Cash (2018) highlights the importance of rigour in design research for answering key research questions and producing real impact in design teaching and practice. Given the widespread use PA has found in engineering design education research, a comprehensive review of this research can facilitate a critical assessment of the methodology and contribute to improved rigour of future protocol studies in engineering education. As such, in this paper we systematically review the use of PA to study engineering design processes, taking a special interest in novice designers in learning contexts. Our goal is to not only understand how PA has been used but also assess its usefulness and limitations and to identify opportunities for improving the method and/or extending its use in new design learning contexts. Specifically, this study has three objectives:

1. Describe how the PA methodology has been used and adapted in engineering design education research, focusing on protocol creation and analysis.
2. Evaluate the range of research questions in engineering design education that PA has been used to answer.
3. Identify any drawbacks and limitations to using PA in this setting and reflect on opportunities and new research directions

The rest of the paper is organized as follows. First, we describe the systematic literature review methodology that was followed (section 2). We then provide a detailed overview of how verbal protocols have been created and analyzed (section 3) and the main research questions the methodology has been used to answer (section 4). Finally, section 5 offers our commentary on the limitations of the method and future directions of PA in engineering education.

3.1 Data collection

3.2 Protocol analysis

Once audio/video recordings are collected, the next step in the process is transcription. Ericsson and Simon (1984) recommend that all verbalizations of a session are recorded and later transcribed verbatim to keep the data in its most raw form (p. 4). While transcribed text facilitates the analysis, it is widely accepted that access to the audio/video recording throughout the analysis can be beneficial as it allows researchers to capture the most comprehensive account of the design session. For example, Wells et al. (2017) used a video to provide a time-stamped recording of the entire session which was later used for easy access to particular parts of the design session. High-quality transcripts together with source audio/video data form the dataset on which PA can be performed.

4.1 Attributes of novices’ design processes

PA has been used to answer a wide range of research questions related to student design processes. A considerable number (25 of 45) of the reviewed studies use PA to understand how novice designers behave at a specific stage of the design process – be it problem formulation, idea generation, modeling, and testing, etc. – and to investigate how these processes might be affected under different conditions or interventions.

Some of the research (8 of 45 studies) has focused on the problem formulation stage and the impact of the design task description and complexity on students’ design work. For example, Lemons et al. 2010b were interested in learning if students can effectively interpret a design task. Protocols were coded and analysed to follow the path each student took to design their solution, finding that the interpretation of the problem would greatly affect their ability to produce a useful solution. Another study investigates if students are implicitly and explicitly decomposing the problem during the idea generation phase (Liikkanen & Perttula, 2009). Codes were assigned to protocols that identified instances of recognition, implicit decomposition, searching memory, explicit decomposition, analogical inference, evaluation, and output. The study found that students rarely needed to explicitly decompose a problem, as they were able to successfully and implicitly decompose the problem in a previous step. A related factor under investigation has been design problem complexity. For example, Cardella et al. (2002) evaluated students’ demonstrated design processes under different problem complexity conditions and found that more complex problems may be related to an increased level of evaluation of solutions generated by the students. Similarly, Williams et al. (2013) used the FBS coding scheme to investigate how small changes (e.g., changing a single word) in the description of the design task affect the design process of student designers.

Following problem formulation, students transition to the concept generation phase of the design process. Moore et al. (2017) aimed to determine which kind of thinking (Type I or Type II) has the greatest influence on idea generation. They hypothesised that Type I processes (those processes associated with fast, impulsive, and intuitive thinking) would produce more novel ideas than those of Type II processes (those processes associated with slow, analytic, and methodical thinking). Using the FBS and two other coding schemes, they associated design actions with one of the two thinking types, and found that it is through a combination of Type I and Type II thinking that designers generate novel ideas, rather than one single process on its own.

Gero et al. (2013) compare the design cognition of student designers using three different idea generation techniques (brainstorming, morphological analysis, and TRIZ), which differed in their degree of structuredness. In their study, pairs of students participated in an activity where they were asked to use only one of the techniques to generate solutions. Using the FBS coding scheme, they found that the structuredness of the technique affects the design cognition of student designers, with the more structured techniques (TRIZ) influencing the amount of time students spend reasoning about the problem compared to reasoning about their solution. Pertulla and Liikane (2006) examine a similar research question – does showing examples affect the idea generation phase of students? PA was used to evaluate the connections (i.e., following up on an idea, modifying an idea, or combining ideas) between ideas generated by individuals who were shown four examples of a design prior to the activity, compared to individuals who were not. Their analysis showed no significant differences between the two groups, aside from the number of new categories that were evaluated.

Two of the reviewed studies use PA to investigate the use of analogies in design. Ball et al. (2004) look at the role of spontaneous analogising with professional engineers and graduate engineering students. Their research used PA to identify the portion of time that study participants devoted to using schema-driven and case-driven analogies as they designed an Automated Rent-A-Car facility. In another example, Cheong et al. (2014) assess the use of biological analogies in creating a solution to an open design problem. The students were given a biological analogy and asked to design a solution, using the analogy as the foundation.

Protocol analysis has also been used to study how students use prototyping, modelling, and sketching in their design process. For example, Lemons et al. (2010a) use both content analysis and protocol analysis to identify the benefits of model building on students’ design solutions. They find that models help students generate and evaluate their ideas, expose flaws, and act as visual aids for their solution. In their study, students were explicitly encouraged to build models of their solutions (using LEGO blocks) to an open-ended design task. Initial content analysis helped create the coding scheme for the protocol analysis, which was used to identify students’ idea generation, metacognitive strategies and other cognitive actions. Cardella et al. (2006) investigate the use of external representations (problem statements, diagrams, equations, etc.) during a design task. In this case, each protocol was coded for instances of looking at a provided diagram, writing, reading their own writing, looking at sketches, or making calculations. Results were used to understand how students were interacting with these important components of designing. Taking an interest in the role of prototyping in student design teams, Bailey and McFarland (2018) use PA to capture student design teams’ characteristics – what phase of the design process they were on (using Atman’s coding scheme) and the extent of collaboration in the team – as they worked on a design task. They find that student teams typically work together to think of multiple solutions and then split up to develop those solutions further.

As designers are rarely working alone, some research has also examined the collaborative element of the design process in students. Studying senior mechanical engineering students working in groups, Sauder and Jin (2013) investigate and measure the relationship between collaborative stimulus and cognitive processes. Their coding scheme identifies instances of prompting, seeding, correcting, and clarification of ideas to the group working on the design challenge. They find that participants’ past experience influences the process of developing shared designed components, and how those components are explored together in the group. Along a similar line, Cash and Maier (2016) use the FBS coding scheme and additional codes to categorize the kinds of gestures used during a design session. Analysis of recordings and protocols of graduate students designing a remote-controlled camera cradle shows that gestures contribute to a shared understanding of a concept or idea. Gestures were coded as being reflective (towards themselves) or directed (towards others), with both kinds having an influence on how the group generated ideas and eventually made important design decisions.

4.2 Comparisons with and within novice designers

Many of the reviewed studies (16 of 45) aim to understand how novice designers compare to each other (based on, for example, discipline or academic level) and to expert (or professional) designers. Using PA, differences have been identified in a variety of studies that highlight important gaps in novices’ understanding of design. Based on these differences, educators are able to make changes to curriculum and learning activities to improve novice performance in a way that resembles expert design thinking (Mentzer et al., 2015).

One of the earliest examples of a protocol study using this design is the work by Atman and Bursic (1996), which investigated if reading a design textbook influenced students’ design process. While the study found no differences in the students’ solution quality, those who read the design text did spend more time solving the problem, generated more solutions, and transitioned between design stages more than those who did not. Atman and colleagues later published an in-depth case study of four participants of their protocol studies, who participated both as first-year students and later on as seniors (Cardella et al., 2008). The case studies provided rich insight into the effect of an undergraduate engineering education on the students’ design processes. In a similar fashion, Williams et al. (2012) use two different design tasks to investigate the differences that might exist in the same set of students before and after they complete an introductory design course. Using the FBS coding scheme to code their protocols, they found significant differences between the performances of students, specifically related to the number of instances of identified functions and developed descriptions of design artefacts.

Some studies (4 of 45) in particular have focused on the design behaviour of first-year engineering students compared to that of high school students, with the goal of assessing the needs of incoming students and the effectiveness of existing design training at both levels. For example, Lammi and Gero (2011) compare the design cognition of mechanical engineering undergraduates to high school seniors and find the latter were able to generate more ideas but did not analyze them further. In contrast, the undergraduate students placed more emphasis on the process of design analysis. Using a previous dataset (from one of Atman’s studies), Mentzer et al. (2015) make a similar comparison between both high school seniors and experts, as well as between high school seniors and high school freshmen. The researchers found the total number of references to a variety of design activities and identified several gaps and opportunities for potential improvements in engineering education, including interventions at the secondary education level. More recently, Wells et al. (2016) use PA to show that there is no significant difference in the design cognition of high school students who had formal pre-engineering education compared to those that did not.

Finally, some studies (6 of 45) have directly compared student and expert designers. For example, Song and Becker (2014) compare novices and experts during the problem decomposition phase and find that experts are able to better decompose and recompose the problem. Their study also highlights the need for students to be given the opportunity to work collaboratively in design projects, as first-year students were observed to struggle with this work environment. More recently, Dixon and Bucknor (2018) aim to understand the use of design heuristics used by both novel and expert designers. The protocols were analysed to reveal the types and frequencies of design heuristics that were used during the task.

5.1 Overview of review findings

Our review of the literature described PA as an adaptable method that can accurately capture the design processes followed by engineering students. In the reviewed studies, the use of PA has contributed to significant insights for engineering education, including how engineering students approach a design problem (Cardella et al., 2002; Lemons et al., 2010a; Liikkanen & Perttula, 2009; Williams et al., 2013), how they generate ideas and model/prototype design solutions (Bailey & McFarland, 2018; Ball et al., 2004; Cardella et al., 2006; Cheong et al., 2014; Gero et al., 2013; Lemons et al., 2010a, 2010b; Moore et al., 2017; Pertulla & Liikane, 2006) and how they work collaboratively in design teams (Cash & Maier, 2016; Sauder & Jin, 2013). The method has also been used to evaluate the effectiveness of specific teaching interventions, design courses, or even entire undergraduate degrees (Atman & Bursic, 1996; Cardella et al., 2008; Williams et al., 2012). These and other studies have served to compare the design processes of engineering students in different disciplines, as well as with high school students and expert designers (Dixon & Bucknor, 2018; Lammi & Gero, 2011; Mentzer et al., 2015; Song & Becker, 2014; Wells et al., 2016). In particular, Atman and colleagues’ data sets have shown that, when collected correctly, a number of diverse perspectives can be used to analyse protocol data, providing both quantitative and cased based accounts of design (Atman, 2019; Cardella et al., 2014), advancing our understanding of students’ design cognition.

We note that a considerable amount of research related to novice designers using PA has been done by a small number of networks of collaborators. This is also evidenced by the two dominant approaches to analyzing protocols – Atman’s (Atman & Bursic, 1998) and FBS (Gero & Kannengiesser, 2004). As such, many of the reviewed studies are based on similar datasets, problems, and student populations. We thus recognize these datasets, authors, and their findings are over-represented in our sample of papers and consequently in our discussion. This realization leads to a number of questions and future research opportunities. First, what is the overall research methods landscape in engineering design education research? And how prominent is PA in this landscape? Second, and following from these questions, we wonder if there are instances when PA is used but another research method might be more suitable, and vice-versa.

While answering the questions posed above is beyond the scope of this review, we nevertheless can comment on a number of limitations that PA poses for engineering design education researchers, as well as identify some research opportunities to address those limitations.

5.2 Limitations in collecting and analyzing verbal protocols

Despite efforts to improve the efficiency of collecting and analyzing verbal protocols, it still remains a lengthy and time consuming process (Lemons et al., 2010a). As such, PA is not an appropriate tool for routine student assessment or for large populations in a single study (Atman & Bursic, 1998). Researchers may opt to limit the number of participants in their study population, thus reducing the studies’ statistical power (Coso et al., 2010; Daly et al., 2018; Moore et al., 2014; Patel et al., 2018). In addition, conclusions from the data become less generalizable, requiring further research efforts to validate the claims.

Protocol studies are often hosted in small, quiet rooms away from distractions. Unfortunately, this constraint, which ensures quality audio/video for later transcription, also limits the diversity of study designs used for large group settings. For example, hackathon-like design activities, which expose students to ill-structured and open-ended problems (Rennick et al., 2018), offer a significant opportunity to observe student design behaviour (Hurst et al., 2019); however, their format poses a significant number of challenges to a protocol study, mainly the number of students in conjunction with a loud and hectic team design environment, prohibiting the ability to collect useful verbal recordings.

Because of constraints related to conditions under which verbal protocols can be collected, the design tasks studied with PA may not be representative of those engineering novices commonly engage with. As noted earlier, many of the reviewed studies used problems that were related to mechanical engineering. These problems do not necessarily require participants to use an interdisciplinary approach to problem solving, which is more typical in an authentic setting. Further, setting time limits on activities does not always allow participants to progress through an entire design process (Roberts et al., 2007). Although there is an attempt at making these problems as realistic as possible, there is still a call for future investigations into more complex design situations (Cash & Maier, 2016). Professional and practicing engineers may work on design projects over long periods of time, so design sessions that are limited to a few hours may not be representative of actual design behaviour. Similarly, using an experimental design in which participants are encouraged to use a specific process, for example by providing prototyping material to them (Bailey & McFarland, 2018), automatically influences their design approach. As such, in many of the reviewed studies the interventions used may prohibit the ability to observe natural design behaviour.

5.3 Opportunities and future research directions

Researchers can take advantage of technological advancements that can improve the efficiency of collecting, processing, and analyzing verbal protocols. Smartphones and tablets have the ability to record both audio and video files, which can later be sent to researchers and used for analysis. This reduces the need for the research team to facilitate recording sessions and increases the amount of data that can be collected in a single setting. As technology becomes more accessible, there is potential for students themselves to capture their own design process. Further, many transcription services now use AI to turn audio and video files into transcribed text. Using such tools limits the number of hours required to transcribe protocol data manually and significantly speeds up data processing. Finally, advances in natural language processing provide exciting new opportunities for automating (at least in part) the coding of protocols, as shown for example by Nespoli et al. (n.a.).

PA studies in engineering design education research have used a number of common coding schemes, most commonly Atman’s (Atman & Bursic, 1998) and FBS (Gero & Kannengiesser, 2004), but there have also been cases when researchers developed new coding schemes or combined and adapted existing ones. Gero (2010) argued for the implementation of common coding schemes (as discussed in section 3), which would allow for comparisons across different research studies. Although Mentzer et al. (2015) did not use the FBS ontology argued for by Gero, their work is an example of how this approach can be used. By utilizing Atman’s coding scheme on their own protocol data and a previous data set collected and coded by Atman, they showed that a comparison across different data sets is possible. In future research studies, using this approach will only provide more in depth findings from protocol studies, as we will build on the available research rather than starting anew.

Finally, although the research that has been done using PA has yielded a number of important insights, there is a need for triangulation of those results with other methods (Atman, 2019). Hay et al. (2017) note that one challenge for the field moving forward is to test results of protocol study findings with other methods that are more suited to large population sizes (e.g., interviews, surveys, etc.). Using this approach, we can ensure that we are understanding the processes used by students and practitioners alike in the most comprehensive way possible.