Functional Modeling Supports System Representation

Mental models are the cognitive scaffolding that enables people to successfully negotiate their environments, objects in those environments, and the interactions between people and those objects. Senge describes mental models as deeply held cognitive structures that reflect how we perceive the world works [1] and Markman suggests that “mental models of physical systems are internal representations of external systems” [2]. Furthermore, Fein et al. define mental models as “knowledge that the user has about how a system works, its component parts, the processes, their interrelations, and how one component influences another” [3], which most closely relates to the context of engineering and design. It is not uncommon for people to have different mental models of the same product or artifact, and one can even hold multiple models of a single system [4]. The focus of this work is on graduate engineering students’ perceived knowledge of a system through their mental model representations of that system. In this article, the definition of a mental model is taken to follow the definition offered by Fein et al. [3]. These mental models contain information about the function, form, and dynamics of a given system, which is particularly important for engineered systems. In regards to physical systems, mental models provide the necessary framework to negotiate and make judgments about those systems [5]. We believe that through an understanding of how students formulate and evolve mental models of engineered systems, educators will be able to develop educational pedagogies which intentionally foster the development of their mental models. This has implications for how students and professional engineers make decisions, troubleshoot, communicate, and apply knowledge when working with a system.

This article presents results from a study that elicited graduate engineering students’ mental models of common household products before and after learning functional modeling in a traditional lecture-style graduate course and compared these results to graduate students who did not learn function in an otherwise similar course. The mental model elicitation method and scoring process with rubrics are described in detail in the methodology section. Previous research has shown that the approach taken in this study can successfully elicit students’ mental models [6–8]. This work is motivated by an ongoing effort to understand how people form, alter, and communicate their mental models of engineered systems. Functional modeling and functional decomposition are not universally taught to engineering students. The results of this effort may reveal that learning functional modeling improves a designer’s ability to reason, represent, and communicate about an engineered system, which could inform curricular content on the engineering design process.

Through this investigation, the authors aim to understand the role of function in systems thinking and systems understanding. The research questions guiding this work are: (1) What are the effects of functional modeling instruction on externalized mental model representations of systems? (2) What aspects of systems thinking are supported by learning functional modeling? and (3) What impact do practice effects (improvement on a task due to repeated evaluation) have on mental model representations over time using sketching as a direct elicitation method? It is hypothesized that learning functional modeling will improve scores on the mental model tasks (H1) because functional modeling provides a structured framework that will help with the mental accounting required to fully represent a system. Specifically, functional modeling is hypothesized to improve component-level systems thinking (H2) since functional modeling encourages tracing energy and material through the system step-by-step. Lastly, no practice effects are expected to be present (H3) considering that participants did not receive any form of feedback on their mental model representations.

Mental models offer a unique avenue for the exploration of students’ capability to understand engineered systems and their ability to communicate their knowledge about those systems. There is a need to understand how engineers and designers think about systems and what they can do to improve their systems thinking ability. A brief history of research on mental models and the relationship of mental models to systems thinking are described in Sec. 2. The context of the study and an outline of the procedure are explained in Sec. 3. Subsequently, the results are showcased in Sec. 4 followed by a discussion in Sec. 5 with a final summary of this work in Sec. 6.

Mental model research includes a variety of related topics including naïve physics (untrained reasoning of fundamental physical phenomena) [9–15], engineered systems [16–18], human–computer interactions [3], shared models [19,20], and misconception [18,21–23] to name a few. Studying mental models is a lot like looking at a dirty mirror, where any elicitation of a mental model is simply a representation and not the conceptualization that exists in the mind. Considering this analogy, much of the research on mental models helps to clean that mirror and elicit mental model representations that are relatively close to the actual conceptualizations. Mental models are historically difficult to elicit [24], especially complex ones often found in an engineering context. There are two broad approaches to mental model elicitation: direct elicitation (e.g., diagrams, sketches, symbols, arrangement of physical objects) and indirect elicitation (e.g., written text, interviews, diagnosing faults in a system [25], consensus analysis) [26]. For the study presented in this article, direct elicitation through sketches was implemented because it provided a relatively straightforward task for the participants, took less time than conducting/coding interviews, and was closely related to similar research that successfully used the same approach [27].

Much of the work on mental models indicates that people gravitate toward cognitive efficiency [28–30]. People create minimalist mental models with only enough detail to complete a task, which results in either incomplete or inaccurate mental models at various levels of abstraction. In some cases, this cognitive efficiency is actually caused by perceptual illusions that create erroneous mental models of physical phenomena such as misjudging the water level in a tilted cup [10]. A study involving mostly business graduate students demonstrated that even highly educated individuals lack mental models robust enough to capture basic concepts of system dynamics [31]. This was achieved by asking participants to indicate the volume of water in a bathtub with an inlet, outlet, and periodic filling states, where results showed poor performance on the task suggesting that people largely have erroneous mental models of commonly encountered systems [31]. Other research articles have investigated erroneous mental models of stock and flow diagrams [32], found no correlation between years of employment and an aptitude for systems thinking [33], and that undergraduate students generally lack the ability to effectively reason about systems [34]. All of this considered it is crucial that researchers and educators understand how we can improve engineers’ mental models of systems and provide them with useful frameworks that support the kinds of analytical reasoning and communication skills necessary for success in engineering industry.

Researchers have made efforts to define the term systems thinking. A particularly robust definition of systems thinking was offered by Arnold and Wade as “synergistic analytic skills used to improve the capability of identifying and understanding systems, predicting their behaviors, and devising modifications to them in order to produce desired effects” [35]. They came to this definition through an extensive literature review of published articles on the topic of systems thinking [36] and argue two primary systems thinking modalities: gaining insight and using insight. In a different article, Valerdi and Rouse define systems thinking as “a framework that is based on the belief that the component parts of a system can be best understood in the context of relationships with each other and with other systems rather than in isolation” [37]. This definition suggests that understanding the components of a system is not enough to facilitate systems thinking or to reason about how a system will behave. Hopper et al. similarly argue that component identification is insufficient even when the connections between components are understood, and that an understanding of causality between the components and overall system dynamics is necessary [38]. While it is important to have knowledge of system components, this knowledge does not give the whole picture. Other work has described how systems thinking is not a cognitive ability that comes naturally to most people [37] but is instead a learned set of analytical skills. To quote Richmond, students should ideally be able to “see both the forest and the trees” [39].

Literature discussing the key elements for robust systems thinking was also reviewed. A taxonomy adapted from Bloom’s Taxonomy [40] characterized systems thinking in a similarly structured hierarchy [41]. This hierarchy, called the Taxonomy of Systems Thinking Objectives, is broken down into recognizing interconnections, identifying feedback, understanding dynamic behavior, differentiating types of variables and flows, using conceptual models, creating simulation models, and testing policies. This categorization provided insight into how to score the mental model representations in the study presented in this article to focus on aspects of systems thinking. Specifically, the hierarchy led to the “low-level systems thinking” category on the scoring rubrics from “recognizing interconnections” and “differentiating types of variables and flows.” Others have attempted to create categorical hierarchies for systems thinking competencies [1,41,42]. In addition, previously published research on a functional modeling scoring rubric [43–46] provided additional insight. Functionality repeatedly showed up in the literature as a primary element of systems understanding. In particular, the creation of a black-box model captures energies, materials, and signals that are crossing a system boundary, which led to the high-level systems thinking category on the scoring rubrics. Nagel et al. provided the necessary groundwork for a meaningful functional modeling intervention in a traditional lecture-style graduate course by exploring different methods for teaching functional modeling in an academic classroom [44,45,47,48]. These key elements led the authors to implement a scoring rubric that considered high-level systems thinking, low-level systems thinking, and component architecture to assess mental models of engineered systems.

The study presented in this paper is strongly informed by Lawson’s study on mental models of bicycle functionality [27] where participants showed consistent structural and functional errors in the drawings of bicycles done by bicycle owners and professional cyclists. Participants were asked to either draw the frame, pedals, and chain on a simple drawing of a bicycle or to select the correct orientation of these elements from a provided list. Of nonexperts, over 40% of participants made errors on either task [27]. The participants in Lawson’s study also overestimated the correctness of their models, with the expert cyclists more confident than the nonexperts. Expert cyclists outperformed nonexperts as expected [27]. Rozenblit and Keil suggest that people are overly confident about their understanding of mechanical systems when components are visible [49], as they are on a common bicycle. The study being presented in this article similarly required participants to draw the components and connections within a system to elicit their mental models of that given system. As a key distinction from Lawson’s work, the study presented in this article investigated the effects of learning functional modeling on the completeness of system representation.

Based on the reviewed literature, the research team selected common household products with either removable parts (such as a clothes dryer’s lint collector) or partially visible components (such as the heating coils inside of a hair dryer) to measure participants’ system understanding. It is reasonable to assume that graduate-level engineering students would have a working understanding of how these common products work. The authors tasked participants with sketching their mental models of the given products (direct mental model elicitation) as opposed to eliciting their mental models through a computer simulation [3], allowing for efficient data collection and avoiding expensive software development. Sketching is an accepted method for direct mental model elicitation and serves as a quick and inexpensive way of accessing mental models in the field of design research [24]. The mental model tasks implemented in this protocol study could be easily adapted to many other products or engineered systems, therefore providing flexibility for researchers interested in running their own studies or educators seeking a tool to measure students’ understanding of engineered systems.

The following methodology section describes the process used to successfully elicit students’ mental models of three common household products and consequently measure their ability to communicate their knowledge about those systems. The function and no-function conditions are described with a detailed account of the functional modeling material taught as the experimental intervention. The university context and experimental procedure are provided with a summary of the scoring technique used, which is presented in abbreviated form in the  Appendix.

This section describes the university context, study participants, function intervention material, the mental model elicitation instrument, and the mental model scoring process.

The students were enrolled in a graduate-level engineering design course at a competitive public southeastern United States research-focused university, and voluntarily consented to participation in this study for extra credit in the course. In this project-based course, students worked in teams to practice using early-stage design process techniques. Data from the function condition and the no-function condition were collected during the spring semesters a full year apart due to the constraints of the course. Both conditions could not occur within the same class and the course was offered only yearly. Data from the function condition were collected during the first year, whereas data from the no-function condition were collected the following year.

Participants in this study included 51 engineering students, with 31 in the no-function condition and 20 in the function condition. The function condition was comprised 20 participants who completed both data collections, with three self-reporting as undergraduates, two as Ph.D. students, ten as nonthesis master’s students, and five as thesis-based master’s students. For the no-function condition, all of the 31 student participants in the study were graduate students with two self-reporting as Ph.D. students, 25 as nonthesis master’s students, and four as master’s with thesis students. For the no-function condition, all 31 students participated in both data collections.

The data presented in this paper were collected at two points in the semester as in-class activities three weeks apart for both the function condition and the no-function condition. Table 1 overviews the differences between both conditions.

During the three weeks between data collections, participants in the function condition learned about functional modeling in two lectures and completed two homework assignments (1 – create a functional model of a hand mixer and 2 – create a functional model for their semester-long project). The course content on functional modeling was taught by a Ph.D. student with expertise in functional modeling and decomposition. Participants received feedback on their functional models in the form of grades, comments from the course instructor, and guidance from the Ph.D. student that taught the functional modeling material.

Functional modeling was taught after the first mental model data collection. Participants were first introduced to form versus function, a dyad where form describes physical attributes (e.g., shape, color, size, components, etc.) and function describes the purpose of those attributes and components (e.g., energy or material transformations, transmission, etc.). Next, participants learned how to create energy-material-signal (EMS) black-box models of various common products with an emphasis on verb–noun pairs [50–52]. Participants were then instructed to identify the individual components in a product (in this case a can opener, a leaf blower, and a hand mixer) and assign verb–noun subfunctionality to each component with inputs and outputs appropriately labeled. From this list, energy, material, and signal function chains were generated so that all subfunctionality was accounted for and all input–output pairs matched accordingly. Participants were then taught to recognize synonymous component-level subfunctionality in their different function chains as points where separate function chains should intersect to build an aggregated final functional model. In a lecture, it was emphasized to check that functional model inputs and outputs matched black-box inputs and outputs exactly and to ensure that all subfunctions were verb–noun pairs.

This structured method of teaching functional modeling draws heavy inspiration from the FAST method [53], the flow-based method [51], and the hierarchical method [50]. In this study, participants were taught to list components before identifying subsystems, which is a slight deviation from traditional methods of teaching functional modeling, where the emphasis is typically more top-down than bottom-up (e.g., starting with the black-box model). This method was chosen because the authors believe students learn functional modeling better through this approach based on collective experience teaching the subject and prior studies on the topic. Furthermore, starting from components makes sense when considering the importance of component identification in systems thinking as a low level of abstraction, then working to higher levels of abstraction such as overall system functional, system dynamics, or a systems’ interactions with its environment. During lecture and homework assignments, students were also encouraged to use the functional basis [50,54,55] to form their functional models, but the use of the functional basis was not required. Convention minutiae taught in the lecture follow the conventions described by the published functional modeling scoring rubric [43–45].

Lectures were given with projected slides, and the lecture material was provided digitally as a reference for participants to use while completing assignments. After obtaining consent, students of the Function Condition were given 30 min at the start of class to complete the experiment packet.

In between data collection for the no-function condition, participants studied the engineering design process generally learning about product opportunity gaps [56] and customer needs assessment [57,58]. Material on functional modeling or functional decomposition was intentionally delayed until after the study for the no-function condition. In addition, design methods that might share a resemblance to the process of generating a functional model, involve analogical reasoning, or encourage knowledge transfer (such as creating activity diagrams or the topic of bio-inspired design) were also delayed until after the second data collection. The research team has no reason to believe that any material covered during these three weeks for the no-function condition would interact with the mental model tasks. After obtaining consent, students in the no-function condition were given 30 min at the start of class to complete the experiment packet.

The following describes the measures used in this study. This includes a description of the mental model elicitation instrument and the scoring method used to determine mental model completeness. A sample of the elicitation task and scoring rubric can be found in the  Appendix.

To elicit mental models of a hair dryer, a clothes dryer, and a vacuum cleaner, the research team relied on the method used by Lawson that required participants to sketch their knowledge of a system and its components [27]. This approach is considered a direct elicitation method for mental models through sketching. Participants were tasked with filling out simple outlines of a hair dryer, a clothes dryer, and a vacuum cleaner provided as a paper packet (see  Appendix).

The packet also included a completed mental model example of a common toilet. The toilet example is shown in Fig. 1. This example was created by the research team and was included to ensure that participants knew what was expected of them on the activity since previous research indicated that confusion about the instructions may have led to inconsistent solutions in the data [6,59,60]. The toilet example was carefully chosen because it does not share any components or functionality with the hair dryer, clothes dryer, or vacuum cleaner. This was done to avoid biasing the results. For the three tested systems, participants were also asked to answer a few questions about their knowledge of the product and experience using it:

1. What is the product commonly called?

2. Have you ever used this product? (Circle) Yes/No

3. Do you use this product monthly? (Circle) Yes/No

4. Have you ever taken this product apart? (Circle) Yes/No

These questions allowed for a preliminary assessment of the impact of prior disassembly and to measure participants’ experience with the given products. Each study packet was identical for both the function condition and the no-function condition, and identical for the first and second data collections completed three weeks apart.

Data were scored using an adaptation of previously published scoring rubrics that accompany each product in the mental model tasks [6]. These rubrics have been used in other related studies [7,8,61]. For most rubric questions, a score of 0, 0.5, or 1 was awarded based on the completeness of their solution (some questions only allow for a score of 0 or 1 such as the component questions because a component is either present or not present). Rubric questions were created by the authors based on the reviewed systems thinking literature and the functional modeling scoring rubric [46], were customized based on each engineered systems’ architecture, and were divided into four categories: high-level systems thinking, component-level systems thinking, low-level systems thinking, and subsystem interrelations, where each category is defined as such:

• High-level systems thinking: Addresses materials, energies, and signals crossing the system boundary.

• Component-level systems thinking: Inclusion/Exclusion of system components.

• Low-level systems thinking: Scoring for connections between components.

• Subsystem interrelations: Considers interactions between subsystems (e.g., air flow across heating coil).

The full mental modeling scoring rubrics and elicitation tasks can be found here [62]. Example rubric questions are provided below:

1. Is energy conserved across the system boundaries? (High-Level Systems Thinking)

2. Is there an electric plug of alternate power source? (Component Identification)

3. Is the fan/air moving device connected to the motor, engine, or similar device? (Low-Level Systems Thinking)

Total mental model scores were calculated by summing the number of points awarded to a given mental model of each product. Average mental model scores were calculated by averaging all of the participants’ scores on each of the three products on either the first or second data collection. Since each product has a slightly different number of necessary components, each rubric has a slightly different number of questions. For example, the clothes dryer rubric has the most questions because it is the most complex system of the three products with more necessary components than the hair dryer or vacuum cleaner. During analysis, scores were normalized to allow for comparison of results across all three elicited mental models. Data were randomized so that raters did not know if they were scoring a mental model from the first or second data collection to remove any bias during the scoring process. In addition, a second rater evaluated a portion of the data and was also not aware if the mental models were from the function condition or no-function condition to further eliminate bias and ensure inter-rater reliability.

The results of this study evaluate the impact of the functional modeling intervention by comparing mental model scores between the no-function condition and the function condition. Data were scored by a second-year Ph.D. student studying design theory and methodology and an undergraduate research assistant with extensive experience in functional modeling. Inter-rater reliability scoring was performed on 30% of the mental model data (10% of each product) across all three systems for the function condition and just under 40% of the data for the no-function condition. The data were anonymized such that the undergraduate research assistant did not know whether the mental models were from the pre- or post-data sets. For the no-function condition, analysis shows a percent agreement of 79.9% and a Pearson’s correlation of 0.94 on total mental model scores indicating a strong correlation. In addition, a value for Cohen’s kappa [63] of 0.66 between the two raters indicates substantial agreement, where Cohen’s kappa considers the possibility of random agreement. For the function condition, analysis shows a percent agreement of 78.4% and a Pearson’s correlation of 0.72 on total mental model scores indicating a strong correlation. Finally, analysis shows a Cohen’s kappa [63] of 0.60, which indicates moderate agreement. The inter-rater analysis results for both the no-function condition and the function condition suggest good agreement and reliable scoring. Disagreements in this study likely occur from the subjective interpretation of the hand-drawn student solutions.

Mental model scores from the first and second data collections for both the no-function condition and the function condition were averaged. The average scores are reported as percentages so that comparisons can be made between the three products (hair dryer, clothes dryer, and vacuum cleaner) since the rubrics for each have a slightly different number of rubric questions based on the number of components in each. Figure 2 is the average mental model scores as a percentage for the no-function and function conditions from the first and second data collection on each of the three products. Error bars of ±1 standard error are included.

Participants in the no-function condition showed no significant changes in average mental model scores between the first and second data collections. Participants in the function condition, however, showed significant improvement in the clothes dryer and vacuum cleaner mental model tasks. To confirm these differences, Kruskal–Wallis tests were performed on data from the first and second data collections for each product (Table 2).

No significant differences were found for the no-function condition between the first and second data collections on all three products (Table 2). Significance was found for the function condition of the clothes dryer and vacuum cleaner tasks. For the hair dryer task, the observed means after intervention were 69.40% and 78.00% in the function condition. The research team speculates the lack of significance on hair dryer mental models might be caused by ceiling effects on the hair dryer mental model task considering it is the easiest to complete and the participants consisted of graduate students. The results of this study, however, do not show evidence that the functional modeling intervention significantly impacted participants’ mental models of hairdryers. Using Cohen’s d, the no-function condition showed small effect sizes for all three products whereas the function condition showed large effect sizes, which gives additional confidence that the intervention influenced participants’ mental model scores. In addition, a single-factor ANOVA was performed on the differences in score on each of the three products to determine whether or not improvement was consistent across each of them (F(2) = 1.443, p = 0.245). For differences in the score, the data for each product passes a Shapiro–Wilk test for normality. Since the dataset used to calculate differences in score does not meet normality, however, a Kruskal–Wallis test was also used giving H(2) = 2.621, p = 0.270. In either case, this result is not statistically significant, which indicates that no difference in improvement was detected for each of the three products. Finally, no significant differences were found in the pre-data on each product between the no-function condition and function condition as a measure of consistent sampling.

Further analysis was completed that separates results from each portion of the mental model rubrics for the function condition. This includes the high-level, component, low-level, and system interrelations rubric sections. This was done to explore whether improvements could be attributed to any specific aspect of systems thinking. These results are provided in Fig. 3. As shown, improvements on each portion of the rubric are consistent with average improvement (Fig. 2) across the three different tested products. The results of this analysis do not conclusively indicate that any specific aspect of systems thinking was improved, but rather show consistent improvement across the different portions. For low-level systems thinking on the hair dryer task, analysis shows a statistically significant improvement using a Kruskal–Wallis test (H(1) = 6.956, p = 0.008), but this result is not significant on the clothes dryer (H(1) = 1.616, p = 0.204) or the vacuum cleaner (H(1) = 1.582, p = 0.208). This analysis serves as an example that no specific aspect of systems thinking was observed to be attributed to the average mental model score improvements for the clothes dryer and vacuum cleaner mental models (Fig. 2).

Beyond mental model score improvement, aspects of systems thinking supported through the functional modeling intervention were explored. On the component portion of the mental model rubrics, five necessary components were present on all three rubrics: power source, fan or similar, motor or similar, on/off switch, and internal wiring. These five components were considered as a group during data analysis to determine whether functional modeling was supporting the participants’ ability to realize the analogous similarities between the three products. In other words, it was expected that a participant who included a fan on their hair dryer mental model would realize that the clothes dryer and vacuum cleaner also required components for moving air. To achieve this, the number of times a participant included one of these five components in all three of their mental models was summed for all participants. These sums were then averaged as an indication of how the no-function condition and function condition were able to transfer knowledge between their mental models of the products in their representations. These averages are shown in Fig. 4 with ±standard error bars.

There is no difference in the no-function condition between the first and second data collections. A large difference is observed for the function condition. Using Cohen’s d, the no-function condition had an effect size of −0.21 (small) and the function condition had an effect size of −0.88 (large). This result is statistically significant using a Kruskal–Wallis test (H(1) = 6.126, p = 0.013) with means of 1.65 components before intervention and 2.65 components after intervention for the function condition and not significant for the no-function condition (H(1) = 0.726, p = 0.394). This indicates that the functional modeling intervention is supporting component-level knowledge transfer across the three products. No significant differences were found between the first data collection for each condition.

Participants’ self-reported prior experience disassembling any of the three products was also considered. Since no differences were found between the no-function condition’s and the function condition’s pre-data, all data from the first data collection were considered simultaneously. All mental model scores where a participant indicated that they had previously taken the product apart were averaged and compared to the average score of mental models without prior disassembly experience. This resulted in 165 mental models without prior disassembly experience and 18 with disassembly experience. The results of this analysis are shown in Fig. 5.

As shown, participants that self-reported as having prior experience disassembling the product had higher mental model scores on those products than those without prior experience. This result is statistically significant through a Kruskal–Wallis test where H(1) = 6.486, p = 0.011, and a Cohen’s d of 0.69 for a large effect size, indicating that product disassembly is correlated with higher mental model scores. Forthcoming work has started to explore this result in greater detail with consideration for the large difference in sample size observed in this data.

The presented results show that participants who received a functional modeling intervention had improved mental models on two out of three common household products. There were no notable events or materials between data collections which otherwise would have significantly increased their knowledge of the systems or their ability to represent them. Therefore, we speculate that functional modeling provides a framework from which to consider, communicate, and represent their knowledge resulting in more complete mental model elicitations.

We believe that functional modeling is encouraging systematic representation that involves accounting for the conservation of energy and material. Through the process of functional modeling, the components, connections between the components, and where they fit into the system must all be considered. From the literature review, this process aligns with many of the different aspects required for robust systems thinking such as those described by the Taxonomy of Systems Thinking Objectives [41]. The process of creating a functional model has similarities to systems thinking in that it requires reasoning at various levels of abstraction simultaneously. Through functional decomposition, a designer is compelled to mentally trace the paths of materials and energies through the system, which likely encourages recognition of material and energy transformation, awareness of how components influence materials and energies, and identification of subsystems that are dependent on each other. The mental model scoring rubrics specifically target these aspects of systems thinking. Based on the results of this study, functional modeling seems to provide a framework to communicate knowledge held within a designer’s mental model of the given system. The results do not suggest a full rejection of the first hypothesis, which stated that learning functional modeling will improve scores on the mental model elicitation tasks. Significance was not observed on the hair dryer task after the intervention, which the authors speculate may be related to ceiling effects and the difficulty of the task itself. Results may be different with participants sampled from a different population of students.

To further illustrate the change in mental models for participants that received the functional modeling intervention, Fig. 6 shows an example of hair dryer solutions by a single participant from the first and second data collections of the function condition. This participant is a typical example of what improvement consistently looked like in the data. Notice the inclusion of a motor, internal wiring, connections between different components, and the explicit indication of heat being applied to air in the second model (Fig. 6, right) compared to the first model (Fig. 6, left). These improvements range from component-level systems thinking (motor inclusion) to high-level systems thinking (explicit indication of heat transfer to air).

In Fig. 6 (left), the mental model representation before the functional modeling intervention earned a score of 52% on the associated mental model rubric. After learning functional modeling techniques, the participant earned a score of 96% (Fig. 6, right) for an improvement of 44%. This participant only lost a point on their second hair dryer mental model because they did not clearly indicate a device to control the speed of air. Instead, they refer to “settings” and wrote “air heated/cooled depending on settings,” so received a point for control of the heating element. These improvements are not attributed to any specific aspect of systems thinking but are evident across all four sections (high-level, component-level, low-level, and subsystem interrelations) of the associated mental model rubric. This broad improvement is typical for the function condition across all three products, but it is not evident for the no-function condition.

Results from this work show insufficient evidence to reject the second hypothesis (which stated that functional modeling is hypothesized to improve component-level systems thinking), where data analysis shows improved component identification (Fig. 4). Results from this function condition showed that functional modeling supported component-level systems thinking, especially when those components recurred on all three of the given products with statistical significance between the two conditions. This might further suggest that functional modeling plays a role in a type of systems thinking, namely knowledge transfer. Functional modeling might facilitate knowledge transfer by encouraging a process of reasoning that reveals similarities between different systems when they are reduced to verb–noun pairs. In other words, it might be easier to recognize similarities between different systems if one is only considering basic functionality, which could lead to realizations such as “the fan must be connected to a motor in order to move.” Published literature supports this idea through explorations of the relationship between function, analogy, and engineering design [64].

No evidence was observed of practice effects on the mental model tasks using the scoring methods described. In other words, there is not sufficient evidence to reject the third hypothesis. This result provides further evidence toward the validation of the methodology described in this article and supports future work using the mental model elicitation method developed for this work. Application of the elicitation method may include research on different products, more complex engineered systems, or in different disciplines. We believe that these mental model elicitation tools will be useful for educators to baseline student understanding of systems, measure student learning, aid student communication of understanding, and provide the means for students to assess their own growth and development.

The results of this study do not imply that learning functional modeling has changed the participants’ mental models of these products. Rather, the collected data are a representation of the cognitive conceptualizations and not the conceptualizations actually held in the mind. The authors argue that this work may have instead allowed the participants to give clearer representations of their knowledge about the products. In other words, we believe that functional modeling may have improved their ability to retrieve, represent, and communicate their knowledge. Results are further limited by the population sample which consisted of graduate engineering students from just one engineering program. Results could change if this methodology was implemented on undergraduate engineering students, students broadly, or industry professionals. These endeavors are left to future work.

The study presented in this article shows that a functional modeling intervention is correlated with improvement in mental model scores of two out of three common household products. The clothes dryer and vacuum cleaner showed significant improvements in mental model scores after learning functional modeling. Furthermore, the functional modeling intervention seems to support the inclusion of components common across the products, which may indicate improved knowledge transfer. Functional modeling provides scaffolding for systems thinking through a systematic process that accounts for components, how those components are connected, and overall system functionality. Functional modeling likely supports the ability to communicate knowledge, retrieve knowledge, and/or interpret existing mental models of engineered systems that provides a foundation for systems understanding and communication.

Future work could involve investigating whether functional modeling improves students’ ability to communicate their mental models or rather just provides a framework with which to retrieve knowledge from their mental models. People may also have mental models of these products that are distinct from their understanding of the system such as innate contextual models or models for a user might manipulate the artifact based on prior experience. To address these questions, qualitative methods (indirect mental model elicitation) may be more suited. Other avenues of interest include an investigation into the effects of product dissection activities on mental model representation. Research specifically focused on this is currently underway by the authors of this article based on the results of this study.

Systems understanding is crucial in both engineering industry and academia, and elicited mental models offer a unique glimpse into students’ understanding, perception, and communication of systems. It would also be beneficial to apply this study’s methodology to engineering professionals in industry to draw conclusions about the differences/similarities between novices and experienced designers. This vein of research could inform what types of skills are most critical to teach engineering students to foster their systems thinking skills to best prepare them for engineering industry.

This study provides evidence that functional modeling taught as a part of engineering curricula allows students to better interpret systems and successfully communicate their ideas about those systems. The methodology presented in this work can be applied to other studies on mental models and be adapted to study different aspects of systems thinking. Recall the definition of a mental model by Fein et al. [3] described in the introduction of this article: “knowledge that the user has about how system works, its component parts, the processes, their interrelations, and how one component influences another” [3]. Functional modeling skills provide a framework to interpret a system through a lens that closely considers “the processes” and “their interrelations.” Overall, the research presented in this article provides a foundation for understanding the value of teaching functional modeling and decomposition to engineering students as we continue to investigate how students gain the necessary skills to become successful and effective engineering professionals.

The authors would like to thank the students for their voluntary participation in this research study. This work is supported by the National Science Foundation through Grants 1525170 and 1525449. In addition, this material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE-1650044. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

There are no conflicts of interest.

The datasets generated and supporting the findings of this article are obtainable from the corresponding author upon reasonable request.

Please note that the following hair dryer (HD) mental model rubric and instrument have been provided in abbreviated form. For the rubric, only the questions have been provided. The detailed explanations, section labels, and scoring bins have been omitted to minimize space. Please contact the authors if you intend to implement the rubric or mental model instrument.

High-Level Systems Thinking

1. Is energy conserved across the system boundaries? (0, 1)

2. Is energy changed, converted, or transferred AND conserved within the system? (0, 0.5, 1)

3. Is material conserved across the system boundaries? (0, 1)

4. Is material changed, converted, transferred AND conserved within the system? (0, 0.5, 1)

5. Are signals used appropriately throughout the system? (0, 0.5, 1)

6. Are correct inputs recognized? (0, 0.5, 1)

7. Are correct outputs recognized? (0, 0.5, 1)

8. Overall, does the model represent functional understanding of the system? (0, 0.5, 1)

Component-Level Systems Thinking

1. Is there an electric plug or alternate power source? (0, 1)

2. Is there a fan, air compressor, or air moving device? (0, 1)

3. Is there a heating element? (0, 1)

4. Is there a motor, engine, or similar device? (0, 1)

5. Is there an On/Off or power switch? (0, 1)

6. Is there a component for the control of the heating element? (0, 1)

7. Is there a component for the control of the fan/air moving device? (0, 1)

8. Is the internal wiring complete/present? (0, 0.5, 1)

Low-Level Systems Thinking

1. Is the motor, engine, or similar device powered? (0, 1)

2. Is the fan/air compressor/air moving device connected to the motor, engine, or similar device? (0, 1)

3. Is the heating element powered? (0, 1)

4. Is the motor, engine, or similar device properly regulated? (0, 0.5, 1)

5. Is the heating element properly regulated? (0, 0.5, 1)

Subsystem Interrelations

1. Does the model account for the movement of air through the system? (0, 0.5, 1)

2. Does the model account for the transfer of heat to air within the system? (0, 0.5, 1)

3. Does the model account for the control of electricity within the system? (0, 0.5, 1)

4. Does the model account for varying modes of operation? (0, 0.5, 1)