Validation Study of a Computer-Aided Design Augmentation Tool for Cost Estimation¹

A systematic engineering design process employs a variety of tools to structure the approach and guide engineers in the development and analysis of solutions [1–5]. Computer-aided design (CAD) software is one such tool with examples of catia, proe creo, solidworks, and others that are used for both prototyping preliminary design concepts as well as creating production quality solid models and engineering drawings. While traditional CAD packages provide a variety of tools for geometrical and mechanical evaluation of designs, late-stage concepts such as manufacturability and environmental impact are often not tightly interconnected [6–8]. Consideration of manufacturing aspects is either done outside of CAD using traditional design for manufacturing and assembly (DFMA) methods [9–12] or ignored altogether during the design process. A recent study explored whether these DFMA methods, specifically focusing on additive manufacturing, resulted in fixation on features prematurely [13]. It was found that practicing engineers, without using CAD tools to supplement the evaluation process, fixated on subfeatures without consideration of other system requirements, causing project manufacturing challenges.

Supplementary CAD tools have been developed by both professional and academic users of CAD software to support design for manufacturing [14]. Some tools are focused on analyzing completed parts for manufacturability, providing scores for accessibility or machinability [15], while others focus on providing a final cost estimate for the final part model [16]. DFMA tools can be classified as knowledge based or history based. Knowledge-based approaches seek to provide design guidelines and offer recommendations to change the product to make it more manufacturable [17–20]. Alternatively, a history-based approach uses past manufacturing or assembly costs to create predictive models based on product complexity [11,21–23], features [9], or assembly movements [24–26].

One example of previous work to support DFMA includes prediction of late-stage production information such as assembly time and market value using early-stage information such function structure models and assembly models [27–29]. These predictions are generated using artificial neural network models and require a representative sample of products for training prediction models. Alternatively, researchers have integrated manufacturing information into CAD/CAM environments to guide designers toward using geometrical features that are more suited to the manufacturing capabilities available for production. A feature library is imported into the CAD environment that includes a hierarchical list of features linked to machining information, functionality, design limitations, and design for manufacturing (DFM) guidelines [30]. In the field of architectural design, researchers have proposed entirely new CAD environments that emphasize manufacturing and environmental concerns, by providing rapid feedback of relevant scientific data. This allows designers to evaluate the effects of their design decisions on a variety of metrics including airflow, energy consumption, lighting, and environmental impact through life cycle analysis [31].

Commercially available solutions such as dfmpro³ offer addons that analyze part geometry and features to provide suggestions for manufacturing-focused improvements. These addons provide feedback that is easy to understand and implement with little impact on the modeling process. Alternatively, more comprehensive solutions such as aPriori⁴ use a broader scope of engineering information (CAD models, BOM, materials, volume, factory capabilities) to provide decision support including costing information. While these tools take a more holistic approach to the design for manufacturing, they are less suited for rapid feedback in conceptual design. DFM concurrent costing⁵ is a tool that is integrated into various software packages that require user interaction to answer questions about the parts in question. It is primarily used to estimate a completely modeled part, rather than to illustrate how a modeling action changes the cost. easykost⁶ is a solution that compares designed products to a database of available products and their associated costs. Finally, leandesigner⁷ automatically identifies regions of the solid model that are in conflict with the predefined design for manufacturing rules (expert system) and estimates the manufacturing cost based on general geometric properties (dimensions, mass, and volume). While these tools are presented in the literature as design enablers, their utility is not explored to understand how these tools affect the design and modeling processes through experimental user studies.

For design tools to be useful in practice, they must be validated not only for the accuracy of the information they provide but also for their usefulness to designers. Moreover, the tool should be logical, use meaningful and reliable information, and must not bias the designer [32]. Unlike descriptive mathematical models, engineering design tools are generally prescriptive, meaning they aim to predict future outcomes. Additionally, early-stage design tools are often subjective, meaning different users may draw different conclusions from the use of a tool. This may be due to the level of uncertainty present in the input from human designers and the qualitative nature of the design. As such, design tools are experimentally tested to confirm that they produce the expected outcomes and to ascertain whether designers find the tool useful. The comparison could be subjective and qualitative or may be objective and quantitative.

Experimental user studies have been conducted to explore how different design tools influence the final outcomes. This could range from studying how the size and shape of a morphological matrix can affect how designers explore the design space [33], how the user interface of a CAD system influences the errors made in modeling activities [34], or how different CAD systems result in different modeling approaches for users [35]. Even with these approaches, the studies are limited to comparing between different representations, different interfaces, or different CAD systems. The literature is notably lacking with respect to studying how a design tool changes how users create models.

Toward filling this gap, this article will discuss the experimental validation of a CAD tool developed by a space and aeronautics defense contractor. The tool aims to provide rapid feedback to designers about manufacturing time. A history-based estimation of manufacturing time generated from a representative set of parts is used. It is assumed that parts are produced using traditional machining or milling techniques. Finally, it should be noted that the accuracy of the tool has been verified by the industry partner. The experiment discussed here focuses on the effect of tool use. For brevity, the tool will be referred to as the DAX tool [36].

The DAX tool is developed to assist designers in creating CAD models by providing on-demand prediction of different types of costs associated with the production of the part being designed. In this case, we are specifically interested in the labor hours (as predicted by the tool) needed to manufacture a part. Three conditions of tool availability are studied: with tool, placebo tool, and no tool. The tool is based on a historically trained surrogate model that considers the connectivity graph (boundary representation model) of the part that is modeled. The placebo tool should behave like the DAX tool but lack the added analytical value provided by the live tool. Specifically, the placebo tool presented cost estimation back to the user as a function of the overall mass of the part modeled. Both the “tool” and the “placebo tool” have identical interfaces where the source of the cost estimate is hidden from the user. The final “no tool” condition provides no-cost estimation feature to the participants.

To investigate the effects of the DAX tool, a between-subject replication experiment was designed. Two independent variables were modulated: (1) the type of tool being used and (2) the type of modeling activity performed by the participant. With three tool types (no tool, placebo, and live tool) and three modeling activities (part creation, lightweight redesign, and interface design), nine total conditions were explored. For each activity, participants were asked to reduce part mass and minimize the expected manufacturing time. The manufacturing time (cost) and mass reduction are generally conflicting objectives for a machined part. This was selected as a representative scenario that could capture tradeoffs experienced by designers. A sample of the instructions provided to the participants is found in Table 1. Other detailed task instructions given to the participants are omitted here for brevity. Participants were randomly assigned to one of the nine conditions while making sure participants do not repeat options from either of the variables. This was done to mitigate the effects of the task sequence.

The DAX tool processes CAD data to provide estimates of manufacturing time. The solid modeling data are sent, upon request from a solidworks macro, to an external server that generates cost estimates. The estimates are displayed in a graph within a browser window as shown in Fig. 1. All results from each request are displayed so that the designer can see the impact of the changes between estimate requests. The mechanism of cost estimation is derivate of the previous work for graph/complexity-based assembly time estimation [22,23,37,38]. The internal components of this mechanism are proprietary and not detailed here.

Participants were recruited from engineering programs at Clemson University through a “invitation to participate” survey, which was subsequently used to filter candidates based on their experience with solidworks. A total of 104 students responded to the initial call for participation by completing the survey, 98 of whom were qualified for participating in the study. The qualified participants were scheduled for an experimental session based on their reported preferences. Each experimental session included a single-part modeling activity. Of the 98 students contacted, 72 students appeared at their scheduled experiment sessions and completed at least one session of the study, though some data files were corrupted, reducing the number of data sets. Data from 67 unique participants were fully analyzed. Among these, 20 participants completed only 1 session, 10 participants completed only 2 sessions, and 37 participants completed all 3 sessions. A total of 151 sessions were completed. Finally, participants were provided an incentive to complete the tasks in the form of cash value gift cards ($25 gift card for completing the first session and $75 for completing the subsequent two sessions). The study was reviewed and approved by the Clemson University institutional review board.

Due to restrictions stemming from the COVID-19 pandemic, the experiment was conducted virtually using remote access through citrix workspace. As a result, most participants likely completed their sessions on small laptop screens and with the added layer of remote desktop connection. While this provided challenges in terms of consistency of hardware across participants, it allowed participants to use a familiar computer and allowed for scheduling flexibility. Unique virtual sessions were convened through MS Teams to record the sessions and to communicate questions and instructions. A batch script was created to launch solidworks (for the activity), camtasia (for recording workstation screen), and a python script to log cursor location and button presses. If the experiment condition included one of the two tools (live or placebo), participants directly launched the solidworks macro to request tool updates. When the participants completed the activity, they informed the experimenter, who then confirmed that the necessary files (a solidworks part file, a screen recording of the session, a text log of mouse/keyboard activity, and a *.jsonl file with log of tool use) were saved in an appropriate location. These files were transferred from the local workstations to a shared drive location accessible only to the researchers and anonymized.

The planned experiment procedure included collecting four key types of raw data from the sessions: a solidworks part file, a screen recording of the session, a text log of mouse/keyboard activity, a *.jsonl file with log of tool use. The part file, screen recording, and the text log were collected from all participants, whereas the *.jsonl file was only collected when participants were assigned either the live tool or the placebo tool. The raw data needed to be processed into the outcome metrics needed for analysis. The procedure for generating each of the metrics (modeling time, part mass, tool use, and labor hours) is discussed in this section.

Modeling time is intended to measure the time taken for completing the assigned activity. The simplest measure of modeling time is the length of the video recorded during the experiment sessions. However, the video length includes time for initial setup before the modeling activity is started. Additionally, due to the remote nature of the experiment, participants may have stepped away from the computer or take a break from the activity and do something else. While participants are advised to complete each single session in one sitting without distractions, it was not possible to control the environment as most participants joined the remote sessions from home. Therefore, the text log of keyboard and mouse activity was used to identify instances where participants were inactive. Any period of “no activity” lasting longer than 3 min was tagged and removed from the total modeling time measure. Finally, in sessions where participants were given either the placebo tool or live tool, they spent time setting up the tool before beginning the modeling activity. In this case, calculating the exact amount of time spent in setup was difficult to assess from the text log of keyboard and mouse activity. Therefore, 20 sessions (out of 96) were randomly selected and reviewed to estimate the time taken for tool setup. To emulate the individual differences in participants, the average setup time was made fuzzy by applying a randomizing factor (up to ±25%), resulting in a setup time between 3 and 5 min. The maximum value of this change was informed by the variance in setup time observed in the 20 sessions. Once all the adjustments are applied, the resulting time is recorded as the modeling time.

Next, the part mass is extracted from the solidworks files. A solidworks macro was created to open each part file, check the unit system, and record the part mass, volume, and surface area. In cases where the participants used an incorrect unit system, the instances were tagged. These models were then scaled to reflect the correct units before recording geometric properties. An initial review of the mass, volume, and surface area showed that volume and surface area trend closely with mass. As a result, only the part mass is used for analysis.

After recording the part mass values, the part files were converted into STEP files. The live DAX tool was then used to estimate the labor hours needed to manufacture each part. The tool provides four estimates for each part with confidence levels of 90%, 75%, 50%, and 25%. Predictions are more conservative as confidence level increases. As such, 90% predictions are more conservative, while the 25% predictions are more optimistic. For the analysis presented in this study, 25% and 90% are used to compare the low end and high end of prediction labor hours.

Finally, the *.jsonl files were processed to identify the number of times a tool was used during the modeling activity. This was done by counting the number of lines in the *.jsonl file associated with the session. In cases where duplicate lines occur in succession, the duplicates were removed from the count. This is because duplicate lines in the *.jsonl files are likely due to technical issues or participant misuse of the tool. A few participants appeared to impatiently “click” the cost estimation tool multiple times before the estimate graphs were updated. It should be noted that if duplicate lines are found with other lines between them, they are not removed. This is because participants may have reverted to a previous state of the model, which is a common phenomenon in CAD and not likely to be user/technical error with the tool.

This section summarizes the data collected from the initial survey and provides an overview of the data collected from the experiment sessions. All data collected were reviewed to identify any missing or corrupted files resulting from technical issues or user errors. Ultimately, data from 67 unique participants were analyzed. Among these, 20 participants completed only 1 session, 10 participants completed only 2 sessions, and 37 participants completed all 3 sessions. A total of 151 sessions were completed. It should be noted that analyses presented in this article only consider the experiment sessions that were successfully completed, and no data were lost due to technical issues or user error.

Student responses to the survey were reviewed to understand the demographics of students participating in the study. It should be noted that classical demographics are not collected (e.g., gender, age, ethnicity), instead characteristics of participant that are expected to be relevant to the study are recorded. The initial survey asked students about three topics in addition to scheduling for sessions: (1) information about their engineering graphics course or equivalent, (2) their experience using CAD software in an academic setting, and (3) their experience using CAD software in professional settings.

Participants were asked if, and when, they completed an engineering graphics course. Responses ranged from Spring 2014 to Summer 2020, with 43 of the 68 participants having completed the course between 2017 and 2019. Participants were also asked to report their grades in the course, which shows that 58 of the participants received an A (90% or above) in the course, 7 received a B (80–90%), 1 received a C (70–80%), and 2 did not report a grade.

The initial survey included questions about participants’ experience with CAD software in academic and professional environments. Table 2 shows a count of participants who had some experience using select CAD software. Values under the “Academic” column represent the number of participants who have at least one semester of high school, technical college, or university experience using the respective software. Similarly, values under the “Professional” column represent the number of students who have used the respective software in a professional setting for more than 3 months.

All participants had experience using solidworks in an academic setting, with 75% of them also having used it in a professional setting for more than 3 months. The next most popular choice among students was autocad and/or Autodesk, with 65% and 38% of students having used it in academic and professional settings, respectively. It should be noted that the institution where the experiment was conducted offers an engineering graphics course with autocad, which is required for students enrolled in certain programs and a solidworks class that is required of mechanical engineering students. catia was used by 25% of participants in an academic setting and 19% in a professional setting. ptc creo was used by 19% of participants in both academic and professional cases, whereas 15% of participants used siemens nx.

Each participant also reported how long they have used different CAD software in professional settings. As solidworks is the software used by all participants, only data for solidworks are shown in Table 3.

Nearly 60% of participants have 1 year or less experience using solidworks in a professional setting. This can be largely attributed to the use of solidworks during internships and co-ops. Alternatively, participants with more than 1 year of professional experience are more likely to be graduate students and/or students with technical degrees. Finally, it should be noted that survey responses are self-reported data, and therefore, it is assumed that participants understood the questions and responded honestly. To mitigate linguistic and semantic issues, the survey was pilot tested with domestic and international graduate students to ensure that it was easy to comprehend.

Following the postprocessing, the modeling time, the final part mass, the frequency of tool use, and predicted labor hours for manufacturing the part are recorded and analyzed. Table 4 shows the mean (µ) and standard deviation (σ) of the outcomes based on the type of activity. All values for modeling time are provided in minutes. Similarly, all values for labor hours are provided in hours. Tool use values are simple counts, and part mass is provided in grams. It should be noted that the interface design activity used significantly larger parts; hence, a “k” is added to the mass value to indicate those values are in kilograms.

As shown in Table 4, the mean values for all the metrics are different based on the activity. This is expected because the activities were designed to reflect the different types of tasks that designers encounter in their work. Part mass and labor hours show the largest differences between activities. Meanwhile, the modeling time and tool use metrics are less clearly different.

In addition to the different activities, the study also uses three types of tool assignments: no tool, placebo tool, and live tool. Table 5 shows mean and standard deviation data for all sessions evaluated based on the type of tool used. Part mass is not considered here because the different activities are expected to have significantly different part mass (factor of 100), resulting in an average by tool type to yield values that are not meaningful for analysis. This is also true for labor hours, albeit to a significantly small extent (factor of 10).

As shown in Table 5, no clear differences are observed between the tool types when considering modeling time. For tool use, the mean frequency of use is higher for the placebo tool; however, the high variance in data suggests that the difference in mean values may not be significant. Further analysis of the effects of tool type and differences based on the type of activity are discussed in the following sections. Finally, it should be noted that data presented in Tables 4 and 5 include potential outliers, which are also discussed in subsequent sections.

Prior to the statistical analysis, the data were reviewed for outliers. Outliers were identified using the scaled median absolute deviations approach. While the total number of sessions is high (151), replication for each unique condition was unbalanced, ranging from 12 to 20. Additionally, human behavior data are normally widely varied and prone to outliers due to the inherent subjectivity and complexity of participants. As such, not all statistical outliers were removed from the data, rather only the extreme outliers were removed. These are defined as values that, upon removal, change the mean by more than 10% of the original value. The search for outliers was performed in the subsets sorted for analysis, not simply for the entire data set. For each subset, one outlier was removed at most. These outliers were often two to three times larger than the second largest value. No outliers were observed on the lower end of the data.

In addition to the search for outliers, the dependent variables are also tested for statistical independence among them. Correlation tests show that the modeling time was statistically independent from all other dependent variables (maximum R-value of −0.05). Product mass was moderately correlated with the manufacturing time (R = 0.72, 0.86). This is expected because the estimation of manufacturing time is dependent on part mass and geometry. Additionally, the two manufacturing time estimates are moderately correlated with each other (R = 0.79). This is not unexpected as the manufacturing time estimates are computed using the same inputs but with different confidence intervals.

The session data are sorted into two groups: (1) sessions where participants were given no tool and (2) sessions where participants were given either the live tool or the placebo tool. Mean values are computed for each relevant outcome metric as presented in Table 4. Mean values are found in the rows for each metric. The conditions that performed better, on average, are given in bold. Additionally, p-values comparing “no tool” and “with tool” conditions are presented in Table 6. Note that PC, LW, and ID refer to part creation, lightweighting, and interface design, respectively.

The existence of either tool increased the modeling time for all activity types. Part creation activity shows the largest change in average modeling time when using a tool. The slight increase in modeling time for tool use might be due to the time spent by participants calling the tool, waiting for predictions, and reviewing the graph (30–90 s per request). For part mass, using a tool resulted in lighter parts for the part creation and interface design activities, but had negligible impact for the lightweighting activity. Cost estimations with 25% confidence level showed no noticeable differences in mean values. However, cost estimations with 90% confidence level showed positive effects in the case of interface design and negative effects for part creation.

In all cases compared in Table 6, the mean differences identified are smaller than the standard deviations. Therefore, it is suspected that the differences may not be statistically significant. A set of two sample t-tests are conducted to evaluate the significance of mean differences with a significance level of 0.05. All but two resulting p-values are larger than 0.50, with the smallest p-value being 0.11. Therefore, there is insufficient evidence to conclude that the existence of a tool has a significant impact on the outcome metrics considered.

Table 7 shows a comparison of mean values for each of the outcome metrics based on the type of tool and sorted by activity. For this comparison, tool use frequency is also included as an outcome metric. Data in bold identify better performance between the tools. Again, extreme outliers were removed. P-values for each comparison of live and placebo tools are presented.

The placebo tool yielded lower average modeling times for part creation and lightweighting activities. The live tool shows lower modeling times on average for the interface design activity. This suggests that the live tool may be more helpful when participants are creating novel parts compared to when participants are modifying existing parts. In the case of part mass, the live tool resulted in a lower part mass in each of the activities. When comparing tool use per session, the live tool is used more often in part creation and less frequently in lightweighting and interface design, but the differences are small. For predicted labor hours (cost estimation), the live tool yields fewer hours for the interface design activity, while the placebo tool performs better for the lightweighting activity. For part creation, more generous predictions favor the live tool, whereas conservative predictions favor the placebo tool. The differences in mean values fall within one standard deviation from the mean. To test the significance, a set of two sample t-tests are conducted with a null hypothesis assuming no difference in means and a significance level of 0.05. A statistically significant result is found for part creation activity for the cost estimation at 25% confidence (p = 0.04 and shown in Fig. 2).

This suggests that using the live tool may help participants think more about modifying the parts such that they are lighter yet easier to manufacture. However, the data collected in this study do not provide sufficient evidence to support that claim as the differences in part mass values are not statistically significant. The modeling time for the interface design activity resulted in a p-value = 0.07. Moreover, experiments in human behavior tend to show higher variances, so the use of a higher significance level may be justified. All other t-tests resulted in p-values greater than 0.15.

Modeling time is further investigated to understand the correlation between tool type and these metrics. An analysis of variance comparing the lumped modeling times (combining tool types) for between activities showed that modeling times for the different activities were statistically different based on a significance level of 0.05. Next, the modeling times were investigated within each activity. Figure 3 shows boxplots of modeling time. The lightweighting activity is omitted because modeling time in that activity is similar across both tool types. In the case of part creation, the live tool shows a wider spread, skewed toward the higher end, thus suggesting that the live tool is more likely to yield longer modeling times. Conversely, the interface design activity shows that the placebo tool is more likely to produce longer modeling times.

In the part creation and lightweighting activities, the end goal is more discernable to the participants, such as recreating geometry or achieving mass reduction. Alternatively, the interface design activity asks the students to design the geometry with minimal guidance. In this case, a reliable tool that provides useful feedback on design decisions may help participants reach a satisfactory solution. This is further supported by the part mass and predicted labor hour values, which show that the use of the live tool is correlated with lower part mass and lower manufacturing time. Again, insufficient evidence is available to demonstrate statistical significance.

In summary, while the two tools were not found to be statistically different with respect to modeling time and part mass, the live tool was generally more helpful in creating lighter parts and showed quicker modeling times for open-ended activities.

Based on the data collected and analyzed, the expectations set up prior to the experiment are evaluated. First, the tool does not negatively impact design outcomes. Second, more frequent use of the tool improves design outcomes. It should be noted that “design outcomes” in this case are evaluated with respect to modeling time, part mass, and manufacturing time or predicted labor hours. Tool use count is not viewed as a design outcome as it is more a measure of participant behavior.

The live tool adds a set of actions that participants need to perform beyond normal modeling behaviors. Combined with the processing time of the tool, it is expected that the tool use would increase the modeling time. This was seen in an 18% increase for the part creation activity and 5% increase for lightweighting and interface design. Finally, the concept of modeling time was noticeably absent from participant feedback in the exit survey. Participants provided a variety of positive and negative remarks about the tool, but none of the 45 participants mentioned modeling time as a concern. This suggests that while the use of the tool showed an increase in average modeling time, participants did not necessarily perceive it as a noteworthy change.

The existence of either tool (live or placebo) did not significantly affect the final part mass on average. The largest change was observed in the interface design activity, where the use of a tool resulted in a 4% lower mass. Using the live tool resulted in lighter parts (3% lower mass in the part creation and lightweighting activities and 12% lower mass in the interface design activity). Thus, the live tool produced the lowest part mass on average. While these differences were not found to be statistically different, the rank order of average part mass indicates that the live tool does not negatively impact the design outcome in this case.

The two mass reduction values for each participant are compared to identify whether a given mass reduction value is higher, lower, or similar to its counterpart. Results for the lightweight and part creation activities are presented in Table 8. More participants are found to have better mass reduction when using the live tool. Alternatively, more participants are found to have worse mass reduction when using no tool. In the case of the placebo tool, it produces higher mass reduction at a similar frequency to using no tool. However, it shows fewer cases of lower mass reduction. These results suggest that for any given participant, using a placebo tool is not likely to affect mass reduction negatively while using the live tool is more likely to improve mass reduction.

Finally, reviewing participant feedback from exit surveys suggests that participants used the labor hour predictions as a check on their material removal actions. Several participants observed that when certain design changes resulted in a large increase in the predicted labor hours, they re-evaluated the changes to achieve a similar reduction in mass without a large increase in manufacturing time. This suggests that the tool can have an impact on the design decisions, provided participants prioritize manufacturing time or try to find a balance between mass reduction and manufacturing time.

Using the live tool resulted in the lowest average predictions of labor hours, for both optimistic (25%) and pessimistic (90%) predictions in the case of the interface design activity. Conversely, using the live tool resulted in the highest average prediction of labor hours for the lightweighting activity. In the part creation activity, the live tool produced average optimistic predictions lower than the other tool conditions.

For pessimistic predictions of labor hours in the part creation activity, the live tool resulted in an increase compared to other tool treatments. Overall, this suggests that the impact of tool use on manufacturing time is dependent on the type of activity. It should also be noted that manufacturing time and part mass are inversely related to the types of activities used in this experiment. Therefore, if participants prioritize mass reduction, they are likely to produce parts that require a longer manufacturing time.

For part creation and lightweighting activities, increasing tool use trends with increasing predicted labor hours. This trend is reversed for the interface design activity, where the increased use of the live tool shows a reduction in the predicted labor hours. While these are weak trends, they suggest that using the tool more frequently during a design activity may be more beneficial when the designer is creating a novel part with minimal restrictions on geometry. Some feedback from participants further supports this. Participants reported that they “tried to make changes that did not substantially increase manufacturing time” and that the tool helped them see the “efficiency of your material removal.” Moreover, some participants noted that the tool did not “work correctly” during the two sessions, inadvertently identifying the difference between the predicted labor hours from live tool and placebo tool.

This article presented an experimental testing and validation of a CAD tool designed to support design for manufacture by providing designers with an estimate of machining time. To test the effects of the tool on CAD modeling and design outcomes, an experiment was developed with between-subject replication. Three different design activities were created, and three different tool treatments were provided. The goal of the experiment was to test the use of the tool in different CAD activities and compare the effects of the DAX tool with a placebo tool and with a control condition where no tool was provided. Data collected from the experiment were processed to obtain four metrics. Three of these metrics (modeling time, final part mass, and predicted labor hours) are used as a measure of the design outcome.

The live tool was found to increase the modeling time on average for all modeling activities, but these differences were not statistically significant. Participants did not perceive the tool to increase modeling time. Analysis of the final part mass shows that using the tool produces lighter parts on average compared to using no tool or using a placebo. While the change in part mass is not found to be significantly different, participants reported that they used the tool to guide their design decisions with respect to material removal. Finally, mixed results are observed when tool use is compared with manufacturing time. In generative design activities, the tool helped participants design parts that are quicker to manufacture; however, in other activities, the tool either did not have a notable impact on the manufacturing time or increased manufacturing time compared to using the placebo or using no tool. Participants found reviewing the labor hours predicted by the tool helped guide them toward parts that they perceived to be easier to manufacture.

More generally, a methodological approach is presented that can identify how a design automation or augmentation tool can influence the design process. With new tools being introduced, companies are challenged in understanding the impact that these tools will have on their current design processes and activities. The project presented here was initiated by industry partners as a means to evaluate how a proposed tool might influence the performance of their designers, in terms of both effectiveness (part mass, part cost) and efficiency (modeling time). Without a clear means of investigation, such as that presented here, the tool impact is only a speculation based on experience and anecdotal evidence and not much more than marketing material.

While this study shows that the tool did not produce a significant negative impact on the design outcomes, feedback from participants and analysis of the data collected in the experiment provides avenues for potential improvements for the tool. One of the common themes in participant feedback was about the integration of the DAX tool into the CAD environment. While the implementation described in this article provided a simple and usable interface for participants, a more wholistic integration is preferred. This may have been in part due to participants using small laptop screens and not being able to keep the prediction graph, tool user interface (UI), and modeling instructions all in view simultaneously. However, future iterations of the experiment should still provide the tool as a feature integrated into CAD environment so that the tool UI does not interfere with the workspace. In addition to UI improvements, future work should also include a tutorial or help module for the tool. This will ensure consistency in how participants interpret the output from the tool. Moreover, future experiments should also include a within-subject replication component to test whether participants become more familiar and/or comfortable with the tool and how that changes the effectiveness of the tool.

Finally, the scope of the tool can be expanded to cover assembly modeling by providing prediction of assembly time. The tool can also be improved by integrating other manufacturing methods. As previously mentioned, the DAX tool assumes that parts are manufactured using traditional machining techniques. The tool can be expanded to include more advanced manufacturing methods, as well as additive manufacturing. Negative feedback from participants about the tool often cited its lack of applicability for nontraditional manufacturing processes.

Finally, the modeling behaviors can be studied to determine what behavioral sequences are encouraged or discouraged through use of the tool. This is similar to the function modeling work that analyzed the steps designers used in creating systems models [39]. This type of analysis can support a deeper exploration into designer cognition as influenced by augmentation tools.

There are no conflicts of interest. All procedures performed for studies involving human participants were in accordance with the ethical standards stated in the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards. Informed consent was obtained from all participants. Documentation provided upon request. Informed consent was obtained for all individuals. Documentation provided upon request. This article does not include any research in which animal participants were involved.

The datasets generated and supporting the findings of this article are obtainable from the corresponding author upon reasonable request.