Evaluation Framework for Augmented Reality Smart Glasses as Assembly Operator Support: Case Study of Tool Implementation

Augmented reality smart glasses (ARSG) have been identified as relevant support tools for the Operator 4.0 paradigm. Although ARSG are starting to be used in industry, their use is not yet widespread. A previously developed online tool based on a framework for evaluating ARSG as assembly operator support is iteratively improved in this paper with expanded functionality. The added functionality consists of practical recommendations for implementing ARSG in production. These recommendations were produced with the help of five focus groups of industrial representatives working in production. The recommendations were evaluated using case studies at three different companies. The recommendations were found to be detailed and a good support for the process of considering ARSG integration into production. The companies overall found the tool and its recommendations to be relevant and correct for their cases.


I. INTRODUCTION
It is generally believed that the term ''augmented reality'' (AR) was first used in [1], which predicted that AR might one day be used to provide assembly workers with direct access to CAD data, thus reducing error rates and costs. AR is defined as having three properties: combining real and virtual objects in a real environment, allowing real time interactivity, and aligning real and virtual objects with each other, making them seem to be part of the same reality [2]. AR can be implemented in three ways: head-mounted, handheld, and placed in the environment [3], [4]. Head-mounted AR has been identified as the most suitable for operators as it enables them to receive hands-free information in their field of view [5,6]. Head-mounted devices are sometimes referred to as AR smart glasses (ARSG) [7]. In this paper, ARSG are defined as ''a wearable device with one or two screens in front of the user's eyes that can merge virtual information with physical information in the user's field of view (FOV)'' [8, p. 1299]. The fact that ARSG provide information in the operators' FOV has the additional advantage that operators can receive individual instructions in the same work area.
As regards the increased complexity associated with Industry 4.0, AR is identified as a visual computing technology The associate editor coordinating the review of this manuscript and approving it for publication was Hao Ji. that can support Operator 4.0 in performing traditional tasks and in defining new tasks and scenarios [9]. This is why the highest adopters of AR are industrial enterprises [10]. ARSG are estimated to have a compound annual growth rate of 36 % from 2019 to 2026, with the expected catalyst for adoption being the incorporation of AR in Industry 4.0 applications [11]. Even so, adoption of AR is currently still low in industry [12]. One of the factors hindering AR adoption may be a lack of experience with AR systems interaction [13]. However, the main challenges for AR in industry are the software ecosystem and organizational integration [12]. Masood and Egger [12] hypothesize that more knowledge of AR is needed to support decisions on whether to adopt it and that external support may also be needed to build the necessary knowledge base. Reasons for companies' lack of knowledge of AR could be the availability of more cost-effective alternatives and the lack of resources to investigate ARSG [14]. The reasons could also include a lack of knowledge of the capabilities of ARSG in supporting assembly operators [14]. This was the motivation for creating a framework to help those who develop and improve assembly stations for operators to evaluate the suitability of ARSG as a support tool in specific production cases [14]. According to an expert in implementing ARSG for assembly (see ''Expert interview'' below for further details), ARSG should be seen as just one of many alternatives for supporting operators.
Therefore, there is a need to identify when and how ARSG adds value as operator support in assembly.
To support the industry in identifying when to use ARSG as an operator support, a framework was previously developed as an online evaluation tool [14]. The framework is aimed towards integrators of operator support equipment who wants to be able to quickly assess how high they should prioritize ARSG compared to alternative ways to improve production. It is used by answering 15 questions about a production case, with 5 potential follow-up questions in total. The questions have predefined answers and generates a normalized score, indicating how suitable ARSG is as an operator support tool. Depending on which quartile the score landed in, one of four general descriptions of the case suitability was generated. This first version of the framework received positive feedback, but a limitation of the framework is that it does not provide deeper motivations or practical guidance in how ARSG are suitable or not suitable for a case.

II. AIMS AND OBJECTIVES
The aim of this paper is to improve on the previously developed implementation of a framework [14]. The improvement consists of increasing the functionality of the framework by providing further support once a suitable case has been identified, as well as suggesting what to consider to improve less suitable cases. The framework is made available through an online implementation that provides answer alternatives in response to user input. A corresponding set of recommendations is generated and presented.
The short-term gain from this improvement is to accelerate the rate of ARSG integration into assembly lines, which can increase operator efficiency. By providing practical advice on how to continue the evaluation and on the first steps of implementation, this framework will help to improve understanding of the usability of ARSG in specific cases. The previously identified long-term gain of acquiring a better understanding of the strengths and weaknesses of ARSG is further enhanced by the expansion of the framework in the iteration in this paper.
To achieve this aim, the following objectives have been set: • Develop more detailed recommendations for all questions and answer alternatives of the framework.
• Evaluate the detailed recommendations to ensure industrial relevance.
• Evaluate the tool implementation of the expanded framework as a whole in relevant production cases.

III. BACKGROUND
Providing decision support for industry for AR is not a new concept. In 2017, Palmarini et al. presented a process for choosing an AR system for maintenance operations [15]. The process assessed AR feasibility, hardware, development platforms, and visualization methods. It was based on a literature study, gray documents, and expert interviews. In the area of assembly, a similar framework was created to support the process of deciding whether ARSG are suitable for operator support [14]. The framework consists of fifteen questions about a production case. Each question has a range of alternative answers, with each answer associated with a particular score, assigned on the basis of feedback from industrial experts. The total score for the production case is then normalized to a score from 1-100 to indicate how strong the case is for using ARSG as operator support. One of four general recommendations is then presented to the user of the tool, ranging from ARSG having a very low probability of being applicable to a very high probability. Some questions have answer options that result in follow-up questions if the topic may be a critical issue or an obstacle to the use of ARSG, as, for example, when operators work where there is traffic. Any critical issues are presented in a table on the results page.
The table shows the question, the answer option chosen, the follow-up question, its answer option, and a recommendation on how to handle the issue. One area of improvement identified in [14] was to extend the tool functionality to provide further support in the steps following the first evaluation of a production case. This support would be similar to the recommendations on how to handle specific challenges in relation to critical issues. This follow-up is expanded in this paper to include all answer options.
It is very important to have a clear and accurate understanding of the requirements before starting any system development, but in practice this step is often skipped or downplayed. Thus in two case studies, development was affected by limitations in the requirements inquiry [13]. An advantage of the framework presented in this paper is that it consists of specific questions related to the production case evaluated. The tool developed in this research documents the specific attributes of the chosen production case, thereby laying a foundation for possible continued system development and integration.

IV. METHODOLOGY
The method used in this paper is a combination of two methods: a ''method triangle'' and ''five iterative steps'' [16]. Lings and Lundell [17] presented three perspectives on a method: method-in-concept, method-in-tool, and method-inaction. This was conceptualized by Thorvald et al. [16] as a ''method triangle.'' According to Lings and Lundell [17], the method-in-concept is a social construct of how the stakeholders understand the method. They further state that the method-in-tool occurs when the method-in-concept is realized, and the method-in-action consists of different methodin-tools, used in particular contexts. Both social and technical issues need to be addressed in order to transform a methodin-concept into a method-in-action. Lings and Lundell [17] classification of different instantiations of a method serves to highlight the point that a method should be seen from different perspectives, depending on where in the development and application phase it is.
Another method development process, presented by Blandford and Green, consists of five iterative steps in a life cycle approach [18]. Thorvald et al. [16] combined the method FIGURE 1. The connection between the five iterative steps of [18] and the three method steps in the ''method triangle'' from [17], proposed by [16].
triangle with the five iterative steps of Blandford and Green [18], see Fig. 1. This was done to clarify the steps of method development. When combined like this, the first step focuses on the method-in-concept, followed by a focus on both the method-in-concept and method-in-tool in steps two to four.
In this paper, step one (identification of an opportunity or need) is described in the Introduction, and can be summarized as a need to provide knowledge of ARSG for industry.
Step two (development of more detailed requirements) was based on previous literature reviews [8], [19], [20] and extended by knowledge gained from the first iteration. This step is described in more detail in the section Straw Man Base.
Step three (matching opportunities, needs and requirements) was performed using focus groups. The method used is described in the section ''Focus Groups.'' The implementation of focus groups is described in the section ''Focus Group Interviews.'' Step four (development of the tool) followed, where the focus group interviews generated updated content to be added to the straw man base.
Step five (testing of the method) is done by using the finalized tool in case studies and evaluating the results.
The tool-implementation of the framework in this paper is based on the Cognitive Load Assessment for Manufacturing (CLAM) method developed by [16]. The CLAM method assesses the cognitive load for assembly operators. The method-in-tool was developed as an online web tool [16]. For this paper, an online web tool is used as the method-in-tool.
Palmarini et al. [15] described a process for choosing an AR system for maintenance. They found that surveys, questionnaires, and case studies were suitable validation processes. Their case studies compared the choices of experts and non-experts using the process developed [15]. In this study, iteration case studies will be used. Experts will use the tool and discuss the recommendations it presents in their cases. This will provide insight into how well the recommendations compare to the views of experts.

A. FOCUS GROUPS
Tremblay et al. [21] shows that focus groups have gained increased attention in information systems. Based on Stewart et al. [22], Tremblay et al. [21] present four reasons for the suitability of focus groups in design science research projects. These reasons are flexibility, direct interaction with respondents, large amounts of rich data, and building on other respondents' comments. Focus groups were therefore seen as a suitable data collection method in this project. Participants for the focus groups were decided to be people with experience from planning, implementing, or in other ways making decisions regarding operator support tools for production. It was not seen as necessary to have experience from AR-systems since the focus of the framework is on how to integrate ARSG as a support tool and not on how to design and create ARSG or AR-interfaces. The questions relate to production cases and limitations they put on which tools can be integrated and how.
There are five characteristics of focus group interviews: ''(1) a small group of people who (2) possess certain characteristics, (3) provide qualitative data (4) in a focused discussion (5) to help understand the topic of interest'' [23, p. 6]. According to [24], it is not useful to provide a universal sample size recommendation for focus groups, and ultimately, saturation will be determined during data collection. They do, however, argue that the parameters they identified can help to guide in identifying and justifying sample sizes a priori. The parameters are study purpose, type of codes, group stratification, groups per strata, type of saturation, and degree of saturation. By analyzing 40 focus groups, [25] found that at least 80 % of all themes are likely to be captured with a sample size of two to three focus groups, and 90 % of themes are likely to be identified with three to six focus groups.
In traditional focus groups, participants congregate at a specific location. However, it is also possible to have the sessions online, an approach that is particularly suited to participants who have busy schedules and are geographically dispersed [26].
In this study, the a priori sample size for focus groups was set to three groups. This number was chosen partly based on the parameters provided by [24]. The aim was to have at least three participants in each group. The chosen format was synchronous online focus groups [26] because the target group are experts within geographically dispersed companies, who have busy schedules and are hard to book for physical meetings. Twelve to fifteen questions are ideal to avoid an online focus group degenerating into a survey [26]. Since the framework consists of fifteen questions, it was deemed appropriate to run through all the questions in one session. The five follow-up questions were seen as part of their corresponding questions and part of the same discussion.  The hatched sections are the parts of the tool that have been not been updated from the previous version. All the steps are described in more detail in the following sections, but a brief summary is given here. There were three sources of input for the straw man base (step 0): experience within the field of AR built up by the authors, previous literature reviews, the updated literature review for this paper, the previous iteration of the tool, and the feedback from users. This input was then implemented (step 1) in an improved version of the tool with detailed recommendations, acting as a straw man on which to improve. This straw man was then presented to the focus group participants to provide a basic understanding of the concept of the framework (step 2). The feedback from the focus groups and expert interview was then analyzed and used to update the straw man base. This updated version was evaluated qualitatively by case studies (step 3).
The first objective of this study is to identify relevant recommendations for integrating ARSG into assembly production. Knowledge from experienced integrators in industry was considered a relevant form of data for this purpose.

B. STRAW MAN BASE
As in the first iteration of this framework implementation, a straw man formed a basis for further refinement [14]. The straw man is just meant to start discussions by providing an example set of suitable content and therefore it does not need to be evaluated beforehand. The literature reviews done prior to the first iteration were also relevant in this iteration. A meta-analysis of previous literature reviews related to AR in manufacturing, assembly, and maintenance identified relevant aspects of ARSG as operator support [19]. The perspectives identified in that literature review were expanded on in two follow-up reviews. One of these perspectives, the operator perspective, was explored in depth in a literature review covering assembly instructions, human factors, design, and how to validate a design, depending on whether it was intended to support live production or offline training of operators [8]. Two other perspectives, manufacturing engineering and technological maturity, were explored in depth in a separate literature review [20]. The review from a manufacturing engineering perspective explored the topics of authoring a system, requirements for the surrounding infrastructure, and validating a system. The review of technological maturity explored enabling technology, technological demands, ARSG as a product, and tracking technologies. Knowledge derived from these literature reviews provided a theoretical basis for ARSG implementation as assembly operator support and was used, together with experience, as a starting point for the first iteration [14].
The first iteration of the tool resulted in an interactive web page on which fifteen questions with fixed and weighted answer alternatives were presented to users. Each answer alternative had a hidden numerical score, indicating the case's suitability for ARSG depending on the chosen answers. Five of the questions had some answer alternatives that could indicate a possible critical issue, leading to a follow-up question being asked. If such issues were identified, the result page would present a list of them along with a general recommendation on how to handle each issue. A score from 0-100 was then presented indicating how suitable the case was for ARSG. One of four general recommendations was also presented, depending on the quartile in which the score fell. This was then followed by a five-question survey of the tool itself, which is not a part of the framework.
For this iteration the theoretical base and the framework were revisited to determine relevant areas for expansion. The tool-implementation described above was used as a base to expand upon, as seen in the implementation section of Fig. 2. The questions, their answer alternatives, and their scores were left unaltered. However, a set of recommendations for each answer alternative was created and added to the tool implementation. The recommendations were based on the previous literature reviews, the authors' seven years of experience in AR for industry, and the knowledge gained during the first iteration.
The straw man base was sent to the focus group participants prior to the focus group meetings. This gave the participants a base to which they could relate the scope and purpose of the framework and how the tool presented it. They then took part in the focus group meetings and gave their feedback on how to change and improve the recommendations. The questions, their answer alternatives, score, and general design of the tool were locked so that they could not be modified during this study. Because the tool was not updated between each iteration of the focus groups, each group had the same starting conditions for the discussions.

C. FOCUS GROUP INTERVIEWS
Five focus groups were assembled with 2, 5, 3, 2, and 4 participants, respectively. The sessions were held online. The participants represented eight different companies and one university, see Table 1. Fig. 5 shows the procedure for each focus group. The first step was to ensure consent to record the session. This was followed by introductions, with the discussion leader starting to get all participants acquainted with each other. Then each question from the framework was presented along with the answer alternatives. The discussion leader gave a brief explanation of the purpose of the question and invited the participants to present their views on the question, both from the perspective of their particular company and also from a broader perspective. The focus of the discussions where on what would need to be considered if ARSG were to be used like 'any other production equipment' and what information the participants would need. All participants had previous knowledge on at least a basic level of what AR and ARSG are. If there was any uncertainty regarding what ARSG can handle or how they function the participants were free to ask and also did so. The discussion leader is experienced in both research and practical use of AR and ARSG and was able to answer all questions that arose.
After all questions were completed, the discussion leader summarized the highlights of the discussion, and the participants were allowed to comment. The last step, finalization, described what would happen afterwards, namely that the results would be summarized and analyzed, and the participants would be given access to the collected data.
All groups had time to discuss all questions. There was some overlap in the recommendations suggested by the groups, but no opposing recommendations were identified. Besides discussing the questions, the groups also discussed specific details of their own cases, feedback more specifically related to the tool design, and general views on the topic of ARSG. In group 3, one feedback was that the questions ''made you stop and think.'' In group 2 some feedback related to that the tool design and question formulations were somewhat unclear in some cases. Participants in group 2 felt that it was not clear that the tool focuses on assembly and not, for instance, processing. They also wanted it to be clearer at which level of abstraction the tool was to be used at (per station or per factory for instance). The negative feedback was used to improve the design by updating the tooltips and clarifying the introductory text of the tool to remove the identified uncertainties.

1) EXPERT INTERVIEW
After all focus group interviews had been completed, there was an individual interview with an ARSG expert from industry. The expert is a senior manager at a multinational manufacturing company with around thirty years of experience working with operator instructions, and has had the leading role in testing ARSG as assembly operator support at several manufacturing facilities within that company. This interview was used as a complement to the focus group interviews and was seen as a significant contribution due to the expert's unique seniority in the company and vast experience in the field of practical integration of ARSG in industry.
The expert indicated that it is important that the ARSG fulfill the requirements set by each company in regard to usability and safety. The goal should be that ARSG should be ''like ordinary glasses'' so that they can be used for long periods. For short-term usage, like control tasks done periodically, this requirement is less of an issue. What can be shown and how navigation and interaction can be performed are also important challenges. Translating 3D information to the ARSG interface can be a big challenge when there is little 3D information for the product. At the expert's company, 3D information was therefore more focused on the manufacturing tool side since they had better documentation on those tools. Presentation solutions cannot be standalone: they must connect to the current production system (i.e. digital seamlessness). The expert's company is currently investigating a multitude of general operator support tools.
The expert mentioned that it was common for there to be an initial sense of euphoria when starting discussions with teams considering ARSG. However, this was followed by a lowering of expectations when they realized that ARSG must be integrated into the production system. Therefore, it is important not to view ARSG as a solution to all problems, but rather as one of many tools available.
The main advantage of ARSG, according to the expert, is that they are hands free and mobile, as mentioned in a previous section. At present, there is not a strong case for continual use by operators in production due to the current weight of ARSG and operators being annoyed when shown too much information, for example, when they already know what to do. One helpful way to alleviate this is to have ARSG that can be ''flipped up'' so that operators can easily disengage them. At the same time, there is still a need for  instructions and there is often limited space in production lines to place information equipment such as TV screens. Interaction with information equipment such as keyboards, can also present a challenge. Static production lines versus moving lines can make projector solutions (spatial AR) more feasible.
The experience of the team the expert has worked with is that although there is a big focus on 3D and complex presentations of information, they prefer to look at finding solutions for more practical situations. For example, the expert mentioned that many operators work is set out on printed papers that they need to carry with them, and that screens are sometimes placed where it is hard to look at them. The focus of the company's work with ARSG was thus on finding solutions that would allow operators to see simple information and navigate in the interface without disrupting their hands-on work on the main task. The expert believed that promotional material for ARSG focuses too much on impressive 3D visualizations, which raises expectations and is disconnected from what is useful in practice.
The strongest cases for ARSG are in education for assembly operators and in maintenance. Dynamic instructions are an interesting aspect, allowing individual operators to see adapted instructions depending on errors, for example.
However, integrity issues need to be considered before this can be implemented. Vision recognition in the ARSG can be very useful if the interaction is to show correction information.

2) REFINEMENT
The input from the focus groups and expert interview were analyzed after all of them were completed. Each focus group session was replayed and all recommendations made by participants during the session were added as data to the corresponding question. Table 2 gives an example of the data collected from the groups for question 3 in the operator section. Some recommendations clearly belonged with another question, and were then transferred to that question. When all focus groups had been summarized, each question was analyzed in turn. All groups were analyzed in parallel for each question to ensure no conflicting recommendations existed. Then the data gathered in Table 2 were condensed into more direct practical recommendations. An example of this is presented in Table 3 for question 3 in the operator section. Thereafter the recommendations were mapped to relevant answer alternatives. The mapping was done by first summarizing all recommendations from the groups into one list, resolving any duplicate entries and combining similar ones. For instance, group 2, 3, and 5 all brought up that fluids can be harmful to ARSG. Group 3 also mentioned non-fluid particles as an issue. These three points were combined to warn against fluids and particles. Each of the four answer options to the question were then sequentially given a combination of the recommendations. The combination depended on the context given from the discussions. The resulting combined recommendations consisted of a set of sentences. In the final step, they were formulated into cohesive paragraphs for each corresponding answer alternative to present a concise but easily readable text. Table 4 is an example of recommendations for the different answer alternatives for question 3 in the operator section. Note that there are large similarities in the recommendations for the last three answer options. For answer options 2-4, the differences lies in the first sentences since the frequency of the need for safety glasses affects how they can be combined with ARSG. Regardless of frequency, the other recommendations are relevant. Once all answer options to all questions had recommendations, the recommendations and summaries of the discussions were sent out to all participants. They were given opportunity to raise any critique they had of the summaries and the recommendations. No critique was reported.

VI. EVALUATION
The framework consists of fifteen questions and five followup questions. Including the different answer alternatives, there are a total of 82 answer alternatives for which the tool gives a recommendation to the user, discounting the ''I don't know'' answer alternatives. It is not feasible to assess all possible permutations, so it was decided to assess the tool implementation by using a set of industrial cases. A criterion for choosing cases was to have a diverse set of industries represented to provide a broad view of the tool's validity in a range of assembly cases.

A. CASE STUDIES
Three case studies were done to verify industrial relevance, and they all followed a specific procedure. Meetings were booked with company representatives at their location. This was done to reduce the time investment for the company, to have close access to the production case for reference, and to have the industrial representatives in a familiar environment to allow full focus on the tool evaluation. A physical meeting was possible as all industrial representatives in each case were part of the same company.
The meeting began with a brief introduction to establish rapport and familiarity with the process. After consent was established, a voice recording of the meeting was started for later analysis. Then the researcher presented the tool to the VOLUME 9, 2021 participants and walked through each question. The participants discussed among themselves and decided on the answer to each question, with the researcher available for clarification when needed. Once all questions had been answered, the results were presented and explained by the researcher. Each critical answer and recommendation, if any, was walked through and discussed, followed by the detailed recommendations for each question. For each recommendation, there was a summary discussion about what the recommendation said and how well it coincided with the participants' views. They were actively encouraged to raise their concerns or opposing views, and to elaborate on the case and the recommendation. After all recommendations had been discussed, the participants were presented with the survey of the tool on the last page and filled it out jointly. There was then a debriefing and finalization of the meeting. The recorded material was later analyzed to compare the feedback from the participants for each recommendation and see what improvements were needed.
The first case involved a global manufacturer of sheet metal machinery. Two industrial representatives used the tool and discussed it. The estimate before starting with the tool was 64 (see the middle of Fig. 3). The score after running the tool was 77. The participants reached consensus on how to respond to each question in the tool. In general they agreed with the recommendations presented by the tool. A strength they identified was that the recommendations were condensed to key points, making it easier to get an overview by reading through the recommendations. The recommendations were also practical and relevant, and explanations were provided about how to consider and proceed with ARSG in their case.
The second case involved a global manufacturer in the automotive sector. Three industrial representatives used the tool and discussed it. The score from the tool was 77 (the initial estimate was 50). The participants also reached consensus on how to respond to each question in the tool. A critique that was raised was that the answer alternatives regarding the number of errors (very rarely, rarely, somewhat often, often, and very often) were a bit vague and diffuse. There was consensus in the group that the recommendations were good and relevant overall.
The third case involved a Swedish manufacturer of grainhandling equipment. Two industrial representatives used the tool and discussed it. The score from the tool was 84 (the initial estimate was 63). The participants reached consensus on how to respond to each question in the tool. Some comments from the participants were that the recommendations were very detailed and explained in a good way. According to the participants, the tool gave a good picture of whether ARSG would be suitable and especially why and how.
In summary, all participants in the three case studies generally agreed with the recommendations that the tool presented for their respective cases. They found the recommendations relevant and useful for considering if they should integrate ARSG in their cases and how to start this process. The critique from group two is understandable, but it referenced a deliberate design decision to make the framework suitable for general cases. It is not possible to set a specific percentage for error rates because what is acceptable in one industry may be unacceptable in another. The three cases were from three different companies, active in three different sectors, and in three different cities. Even so, they all independently expressed a positive view of the recommendations presented by the framework.

VII. DISCUSSION
The results and validity of the results and of the method are discussed in the following section.

A. VALIDITY OF RESULTS
A set of case studies as manufacturing companies was used to ensure that the results are relevant. Two possibly severe critiques were identified. First, by only evaluating a subset of the possible answer combinations, the evaluation does not ensure full coverage. Secondly, having a researcher meet the company representatives in person created a risk of social factors interfering, such as being accommodating or intimidated and adjusting the interaction accordingly.
The first point was addressed by considering the spread of the chosen companies. The companies were chosen from different areas to ensure diversity in testing. While located relatively close geographically, they were in three different cities and were active in distinctively different markets. The risk of the companies influencing each other is thus minimal. The second point was addressed by taking precautions to minimize the risk of noise in the data. As described in the evaluation section, the assessments were done at the companies' location to ensure that the participants were in a familiar environment so they could focus on the assessment.

B. VALIDITY OF METHOD
The method used in this paper is based on the development process proposed by [16] and is a form of design science. While design science has utility as its goal, it must be differentiated from routine design or system building [27]. Hiver et al. created seven guidelines to support the process of evaluating whether a research project is to be considered design science [27]. Since this paper has followed the same method as in the previous iteration of this tool [14], similar conclusions regarding adherence to the guidelines have been drawn in this paper; namely that the research follows the guidelines and that design science is a relevant and valid method. The main difference, and also the main critique, is that the tool can be seen as less novel than in the first iteration [14]. There are two main arguments for why this iteration is still considered novel research. The first argument is that the framework has been expanded to encompass a new aspect of ARSG implementation, that is, general practical advice on how to proceed. The second argument is that the evaluation in this iteration has used case studies, a new form of data collection that has produced more in depth discussion of specific application cases. Thus this research is considered as relevant and valid design science.

VIII. CONCLUSION
There were three research objectives for this paper. The first objective was to develop more detailed recommendations for all questions and answer alternatives of the framework. The straw man base for the framework expansion was created by revisiting previous literature reviews, the first iteration of the framework, and the accumulated experience of the authors. The recommendations were then successfully integrated into the framework and implemented in the web-based tool previously created. Thus the first objective was achieved.
The second objective was to evaluate the detailed recommendations to ensure industrial relevance. Relevant recommendations were identified by consulting industrial experts from a number of different manufacturing companies using focus groups and an individual interview with a leading expert in the field. This resulted in a set of recommendations of relevant aspects to consider when integrating ARSG as an operator support tool in assembly.
The third objective was to evaluate the tool implementation of the expanded framework as a whole in relevant production cases. The tool was evaluated in three case studies involving a diverse set of manufacturing companies. The results showed that the company representatives found the tool easy to use and the questions relevant for evaluating the suitability of ARSG for their specific cases. They generally agreed that the recommendations were correct and relevant, indicating an added functionality of the tool: further practical guidance once a suitable case has been found or better understanding of why a case might not be suitable. Thus this evaluation of the tool as a whole showed that the second objective had been achieved by improving the framework with added functionality. This framework now provides more in depth understanding of how and what to consider if ARSG are to be implemented as an operator support tool in assembly.

IX. FUTURE WORK
This paper reports on the expansion of a previously developed framework-based tool. It provides further support in the initial steps when deciding whether ARSG can be used in a specific assembly case, and indicates what to consider in regard to implementation. Evaluation was done using studies of real assembly cases at three different manufacturing companies. More case studies will yield more data to further enhance the accuracy of the tool. Currently the tool is only available in Swedish, which means it can only be used by industrial representatives who know that language. Translating the tool into other languages will allow it to be tested on a more varied set of assembly cases. Other future work includes broadening the focus to include machine operations and evaluation at a factory level.