Process of Materials Picking Using Augmented Reality

The thesis describes the research conducted on the effectiveness of purposed Augmented Reality solutions for material identification in the production system. The research was carried out at the Smart Factory laboratory of the Poznan University of Technology and was divided into two stages. In the first stage, the research has been conducted to identify both of the best variants of application and the AR smart glasses which were used in second stage. The time of identification and completion of production parts obtained using the AR technique and the traditional technique were compared during the second stage of the research. The results of the second experiment show that the AR method is efficacious for identifying materials in a production system; however, the traditional method time is shorter. This proves that the traditional method is more effective and therefore AR solutions need to be improved.


I. INTRODUCTION
The majority of modern production companies introduce an paradigm for the increasing digitization and automation of the manufacturing environment known as 0 [1]. One of the elements they introduce are the Smart Factories [2]. Human workforce, integrated into manufacturing systems in Smart Factories, must be flexible and adaptive [3]. It forces new methods of employee training as well as improving their work on a production line, which can be achieved by using Virtual and Augmented Reality (VR and AR) [4]. The definition of the AR technology includes constant connection of real-world and digital objects as well as an interaction between the user and the virtual objects [5]. The AR solutions can be based on cellphones, tablets [6], smart glasses, or headsets [7] such as Microsoft HoloLens.
AR applications can be used, among others, in a manufacturing company during the process of picking materials and orders in warehouses. The picking process plays a crucial role in supply chain management from the perspectives of production system and the physical distribution activities. Order picking is laborious, in addition, its costs can be even a half of the total warehousing costs [8].
The article shows the research results which allow to determine the effectiveness of the application of various AR technical solutions. The technical components which The associate editor coordinating the review of this manuscript and approving it for publication was Lei Wei . obtained the best results have been used for a research comparing the times of identification and completion of production parts achieved with the AR technique and the traditional technique. The research was carried out in the Smart Factory laboratory at the Poznan University of Technology A. INDUSTRY 4.0 The fourth industrial revolution, shortly Industry 4.0 [9] is usually identified with broadly understood digitalization [10]. Industry 4.0 is considered as a new industrial stage in which vertical and horizontal manufacturing processes integration and product connectivity can help companies to achieve higher industrial performance [11]. This concept assumes an improvement in the efficiency of production processes, an increase in production flexibility, a more efficient use of the company's resources, and an increase in the autonomy of employees, as well as the development of self-organizing and autonomous cooperation networks. The previously mentioned benefits can be obtained by digital integration of information and communication technologies (ICT)especially the Internet of Things (IoT) [12], the so-called Cyber-Physical Systems (CPS), [13]- [15] and manufacturing operations systems which are controlled and monitored by computer algorithms. Industry 4.0 assumes the preparation of a computerized and intelligent manufacturing environment guaranteeing flexibility and high efficiency of production, integration of different activities, and effective communication between a client and a producer, as well as between the producer and suppliers [16]. The concept of Industry 4.0 joins technological achievements from recent years with a vision of future intelligent and automated production systems, in which a real world is connected with a virtual one, ensuring more efficient use of available information [17]. The efficient use of information can be achieved by using Augmented Reality technology, which has started to be considered as one of the most interesting technologies companies should invest in [18].

B. VIRTUAL, AUGMENTED AND MIXED REALITY
Virtual, Augmented, and Mixed Reality use similar visualization techniques and hardware such as HMDs or smart glasses to obtain different goals. The main goal for VR applications is to make a user feel the sense of presence in computergenerated environment, this sensation is also called immersion [19]. It can be achieved in many ways, e.g. tracking users position and hand gestures, and allowing them to interact with virtual world. It is especially important in education [20], as user engagement can have positive effects on learning transfer. User tracking can be realized by using various tracking systems [21], basing on different physical phenomena such as ultrasound, magnetism, and visible or infra-red light.
Augmented Reality is used for displaying computer generated visualizations on real-world objects as a way to ''augment'' the real-world [22]. One of the most popular methods of recognizing real -world objects is the use of barcodes or 2D markers. Other methods include, among others, using RFID, GPS, and 3D object data. The method of interaction with virtual objects differs depending on the used device and its sensors. In the case of standalone devices (not connected to a computer), which are mostly used in production practice due to limited computing power, the interaction is usually limited and graphics are simplified.
Some studies distinguish AR from assisted reality and uses term assisted reality as form of AR in which user's field of view is enhanced by displaying 2D information and Mixed Reality as mature form where virtual and real objects are no longer distinguishable for the user [23].
Nowadays, VR, AR, and MR applications are widely used in entertainment, education, and industry. In literature there are many examples of using these technologies, for instance, in training and education [24], engineering [25]- [27], medicine [28], high school education [29] as well as tools supporting supervising and controlling of production processes [30]. Simplification of information exchange [31], reduction of downtime [32], display of Key Performance Indicators (KPI) and current data [33], access to resources and knowledge [34], operation and servicing machines [35], [36] etc. are examples of well described AR use. This wide range of uses is possible because AR / MR solutions can exchange data with other IT systems and display complex information [37]- [39].

C. ORDER PICKING AND MASS CUSTOMIZATION
The process of retrieving products from storage or buffer areas -order picking [38] can be carried out in many ways. The simplest method of order picking is done by the operators who walk around the site and collect units while the most advanced methods are based on a fully automated system with mechanized picking. Each of these methods is suitable for one or more uses, but it is the most laborious in warehouses with manual systems and very capital-intensive in warehouses with automated systems [8]. The method can influence for more than a half of the total warehousing cost and plays a crucial role in supply chain management [40]. In most manual systems, order picking is realized with a paper list, barcodes scanners, and pick-by-light or pick-by-voice systems. Using order picking system based on AR can improve this process by reduction of time for searching information when the data is displayed in the user's visual field [41]. This can be especially useful when the enterprise pursues a strategy of Mass Customization (MC), which is focused on fulfilling individual client requirements by production companies [42]. MC allows to satisfy the expectations of every client by adjusting a product to the client's needs [16] and thanks to this, different product variants are produced and the picked objects change rapidly, thus constant connection between a worker and database containing currently assembled products is especially important.

II. SMARTFACTORY LABORATORY AND THE SUBJECT OF RESEARCH
The production system at the Smart Factory laboratory of the Poznań University of Technology was built in order to conduct research for various technical and organizational solutions in accordance with the Industry 4.0 concept. The laboratory equipment allows for simulation mapping of various processes taking place in the real production system. The research conducted in the laboratory focuses on: development of systems supporting the design and configuration of customized products, application of virtual reality and Augmented Reality solutions in production and training processes, development of methods and IT systems supporting production planning and control, and application of solutions aimed at supervising and controlling the materials flow, e.g. RFID, RTLS. The main element is an automated assembly line, while the management of the production system is carried out in an IT system called 4Factory. The functionalities of the system enable production planning, material flow supervision, and production line control ( Figure 1).

III. SUBJECT AND PURPOSE OF RESEARCH
The subject of the research is the process of picking the parts required for the implementation of assembly operations. A pick and deliveries list for individual production stations is created as a result of material demand determined on the basis of the production schedule and the construction structure of products. On this basis, logistics employees assemble the required parts and their quantities, and then deliver them to the flow racks at production stations. The materials for production are stored on a separate storage rack with specific locations. The material flow (entry and withdrawal of material) is managed in the warehouse module of the 4Factory IT system. The presented process is carried out by logistics employees with the use of paper documentation which is the completion order. On the basis of the information contained therein, the employees identify locations in the resource warehouse, verify the type of parts, and count the required number of them. This sequence is repeated for each type of part being the subject of completion, which directly affects the time of this process accomplishment. The employee decides the order of picking the parts, and he also independently verifies its correctness.
The study which allows the determination of the effectiveness of the use of augmented reality solutions in the process of picking materials was carried out in two stages. The first stage concerned the selection of the AR system elements which allow to obtain the best reading efficiency / identifying data on the warehouse rack. The results of these tests were used in the execution of the second stage, which concerned the accomplishment of the process of picking parts in simulated conditions. The research concerned the comparison of two variants of the picking process accomplishment: with and without the use of AR solutions. Time and correctness of the picking process were determined as the evaluation criteria.

IV. RESEARCH ON SELECTION OF AR SYSTEM ELEMENTS
The research concerned the selection of the Augmented Reality system elements allowing for the greatest effectiveness of identifying markers placed on the shelves of a warehouse rack. The elements subject to research were: AR glasses, AR applications, marker type, and marker size.
The research plan included conducting experiments that allowed to establish the correlation and selection of the elements allowing for the greatest effectiveness of action.

A. AR DEVICES CHARATERISTICS
Two devices classified as Augmented Reality Smart Glasses (ARSG) were used in the research. ARGS are head-up transparent display integrated with a wearable miniature computer [43], that present information at eye level, making them an ideal user interface for an industrial operator [44]. The first device is an Epson MOVERIO BT-300 which has an additional touchpad. The second device is the Vuzix MT300. The main parameters of both devices are summarized in Table 1.

B. APPLICATION CHARACTERISTICS
Two applications have been prepared in the Unity 3D environment. The applications differ in the method of retrieving information in AR glasses. They were connected with the database of the warehouse module of the 4factory IT system. The scope of information displayed for a given location in the warehouse marked with a marker includes: name and designation of the location, name and designation of parts in a given location, number of parts.
The first application (SmartFactory107_3_Constant) after the marker identification allows the information to be displayed immediately (Figure 2). The second application (SmartFactory107_3_Close) forces the user to click on the object (a cube), which is displayed in a moment of the marker recognition to display the information. In order to hide the information panel, the user has to use the ''close'' button in the application window ( Figure 3).
The Vuforia plug-in was applied to execute the application. It was used intending to recognize markers previously entered into the database.

C. AR TARGETS
Two types of markers were selected for the purpose of the research. The markers were in the form of a graphic code. The first type was based on a two-color image in the form of light graphic objects with dark outlines. The second type of a marker was based on graphic elements of various shades from white to black (Figure 4).  The selected types of markers were verified to assess the correctness of assigning data to them. The verification process was carried out while entering the Vuforia database on the website. According to the system of characteristic points assessment, the tested markers were rated 4/5 and 5/5. Figure 5 shows the points recognized on the marker, each of the pluses marked in yellow is one characteristic point recognized by the algorithm.

D. RESEARCH IMPLEMENTATION
The research was carried out on three participants. Each of them identified the markers placed on the storage rack in a strictly defined order in three repetitions. On three shelves of the rack there were 4 markers identifying the location and indicating through the application which parts were there. (Figure 6). On the basis of preliminary experiments, the minimum distances from the rack from which markers of a particular size should be read to obtain the best reading for the particular AR glasses were established. The distance for a small marker was 30 cm and 60 cm for a large one. In order to ensure repeatable research conditions, constant lighting conditions with a value of 430 lx of artificial lighting intensity were provided. The required marker reading time was also assumed. The reading of the marker had to be done within 20 seconds. Exceeding this time was tantamount to no marker reading, and the participant moved to the next marker.

E. RESEARCH RESULTS
The evaluation criteria for the simulation tests were the effectiveness and time of reading the markers which were placed on the shelves of the storage rack. The results are presented in Table No. 2.
The MT300 glasses obtained an average of 43% accuracy in reading markers taking into account two types of software. Also, the highest average reading time of markers was obtained for this device. The obtained result, regardless of the type of software used, is not much lower than the determined upper limit of the reading time (20s). The BT-300 glasses obtained much lower average reading time results. There were also no significant differences in terms of the type of software. VOLUME 9, 2021 On this basis, it was decided to carefully analyze only the results obtained for the BT-300 device. Due to the fact that there was a significant number of unread markers in the case of MT 300 glasses, they were omitted in further analyses.
An in-depth analysis of the main effects results for the time variable in terms of marker type, reading angle, and marker size indicates that:  The analysis of the graph shows that there is no interaction for these combinations (Figure 9).

V. IN THE CASE OF THE SOFTWARE NO. 2 (FIGURE 10)
-Angle * Marker Size: the shortest reading time was obtained when a large marker was placed on the subject's line of sight, a little better result was noticed for a small marker placed at the same level, -Marker Type * Marker Size: the shortest time was obtained for a large marker No. 2, and the longest time for a small marker of the same type; in the case of Marker Type No. 1, the shortest time was obtained for a large marker and a little longer for a small marker, -Marker Type * Angle: for the Marker Type No. 2 located on the subject's line of sight, the obtained time was the shortest, when changing the marker type, the time was longer. Based on the Pareto chart, it can be concluded that in the case of the software No.1, the Marker Type factor is a statistically significant factor. It means that its impact strength at the ''−'' and ''+'' levels differs significantly ( Figure 11a); however, for the software No. 2, it can be concluded that the factor B Angle is a statistically significant factor. It means that its impact strength at the −i + level differs significantly (Figure 11. B). The charts below show the results of average marker reading times depending on the variants of the AR system elements (examined factors). In the case of the software No. 1, the graph below shows how significant impact the Marker Type has on reading time. The average times on the left side of the cube are no more than 2s, while those on the right are no less than 6s. Larger differences of the average times can be read for Marker Size. The results for the small marker were obtained in the range of 1s -8.5s, and for the large one in the range of 1s -7s. It confirms that the Marker Type factor is statistically significant and the Marker Size factor has a random impact ( Figure 12).
For the software No. 2, the results show the significance of the Angle factor. The obtained reading times of the markers on the line of sight do not exceed 6.5 [s], while the average time for the markers located below the line of sight is not less than 15 [s] (Fig 13). It confirms that the Angle factor is statistically significant, and the Marker Type and Marker Size have a random impact.
The analysis of the obtained reading efficiency results and correlation between the examined factors (elements of the AR system), taking into account the specific conditions related to the implementation of the material picking process, became the basis for the selection of elements of the AR system. The elements selected for further research are: BT300 device, software: No. 1, marker type: No. 2, marker size: large, Thanks to the elements mentioned above it is possible to read the largest number of markers and obtain the shortest output time.

VI. RESEARCH ON MATERIAL PICKING PROCESS
A. DATA GATHERING Selected hardware and infrastructure elements were tested in the process of materials picking. The research concerned the comparison of picking processes with the use of AR solutions and those carried out without such support. The evaluation parameters were: the correctness of completing types of parts, their number and picking time. The subject of the research was a storage rack, on the shelves of which containers with parts were placed. The given part was located in only one location. The task of the picker was to take the required number of a given part on the basis of the picking list and put it into the collecting container. In the normal process, the survey participant received a paper-based picking list and identified the parts on this basis (Figure 14a). In the process carried out with the use of AR solutions, the picking list was displayed on the glasses screen. The application made for the needs of the research also allowed to indicate, on the basis of the marker readout, which part is in a given location of the storage rack. After completing a given part, the user went to the next position on the picking list. The study also maintained the same environmental conditions, distances, devices and the method of recording the results as in the case of study no. 1 (Figure 14b). The research was conducted on a group of 20 people (10 for the first and 10 for the second). The group consisted of people aged 25-30, in which there were 10 men and women. Each of the survey participants carried out 20 of the same picking orders. These orders varied in part types (from 1 to 16 part types) and the number of items to be picked (up to 1 to 55 parts). In order to eliminate the factor of remembering the location of parts on the shelf, each order was followed by a change of their location on the shelf.
The evaluation of the completion of the completion orders consisted in determining: -correctness of part types, -the correctness of the number of parts, -picking time.

B. RESULTS AND ANALYSIS
The results of the average times indicate that for most of the picking orders, participants using the AR solution completed them faster ( Figure 15). As a result of the execution of orders, there were no errors related to the picking of incorrect parts and their number of pieces. The analysis of the results shows that the application of the AR solution resulted in a smaller scatter of the obtained picking times ( Table 3).
The picking orders included a different number of types of parts and their number of pieces. The analysis of the results shows that shorter average picking times were obtained with the use of AR for orders with a greater number of parts types ( Figure 16).   The use of the AR solution also shows greater efficiency with an increase in the number of parts to be picked ( Figure 17). This may indicate that AR solutions contribute to a better perception and faster identification of construction and task details necessary to be implemented in real conditions.
The presented results indicate that the implementation of the material picking process using AR solutions is effective. The research results also show the advantage of the AR solution with an increase in the number of types of parts and their number of pieces. The use of AR techniques contributes to more effective identification of parts, which directly contributed to shortening the picking time. Based on the results, it can also be concluded that the use of AR solutions has a positive effect on the stability and repeatability of the completion process.

VII. CONCLUSION AND FUTURE WORK
The progressive customization causes that the variants of the components of finished products are increasing. For this reason, effective production logistics processes are an increasing challenge in the operation of production systems. The efficiency of the production processes depends on the effectiveness of these processes. The presented solution based on the augmented reality solution shows its great potential in the implementation of materials picking processes. The test results show that with the increase in the number of types of parts and the number of pieces, it is possible to shorten the completion time of picking orders. The results also indicate the stability and repeatability of the processes carried out using AR. It is worth noting that the research was carried out with the use of standard AR equipment, which indicates the high economic potential of these solutions. The tests were carried out in laboratory conditions and must be repeated for real logistic processes. However, conducting these tests requires corrections in the identification and communication software. They will also require the selection of appropriate technical infrastructure enabling application in industrial conditions. ACKNOWLEDGMENT KRZYSZTOF ŻYWICKI received the master's degree in mechanical engineering, in 2000, with a focus on production systems and the Ph.D. degree, in 2009. He is currently an Assistant Professor with the Faculty of Mechanical Engineering, Poznań University of Technology. He is the author of many implementations in the area of production planning and control and also using modern IT solutions. His current research interests include wide scope of the production management and improvement, the concept of Industry 4.0, and in particular the idea of digital twins.
PAWEŁ BUŃ received the master's degree in machine diagnostics and measurement systems from the Faculty of Mechanical Engineering, Poznań University of Technology, in 2013, where he is currently pursuing the Ph.D. degree in mechanical engineering. Since 2015, he has been a Research Assistant with the Poznań University of Technology. His research interests from the beginning of his work with the Poznań University of Technology focused on the use of virtual and augmented reality systems, position tracking, visualization, interaction methods and its impact on the immersion, and knowledge transfer.