Augmented Intelligence: Surveys of Literature and Expert Opinion to Understand Relations Between Human Intelligence and Artificial Intelligence

Augmented intelligence (AuI) integrates human intelligence (HI) and artificial intelligence (AI) to harness their strengths and mitigate their weaknesses. The combination of HI and AI has seen to improve both human and machine capabilities, and achieve a better performance compared to separate HI and AI approaches. In this paper, we present a survey of literature to understand how AuI has been applied in the literature, including the roles of HI and AI, AI approaches, features, and applications. Due to the limited literature related to this topic, we also present a survey of expert opinion to answer four main questions to understand the experts’ implications of AuI, including: a) the definition of AuI and the significance of HI in AuI; b) the roles of HI in AuI; c) the current and future applications of AuI in research, industry, and public, as well as the advantages and shortcomings of AuI; and d) end users’ view of the application of AuI. We also present recommendations to improve AuI, and provide a comparison between the findings from the surveys of both literature and expert opinion. The discussion of this paper shows the promising potential of AuI compared to separate HI and AI approaches.

to improve and amplify, rather than to replace, the human cognitive power [3]- [5], which is to make a single human or a group of humans [1] stronger in simple words [6]. Hence, while AI is a machine-centred intelligence system, AuI is a human-centred intelligent system that allows humans to amplify existing or create new skills and capabilities, allowing them to even exceed their potentials while offering new experiences [5], [7]- [9].

A. AUGMENTED INTELLIGENCE FOR ADDRESSING THE SHORTCOMINGS OF ARTIFICIAL INTELLIGENCE
Traditionally, the goal of AI is to create advanced computers and machines that possess intelligence equal to humankind's [10], such as passing the Turing test [11]. Various AI approaches, particular machine learning approaches (e.g., support vector machine and random forest) have been applied to perform various tasks that require HI (e.g., classification and regression) in various applications (e.g., data mining and traffic signal control) and sectors (e.g., transportation, manufacturing, and healthcare). Ever since Google's DeepMind AlphaGo used deep learning [12], [13] to defeat Lee Sedol, who is one of the best players at strategy board game Go in 2016 [14], there has been new and refreshed enthusiasm in the world of AI to turn non-smart systems smart, and the smart ones smarter, and even achieving a better-than-human intelligence. With the advent of advanced enabling technologies, such as big data, Internet of things, and cloud computing, AI is expected to play major roles in next-generation technologies, such as driverless vehicles and smart everything.
Nevertheless, AI possesses shortcomings. For instance, it is naturally black-box that lacks transparency. So, how exactly AI makes decisions and interpretations is unclear, and so it is challenging to identify and quantify the errors of the results, as well as to troubleshoot. The root cause of the errors could be insufficient training data, inappropriate configuration, and so on. Imagine the life or death decisions made in medical diagnoses, the non-transparent decisions made by AI is largely unacceptable. The integration of HI and AI also means that the traditional shortcomings of AI are inherited to AuI. Examples are the complexity of the AI architecture, the need of a dedicated hardware to run AI, and the need to increase the energy efficiency and speed of AI algorithms [15]. Hence, the effects of the shortcomings of AI to AuI warrant a thorough investigation.

B. CONTRIBUTIONS OF THIS PAPER
The contributions of this paper is to understand the AuI implementation by answering two research questions: a) how has AuI been applied in the literature?; and b) what kinds of implications does AuI make based on experts' opinion? To answer question (a), we conducted a secondary research to gather examples of how AuI has been applied in the literature. The procedure was to gather information from published papers related to this topic in major indices, including Web of Science (www.webofknowledge.com) and Scopus (www.scopus.com), and then to obtain these papers from digital libraries, such as IEEE Xplore (ieeexplore.ieee.org) and ScienceDirect (www.sciencedirect.com). To answer question (b), we conducted a primary research, particularly an expert interview (or expert survey), to gather experts' opinion on the implications of the application of AuI. The procedure was to request experts to answer a questionnaire. Comparison of findings between the literature review (secondary research) and the expert survey (primary research) were made to identify recommendations for improvement.

C. ORGANIZATION OF THIS PAPER
The rest of this paper is organized as follows. Section II presents background. Section III conducts the secondary research (or the literature review) that presents the application of AuI to a diverse range of applications, such as braincomputer interfaces, speech recognition systems, and so on, in the literature. Section IV presents the development and deployment of AuI. Section V conducts the primary research (or an expert survey) that presents experts' opinion on the AuI applications. Section VI presents a comparison of findings between the literature survey and our expert survey studies. Section VII presents recommendations to improve AuI. Section VIII concludes this paper.

II. BACKGROUND AND MOTIVATING THE NEED FOR AuI
This section presents background, including a comparison between HI and AI, and the implementation approaches for AuI, including the roles of HI and AI in AuI, the AI approaches in AuI, and the AuI features.

A. COMPARISON OF HUMAN INTELLIGENCE AND ARTIFICIAL INTELLIGENCE
Both HI and AI have different strengths and weaknesses, in terms of the capabilities of learning knowledge, making decisions, and performing tasks, as summarized in Table 1 and explained in the rest of this section. Understanding the strengths of both types of intelligence helps to capitalize them in AuI, and understanding the weaknesses helps to address them in AuI.

1) LEARNING KNOWLEDGE
HI possesses a higher capability to learn knowledge. Firstly, HI achieves a higher capability to learn under high-noise environment, whereby a stenographer has shown to achieve a higher rate of accuracy compared to a dictation software when a speech is either mixed with accent, rife with semantic, or made under noisy environment [6]. Secondly, HI achieves a higher capability to learn based on unquantifiable information and situations for making intuitive decisions [16]. Thirdly, HI achieves a higher capability to learn based on arbitrary information and situations, where training data and experience may be lacking. Meanwhile, the quality of AI is largely dependent on the quality of the training data set, which must be properly labelled, complete, unbiased, and not overlap with test data set [9]. In addition, gathering personal VOLUME 9, 2021 information (e.g., medical information) can be challenging, and so there could be insufficient and incomplete data sets. In short, HI can perform both well-defined and non-welldefined tasks flexibly, while AI can perform well-defined tasks only [3]. For instance, the DL approach trained to win the Go game cannot be easily adapted and deployed in driverless vehicles. Fourthly, HI possesses a higher capability to learn soft skills, including artistic, creativity, flexibility, imaginative, intuition, and sensibility. The soft skills are human's right-brain capabilities important for creating innovations and designs, and handling uncertain and unforeseen situations [5], [6].

2) MAKING DECISIONS
Both HI and AI have difficulty in making unbiased decisions. While biases in HI can be affected by poor behaviours and habits, such as entrenchment, short-termism, stereotypes, and favouritism [2], biases in AI can be caused by the influence of biased data and its restrictions [17]- [20].

3) PERFORMING TASKS
Both HI and AI have shown to achieve a low capability to recognize a large number of face images [6]. HI possesses a higher capability to recognize: a) patterns because HI can understand relational inferences in patterns [15] although AI can visualize multi-dimensional data [1]; and b) emotions. On the other hand, AI possesses a higher capability in: a) memorizing; b) computing, which processes a massive amount of data generated by a wide range of entities, as well as probabilistic and analytic tasks, in the big data era [2], [21]; and c) works performed in a consistent and efficient manner despite under long hours and high speed. Both memorizing and computing skills are human's left brain capabilities. In other words, AI possesses a higher human's left-brain capabilities, and HI possesses a higher human's right-brain capabilities.
The shortcomings and the unfavourable characteristics of AI (see Table 1) lead to the need for ''human touch'' [9], and hence motivating the need for AuI.

B. IMPLEMENTATION APPROACHES FOR AUGMENTED INTELLIGENCE
This section presents various approaches applied to implement AuI, including the roles of HI and AI in AuI, the AI approaches in AuI, the approach for assessing the confidence of HI and AI decisions, the approach for modelling human perceptions, and the approach for adjusting human and machine participation based on workload. Figure 1 presents the AuI attributes.

1) ROLES OF HUMAN INTELLIGENCE AND ARTIFICIAL INTELLIGENCE IN AUGMENTED INTELLIGENCE
HI and AI play three main roles in AuI, namely: a) the HI-AI approach enables HI to advise AI; b) the AI-HI approach enables AI to advise HI; and c) the combined HI and AI decisions approach uses both HI and AI simultaneously. The first and second approaches are a cascading decision-making model, and the third approach is a parallel decision-making model.
Firstly, in the HI-AI approach, HI serves as the starting point to provide inputs to the AI decision-making engine. This helps to increase the context awareness and correctness of a system as shown in Figure 2. Without explicitly programmed, AI uses real-time private information provided by human and sensors to generate local decisions. As HI can be inconsistent and affected by bias [22], human interpretability of inference criteria can be used to exclude biases, such as racism [16]. There are three main types of human inputs. Firstly, the changes of brain activities measured based on electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) signals while performing cognitive tasks with different difficulty levels. However, this technique requires a human to be tethered by wires [23], which is inconvenient. Secondly, measurements received from body sensors, such as Microsoft Band that provides heart rate and skin temperature [Ref]. In addition, HI provides labelled data and features, as well as human feedback, for training AI [10]. Thirdly, crowdsourcing that enables common individuals to provide information [24]. The HI-AI approach has been adopted in [25]- [28].
Secondly, in the AI-HI approach, AI serves as the starting point to generate knowledge and provide inputs or guidance to human as the decision-making engine. This helps to improve the correctness of the decisions made by human as shown in Figure 3. This enables human to learn and exchange implicit knowledge among themselves without social interaction and with reduced effort in complex scenarios. This approach regards human as more reliable. The final decision  can be fed to the AI for further enhancement [9]. As examples, AI is used to advise physicians during surgical operations. The AI-HI approach has been adopted in [2].
Thirdly, in the combined HI and AI decisions, both HI and AI inputs are taken into consideration simultaneously for making the final decisions. This approach has been adopted in [16], [22], [29], as seen in Figure 6 (see Section II-B3).
The above different approaches can be used to design different AuI approaches applied to different purposes. As an example of debugging in software engineering, the HI-AI approach receives guidance from HI to help AI in fixing errors, while the AI-HI approach receives guidance from AI to help HI in fixing errors [30]. In addition, these approaches can form a feedback loop so that the final outputs of the approaches can be verified either by human or machine, and then used in helping to improve AI as time goes by. Further research can be pursued to investigate coordination between HI and AI that run separately, so that both human-related and AI-related subtasks, which have different resource requirements and constraints, are well handled.

2) ARTIFICIAL INTELLIGENCE APPROACHES IN AUGMENTED INTELLIGENCE
There are two popular AI approaches, namely deep neural network (DNN) [21] and convolutional neural network (CNN) [15], applied to AuI in the literature. DNN, which has been applied in [21], is a feedforward network that has three main types of layers. Firstly, an input layer receives inputs (or data sets) and passes them to hidden layers. Secondly, multiple hidden layers, which are located between the input and output layers, learn through repeated iterations of data sets. Each neuron uses a weighted function and an activation function to perform nonlinear transformation. Thirdly, an output layer generates outputs. Figure 4 shows an example of the DNN architecture.
CNN is an excellent tool for computer vision tasks [12], from identifying human eyes or facial gestures to recognizing and classifying heart diseases [15], [35]. CNN has four main operations as shown in Figure 5. Firstly, convolution extracts features from a pixelized input image using filters (also known as kernels or feature detectors) to produce different feature maps (also known as activation maps or convolute features). Secondly, rectified linear units (ReLUs) perform element-wise and non-linear operations on the feature maps. Thirdly, pooling or sub-sampling reduces the spatial size of the feature maps by extracting dominant features via either max pooling or average pooling. The preceding steps are conveniently called feature learning, and they may be repeated to produce multiple dominant features. Fourthly, classification, which is implemented using a fully connected neural network, is the final operation that flattens the dominant features into a single vector of values used to classify the input image.

3) ASSESSING THE CONFIDENCE OF HI AND AI DECISIONS
The confidence scores of HI and AI decisions are estimated for the AI-HI approach to reduce human intervention in [22]. The confidence score represents the perceived probability of a decision being correct in the presence of uncertainty. In general, AI generates a decision accompanied with an AI confidence score. The AI decision is final if it has a high AI confidence score, otherwise it is passed to HI. Although HI is seen to be more reliable and can correct mistakes, the HI decision can be rejected if the HI confidence score is lower, whereby the AI decision is preferred in this case. By estimating the HI confidence score based on the AI decision, the final decision can be made without involving HI if the HI confidence score is lower, which helps to reduce human intervention.
The proposed approach is achieved using Figure 6. The database stores samples of real-life data and decisions, which may be distorted with different noise levels, made by a group of experts. Based on the samples from the database, HI maps between a sample and a HI confidence score to optimize a DNN with fully connected layers using gradient descent. In other words, the DNN approach estimates the confidence score of its decisions, whose distribution is modelled using the Bayesian probability. The output layer has a single neuron to regress the HI confidence score. AI uses the maximum softmax probability to estimate the AI confidence score. Subsequently, the HI and AI confidence scores are fed into an alignment algorithm to reduce the effect of misaligned confidence scores between HI and AI so that both scores reflect similar results. Finally, decision rule accepts either the AI or HI decision.
The proposed approach is used to recognize faces and license plate characters, respectively, and it has shown to reduce human intervention while providing more accurate decisions as compared to separate HI and AI approaches.

4) MODELLING HUMAN PERCEPTIONS
Human actions are affected by individual perceptions of the underlying concept that cannot be tuned or controlled like machine agents (e.g., sensor nodes). Based on the cumulative prospect theory [31]- [33], Prelec reweighting function [34], which is a probability reweighting function, is used to describe the boundedly rational human behaviour or perception under risk. Since human perception can be affected by noise, the Prelec reweighting function is investigated under the additive and Gaussian noise, and it has shown to not capture all patterns of beliefs that mimic the optimal behaviors of agents. Therefore, when the AI-HI approach is used, the local decision of AI should be generated under low noise environment, otherwise the HI-AI output is preferred.

5) ADJUSTING HUMAN AND MACHINE PARTICIPATION BASED ON WORKLOAD
The levels of human and machine participation in a human-machine collaborative task can be adjusted dynamically to ensure that the human's workload (or the cognitive resource) falls within the optimal range.
For instance, in [23], the level of autonomy (LOA) is calculated to adjust the autonomy of machines (i.e., UAVs), whereby a higher LOA value represents a greater participation from machines, and vice-versa. In particular, the LOA value increases with the busy level of the human operator so that the machines share more workload. In [23], a single human operator collaborates with a team of UAVs to search for multiple targets in an urban environment, and so the LOA value is calculated to adjust the participation from machines based on the busy level of the human operator. The human operator controls the UAVs to search for the targets when the LOA value is low, and the UAVs search for the targets autonomously when the LOA value is high. The LOA value is based on the: a) mission complexity that includes the types  of tasks, the number of UAVs, and the number of targets; b) environment complexity that includes weather and wind aspects affecting the mobility of UAVs, and the surrounding complexity; and c) human interaction that includes the operation frequency, which increases when the human operator becomes busier.

III. APPLICATION OF AUGMENTED INTELLIGENCE
This section presents the application of AuI to a diverse range of applications. Examples of applications include brain-computer interfaces [15], speech recognition systems [21], social edge intelligence [15], telerobotic [36], games [37], public e-service [38], and so on.

A. BRAIN-COMPUTER INTERFACES
Brain-computer interfaces enable users operate external devices by thinking, such as controlling robots and devices in smart homes and Internet of things. Traditionally, an EEG cap is used, whereby many electrodes are attached to a user's head, which can be inconvenient, to detect brainwave signals and identify imaginary motor commands. In addition, the use of brainwave signals has low classification accuracy [15].
In [15], biosensors are placed on unobtrusive locations on a user's face to measure physiological signals, such as EEG, EOG, and EMG signals, in order to detect various eye movements and facial expressions. This allows users to operate external devices using their own eyes and facial gestures, rather than brainwave signals only. The HI-AI model incorporated with CNN receives six types of physiological signals. Each physiological signal is collected within a short period of time using a sliding window, transformed into a power spectrum, and then fed into the CNN through the input layer. The CNN generates commands representing the human intentions. The proposed AuI approach helps to generate commands based on a large, diverse set of physiological signals.

B. SPEECH RECOGNITION SYSTEMS
Speech recognition systems enable speech prediction. Traditionally, speech recognition systems use acoustic features without taking into consideration the users' context, such as users' sentiment and surrounding environment, which reduces the recognition accuracy and the human touch in responses (or being robotic).
In [21], the HI-AI model incorporated with CNN receives speech, as well as sentiment and environmental information, to improve the recognition accuracy. Specifically, the model receives three main types of inputs: a) the received speech; b) the sentiment value of a sentence, which is generated by a sentiment analyser, that represents feelings, particularly how positive, neutral, or negative the sentiment is; and c) the environment value generated by sensors (i.e., motion and light sensors) in a wireless sensor network. The proposed AuI approach has shown to improve the recognition accuracy compared to the traditional system that uses speech only. The recognition accuracy has shown to increase with the number of hidden layers (i.e., from one to five).

C. SOCIAL EDGE INTELLIGENCE
Social edge intelligence collects human inputs through crowdsourcing, explores the wisdom and collaboration of common individuals -without involvement from third parties VOLUME 9, 2021 and regulatory bodies [24] -and then uses AI to accomplish tasks. As an example, social edge intelligence uses eyewitnesses' on-site edge devices to collect images of damaged scenes through crowdsourcing, and then uses AI to assess the damage level caused by a disaster (e.g., earthquake, forest fire, and flood), which helps to guide rescue teams and UAVs to areas of interest. As another example, social edge intelligence uses eyewitnesses' on-site edge devices to collect images of traffic accidents through crowdsourcing, and then uses AI to identify locations with high rate of accidents.
HI is essential in social edge intelligence to improve accuracy, particularly in image classification (e.g., identifying fake or irrelevant scenes in images). Specifically, HI addresses four main challenges that can cause misclassifications: a) the close-up angle of an image can be misclassified as severe (e.g., damage from a disaster); b) noise (e.g., dust) captured in images, or those with low resolution; c) unvetted images that may be fake or irrelevant; and d) images with low-level features that do not capture the high-level context of the scenes (or the story behind the scenes).
In general, social edge intelligence is a HI-AI model with three main components. Firstly, a crowdsourcing platform collects human inputs, which are based on HI. It is important to encourage humans with different privacy requirements to provide responsive and high-quality inputs in crowdsourcing. So, the privacy of the human contributors must be ensured, and incentive must be given to human contributors who report their private information (e.g., location). Secondly, AI for mobile applications (e.g., mobile sensing, smart homes, and intelligent transportation systems). Thirdly, edge devices (e.g., IoT devices, smartphones, and UAVs) with sufficient computational and memory capabilities, as well as the energy level, to accomplish tasks. AuI approaches have shown to improve the overall system performance (e.g., a reduced response delay incurred by crowdsourcing and an increased response accuracy [15]).

D. INDUSTRY AND MANUFACTURING SYSTEMS
Efficient industry and manufacturing systems strive to produce high-quality products at the lowest possible cost, which is essential to improve competitiveness. This section presents the use of AuI in new product designs, packaging, telerobotic systems, ramp-up, fault detection, and training human employees.

1) NEW PRODUCT DESIGN
In [28], AuI is used to design new products using: a) HI to introduce creativity and fuzzy reasoning without the need for a complete model of the complex engineering processes; and b) AI to provide accurate computation and run repeated and procedural tasks with reduced time and cost. The HI-AI model has three main steps: a) case-based reasoning (CBR) uses AI to find similar designs in database in order to provide initial population (or potential solutions) for the next step; b) genetic algorithm (GA) is an AI approach that performs crossover, mutation, and removal to discover the ideal design using the population generated by CBR; and c) HI monitors the evolution in GA, and modifies variables, constraints, the objective function, and search strategies dynamically. The proposed AuI approach helps human designers to incorporate their knowledge, experience, and inspiration into designs.

2) PACKAGING
In [26], AuI is used in 2D packaging to maximize the number of objects with irregular shapes and sizes in a rectangular 2D container, in which the objects must not overlap. Minimizing empty spaces in a container improves the overall space efficiency, which reduces the overall shipping and delivery costs. Traditionally, although HI has been the main way to achieve the optimal space efficiency, it is time consuming.
In [26], the AI-HI model has two main steps: a) AI-based greedy algorithms, including biggest place first, biggest length first, smallest place first, and smallest length first, arrange objects in a container; and b) HI refines the outcome of the greedy algorithms. The proposed AuI approach has shown to increase space efficiency.

3) TELEROBOTIC SYSTEMS
Telerobotic system combines a human's and a machine's commands to generate a single joint command to control robots (e.g., a robotic arm), which is useful in manufacturing plants and hazardous environment. The joint command must not exceed the maximum operating parameters of the system, such as the maximum velocity and the maximum acceleration. The joint command moves the robotic arm, and the movement is observed by both human and machine. Both human and machine adjust their commands based on the observed movements.
In [36], human uses a Spaceball to generate a kinetic-based command based on the movement generated in the previous time instant to control the position of a robotic arm, which is a challenging task for a machine. Meanwhile, a machine equipped with sensors controls the force applied by the robotic arm, which is a challenging task for a human. Hence, both position and force are controlled simultaneously, so that the robotic arm can avoid unexpected obstacles and ensure a smooth movement. The system output (i.e., the observed movement) is measured to provide a reference so that both human and machine can adjust their actions based on the outcome. The proposed AuI approach has shown to avoid unexpected obstacles, achieve accurate position tracking, and ensure a smooth movement, while reducing human's involvement and skill requirements.

4) RAMP-UP
Ramp-up is a manufacturing phase that brings up the production system to its required operational performance when it is used for the first time or during reramp-ups following breakdowns or changeovers. The ramp-up process is unpredictable and requires a long time, and so it can become a bottleneck.
Traditionally, during a ramp-up, human operators adjust and improve system performance manually in a trial and error manner. Although human operators have prior knowledge of the relevant (or potentially optimal) actions, they are normally unclear about optimal actions for different kinds of states.
In [39], the HI-AI model uses reinforcement learning (RL), which is an AI approach. The RL approach is enhanced through receiving human inputs and exploring a set of candidate actions following a human operator's preference and contraints. In general, RL performs exploration to select different kinds of actions under different kinds of states (or operating environments) to improve its knowledge as time goes by. Traditionally, exploration selects random actions to understand the suitability of most candidate actions under a particular state. This explains the increased ramp-up time, particularly when the number of possible states is large. Hence, by identifying a set of appropriate candidate actions, the enhanced approach helps to reduce the time required to explore and identify optimal actions under different states, while avoiding inappropriate actions that can cause issues. At any time instant, RL enables an agent (i.e., the production system) to observe states (i.e., the human operator's observations and the machine status, such as the weight of the products and the duration of the filling process), learn knowledge, and then select the right actions (i.e., applying the right level of pressure, and aligning the gripper) to increase its accumulated reward (i.e., metrics related to the quality of the products and the efficiency of the process) as time goes by. The proposed AuI approach has shown to increase the exploration efficiency, which reduces the ramp-up time.

5) FAULT DETECTION
Fault detection detects and diagnoses (or identifies the root cause of) faults, as well as performs aversion, in large chemical and manufacturing plants in a real-time manner. The outcomes of the fault detection system notify human operators to take the right corrective measures in order to ensure that operations are back to normal again. Traditionally, it has been challenging for human operators to assess a massive amount of data generated by a large number of sensors and instruments in the plants with the least possible delay. Meanwhile, although AI can process the massive amount of data, it has three main shortcomings because it: a) cannot detect novel faults not found in the training data set; b) cannot detect faults accurately since there are a significant lower number of faulty operations (or labels) compared to normal operations in the training data set; and c) cannot diagnose faults. However, reducing the delay incurred in detecting faults is important to reduce loss.
In [25], the AI-HI model is incorporated with two AI approaches to detect and diagnose faults with increased accuracy. Firstly, random forest, which is a supervised learning approach, uses a massive amount of sensor and instrument data to predict the probability of the type of faulty operations accurately. Secondly, principal component analysis (PCA) is a semi-supervised learning approach that: a) uses the massive amount of sensor and instrument data to isolate novel faults; and b) identifies the variables contributing to the faults in diagnosis. The advantage of PCA lies in the training data set, which requires normal operations only to detect faults, whereby operations that deviate from normal operations are isolated. Subsequently, HI (i.e., human operators) reviews the outcomes of both supervised and semi-supervised learning approaches, and refines them to provide final outcomes of faults and their root causes, which are then used to retrain AI. The proposed AuI approach has shown to increase the accuracy of fault detection and diagnosis.

6) TRAINING HUMAN EMPLOYEES
Training is generally aimed at transferring knowledge from experienced employees to inexperienced employees. One main challenge is that not all knowledge can be transmitted in oral and written form, particularly experiential and implicit knowledge in handling non-nominal cases and errors. Such knowledge is generally not documented in manuals and is difficult to be described and taught. Hence, designing a complete AI-based automated system for training is both challenging and complex as different behaviours and personalities must be modelled.
In [16], the HI-AI model incorporated with RL has two main stages. Firstly, the proposed training system with RL undergoes the training phase that learns knowledge through two ways: a) observes and captures the experienced employees' actual process execution; and b) uses rules and optimization. Implicit knowledge is learned through imitation, whereby the experienced employees' behaviours are matched with some known policies (or sequences of actions). Secondly, during the training session of the inexperienced employees, the proposed training system provides feedback (e.g., the quality and speed of the execution process, and so on) to inexperienced employees based on the learned knowledge. The proposed AuI approach helps to facilitate the training process of inexperienced employees.

E. HEALTHCARE
This section presents the use of AuI in disease detection and drug dosage estimation.

1) DISEASE DETECTION
In [29], AuI is used to detect colorectal polyps in colon cancer screening. Colorectal polyps are abnormal growths of cancerous tissues that appear inside the colon. Traditionally, highly trained medical professionals are involved to distinguish between true and false colorectal polyps in computed tomographic (CT) images. However, this incurs high medical cost and increases the need for medical professionals.
In [29], the combined HI and AI model detects colorectal polyps in CT images. While AI uses computer-aided detection software, HI uses human vision. Subsequently, α-integration combines both HI and AI decisions [40]. The α-integration approach blends multiple positive measures, such as the probability distributions of stochastic models, in an optimal manner by minimizing α-divergence. Two main parameters in α-divergence are: a) the α parameter determines the characteristics of the integration; and b) the weight parameter determines the level of importance of each positive measure. Gradient descent is used to identify the optimal α and weight parameters. The proposed AuI approach has shown to increase the detection of colorectal polyps compared to separate HI and AI approaches.

2) DRUG DOSAGE ESTIMATION
Scientists and clinicians have been facing difficulties to ascertain the appropriate dosage of medication that patients are likely react to treatment. Traditionally, linear parametric models have been commonly used for estimating dosageresponse relationships; however, there are limitations, such as the lack of concern to the effects and complexity of a patient's physiology [41].
In [2], the HI-AI model determines and infuses the optimal quantity of intravenous (IV) fluid and antibiotics to treat sepsis, which is one of the leading causes of death for ICU patients. An inappropriate quantity of the IV fluid can cause adverse effects. There are two main steps. Firstly, a physician (HI) determines and prescribes the IV dosage. Secondly, AI refines the dosage given by HI to optimize an objective function based on various factors, including the prescribed IV dosage, demographic, blood pressure, and so on, in order to maximize the chance of survival. The proposed AuI approach has shown to increase the survival rate by up to 22% among patients diagnosed with sepsis.

F. SELF-DRIVING CARS
With many people losing their lives to road accidents every year due to human factors (e.g., driver's distraction and misjudgement), there have been an increased interest in automated self-driving cars.
In [27], a driver advocate system based on the HI-AI model monitors drivers, learns their behaviours, and interacts with them to prevent accidents. There are two main steps. Firstly, based on HI, the physical parameters of the car, such as location, speed, acceleration, and position, are gathered. Secondly, the physical parameters are fed into AI so that it learns to adapt to the unique driving behaviour of each driver. The proposed AuI approach helps to smoothen the driving experience, such as to avoid calling out early warnings that can irritate drivers.

IV. DEVELOPMENT AND DEPLOYMENT OF AUGMENTED INTELLIGENCE
Various authorities and professional bodies have drafted policies to support the design, evaluation, and implementation of AuI in order to minimize unintended consequences.
As an example, in healthcare, the American Academy of Dermatology publishes a workflow for developing AuI applied to dermatology [4]. The workflow involves practicing physicians and has four main steps. Firstly, the data collection stage collects and labels a high-quality data set representing the target population. Secondly, the model development stage uses input data and known output data sets to train an AuI model, and then uses independent test data set to validate the model and evaluate its safety, accuracy, and effectiveness. Thirdly, the model deployment and monitoring stage deploys the model in the real-world setting, and monitors and validates it continuously. Fourthly, the new data stage uses new data to refine and improve the model in order to maintain a continued assurance of satisfaction. The workflow ensures thoughtful, high-quality, and clinically validated designs for AuI implementations in order to improve patient outcomes and provider satisfaction.
Apart from the policies, professional practices are essential, such as encouraging education for all stakeholders to understand the advantages and limitations of AuI, and exploring legal implications of AuI.

V. EXPERT INTERVIEW
The main objective of our expert interview is to understand the relations between HI and AI. Since the main focus is to generate new knowledge, rather than to quantify knowledge, we have chosen qualitative interviews. This section presents research questions, methodology, as well as interview results, summary, and discussion of results.

A. RESEARCH QUESTIONS
There are four main research questions as follows:

RQ.1 What is AuI and the significance of HI in AuI?
This

B. METHODOLOGY
We conduct expert interview, which is an integral qualitative approach for gathering new knowledge rather than quantifying existing knowledge. Similar approach has been applied in [42]. The questions of the interviews are mostly qualitative and open in nature, and so there is no predefined answer. The expert interview was conducted without gathering personal information like demographics, such as name, age, and gender, and it was about the subject matter, namely AuI. Hence, ethical approval was not required. The experts signed an informed consent form before interviews began. We explain sampling, interview questions, and the interview results analysis procedure, in the rest of this subsection.

1) SAMPLING
The interviews were conducted with 18 experts from the AI and AuI fields. The participating experts have at least three years of experience in AI and AuI, and they do not know each other. Experts were selected to achieve a balanced coverage of both academia and industry. When asked how experienced the experts thought they were with AuI, 75% of the experts regarded themselves as aware of it, 17% selected never heard of it, and 8% selected planned to use it in my future research/ products as shown in Figure 7. The high awareness of AuI, together with the lack of its application, show AuI is still at its infancy. The sample size is sufficient to provide statistically significant conclusions based on the immediate answers that come to the experts' minds when thinking about AuI [43], [44].

2) INTERVIEW QUESTIONS
We conducted the semi-structured (or open-ended) interview via video conferencing (or online interview) or voice recording (or offline interview) between April and July 2021. The interviews were recorded with consent. Each interview took around 30 minutes. We provided minimal instruction without preliminary briefing about the topic to the experts to minimize biases during the interview. Experts can talk very openly when answering the questions due to the open-ended nature of the interview. Based on our research questions RQ.1−RQ.4, we designed 14 interview questions segregated into 4 categories. Firstly, the understanding AuI category, which answers RQ.1, understands the background of the experts, particularly their experience and understanding with AuI. Secondly, the HI in AuI category, which answers RQ.2, understands the advantages of HI in AuI. Thirdly, the AuI approach category, which answers RQ.3, gathers information about the current and future applications of AuI. Fourthly, the end users' views category, which answers RQ.4, gains more information from the end users' perspective based on the experts' opinion. The questions are shown in Table 3.

3) INTERVIEW RESULTS ANALYSIS
The interview results are analyzed following four main steps as follows [42]: S.1 Data preparation transcribes the interview responses word-by-word from recorded interviews. S.2 Definition of coding categories identifies the main categories and their subcategories to generate a category system. The main categories and subcategories are based on the categories of our interview questions in response to the research questions RQ.1−RQ.4.
The category system was updated continuously to include new subcategories when analyzing the interview results. The resulting category system has 4 main categories (apart from identifying experts' experience level with AuI) and 11 subcategories as shown in Figure 8. S.3 Codification assigns the text fragments of the important transcribed interview responses to one or more subcategories in Figure 8. The category system was updated to include new subcategories when text fragments were unable to be categorized. In total, there are 1850 text fragments. In general, 22.3% of the text fragments are coded to the understanding AuI category, 23.7% to the HI in AuI category, 46.8% to the AuI approach category, and 7.2% to the end users' views category. S.4 Interpretation categorizes a list of text fragments based on categories and subcategories. Due to the large number of text fragments, paraphrasing categorizes the text fragments in each subcategories in order to reduce the data set size further. Table 4 presents some examples of paraphrasing. Subsequently, we analyze the paraphrased text fragments to gather the experts' opinion of a particular subcategory. We relate the experts' opinion to the relevant points in the literature presented in Sections II-IV.

C. INTERVIEW RESULTS, SUMMARY, AND DISCUSSION OF RESULTS
This section presents the interviewees' responses to the research questions RQ.1−RQ.4 (see Section V-A). The percentage of interviewees providing the same point is given in parentheses.

1) RESPONSES TO RQ.1: THE UNDERSTANDING AuI CATEGORY
This section gathers information from experts to understand the definition and significance of AuI.

a: DEFINITION OF AuI
The experts explained AuI, which was also called hybrid intelligence (5.6%), as shown in Table 5. The definitions are segregated into three groups: a) balanced-centric highlights the significance of both HI and AI; b) human-centric; and c) AI-centric.
The explanation is human-centric in nature despite the partnership between HI and AI. Overall, the experts believe that AuI does not replace HI (16.7%). Instead, AuI reinforces the role of HI through intelligence amplification or the enhancement of human capabilities, particularly the cognitive capability (38.9%). Just like how a computer increases its RAM to improve its computational speed, human can enhance capabilities using AuI. For example, AI performs preprocesses so that HI can make decisions with judgement at reduced time. Ultimately, AI is seen to provide recommendations, and HI makes final decisions (11.11%). Some experts opined that HI has been playing its role since AI was first coined. This is because HI has been guiding the design and application of AI. In recent years, the birth and growth of AuI is expected to increase the role of HI.
To a lesser extent, AuI is described as AI-centric to benefit AI, specifically to: a) enhance AI applications using human strengths, such as incorporating the human capability of imagination into a computer; b) remove negative human aspects, such as bias, fatigue, and distraction that can misinterpret data; and c) enable HI to provide feedback to AI for learning.
One of the experts pointed out the importance of both HI and AI, which has a balanced-centric. Specifically, ''. . . when HI is the least important in AuI, then we are very much pleased with AI. On the other hand, when HI is the most important in AuI, then we never achieve automation.''

b: SIGNIFICANCE OF AuI
During the interviews, the experts shared three main points explaining the significance of AuI. Firstly, human and machine learn differently. Human learns through experience and reinforcement. According to an expert, ''Although machines can learn through machine-based reinforcement, the human-like learning of reinforcement is significant.'' Secondly, we are living in an environment in which AuI can be naturally applied. One of the experts pointed out that ''. . . everything out there is the combination of system plus judgement. System is the set of environment (or tasks) created through mostly repetitive procedures, and once it is ready, HI can make judgement. For example, AI can gather data (e.g., X-ray and diagnosis data (11.1%)) and past reports, and perform analysis, then doctors use HI to make judgement. Human has been using HI in very little time (e.g., 1−2% of the time), and all other time is spent in getting the system ready, which can be performed using AI. Doctors can spend more time to attend to patients (11.1%).''   Thirdly, the combination of HI and AI has a great potential to achieve a higher performance compared to separate HI and AI. One of the experts shared that ''I am currently working on a medical project that provides the fatality of covid infection (or covid spread) in humans based on the lung condition. A medical specialist could detect 78% of the time, while AI-enabled software could detect 83% of the time. When both are combined, the accuracy rate has crossed 90% while it is still undergoing training.'' Another expert shared that, ''in a clinical study for detecting lymph node cancer cells, AI has a 7.5% error rate and human pathologists has a 3.5% error rate, and the combined inputs from AI and HI has a 0.5% error rate.''

2) RESPONSES TO RQ.2: THE HI IN AuI CATEGORY
This section gathers information from experts to understand the functions of HI in AuI. When asked how important is HI in AuI, most experts selected at least level 3 out of 5 levels as shown in Figure 9. Some experts mentioned that HI guides and trains AI (22.2%) and drives AI in the right way, and HI plays important role in programming advanced and complex AuI algorithms (11.1%). The experts have shown interest in AuI, and we received positive responses, such as ''I think for critical systems (e.g., healthcare), human interpretation is needed because it affects life'', ''I think AuI has a lot of potentials, yet we don't benefit from it fully as we haven't discovered its potential yet'', ''This field is still new and my VOLUME 9, 2021   research mainly focus on the effect of previous experience'', and ''I wish to learn more about AuI, and AuI is better than AI if it is matured''.
Interestingly, one of the experts selected level 1 representing the least importance. The expert highlighted the importance of the human aspects and the need to cater for different levels of human's intelligence through ''. . . augmented intelligence needs to be designed based on human factors so it can become easily usable for different levels of intelligence. For example, nowadays everybody can use a cell phone so technologies need to be adjusted for human intelligence.'' There are six main functions of HI in AuI as shown in Table 6. Of particular interest are the examples given by the experts highlighting the indispensable functions of HI in AuI in our daily life. In the field of healthcare, one expert shared that ''AuI has been widely used since medical professionals use AI to provide medical treatments, which is a great example to define and explain the AuI concept''. Another expert shared that ''Doctors can take account of human factors (e.g., the emotion of patients) and prescribe based on it.'' In the area of computing, HI has been used with AI. For instance, email users correct AI decisions when legitimate emails go into the spam folder.

3) RESPONSES TO RQ.3: THE AuI APPROACH CATEGORY
This section gathers information from experts to understand the current and future applications of AuI.
There are mixed feelings among the experts regarding how widely AuI has been adopted as shown in Figure 10. Some of the experts opined that automated systems require directions or inputs from humans to function properly (28%). This may explain why, in regards to AuI, most experts selected aware of it (70%) as shown in Figure 7, yet 10% and 20% selected levels 4 and 5, respectively. When asked how successful is the use of AuI, 50% of the experts selected at least Level 3, and the rest selected Level 2 as shown in Figure 11.

a: CURRENT APPLICATIONS OF AuI
We seek opinion from experts regarding the current applications of AuI as shown in Table 7  rejected by the physician. The feedback of the physician helps us identify false positives and false negatives, and improves the overall diagnostic ability of the application and trust of the patients on medical care''. As another example, ''. . . measuring brain activity with EEG. Now, there are some special augmented intelligent tech products, which you can use. For example, glasses and then these glasses enable you to have a 3D image of the brain activity, instead of a 2D version of the data which can be quite useful for research''. 70% of the experts believed AuI has at least the level 3 adoption, and they have expressed interest to use AuI in their research or products. Examples are as follows: Next, we seek opinion from experts regarding the future applications of AuI, which is also shown in Table 7. When asked how beneficial AuI can be in their future research or products, 86% selected at least level 3 out of 5 levels as shown in Figure 12.
Generally speaking, AuI can be used when AI is used, and when there is a need for increasing the cognitive capabilities (e.g., reducing bias) of a person. With data being the new oil, collected data can further enhance AuI in the future. The experts have mentioned a wide range of future applications of AuI, particularly data-driven applications, including those in the fields of education, entertainment (e.g., games), This section gathers information from experts to understand the end users' views about AuI. Specifically, we asked the experts to put themselves in the shoes of end users and explain how they feel about AuI. The experts shared four perspectives as shown in Table 8.
Two of the experts pointed out that end users would not feel the presence of AuI. One of the experts futher explained that ''I think end users don't get to see it yet, or they see it as 'automation', not AuI. The moment they will get to know, the first reaction would be of mistrust, they want to 'confirm' the results from the human expert.''

VI. COMPARISON BETWEEN LITERATURE REVIEW AND EXPERT SURVEY STUDIES
This section explains the similarities and differences between the findings from the surveys of both literature and expert opinion. We explain based on our research questions RQ.1−RQ.3 in Section V-A. Meanwhile, comparison is not The research question ''What is AuI and the significance of HI in AuI?'' (RQ.1) aims to understand AuI the definition and significance of AuI. The definition of AuI is human-centric based on both literature and expert surveys, although AuI advocates the partnership between humans and machines [1]. Therefore, in general, AuI is regarded as an extension of AI with HI, which helps existing AI to incorporate human capabilities (e.g., emotion) and values (e.g., ethical and societal values) to address the shortcomings of AI. Instead of replacing HI, both literature and expert surveys agree that AuI improve and amplify, rather than to replace, HI. Using AuI, AI uses its strength to address the weaknesses of HI, and vice-versa. While AuI is seen as a new paradigm in the literature, some experts opined that HI has been playing its role, particularly in training and guiding AI, explaining the indispensable role of HI since the inception of AI. This point indicates that AuI can be naturally applied in existing AI systems.
Based on both literature and expert surveys, AuI is significance and essential. This is because: a) human and machine learn differently; b) AuI can be naturally applied in the human environment; and c) the combination of HI and AI has a great potential to achieve a higher performance compared to separate HI and AI approaches. The research question ''What are the roles of HI in AuI?'' (RQ.2) aims to gain more understanding of how HI can benefit and improve AuI. Based on the literature, there are two main functions of HI in AuI, which are aimed at addressing the shortcomings of AI in learning knowledge (i.e., learning soft skills, knowledge under high-noise environment, as well as unquantifiable and arbitrary information and situations) and performing tasks (i.e., recognizing emotions).
The experts have given additional points about the roles of HI in AuI, which are more diversified as compared to those in the literature. Based on the expert survey, there are additional six main functions. Firstly, HI incorporates human qualities (i.e., business acumen, feelings, empathy, intuition, common sense, and human touch, as well as uses prior knowledge in new and unknown environments). While both HI and AI have difficulty in making unbiased decisions due to the influence and restriction caused by biased data in the literature [17]- [20], experts believe that HI plays a role to reduce it in an indirect manner, such as sourcing data for learning from a sufficient number of sources. Secondly, HI performs preprocessing, which highlights the roles of HI since AI was first coined, including cleaning, filtering, manipulating, analyzing, data sets, as well as reducing reliance on a large data set. The third and fourth points highlight the supportive role of HI. Thirdly, HI reduces the high computational power of AI. Fourthly, HI monitors AI. Fifthly, in decision making, HI: a) does not require a large data set; b) accepts or rejects AI decisions based on own judgements; c) improves AI decisions (e.g., increasing accuracy and reducing errors) which are not guaranteed to be optimal; and d) makes a timely decision. Sixthly, HI addresses AI shortcomings, particularly its capability to think independently and out-of-the-box.

C. COMPARISONS RELATED TO RQ.3
The research question ''What are the current and future applications of AuI in research, industry, and public? What are the advantages and shortcomings of AuI?'' (RQ.3) aims to understand the current and future applications of AuI.
Based on the literature, there are a wide range of proposed applications, from brain-computer interfaces to self-driving cars as listed in Figure 1 and explained in Table 2. Meanwhile, based on the expert survey, the current applications are listed and explained in Table 7. The experts extended the current applications of AuI in the literature in Table 2 with what they are aware of, namely biometrics, semantic analysis, social media, audio/ visual, transportation, computer applications, finance, human management, and politics. As for the future applications, the experts suggested applications as shown in Table 7, namely past experience regeneration, critical decision making, assistive technology, and scene and object identification. Based on both literature [22], [29] and expert interview, the combination of HI and AI achieves a higher performance compared to separate HI and AI approaches.

VII. RECOMMENDATIONS
The experts have mentioned six main recommendations to improve AuI as shown in Table 9. Further research can be VOLUME 9, 2021 pursued to investigate these recommendations for enhancing AuI, while taking the end users' views of AuI into consideration.

VIII. CONCLUSION
Augmented intelligence (AuI) combines human intelligence (HI) and artificial intelligence (AI) to reap the benefits of both intelligences while complementing each other's shortcomings. At present, AuI is in its nascent stage, and so more investigations can be conducted to understand its full potential and explore its real-world applications. Based on the literature, there are three main models in AuI: a) the HI-AI model enables HI to serve as inputs to train AI; b) the AI-HI model enables AI to serve as inputs to empower HI; and c) the combined HI and AI model enables HI and AI to make decisions based on HI and AI inputs simultaneously. Popular AI approaches in AuI include deep neural network and convolutional neural network. Examples of features of AuI include assessing the confidence of HI and AI decisions, modelling human perceptions, and adjusting human and machine participation based on workload. Due to the novelty of AuI, we conducted an expert interview to answer four research questions related to: a) understanding AuI; b) HI roles in AuI; c) AuI approaches; and d) end users' views. The expert interview confirms the discussion in the literature, particularly the promising potential of AuI compared to either HI or AI alone. The expert interview also extends the discussion of the literature with end users' views and recommendations for future research and investigation to stimulate further research in this emerging topic.