Contents 
Autism spectrum disorder is a developmental disorder characterized by significant social, communication, and behavioral challenges. Individuals diagnosed with autism, intellectual, and developmental disabilities (AUIDD) typically require long-term care and targeted treatment and teaching. Effective treatment of AUIDD relies on efficient and careful behavioral observations done by trained applied behavioral analysts (ABAs). However, this process overburdens ABAs by requiring the clinicians to collect and analyze data, identify the problem behaviors, conduct pattern analysis to categorize and predict categorical outcomes, hypothesize responsiveness to treatments, and detect the effects of treatment plans. Successful integration of digital technologies into clinical decision-making pipelines and the advancements in automated decision making using artificial intelligence (AI) algorithms highlights the importance of augmenting teaching and treatments using novel algorithms and high-fidelity sensors. In this article, we present an AI-augmented learning and applied behavior analytics (AI-ABA) platform to provide personalized treatment and learning plans to AUIDD individuals. By defining systematic experiments along with automated data collection and analysis, AI-ABA can promote self-regulative behavior using reinforcement-based augmented or virtual reality and other mobile platforms. Thus, AI-ABA could assist clinicians to focus on making precise data-driven decisions and increase the quality of individualized interventions for individuals with AUIDD.
Autism spectrum disorder (“autism”) is a developmental disorder characterized by significant social, communication, and behavioral challenges. According to the National Health Statistics Report [1], the prevalence of children ages 3–17 years diagnosed with a developmental disability has increased considerably from 6.99% in 2014–2016 to 17.8% in 2015–2018. Developmental disabilities can be severe, long-term disorders often including intellectual impairments, physical impairments, or both. Intellectual disabilities, defined by significant limitations in cognition and adaptive functioning, are some of the most common impairments diagnosed during the developmental years, while physical impairments are typically identifiable from birth. Often, these impairments co-occur and individuals diagnosed with AUIDD typically require long-term care and targeted treatment and teaching.
The physical and mental limitations presenting due to AUIDD affect different aspects of life, including personal (self-care, independent living) and social skills (keeping conversations, public speaking), often causing children to learn and develop slower than a typical child. While there is no cure for children with AUIDD, there are several types of treatments such as ABA, occupational therapy, speech therapy, physical therapy, and pharmacological therapy available. Each of these treatments have proven effective in helping individuals with AUIDD achieve a high level of skill development with earlier treatment leading to larger gains. Detecting and diagnosing these developmental disorders early could help apply the needed treatment procedures and facilitate many students’ learning and functioning to improve the behavior of children over time.
Effective treatment relies on efficient observation. However, the process of behavioral observation is time-consuming, requiring clinicians to collect and analyze data [2], use the collected data to identify the function of a problem behavior (or what the child is trying to communicate), use collected data to conduct pattern analysis to categorize problems and predict categorical outcomes, use assessment data to hypothesize responsiveness to treatment, design treatment plans using a hypothesis regarding patient responsiveness to treatment, and collect ongoing treatment data to detect the effects of the treatment plan. For example, behavioral assessments used in ABA, such as the functional behavioral assessment, are used to detect behavior patterns through indirect observations (screening instruments), direct observations, and experimental analysis to identify behavioral function. The identified function is then matched with a function-based behavioral intervention [3]. However, since clinicians rely on human collected data to make decisions, chances of unreliable decisions are likely high, particularly considering varied clinical training that may result in differences from one behavior analyst (BA) clinician to another.
Recent research illustrates the importance of personalized treatment plans for individuals with AUIDD. However, while the treatment and education plans may be individualized, they are often not precise or efficient. In addition, time spent by clinicians collecting and analyzing the data often detracts from providing empathetic treatment. Given the high impact of digital learning platforms, AI, and cloud computing, the purpose of this study is twofold. The first part describes how AI in combination with emerging technologies such as augmented reality (AR) or virtual reality (VR) in the form of a digital learning platform can benefit more autistic children and provide a personalized adoptive learning paradigm. The second part aims to develop conceptual understanding on how using the efficacy of AI in designing and applying an appropriate assessment framework for ABA and clinical interventions can assist clinicians and educators to effectively assess and monitor each child’s behavior and quickly modify interventions to meet his or her specific needs and to account for various differences across environments.
Augmenting Teaching and Precision Treatment
Digital platforms and technologies are argued by many to have a pivotal role in the dynamics of changing landscape of clinical treatment. The main arguments include improved support for treatment by 1) contextualizing and increasing motivation of students and promoting engagement through interactive environments and reward structures, 2) providing a learning experience that caters to pace of patient’s individual learning, and 3) providing continuous and lifelong learning through mobile learning platforms [4].
Several studies have shown promising results in improving the learning performance and boosting motivation to learn using graphical contents and interactions [5]. Teaching strategies and interventions that utilize digital games in mobile devices or tablets have also shown promise for incorporating behaviors management techniques into games. Restrictive and repetitive behaviors and interests (RRIBs) that might occur while playing games could be monitored and treated using embedded automated redirection to other games or levels to prevent interfering behaviors that prevent access to learning opportunities and help promote calmness [6]. However, considering the range of functioning for individuals with cognitive disabilities, further studies involving precision of treatment options to ensure individualization are needed as well as replication of these results across large cohorts of participants [7].
Role of AR and VR
Emerging technologies such as AR, including VR and mixed reality (MR), are at the forefront of recent technology-embedded practices that overlays reality and supplies additional layers to augment the perception of users [8] as well as enabling real-time interaction of real and virtual objects [9]. Recent interest in using AR and VR technologies to aid adults and children with autism spectrum disorder (ASD) provides additional sensory information such as eye-tracking as well as a virtual platform to continuously interact with people and environment around them in a controlled setting while collecting data for future analysis.
Safe and side-effect-free technologies are changing how AR/VR platforms are being found to be beneficial for improving soft skills, behaviors, and improving emotional skills [10]. However, the importance of personalized services to provide augmentation for individual learners has yet to be researched at large. Thus, the need for personalized adaptive learning paradigms is required to improve engagement, autonomy, and to promote individualized preferences for children with cognitive disorders. Data-driven algorithms could make use of these digital technologies to improve data collection while reducing the demands placed on behavioral analysts and educators and the time required to collect and label these behavioral data.
Artificially Intelligent Methods in Behavioral Health
Recent advancements in AI have enabled real-time human action performance [11], facial behavioral analysis [12], speech analysis [13], speech disfluency detection [14], stereotypical motor movement from sensory data [15], and many more. Published research from the past five years shows the use of a wide variety of sensory inputs to predict human behavior, diseases, and cognitive states using AI methods, especially deep learning (DL). Electroencephalograms (EEGs) have been used extensively to study the internal brain states by recording the electrical activity of brain waves for health monitoring [16], predicting diseases such as Parkinson’s [17] and assessing emotional disorders [18]. Even though using EEGs to study autism could have contradictions based on the experimental conditions during EEG registration among subjects, age differences, and diversity of subjects, the abnormal EEG laterization in subjects with ASD can be leveraged to build AI models to predict traits of autism [19].
Prior research using DL algorithms illustrate the successful use of facial videos collected using cameras to estimate the attention and engagement of children with developmental disorders [20]. Similarly, inertial measurement unit sensors that rely on accelerometer, gyroscope, and magnetometers that can collect information about the frequency, intensity, and duration of physical activities have shown to detect stereotypical movements in ASD children [15]. As our children rely more on digital devices such as iPads and mobile phones to read, learn, and interact from an early age [21], it is only natural to research in facilitating access to individualized digital content through a variety of interfaces (such as games, interactive lessons, maps, and more) to cater for the exceptional learners.
Challenges of AI in Behavioral Health
Despite the impressive role of AI in behavioral health, there are two key open challenges that limits its use in clinical decision making: 1) limited amount of labeled data to train AI algorithms and 2) black-box nature of deep neural network models. There are two potential solutions to these problems. First, self-supervised representation learning has been recently used to learn meaningful dense representations from small amount of data. Also, reinforcement learning paradigms can learn to optimize based on an exploration-exploitation paradigm on any defined environment. Second, explainable AI methods can be used to improve the transparency and trust of decision making by generating meta-information to describe “why” and “how” a decision was made while suggesting “what” features influenced the decision the most [24].
Personalized Explainable AI to Improve ABA and Treatment
ABA is an effective and evidence-based treatment to address individual needs for children with developmental and intellectual disabilities. Additionally, ABA treatment programs have been successfully implemented in different settings such as home, communities, school, and other educational centers. The beauty of ABA is that it focuses on data-based decision making, which allows the clinician to tailor interventions and personalize treatment options to an individual’s unique situation and need. However, as previously discussed, the analysis process is time-consuming and has limitations due to its reliance on humans as the data collectors and pattern analyzers, which is subjective given the reliance on professional judgment. These limitations can also be confounded by the ever-increasing demands placed on BAs and educators, such as increased caseloads, time demands, and limited resources. Augmenting and complementing the systematic evaluation of subject’s data using intelligent algorithms is an avenue that warrants further research. Recent literature highlights the importance of online treatments and large adoption of telehealth solutions [24] across the globe.
Recently, AI has been widely used to investigate autism with the overall goal of simplifying and speeding up the diagnostic process as well as making access to early interventions possible [25], in which supervised machine learning algorithms have been studied as useful aids to support decision making in screening [26], diagnosing autism [27], shorten diagnosing time [28], predicting behavior, and constructing predictive models [29], [30]. More recently, there have been a number of developments demonstrating the feasibility of automated facial behavior analysis systems for identifying and prediagnosing of autistic individuals. This research has shown positive advancements in designing the AI-based platform algorithm to detect and evaluate autistic patients based on brain activation patterns and mental disorders [31], faster screening [28], social interaction analysis [32], [33], facial expression [34], [35], and eye tracing [36]. However, few studies have considered the feasibility of AI in a natural setting, such as in homes or schools [37]. Effective treatment of AUIDD relies on efficient and careful behavioral observation and assessment conducted by board certified behavioral analysts (BCBA) through assessment, both direct and indirect. Although current practices for ABA treatment are highly effective, they are often time-consuming and are subject to human errors.
Building an explainable AI-augmented learning and applied behavior analytics (AI-ABA) platform could complement licensed BAs and therapists who rely on direct observation of audio-visual cues and other physiological data available during a session to diagnose and provide feedback to subjects. AI-ABA could make use of facial expressions, extremity movements, speech tone, heart rate, and other available data to build automated pipelines to detect, diagnose, and alert BAs of during treatment sessions with clients. Some of the desirable qualities of the AI-ABA and learning platform are as follows:
the ability to collect cognitive, perceptive, speech, movement, and physiological multimodal data of individuals
the ability to define systematic experiments with dynamic reinforcers in virtual or physical environments
the ability to manage experimental data and results of recurring experiments of individuals
provide early detection of behavioral changes in children and explain why, how, and what features were used for the prediction.
Moreover, AI-ABA platforms can promote self-regulative behavior using reinforcement-based AR/VR or game environments and collected multimodal data. It could also promote creativity and curiosity by designing personalized dynamic environments that involve physical activities and speech interactions while completing set tasks. By supporting games and virtual agents, AI-ABA could promote the development of spoken communication and social communicative behavior such that the collaboration and social skills can be improved by interacting with virtual agents in social settings using natural language technologies.
Figure 1 illustrates the system architecture of an AI-ABA paradigm that complements human intelligence with AI algorithms. General system architecture of AI-ABA must consist of 1) multimodal sensing technologies to collect a variety of sensor data, 2) an AI-augmented treatment personalizer (ATP) to ingest the sensor data and generate personalized treatment plans and individualized educational plans (IEPs) to children, 3) a suite of explainable AI algorithms to support various requirements of the ATP, and 4) integration to various presentation formats or front-end technologies such as AR/VR/MR [38] or tablets [39].
The system architecture of the AI-ABA platform. Multimodal sensory information is collected using both invasive and noninvasive sensors, which is processed by AI algorithms to support decision making in treatment and learning paradigms of BAs. All data are stored securely in the cloud accessible by practitioners. Reinforcement paradigms are set up in a personalized fashion unique to each individual. (Source: VR image: Flaticon.)
By designing hardware–software systems that cater to the need of behavior under study, AI-ABA could be used as a platform for closed-loop real-time multimodal data analysis. For example, data-repetitive behaviors in individuals with AUIDD could be collected using cameras or other sensors. These collected data can be streamed to a computer device that can host AI algorithms that are optimized to run on limited hardware resources using quantization operations [40]. The AI predictions could be used to initiate behavioral correction algorithms that are either deterministic or not. By exposing the results of behavioral correcting algorithms to specific web application programming interfaces (APIs), these results can be viewed on mobile or desktop devices or visualized on a monitor screen. In the following section, we summarize a few behavioral challenges of individuals with AUIDD and how AI-ABA could be a beneficial technology in augmenting treatment and teaching.
Behavioral Challenges and Sensing Technologies
Table 1 summarizes some of the common treated behavioral challenges for children with AUIDD and possible sensor measurements that could improve data collection while reducing the physical and mental burden of arduous observational data collection for BAs during the treatments. Agitation, aggression, and stereotypy seen in children with developmental disorders could reoccur due to the process of reinforcement that parents unknowingly encourage in the household. Hence, it is very important to track behavioral changes in multiple settings and environments to design individualized treatment plans or IEPs for children.
A problem behavior or challenging behavior is any culturally abnormal behavior that could jeopardize the physical safety of the individual or others that often restricts the person from social or communal activities. Self-harm or self-injurious behavior, harming caregivers, or general aggression is seen in children with low frustration tolerance. Throwing items at people, hitting oneself with objects, banging one’s head against an object, and so on are visible behaviors of aggression and agitation in children with problem behaviors. Audible and/or visual behavior such as vocal stereotypy is commonly identified by parents as their children repeat words repetitively. However, despite the many intervention models described in the research, only a few focus on the impact of intervention models in vocal stereotypy and the secondary impacts in other behaviors [41].
Problem behaviors can be evaluated in multiple settings and environments by collecting multimodal sensor data including EEG, gait and limb movements, and body tracking using cameras. Figure 2 illustrates one such method to index mild versus intense actions in humans. By generating a temporal map of body movements, a spatiotemporal long short-term memory-based attention network is used to highlight the areas of the body with rapid movements to classify actions. A kinetic fuzzy intensity analysis network generates an action intensity based on the temporal action map and DL prediction. Attention and agitation can be studied by a temporal analysis of the facial muscle movements and emotional states.
![Figure 2. - Tracking aggressive behavior using a body-tracking AI algorithm is illustrated. Here, forceful movements to the head can be classified as banging the head using a temporal action indexing module [11], as described in our previous research.](/mediastore/IEEE/content/media/6745853/9586378/9586405/ghafghazi02-3086989-large.gif)
Tracking aggressive behavior using a body-tracking AI algorithm is illustrated. Here, forceful movements to the head can be classified as banging the head using a temporal action indexing module [11], as described in our previous research.
Attention span can be studied using a temporal analysis of EEG signals as illustrated in Figure 3(a). Brain attention maps can be generated using DL algorithms to understand the pre and post brain states, leading to a more detailed understanding than traditional methods. Vocal stereotypy can be detected by observing the lower facial muscle movements for repetitive motions. This is illustrated in Figure 3(b). Here, the microexpressions on the face leading to repeated behaviors and sounds can be used as inputs to a DL algorithm to classify vocal stereotypy among other behaviors. We have found the Microsoft Azure Kinect camera to be a well-rounded sensor to record body-tracking data, red-green-blue, infrared, and depth videos, and also stereo audio data. Additionally, we found the Empatica E4 wearable sensor to be reliable to study sleep disorders using heart rate, electrodermal activity, and other sensory data, as illustrated in Figure 4. Integration of data collected via behavioral sensing can lead to deeper understanding of the triggers of problem behavior, such as poor sleep the night before. It can also help to distinguish behaviors triggered by environmental events versus those rooted in biological processes, and facilitate early identification of behaviors via precursors such as heart rate. Precise understanding of behavioral triggers is essential to ensuring effective and efficient treatment.
The attention span of children can be studied by capturing EEG signals or eye-tracking information, whereas vocal stereotypy can be predicted by studying the facial muscle movement patterns over time. In (a), we illustrate a DL model generating a prediction and corresponding brain activation maps highlighting the most attributing region of the brain. Vocal stereotypy (b) can be detected by observing the repetitive motion in the lower facial muscle groups using a facial muscle movement analyzer.
Wearable physiological sensors such as Empatica E4 can be used to collect heart rate data and study variability to carry out sleep staging and sleep quality analysis before and after therapy sessions. Here, the red dotted lines indicate intermittent waking-up states.
Various other actions, such as pacing in a repeated circular motion, body rocking, back-and-forth movement of fingers, tapping objects repeatedly, and so on are used as pointers to understand the hidden behaviors that might help with a diagnosis. Single-subject research designs, focused on individual participants, are often used, even now, to identify and evaluate interventions due to the heterogeneity of the AUIDD population. However, the integration of AI into ABA can allow for collection of larger data sets and analysis of multimodal behavioral data, enhancing research standards and the strength of the research evidence. In reality, finding interventions that can be considered evidence-based practices require large cohorts of subjects and multisession data sets [28]. Hence, scalable architectures such as AI-ABA should be explored as a means to collect data and to promote early detection AUIDD from large subject populations and predictive models facilitating responsiveness to intervention.
The Road Ahead
Systems such as AI-ABA could assists practitioners in making the precise data-based decisions to increase the quality of individualized intervention for individuals with AUIDD. This type of system is likely to have profound effects on the improvement in treatment efficacy and treatment outcomes, through a blend of both real-world and virtual elements that embeds generalization of targeted skills across multiple environments for the various aspects of an individual’s life. Recent studies illustrate the importance of active involvement of children in the intervention process [42], [43]. Additionally, providing a learning environment with effective interaction and communication is another important factor that should be taken into account [44], [45]. Since AI-ABA provides the optimal use of delivering ABA services for children with developmental and cognitive needs, the system could easily be integrated to across environments, such as schools and home, to deliver a more comprehensive treatment program. This smart and connected health via behavioral sensing would not only be highly desirable for existing outpatient clinics but also can be integrated into telehealth platforms to facilitate service access to those living in rural and deprived services areas and can provide a level of system resiliency during periods where human interaction is limited, such as the ongoing COVID-19 pandemic. In addition, applying home-based ABA can help ensure parents involvement with their child training alongside other stakeholders (e.g., teachers and caregivers) to create a more comprehensive support network for service delivery.
AI-ABA will help researchers to focus on the precision within intervention research, reinforcers effectiveness, and individualized treatment models while augmenting part of the data collection and analysis to AI algorithms. For example, a system capable of handling multiple sensory inputs for data capture could in a plug-and-play manner collect data using specific APIs and process them using existing machine learning algorithms. The processed data and results could be used to dynamically influence the virtual environments, learning structures, the precision of treatment plans, effectiveness, and the incorporation of AR or VR digital platforms to promote greater access to intervention and generalization of treatment effects. AR and VR feedback loops could increase learning engagement and raise comprehension of topics, provide interaction, improve communication, trigger imagination, and enhance problem-solving skills, especially when involving spatial skills [38]. A combination of AR and VR technologies with other invasive and noninvasive data collection systems could collect both physiological and behavioral data to study temporal dynamics of behavior in children. Additionally, this could enable just-in-time adaptive interventions by collecting precise data and provide a repository of prior behavior of each client.
ACKNOWLEDGMENTS
This project was funded partly by both the Open Cloud Institute at the University of Texas at San Antonio (UTSA) and the UTSA Brain Health Consortium and Office of the Vice President for Research, Economic Development, and Knowledge Enterprise. Arun Das and Shadi Ghafgazhi contributed equally. We gratefully acknowledge the use of the services of Jetstream cloud.