Hybrid Navigation Decision Control Mechanism for Intelligent Wheel-Chair

The continuous rising of the elderly/disabled population has created a requirement for assistive robotics devices to counter the lack of trustworthy servants. Intelligent Wheelchairs are developed for that particular purpose. Intelligent Wheelchairs differ depending on the interactive modality and most commonly found modalities are speech-controlled. Since these are assistive devices that need to act as human companions, it is necessary to have a dialogue between the device and the user. Even though the wheelchair is fully automated, the user should have control over it at some point. However, this exchange of control should be intelligent and transitions need to be executed in order to safeguard the user. Therefore the purpose of this paper is to propose an intelligent system that would navigate an intelligent voice-controlled wheelchair facilitating the intelligent exchange of control between the user and the wheelchair. This control is not simultaneous and one can override the other only when navigation could lead to collisions. In the proposed method, users can control the wheelchair using fixed vocal commands, and execution of those commands will be performed using the spatial and control parameters. Control of the wheelchair will be exchanged between the user and the wheelchair itself considering specific parameters such as obstacle distance, collision time, the velocity of the wheelchair among others. User control mode has 5 definite vocal commands with classifiers to identify any navigation command into the command model and considers uncertain terms such as ‘little’ and ‘hard’ for ‘Turn’ commands. Command classification had produced a Cohen’s Kappa value of 0.9462 and the classifier for the uncertain terms had produced a Cohen’s Kappa value of 0.7325. Both were acceptable values for those particular classifications. As per the experiment results, the proposed system reduced the vocal command frequency and risk of collisions through proper control of the velocity levels and intelligent exchange of control at given locations.


I. INTRODUCTION
Assistive devices are gradually becoming a necessity in the current context.As an example, mobile apps to support animals, assistive robot translators, robot waiter, elderly supporting robots and a spectrum of support systems are present in the world [1], [2].However, having an intelligent assistive device that could replace a human companion to the The associate editor coordinating the review of this manuscript and approving it for publication was Giacinto Barresi .
best possible extent will be valuable.Therefore devices such as Intelligent wheelchairs are being developed focusing the elderly and disabled community [3], [4].Since empowering diverse groups is the need of the hour, these wheelchairs should behave as close as a humans in the best way possible.That is why it is important to consider the modalities in which humans interact with each other.In human-human communication, there are multiple modalities used such as Speech, Hand gestures and other gesture cues, Gaze, Head movement, BCI, EEG etc.There is a number of unimodal intelligent wheelchairs developed based on these modalities as mentioned in [5], [6], [7], [8], [9], [10], [11], [12], and [13].Considering these modalities, speech-controlled wheelchairs are given priority as it is considered primary human modality.Other modalities are considered complementary to the speech modality most of the time.Among the voice-controlled wheelchairs developed, most of them are based on predefined short vocal commands such as 'Go Forward', 'Turn Left' etc.Some of the systems include the elaborative vocal commands that present information on the destination through landmarks, objects, object attributes [8], [14], [15], [16], [17], [18], [19].In this kind of wheelchair, a map is created with objects and landmarks information and the wheelchair will move to the destination accordingly [20], [21].Another important development on vocal command interpretation is uncertain information such as distance and direction [17], [22].Even though there are voice-controlled wheelchairs that use elaborative commands that would allow them to move to the exact location, a fixed map with object information is necessary.For wheelchairs that have to deal with real-time maps and unknown environments, short vocal commands are the solution [21].However, the amount of vocal commands that need to be executed in a short span of time is high for that scenario.It would be frustrating the wheelchair user who is either disabled or elderly.Also, the execution of those commands in a short span of time in constrained environments is a risky affair for the user.Also, most of these intelligent wheelchairs are either controlled by the user or by the wheelchair, the whole time.Since this is an assistive device that needs to act as a human companion, it is necessary to have a dialogue between the device and the user.Even though the wheelchair is fully automated, the user should have control over it at some point.However, this exchange of control should be intelligent and transitions need to be executed in order to safeguard the user.
Further, as Intelligent Wheelchairs are developed targeting disabled and elderly users, it is important to consider their comfort and capabilities.Since most of the time, these wheelchairs are used by people who have certain disabilities in the lower body.Therefore it is difficult for them to comprehend the actual size of the wheelchair, space behind them, and turning arc of the wheelchair.However, they have a certain idea of the dimensions of the front side of the wheelchair and the space in front of them.the se limitations will ultimately cause the users to make navigation choices and commands that will make the wheelchair take a risky turn or go through a narrow path.This will cause the wheelchair to be either stuck in a place or cause serious injuries to the user.Therefore it is necessary to have a method to detect this issue beforehand and either communicate to the user or take control of the situation.Therefore it is necessary to have an intelligent exchange of control between the user and the system in order to accomplish this.
Therefore the purpose of this paper is to propose an intelligent system that would navigate an intelligent voice-controlled wheelchair facilitating the intelligent exchange of control between the user and the wheelchair.This control is not simultaneous and one can override the other only when navigation could lead to collisions.Also, this intelligent exchange of control is executed considering surrounding environment of the wheelchair.In the proposed method, users can control the wheelchair using fixed vocal commands, and execution of those commands will be performed using the spatial and control parameters.Control of the wheelchair will be exchanged between the user and the wheelchair itself considering specific parameters such as obstacle distance, collision time, the velocity of the wheelchair among others.

II. RELATED WORK
Earlier research literature regarding intelligent wheelchairs can be represented through major technological breakthroughs they had in the process of robotics development.
As an intelligent wheelchair process can be discussed in three main stages; input methods, operating modes, and human factors, technology or the approach used in either of those stages are used to understand the development [23].Some of those include Haptic feedback, Voice recognition, Object recognition, Human Physiology, Collision Avoidance, and Social Issues [24].
There are various types of intelligent wheelchairs developed in the literature based on the input method [25].There are two major types of input methods available in the literature.They are namely unimodal and multimodal wheelchairs.The most common type is unimodal and speech, hand gestures, gaze, EMG, and other input modalities are used as unimodal input methods.However, these wheelchairs have user control as the control mode or the operating mode.For example, the system mentioned in this paper presents a hand gesture model specifically customized for navigation of an intelligent wheelchair in an indoor environment.Also in this system, the hand gesture model covers every navigation scenario and compensates for the gesture variance caused by tremors.Even though the interactive system is intelligent enough to control through hand gestures, hand gestures are ultimately given by the user.Therefore the control of the wheelchair navigation lies at the hand of the user alone.Since the wheelchair users are either disabled or elderly, their comprehension of the environment, dimensions of the wheelchair, and space around it are less accurate.Further, they could make wrong choices of gesture commands and their cognitive capabilities in calculating the time and space could be seriously hampered.Also, the gesture commands the users make could be slower and that could lead to accidents and injuries.The reason behind that is the lack of feedback from the wheelchair.This is true for other unimodal and even multimodal wheelchairs since the control is still at the hand of the user.Due to the lack of feedback from the wheelchair, though the interaction is intelligent, navigation will not be intelligent.
The operating mode of an intelligent wheelchair is described as the autonomous nature of the operation.The autonomous nature of Intelligent Wheelchairs extends from fully autonomous to semi-autonomous intelligent wheelchair and it depends on the user ability and task of the wheelchair [42], [43].If the user cannot plan and execute a path, then the wheelchair should be autonomous and the user should be in the same controlled environment.If a user can plan the path and execute, then only a collision-avoidance system is implemented [23].Ideally one wants to get the user's requirements and abilities into account and try to maximize the user control of the wheelchair.Among semi-autonomous control modes, shared control has been a popular operating mode [26], [27], [28], [29], [30].Following or using a companion to follow is described as a common shared control mode where a vision sensor could provide the visual data of the person or the object to follow [31], [32].However, the lack of user control on this semi-autonomous mode is a glaring flaw.Navigational assistance is another operating mode that enables the semi-autonomous operation of an intelligent wheelchair [33], [34], [35].However, in these shared control systems, transitions between the two control modes are not based on the wheelchair navigation information but rather based on the environment.Also, the two modes operate independently and monitoring of each mode to finetune the navigational experience is lacking.The system mentioned in [24] presents a novel shared controller using the Bayesian approach to enhance the performance of a brain-actuated robot system.In here shared control of robot control and brain actuated control has taken uncertainty of the robot perception, action, and human control into consideration.This method builds probabilistic models of human and robot control commands using maximum a posteriori probability (MAP) for optimum control of the wheelchair.However, in this system wheelchair navigation information such as direction, velocity and spatial information such as distances to the obstacle, angle towards obstacles haven't been taken into consideration.Out of the available shared control intelligent wheelchairs that facilitate the semi-autonomous nature of the wheelchair, there are two types of control strategies.One is user-controlled wheelchairs where the shared control part applies only for navigational purposes.In these types of Intelligent Wheelchairs, the semi-autonomous mode of the wheelchair is achieved by obstacle detection and avoiding algorithms.There is no intelligent exchange of control between the user and the wheelchair [26], [27], [29], [33], [34].Another type of control strategy is focused on the autonomy of the wheelchair and providing less control for the user.There is no monitoring of the control exchange between the two modes and most of the time, completely overrides the user citing safety.These systems work when the user is seriously injured and has no control of their faculties.However, for more robust design and use, the Intelligent Wheelchairs must have a harmonious control strategy between the user and the wheelchair.The reason behind this is that most wheelchair users have certain control over their physical faculties and would want to have more control of their day-to-day lives [34], [36], [37], [38], [39], [40], [41].
Work has been carried out to understand surrounding environment while navigating assistive robots or autonomous vehicles.Different approaches were taken to represent spatial information about the environment.As an example [42] suggests a cognitive model to represent obstacles in a virtual representation of the environment.In this approach vocal description given by the user is used to identify objects and later it will be combined with spatial data received through laser scanner to create virtual enhance map of the environment.In [43], the approach is to create a cognitive map of the environment using sequence data received through a conversation with the user.The conversation between user and the system will be used to filter spatial information of an object and its attributes to identify location of the object in a virtual map.The created virtual map will be amalgamated with spatial map to verify location of the identified objects.
During navigation its more comfortable to use uncertain terms to interpret spatial information.As an example use of ''There is a table three meters in front of you'' than using ''There is a table near you'' to describe an obstacle lying ahead of the walking path can make a person confuse.It is not natural to use exact measurement or numerical values in natural sentences.Humans tend to express measurement using uncertain terms such as ''near'', ''far'' or ''little bit left'', ''hard left'' to give direction or explain distance related spatial values.Methods have been proposed to understand and interpret uncertain information related to spatial measurements [44], [45].Fuzzy membership functions and neuro fuzzy approaches are used to interpret spatial related uncertain terms [46], [47].However, interpreted terms are mainly used to describe the environment or give a voice command such as ''move near to the table'' or ''There is a chair near to the table''.Voice commands, like ''turn little bit right'' or ''take hard left'' are not clearly explained in any of the approaches.

III. HUMAN STUDY TO IDENTIFY VOCAL NAVIGATION COMMAND FREQUENCY IN GUIDED WHEELCHAIR NAVIGATION
50 randomly selected participants were asked to pair themselves.One person was asked to guide the other who is the wheelchair user.The wheelchair used in this study was a joystick-controlled wheelchair and the wheelchair user was blindfolded and had no idea of the navigation path.The guide had to navigate the wheelchair user using vocal commands according to the given navigation path.Wheelchair users would drive the wheelchair according to the vocal instructions given by their guide.Further, whenever there is a deviation from the path or near collision, the guide has the freedom to correct the wheelchair path using vocal instructions.Vocal commands/instructions given by the guide are recorded and also the number of collisions/stoppages/deviations were also recorded with the respective time signatures.The used environment setup is given in Fig. 1.Further how many commands that the guide had to give for one particular navigation path, the frequency of the commands used in a particular movement, and time for the wheelchair user to execute one particular command are calculated.The calculated values are given in TABLE 1 Among the vocal navigational commands given by the users, a few of the frequently used commands included ''go forward'', ''turn right'', ''stop there'' and etc. Considering every navigation scenario that could occur in a domestic setup, five navigational vocal commands were selected as a command library.The five commands in the library are ''Forward'', ''Turn right'', ''Turn left'', ''Stop'', and ''Turn around''.With these commands, if the wheelchair has the necessary intelligence to comprehend navigational conditions, users can easily navigate the wheelchair using speech modality.
The navigation path of the map is divided into 7 regions as shown in the Figure 1.Region 1 includes a few obstacles in the path and also there is a choice of the direction to be made during the navigation.As represented in TABLE 1, the command frequency for the ''turn right'' and ''turn left'' commands are higher compared to other regions.Also, the ''turn right'' command frequency is higher since naturally right-handed users tend to choose the right side.Righthanded users were higher among the participants since they were selected randomly.Also, we could observe that the command frequency for ''forward'' is higher in regions 4 and 5 since the path is straighter.Command frequency for ''turn right'' in region 6 and ''turn left'' in region 3 is higher compared to other regions since those regions have a bend.Similarly, command frequency for ''stop'' command is higher in regions 2, 5, and 7 since the path is narrower or with more obstacles.Also, it is observed that the total frequency for each region is hovering around 97% to 99% which concludes that almost all the vocal commands can be categorized into the five main commands.
The percentages of navigation command types used by the participants after being classified into the main categories used to model navigation command classifiers are represented in Fig. 2. Further, more detailed analysis of navigation commands used by participants are given in Fig. 3 and 4. According to that, the highest percentage belongs to 'turn right/left commands and 'right' and 'left' commands evenly distributed.The lowest percentage belongs to the 'turn around' command since there are fewer opportunities to use that command in the navigation path given to the users.However, there are a higher amount of 'stop' commands which amounts to 11% which is quite astounding for navigation.This is due to the uncertainty and hesitation of the user with the surroundings and their comprehension of the environment given their cognitive capabilities.What results due to this increase is that the user gets exhausted after a while with this start/stop nature of the navigation and uses an extraordinary amount of concentration on avoiding  collisions.Therefore the navigation would not be smooth and the user will be uncomfortable the whole way through.

IV. SYSTEM OVERVIEW
The overall system is shown in Fig 6, and the hardware setup is given in Fig. 8. Data required to identify the spatial information are acquired through Laser Data Receiver (LDR).Laser data received are analyzed in the Laser Data Analyzer to build the map and identify obstacles.Then these analyzed data are processed in the Spatial Information Processor (SIP) to calculate the spatial information such as distances to the obstacle, angle towards the obstacle w.r.t to the navigation direction, etc. Navigational vocal commands are captured through a microphone and captured voice data are analyzed in the Vocal Command Analyzer (VCA).In the VCA, classification is performed to identify executable user vocal commands using the available language library through a recurrent neural network.Wheelchair navigation data such as the velocity of the wheelchair, control mode etc. captured through Wheelchair Control Data Receiver (WCRC) are monitored by the Data Monitoring System (DMS).Further in the DMS, other than control mode and velocity, spatial information of the navigation environment and user vocal commands are also monitored throughout.Wheelchair Control Module is the main control unit of this system and it consists of Control Mode Initiator (CMI), User Control Mode (UCM), and Autonomous Control Mode (ACM).CMI decides whether the control mode needs to be changed between user control mode and autonomous mode.Further CMI initiates the control mode.UCM receives the user command and processes the action required.ACM processes the required navigational action using the spatial data, wheelchair data received through SIP and WCRC.Action Controller (AC) translates the navigational instructions given by the WCM and gives necessary low-level control instructions to the wheelchair navigation controller.

V. EXPERIMENT SETUP
The autonomous control system proposed in this article will be implemented on an Intelligent Wheelchair developed in the Robotics and Automation Laboratory, Department of Electrical Engineering, University of Moratuwa, Sri Lanka.Implemented wheelchair setup is shown in Fig. 10.This wheelchair is capable of interacting with the user through hand gestures and speech.A leap motion sensor was used to capture hand gesture information and a microphone was used to capture the speech of the user.Intelligent systems are developed and run through the industrial computer installed on the wheelchair.Further, for navigation and localization purposes, a laser sensor was installed.Other sensors required for navigational assistance such as ultrasonic sensors, bumper/limit switches were also used.The hardware system of the developed intelligent wheelchair is represented in the Figure 10.
A Jazzy power wheelchair was used as the mechanical structure and motor controllers and drivers are modified specifically for intelligent navigation in indoor environments.The maximum speed of the wheelchair is 10 km/h and it includes 4-speed levels.ARK-3520L industrial-grade computer is installed in this wheelchair which has a 6th generation Intel Core i7-6820EQ processor with 2.8GHz frequency.It also has 32 GB DDR4 2133MHz memory.As the input/output ports it includes 2 LAN ports, 4 serial 118562 VOLUME 11, 2023 Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.In the second layer each word is stored one vector per word.When the vector is called, it converts the word indices to vector sequence.Since the vector sequence is trainable, the converted vectors are used to train the classification model.
After classification in to main navigation command categories, the voice commands are classified into ''turn left'' category or ''turn right'' category.Then it will be forward to the sub classifier to identify uncertain information.The sub classifier store the pre-trained 192 vocabulary with their conditional probabilities and the calculated values will be used to find maximum probability likelihood of the uncertain term in a given sentence.Maximum probability will then be used to categorize the navigation command into sub classes.The maximum input for sentence is 5 words as shown in equation 1 and equation 2 and 3 will be used to identify the subclass category.Classification process is shown in Fig. 5 S i ∈ x 1 , x 2 , x 3 , x 4 , x 5 (1)

B. WHEELCHAIR CONTROLLER CHANGING PROCESS
Wheelchair controller changing process is initiated based on the possibility of collision with obstacles.Algorithm 1 receives real time data such as velocity and spatial information of the surrounding environment process by SIP.Since the wheelchair speed is a parameter with four speed levels Algorithm 1 calculates the remaining time for the collision if the wheelchair pursuits the same path.Based on the calculated values and default respond coefficient(can vary based on the user) CMI changes the controlling access.

C. CONTROLLER TASK IDENTIFICATION
Algorithm 2 is responsible for the controller action identification and direction calculation.Action controller is consists of 5 actions which can be used as system outputted decisions.The actions defined in the Action Controller is shown in

D. DIRECTION CALCULATION
During a navigation process the control of the wheelchair can be changed between user and the system.However when the system is in control of the wheelchair and the act 2 is triggered, the wheelchair may require to change the direction.Algorithm 3 observes the surrounding obstacles to calculate angle between current direction of the wheelchair  and other obstacles.This algorithm takes the dimension of the wheelchair itself to the calculations as shown in Fig. 18.Direction commands can be categorized in to two main sub classes called ''Turn Left'' and ''Turn Right''.Further, those two commands are classified in to three sub classes called little, left/right and hard.Algorithm 4 and 5 take the command and calculate the angle requested by the user.The method of angles selection is represented in Fig. 17.The command can consists of uncertain terms, Therefore after classification into the main five classed, the voice commands with directional information are again filtered by algorithm 6 to identify uncertain terms.The fuzzy membership functions which was used to interpret uncertain terms are shown in Fig. 16.
Algorithm 4 monitors the exchange of control between user and the system.Moreover, it observes the target positioning of the wheelchair.Observed exchanges are recorded as shown in Fig. 25.If the Action Controller assigns system as the controller and WNC changes its direction, control mode initiator monitors possible collisions with obstacles and previous control exchanges between user and the system to decide for suitable action.

E. MAP FORMATION
Spatial information is acquired through laser scanner and the the raw data will be converted into a RGB image.The  Image conversion process is given in Figure 11 and example spatial map and the same spatial map after the movement of wheelchair is given in Fig 13 and 14.The distance measurement method is explained in Fig. 12.The images before and after processing of,each step are given in Fig. 19

A. HUMAN STUDY TO IDENTIFY PROPORTION OF THE AGREEMENT BETWEEN USER AND THE SYSTEM GENERATED COMMANDS
Randomly selected 12 participants were asked to watch the videotape of the human study I and asked to sort given vocal   command segments into five vocal commands defined in the library.With the video of the previous study, participants were FIGURE 17. Spatial cluster representation -After identification of each obstacle centroid will be calculated.Since the wheelchair can be navigated by both user and the system the directions to each obstacle is calculated and updated real-time.When the system receive where to change the direction through voice command such as ''Turn Left'' or ''Turn Right'' the system identifies relevant region for the command and directions include only in that region will be considered for the execution of the action.

Algorithm 3 Direction Deciding Algorithm
Require: : t, v, τ Ensure: : θ 2 using the classification system into five navigational vocal commands in the library.(forward, stop, turn right, turn left and turn around) Both classifications made by the user and the system were then compared using a confusion matrix.Further, vocal commands including ''right'' and ''left'' from the study I were then given to the same set of participants and asked them to sort these vocal commands into uncertain vocal commands such as ''little right'', ''right'', ''hard right'', ''little left'', ''left'' and ''hard left'' according to the intention of the particular command given the context of the navigation scenario in the video.Again the same set of vocal commands was classified using the sub-classifier in the intelligent system and both classifications made by the user and the system were then compared.Results of the human study is shown in Initiate act 1 else Initiate act 3 end if given by them into five defined vocal commands according to the navigation scenarios.Those sorted commands are then compared with the commands classified by the system.Commands are named 1 through 5 correspond to 'Forward', 'Turn Right', Turn Left', 'Turn Around; and 'Stop' respectively.There were 67 vocal commands of command type 1 and 63 were detected as same by the system and 4 were detected as command type 3. Similarly, there were 133 commands of command type 2 were intended by the users and only 132 were detected as such and the other one was detected as command type 1.These small discrepancies are caused by the slightly misheard commands through the microphone and the error percentage is quite small as shown in TABLE 2. According to TABLE 3, the observed Cohen's kappa value for the earlier confusion matrix is 0.9462 with a standard error of 0.0132.Since the observed Cohen's kappa or unweighted Cohen's kappa value is higher than 0.81 it can be concluded that the classification made by the system in determining the vocal commands is satisfactory.Further chance expected values for each command type are positive values according to the frequencies of agreement as shown in TABLE 4 and Proportions of agreement in TABLE 5. Therefore it can be concluded that that the system will classify the vocal commands accurately over time.
Similarly, the vocal commands which were intended by the user as 'Turn Right' and 'Turn Left' were produced in front of the users.Then they were asked to sort those gestures into uncertain commands such as 'little right', 'right', 'hard right', and so on so forth.Then these user observed vocal commands of command types 1 through 6 are compared with the system observed commands.Command types 1 through 6 corresponds to vocal commands 'little left', 'left', 'hard left', 'little right', 'right', and 'hard right'.As represented in the confusion matrix in TABLE 6, there were 58 userobserved commands of 'little left' and the system has only observed 54 out of them as such.3 commands were observed as 'left' and another one as 'hard left'.This deviation is quite small and probably can be considered a minor discrepancy.However, there are some slightly higher deviations in the command types 2, 3, 5, and 6.Since this is a classification of uncertain terms and classification is based on a fuzzy system, this amount of deviation is expected and valid.This is due to the fact that the interpretation is based on human behavior, borders among the uncertain terms are loose.Even though the accuracy is slightly lower, this classification is similar to human behavior.As represented in TABLE 7, Cohen's kappa value is 0.7325 which is slightly lower than 0.81, but acceptable since this is a fuzzy system.Also according to TABLE 8, chance expected values for all command types are positive values.Therefore it can be concluded that this classification will be accurate when repeated for multiple users over time.
The velocity distribution of the system over the distance through the navigation map is represented by the Figure 27.There are four different fixed levels of velocities in the wheelchair system.They are represented as 1 to 4 while 1 being the lowest and 4 being the highest.Further, the distance is represented by regions 1 through 11.According to the distribution, the velocity of the wheelchair is at the lowest in the regions 1 through 3 and starts to increase slightly after that.Reasons for this observation are that those earlier regions have a number of obstacles, bends, and narrow hallways which cause the wheelchair to lower the velocity.After that wheelchair comes to much clearer regions and hence the increase of the velocity.However, it hovers around velocity level 2 until region 10 and beyond that, the velocity increases to further values.This is due to the lack of narrow hallways in the latter regions.
The amount of navigation commands used by the user during the navigation path while the wheelchair was controlled by the user alone is represented in the Figure 28.Also, the number of navigation commands used by the user when the wheelchair was controlled by both the user and the system is shown by the black dotted line in the graph shown in Figure 28.Generally, the number of navigation commands used in the first few regions is higher due to the fact that those regions have a high number of obstacles to navigate through and a high number of turns and choices to make.This number decreases in the middle since the obstacles are less comparatively.However, there is a slight increase observed in the latter regions since the number of turns and other navigational scenarios such as turnarounds   Therefore it can be concluded that the proposed method has lowered the burden of the user significantly and softened the physical exhaustion.

C. EXPERIMENT II -OBSTACLE IDENTIFICATION PROCESS EVALUATION
A fixed indoor environment was defined and set up with objects such as tables, chairs, stools, walls and etc.Then a number of still maps were taken from different perspectives and obstacles in those maps were clustered using the proposed intelligent system.A comparison is made in order to 118572 VOLUME 11, 2023 Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.check whether the objects present in the defined environment were clustered as it is by the system and whether those objects were clustered as multiple objects.

D. EXPERIMENT III -EFFECTIVENESS OF HYBRID CONTROL OF THE WHEELCHAIR OBSERVATION
A path is defined in a fixed environment and a wheelchair user is asked to control the Intelligent Wheelchair with the developed intelligent systems.The way in which the control of the wheelchair exchanged between the user control mode and autonomous mode, change of the velocity levels with time, when and where the exchange of the control modes happened, and why it happened were observed.The control exchange between user and system with respect to time is shown in Fig. 29 and the average use of commands used by participants are shown in Fig. 30.The navigation map used for the human study was divided into 11 regions as represented in the Figure 26.When the wheelchair was operated using the proposed system, how the control mode changed with time during the navigation is represented in the Figure 29.User control mode is given by the value 1 and autonomous mode is given by the value 2. It is clearly observed that when the wheelchair gets closer to obstacles, narrow hallways, and closer to walls control is shifting to autonomous mode.Then when there is a navigation choice is to be made, control shifts back to user control mode.Navigation or the drive of the wheelchair is always initiated by the user whenever it endangers the user itself.After the navigation choice or the initiation is made the system will monitor the environment and wheelchair parameters and change the control mode back to the autonomous mode whenever it is necessary.What stands out in this system is the smooth and harmonious exchange of the control ensuring the maximum safety of the wheelchair user.Further, distance covered through the navigation path with time is represented in the Figure 30.It can be observed that whenever there is a control change back and forth between the user control mode and autonomous mode, the wheelchair stops and execute the change.Also when a bend or a location where a wheelchair has to be turned wheelchair slow down and takes time to execute the action.When these two graphs are compared together it is clear that in the busier regions or obstacle-ridden regions, time taken to cover the distance is higher and the control exchanges are also higher.Further, in those regions, the control mode is mostly the autonomous mode.

VII. CONCLUSION
Intelligent Wheelchairs employ different control modes for their navigation.User control mode which is most prevalent opens up a number of issues with the safety of the user.Available shared control modes in already developed wheelchairs are not addressing the necessary harmony between the autonomous mode and the user control mode.Without focusing on one control mode to mitigate issues with smooth navigation, an intelligent exchange of the control between the user and the wheelchair system is required.Therefore, this paper present an intelligent wheelchair navigation system where the exchange of the control between the user and the system is employed intelligently.For this purpose, not only the environmental parameters such as distances to the obstacles, space available in the path etc.Moreover, the wheelchair parameters such as velocity and orientation are utilized.Further, all of these parameters are monitored in order to execute the control exchanges back and forth between the two control modes.Also, a model of vocal navigation commands was introduced to cover all the navigational scenarios that are encountered in an indoor environment.This model included an intelligent system to classify any vocal navigation command into one of the defined commands.This wheelchair navigation system is implemented on the intelligent wheelchair developed by the Intelligent Service Robotics Group, Department of Electrical Engineering, University of Moratuwa, Sri Lanka.To validate the proposed system, Participants were asked to navigate the intelligent wheelchair using the developed system in a given navigation path.The number of vocal commands used, command frequency, and comparison between user intended navigation commands and system observed navigation commands were analyzed.Further, how the control exchanges happened with time and distance during the navigation and how the velocity of the wheelchair changed during the navigation according to the control exchanges and environmental parameters were also analyzed.It could be concluded that the number of vocal commands that had to be used for the proposed system is significantly smaller.It reduced the mental exhaustion of the user and improved the smoothness and the safety of the user.Further, the user could take back the control in the opportune times and the commands were properly detected by the system with significant accuracy.This allowed the user to be individualistic and independent which is the need of the hour nowadays.This intelligent navigation system could solve a major issue with recently developed intelligent wheelchairs.We observe wheelchairs with intelligent interactive systems which allow human-robot interaction with a number of modalities mimicking human behavior.However, the navigation of the wheelchair has to be both human-like and safe which the intelligent navigation system proposed in this paper can achieve.Not only that this intelligent exchange of control empowers the wheelchair user, and it also ensures the safety and the smoothness of the navigation.

FIGURE 1 .
FIGURE 1. Setup for the human study that was carried out to explore navigation command used by human operator while navigating a wheelchair through a hall way.The frequencies of used navigation commands in each region are shown in a PI charts.

FIGURE 2 .TABLE 1 .
FIGURE 2. Navigation commands types used by the participants after classified into the main categories.These commands were used to model navigation command classifier.

FIGURE 3 .
FIGURE 3. Number of navigation commands and the time of the commands issued by the participants during navigating of the wheel chair in a given path.

FIGURE 4 .
FIGURE 4. Number of navigation commands used in different regions.Minimum, maximum and average number of commands used by different participants are shown in the graph.

FIGURE 5 .
FIGURE 5. Language model for sub classification of the navigation commands.

FIGURE 6 .
FIGURE 6. System overview of the intelligent navigation system which is responsible for understanding user vocal commands, decision making on navigation commands, monitoring the wheelchair navigation and control parameters, and intelligent control exchange.

FIGURE 7 .
FIGURE 7. The range of the 3D laser senor with the height of the object to be detected.The laser sensor has four layers and a range of 0-45 m.A range of 0-10 m is used for the proposed system.

FIGURE 8 .
FIGURE 8. Hardware system overview of the intelligent wheelchair which represents the input/output connections to the industrial computer and the overall hardware connections of the wheelchair.

FIGURE 9 .
FIGURE 9. Recurrent neural network architecture for voice command classification.

FIGURE 10 .
FIGURE 10.Wheelchair setup used for the human studies and the experiments.

Fig. 15 .
Fig. 15.Algorithm 2 takes inputs from the SIP and the CMI to decide relevant action for the situation.If the action requires change of direction Algorithm 2 calculates the angle to change direction based on remaining time for a collision and spatial placement of the surrounding obstacles.

FIGURE 11 .
FIGURE 11.Map creation process -Once the laser scanner perceives the spatial data it'll be converted into a image.Then the image is processed to obtain position of obstacles in the surrounding environment.

FIGURE 12 .
FIGURE 12.Spatial Distances are measured between the position of laser scanner to the centroid of identified obstacle.Since map is not completed to identify obstacle as a whole object, the centroid of clustered obstacle is considered as a object.

FIGURE 13 .
FIGURE 13.Example Spatial Map created using laser scanner data.spatial maps consists of 4 layers for different heights as shown in Figure 7. Then the RGB images are converted to binary images by Algorithm 6.After the conversion binary images of 4 layers are enhance for continuity of boundaries and edges by the Algorithm 7 and 9. Processed images are the amalgamated by overlapping each layer by the Algorithm 10.Then the overlapped image will be enhanced to identify obstacle and free space by Algorithm 8 and 11.Overlapped, enhanced and clutered images are shown in Fig. 21, 22 and 23.Image conversion process is given in Figure11and example spatial map and the same spatial map after the movement of wheelchair is given in Fig13 and 14.The distance measurement method is explained in Fig.12.The images before and after processing of,each step are given in Fig.19(a) to 19(e) and Fig 20(a) to 20(e).

FIGURE 14 .
FIGURE 14. Example Spatial Map after moving forward relative to the wheelchair position of the map given in Fig. 13.

FIGURE 15 .
FIGURE 15.Action hierarchy for wheelchair system to execute during system controlled situations.

FIGURE 16 .
FIGURE 16.Fuzzy membership function for direction deciding process.

FIGURE 18 .
FIGURE 18. Measurements of wheelchair taken by laser scanner to calculate directions.w 1 and w 2 are fixed parameters and other measurements are calculated real time.

FIGURE 19 .
FIGURE 19.Maps of surrounding environment perceive through laser scanner.Each image consist with spatial information about the obstacle in different levels of heights.lowest ground covers by (a) and highest ground is shown in (d) (b) and (c) depicts the middle layers.(e) shows the complete image after overlapping all the 4 layers.Visual image of the environment is given in Fig. 24.

FIGURE 20 .
FIGURE 20.Processed images of maps given in Fig. 24 created using surrounding environment.Each image consist with spatial information about the obstacle in different levels of heights.lowest ground covers by (a) and highest ground is shown in (d) (b) and (c) depicts the middle layers.(e) shows the complete image after overlapping all the 4 layers.Overlapped image is used to enhance and identify boundaries and obstacles.Visual image of the environment is given in Fig. 24.

Algorithm 5 FIGURE 21 .
FIGURE 21.Overlapped image of environment after boundary enhancement to improve continuity.

FIGURE 22 .
FIGURE 22. Improved spatial map to identify obstacle clusters.Obstacle boundaries are more enhanced.

FIGURE 23 .FIGURE 24 .
FIGURE 23.Enhanced overlapped image and its proceed version after cluster identification.

FIGURE 26 .
FIGURE 26.The navigation map which was used for the experiment I.It is divided into 11 regions considering the obstacles and navigation scenarios involved.Participants are asked to drive the wheelchair using the proposed system through this navigation path.

FIGURE 27 .
FIGURE27.Change of velocity level during user navigating the wheel chair in a given path.The system is capable of changing its velocity among 4 levels.

FIGURE 28 .
FIGURE 28.Amount of navigation commands used by the user during navigation path was shown in the graph.The black line show the amount of the navigation commands used by the user when both system and user controlled the wheelchair during navigation.

FIGURE 29 .
FIGURE 29.Exchange of the control between user and the system is during navigation.

FIGURE 30 .
FIGURE 30.Average amount of navigation commands used by the participants during navigating the wheelchair in given path was shown in the graph.
TABLE. 2, 3, 4, 5 6, 7 and 8.The confusion matrix given in TABLE 2 depicts the comparison between the user-intended vocal commands and system-detected vocal commands.As explained in human study I, particpants were asked to sort the vocal commands Algorithm 4 Action Deciding Algorithm Require: : t, v, τ Ensure

TABLE 2 .
Confusion matrix for the human study 1.

TABLE 3 .
Unweighted Kappa analysis for the human study 1.

TABLE 4 .
Frequencies of agreement for the human study 1.

TABLE 5 .
Proportions of agreement.

TABLE 7 .
Unweighted Kappa analysis to compare user and system behavior.

TABLE 8 .
Proportions of agreement -navigation command sub class.