Remote Control of a Wheeled Robot by Visible Light for Support in Infectious Disease Hospitals

The shortage of medical personnel is now a problem and is expected to worsen in the future. Meanwhile, in the case of infectious diseases such as new coronavirus infections, it is very important that nurses and other medical staff treat patients as remotely as possible, which helps to prevent nosocomial infections and is also important for maintaining the medical system. One way to address these issues is to use mobile robots as assistants to automate the transport of patients and supplies. Wireless communication is necessary to control such robots remotely, but doing so with radio waves is undesirable in the medical field because of the impact on patients and medical equipment. Therefore, this paper proposes a teleoperation system for a wheeled mobile robot using visible light with a camera and an array of LEDs, an approach that affects neither patients nor medical equipment. This paper presents the development of visible light communication for this system by introducing a novel blinking pattern for the receiver and novel light-weight algorithm for the transmitter. According to a wide range of experiments, the proposed teleoperation system performs well enough for the remote control of wheeled robots in hospitals.


I. INTRODUCTION
The pandemic of the new coronavirus infection (COVID19) has highlighted the shortage of nurses around the world, and according to a WHO report, there will be a shortage of 5.9 million nurses by 2030 [1]. In Japan, it is estimated that there will be a shortage of around 50 000 nurses by 2025, and the situation is expected to worsen in the future [2]. Furthermore, in the case of infectious diseases such as new coronavirus infections, it is very important that nurses and other medical staff treat patients as remotely as possible, which helps to prevent nosocomial infections and is also important for maintaining the medical system.
One way to solve these problems in the medical field is to use mobile robots as assistants [3], [4]. In such a system, the transportation of patients and supplies is automated by remote control of mobile robots. However, wireless communication is necessary for the teleoperation of robots, but doing so with radio waves or infrared rays is undesirable in medical settings because of the negative impact on patients The associate editor coordinating the review of this manuscript and approving it for publication was Shen Yin. and medical equipment [5], [6]. For wireless communication in facilities where the use of radio waves should be avoided, such as hospitals, visible light communication has been attracting attention [7]- [11].
Visible light communication is wireless communication using electromagnetic waves with wavelengths of 400-700 nm, which is the visible light band [12], [13]. Therefore, unlike radio wave communication, precision equipment and the human body are unaffected, and visible light communication is deemed safe for use in facilities such as hospitals There are two main types of visible light communication, in which the receiver is either a photodiode or a camera (image sensor) [14]. Photodiodes have high-speed response but are affected greatly by disturbance and interference [15]. On the other hand, experiments have shown that although cameras respond more slowly than do photodiodes, the effects of disturbance and interference can be reduced by image processing [16]- [18]. They have applied high-speed cameras as receivers to improve the data rate and communication distance. The high-speed cameras are very expensive. Furthermore the weight is 1kg or more, so they are a little difficult to install on small-type robots that we VOLUME 9, 2021 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ use in this work. On the other hand, to achieve the high data rate with a high speed camera, high frame rate (more than 1000 fps) image capturing is necessary since the data rate is dependent on the camera frame rate. In addition to that, the real time image processing following the high-frame rate is also necessary to achieve a communication application. For example, if the camera frame rate is 1000fps, a single frame has to be processed within a 1millisec. To achieve this image processing in real time, a larger computer needs to be used at the receiver side. Therefore, the hardware environment at the receiver side becomes very complicated. In this paper, a teleoperation system for a wheeled mobile robot using visible light communication with a general 2D camera and a light-emitting diode (LED) array that affects neither patients nor medical equipment is proposed. Because the communication speed with a single LED matrix is not particularly high as a transmitter we use instead an LED array combining multiple LED matrices. The signals from multiple LED matrices can be transmitted simultaneously in parallel, thereby increasing the communication speed. In the proposed system, a camera (receiver) is mounted on the robot side and an LED array is installed on the manipulated side. First, we set the LED array (transmitter) to blink in several predetermined patterns and set the motion of the wheeled mobile robot corresponding to each pattern. Then, by processing the images captured by the camera (receiver), the blinking patterns are recognized and the corresponding actions are executed by the wheeled mobile robot. Image processing algorithms for the receiver side were developed considering applicability, they are light-weight and can be implemented with small-type hardware such as Raspberry pi. This implementation significance leads to improving the applicability of the overall proposal.
The contributions of this paper are as follows.
1. The images from the camera on the receiver side are processed and an L-shaped area is established with which to find the transmitter when it was blinking in order to make it easier to find and track the transmitter. This reduced the loss of area for data transmission while facilitating discovery and tracking on the receiver side 2. A novel transmitter detection method and blinking pattern recognition method for the receiver are proposed. The transmitter detection method is based on frame subtraction accumulation. The blinking pattern detection method is proposed based on the Gaussian model. 3. The algorithms developed to detect the transmitter and understand the blinking pattern are lightweight and can be implemented in real time on small microcontrollers such as Raspberry pi. This blinking pattern understanding method performs better than previous methods The new method described above to study the visible light communication between the LED matrix array and the robot-mounted camera is introduced. In particular, we confirm the effectiveness of the proposed method for the part of the receiver that processes the images from the camera and receives the transmitter's detection and blinking patterns. Furthermore, considering the proposed system installed in a small robot, we implemented the process for receiving driving instructions (commands) on the receiver side on small hardware, conducted a teleoperation experiment on a wheeled robots using the proposed visible light base teleoperation system, and confirmed the usefulness of the proposed communication method. In this system, the robot is operated by blinking the LED array from a remote location, thereby enabling nurses to care for patients remotely, by sending necessary stuff to patients by small wheel robots and moving the wheelchair patients infected with infectious diseases, keeping a certain distance from them. This will help prevent infection of medical staff, especially when the target is a patient with an infectious disease such as the new coronavirus. In visible light-based robot teleoperation, the user (master) commands the robot by blinking the transmitter (LED array). Meanwhile, the user can see both commanding signals of the transmitter and respond from the robot following the signals. So, teleoperation can easily be achieved in a safe manner. This is also an advantage of the visible light-based robot commanding systems.

II. RELATED WORK
There have been previous studies of teleoperated mobile robots to assist with tasks in hospitals. However, most of those robots traveled on a predetermined path and performed a predetermined action [16], [20]- [27], so they did not require much teleoperation. In addition, there have been reports of robots in the medical field that can be operated wirelessly, such as through the Internet [30], and Dallal et al. [31] showed that a hospital mobile robot could be controlled over a wireless local area network. However, in many hospitals, the use of radio waves and infrared rays is undesirable because of their impact on patients and medical equipment [5], [6]. Furthermore, as mentioned above, with the spread of coronavirus infection, the importance of remotely operated robots is increasing in the medical field. For this reason, the aim is to realize the proposed system by visible light communication without using radio waves and infrared rays Visible light communication using LED arrays and cameras has already been studied in road-to-vehicle communication, and the detection and tracking of transmitters has also been studied in that field [16]- [19] To facilitate the detection of transmitters on the receiver side, an area for transmitter detection is provided in the LED array, and in a study aiming at the detection of transmitters [33], the outer frame of the LED array is always lit up for transmitter detection. This method facilitates the discovery and tracking of transmitters. However, in reserving the area for transmitter discovery, the area for sending data is reduced [30], [31]. In a study that aimed to discover transmitters without an LED area for transmitter discovery [34] the differences between multiple frames are accumulated to discover blinking transmitters. However, a drawback is that even if the differences between multiple frames are accumulated, if the relative positions of the transmitter (LED array) and receiver are not constant, then it may be impossible to discover the signal region accurately because of the displacement of the transmitter between frames [32]. The discovered transmitters are also tracked by optical flow [35], which is realized by tracking feature points other than the transmitter region in continuous frames, but accurate tracking is difficult when there are other moving objects around the transmitter. Therefore, it would be difficult to apply that approach to the proposed system. Instead, in the present paper, we propose an L-shaped region for transmitter discovery, which is described in detail in Section IV-A. This reduced the loss of area for data transmission while making detection and tracking easier. In the aforementioned paper [33] on visible light road-to-vehicle communication, the coding accounts for the image blurring due to the long distance. However, because the communication distance is assumed to be about 10 m in the system proposed herein, we reason that unlike in road-to-vehicle communication the influence of image blurring on the code error rate is small. However, there may be noise due to light source interference between neighboring LEDs in the LED array, so in this paper, we also propose a method for reducing such noise by using a two-dimensional Gaussian distribution On the other hand, VLC has been achieved with a single RGB LED and a LFR camera [36]. Their proposal is interesting, but the receiver side hardware environment is very large. Therefore, it is difficult to apply that system for robot commanding targets of this work.

III. PROPOSED SYSTEM
The proposed system is shown schematically in Fig. 1. A wall-mounted LED array transmits driving signals to a wheeled mobile robot that can be controlled remotely by blinking the LED array from a distance. The camera on the wheeled mobile robot receives the signals and controls the motors of the wheeled robot to make it run according to the received signals. The equipment configuration used in the present study is given in Table 1. The LED array of the transmitter (one type used in this study) was self-made by arranging the small LED matrix shown in Fig. 2 in a 4 × 4 configuration as shown in Fig. 3. Here, as shown in Fig. 2, one LED dot matrix consists of 64 LEDs in an 8 × 8 configuration. This LED array is controlled by an Arduino Uno and sends signals. On the receiver side is a Raspberry Pi 3 Model B, which processes the images from the webcam and controls the robot's motors    as shown in Fig. 4. The specifications of the webcam used (C922 Pro Stream Webcam) are given in Table 2.

IV. TRANSMISSION OF TRAVEL DIRECTION SIGNALS BY LED ARRAY A. ASSIGNING SIGNAL AREAS
An image of one of the LED arrays used in this study is shown in Fig. 5. This actual array comprises 16 LED dot matrices, one of which is shown in Fig. 2. In Fig. 5, the LED dot matrix of column i and row j constituting the LED array is denoted as LED i,j (where i and j are integers); the gray area is the signal area for transmitter detection on the receiver side, and the red area is the driving indication signal area. This arrangement has to be changed when another LED array is used.

B. DETERMINING BLINKING PATTERNS OF LED ARRAY
The receiving section on the receiver side receives signals by assessing the differences between consecutive lighting patterns of the LED array on the transmitter side. Therefore, when sending an arbitrary signal, it is necessary to determine the next lighting pattern by considering the previous lighting pattern. In the signal area for detection, if all lights are on in the previous pattern, then they are all off in the next pattern, and conversely, if all lights are off in the previous pattern, then they are all on in the next pattern. By doing so, an L-shaped signal area for detection always appears as a difference on the receiver side, and this feature is used to detect the transmitter on the receiver side The driving direction signal is transmitted as a 9-bit signal by switching the lighting pattern once. The lighting pattern is represented by the 9-bit binary number {b0b1b2b3b4b5b6b7b8}, where 0 is off and 1 is on, and is assigned one bit at a time as shown in Fig. 7. Table 3 gives the signals obtained on the receiver side by switching the lighting state of the LED dot matrix that constitutes the LED array.
In the case of transmitting a 9-bit binary number D as a driving indication signal, if a 9-bit binary number B is   represented in the previous lighting pattern, then the next lighting pattern B 1 is obtained by a bit-by-bit negative exclusion logical OR of D and B, as shown given by eq. (1).
Finally, Fig. 8(a) shows examples of the lighting patterns of the LED array, with the lights on being red and the lights off being white. In this example, the lighting patterns are shown for transmitting {000000000 (2)}, {110111001(2)}, and {101101011(2)} as driving instruction signals when the initial state of the LED array is all lights off. Fig. 8(b) shows the image obtained of the differences between the lighting patterns in Fig. 8(a).
There are many types of lights in hospitals. But, according to our observations, they do not make similar blinking patterns and most of them do not blink. Therefore, we believe that other lights in the environment do not affect the system. Fig 9 shows the image processing flow on the receiver side. The receiver detects and tracks the transmitter and receives the signal by taking the differences between consecutive frames for the video obtained from the camera. Each process is described in detail below.

A. DIFFERENTIAL PROCESSING
The detection of the transmitter is based on the flashing of the LED array and uses the frame-to-frame difference. Two consecutive RGB frames obtained from the camera are shown in Fig. 10. Each image is converted to grayscale from eq.(2) and the Gaussian filter in eq.(3) is applied. Applying the Gaussian filter removes high-frequency noise in the image [34]. Fig. 11 shows the images obtained by applying the aforementioned processes to the images in Fig. 10 Finally, the binarized image from the differences obtained when the LED array is blinking [35] is shown in Fig. 12. The differences were calculated using eq.(4), and the threshold was determined dynamically using Otsu's method for binarization [37]- [39]. Binarization was performed using the threshold value. In the Otsu binarization method, the methodology is based on intra-class variance (σ 2 w (k)). It searches for the threshold that minimizes the intra-class variance which is defined as a weighted sum of variances of the two classes that is shown as in eq (5).
w 0 and w 1 denote the probabilities of the two classes are separated by a threshold k. Here, σ 2 0 and σ 2 1 denote the variances of the classes. The probability of both classes w 0 (k) and w 1 (k) can be determined from the R bins of the histogram that generates with the pixel values versus their occurrences [39]. Eq. (6) and eq. (7) show the calculation of w 0 (k) and w 1 (k) of eq. (5) respectively.
Here, the minimum value of intra-class variance is the same as the maximum value of inter-class variance. By considering the computational easiness, in this paper inter-class variance (σ 2 b (k)) is applied to automatically decide the threshold. The inter-class variance calculation is shown in eq. 8 and eq. 9.
In the eq. (8) and eq. (9), the mean values of the classes are denoted by the µ 0 (k) and µ 1 (k). Furthermore, the overall mean value of the two classes is denoted by µ to . Their calculations are shown in eq. (10), (11), and (12) respectively.

B. TRANSMITTER DETECTION
Because the LED array always lights up the L-shaped area, the L-shaped difference always appears in the difference VOLUME 9, 2021 FIGURE 11. Gray images. image along with the difference caused by the travel indication signal. By finding the bounding rectangle of the L-shaped detection signal area, the bounding rectangle of the entire LED array can be obtained. To realize this in the receiver, we use the following method, the specific processing flow of which is shown in Fig. 13. First, we find all the smallest bounding rectangles of the white pixel area in the binary difference image. After this, we eliminate the bounding rectangles that are square and whose area is less than the threshold value. In this process, a square is a rectangle with a spectral ratio of 90% or more, and the area threshold is 0.5% of the total area of the binary difference image. The resulting bounding rectangle is shown in Fig. 14(a). The largest one from the remaining bounding rectangles is obtained as the bounding rectangle of the transmitter. The final obtained bounding rectangle is shown in Fig. 14(b). Fig. 15(a) shows a cropped image of the LED array detected from the difference binarized image in Fig. 14(b). Because the LED array of the transmitter is a 4 × 4 LED dot matrix array,  it is divided into a 4 × 4 grid when acquiring the blinking pattern ( Fig. 15(b)). The number of white pixels in each grid of the 3 × 3 area excluding the L-shaped area is counted, and the number determines whether the grid is white or black. The number of white pixels in each grid is counted. The difference between the blinking patterns of the LED array is detected as a binary 1 for white and a binary 0 for black, and a running indication signal is received accordingly.

D. SOURCE INTERFERENCE DENOISING BASED ON GAUSSIAN DISTRIBUTION
It is possible that there is noise in the LED image in Fig. 15 due to the interference of neighboring light sources. Therefore, we reason that the pixels closer to the center of each grid are more likely to adequately represent the state of that grid. Therefore, instead of simply counting the number of white pixels in each grid, we weight the pixels in each grid based on a Gaussian distribution in advance. Fig. 16 shows the weighted image based on the normalized two-dimensional Gaussian distribution, created from (3) with σ = 9.0. Fig. 17 shows the image obtained by multiplying the image in Fig. 15(a) with the weighted image in Fig. 16. For each grid in this image, a weighted count of white pixels is performed. If the number of white pixels is more than 50% of     the total, the signal is set to 1. Fig. 18 shows the final Gaussian weighted image and Fig. 19 shows the result of the count. From this result, {110111010(2)} is obtained as the driving indication signal.

VI. EXPERIMENT
The experiments were conducted to verify the detection rate of driving direction signals with and without weighting by Gaussian distribution to eliminate light source interference noise in the acquisition of driving direction signals. Five types of signals (straight ahead, backward, right turn, left turn, and stop) were transmitted 200 times each at different distances between the transmitter and the receiver, and the detection rate of the driving direction signal for each signal   and their average values were measured. According to the signals received by the receiver, the motor of the mobile robot was controlled to go straight, backward, turn right, turn left, and stop, and detection was considered successful when the mobile robot operated correctly according to the signals.  Fig. 22 shows one of the experimental environments. The mobile robot and wheelchair consisted of a webcam (C922 Pro Stream Webcam), a Raspberry Pi 3 Model B, and two motors. As shown in Fig. 2, the transmitter was fixed at a height of 70 cm, and the receiver (mobile robot) was placed on the floor. The assignment of the five signals to be tested in the experiment is given in Table 4.

B. RESULTS
The results for each signal-receiving rate (detection of pattern on receiver side) according to the distance to the receiver are shown in Fig. 23, here each pattern has been tested 200 times and average results are shown in graph; a graph of the average of the pattern detection rates for the five signals is shown in Fig. 24. These results were obtained both with and without Gaussian weighting. Here, the conventional method is that without Gaussian weighting [40].
As shown in Figs 23 and 24, a signal-receiving rate of 100% was achieved for all five types of signals when weighting was applied. In addition, applying weighting improved the detection rate at all the distances verified. However, when the distance to the transmitter exceeded 8 m, the signal-receiving rate decreased even when weighting was applied to remove the interference noise among light sources. However, we reason that the proposed method of restoring data by weighting based on the Gaussian distribution definitely improved the performance compared with the conventional method of restoring data without using the Gaussian distribution. In this study, we established an L-shaped area of the LED matrix array as a device to easily find the transmitter on the receiver side. In our experiments, we succeeded in finding the LED matrix array on the receiver side 100% of the time when the distance between the camera and the LED matrix array was within 6 m. Furthermore, the LED matrix array was found and tracked in continuous frames, which means that the transmitter can be tracked with high accuracy as long as the distance between the camera and the LED matrix array is within 6 m.
The successful detection results for each pattern are also shown in Figs 25 and 26. In the successful case shown in Fig. 25, the robot performed the specified action without VOLUME 9, 2021 FIGURE 26. Example of signal-receiving situation at receipt of robot-commanding patterns (error case: no signal received, Only L shape was detected).
any problem. The distance between the camera and the LED matrix array was 6 m or less, and the robot could be controlled remotely without any problem. This confirms the effectiveness of the proposed new method for detecting the array and acquiring the blinking pattern, especially for detecting the transmitter (LED matrix array) on the receiver side. On the other hand, as the distance between the camera and the LED matrix array increased beyond 6 m, the number of failures increased as shown in Fig. 26. Possible causes of this failure are (i) the decreasing size of the transmitter in the image of the receiver's camera as the latter moved away from the transmitter and (ii) the increase in noise due to external disturbances. In the future, we would like to consider countermeasures by increasing the resolution of the camera and devising new ways to remove noise.
In addition, we conducted a running experiment using the small robot and wheelchair we made this time. The receiving and processing of the running instruction signals on the receiver side and the motor control of the robot were implemented using small hardware (Raspberry Pi), and we succeeded in implementing the proposed image-processing algorithm in real time. Please clarify slow motion video you are referring to video in the Supplemental Materials for confirmation of the robot remote control operations using the proposed method On the other hand, it was difficult to apply AI models to overcome the image recognition problems of this work, since AI models were difficult to execute in real time on the microcontroller.

VII. CONCLUSION
In this study, we proposed a teleoperation system for a wheeled mobile robot using visible light communication, which affects neither patients nor medical equipment. This was proposed as a countermeasure to the shortage of nurses and so that medical staff can provide remote treatment to patients with infectious diseases. We used a camera for the receiver and an LED array for the transmitter. The images from the camera on the receiver side are processed and an L-shaped area is established with which to find the transmitter when it was blinking in order to make it easier to find and track the transmitter. By detecting the differences between consecutive lighting patterns of the LED array from the receiver side, we made it possible to find and track the transmitter and to receive driving instruction signals. In addition, we proposed a method for eliminating interference noise during signal transmission between the light source and the camera based on a Gaussian distribution, and we demonstrated the effectiveness of this method by experiments. Furthermore, we succeeded in the teleoperation of an actual wheeled robot/chair using the proposed visible light communication.