Development of Finger-Mounted High-Density Pin-Array Haptic Display

The presentation of virtual object shapes using a finger-mounted pin-array haptic display is one of the major topics of research in haptics. If this can be realized, the operability of objects and immersive feeling in the virtual space will be improved. For now, previous studies showed that shape recognition performance using such a pin-array display was far inferior from the performance in the real world using a real object. We considered that both the density of contact points and coverage areas are essential to improve the recognition performance. However, the size of the actuator that pushes each pin was a constraint, and the previously developed display could not have a large contact density and coverage area. This study proposes a novel design of a finger-mounted pin-array display that works around the constraint. We adopted a pneumatic drive because the pneumatic actuator, or air cylinder, can be a simple structure and can be arranged in a dense array. Our developed finger-mounted display has a higher contact point density and a larger coverage area than any other previously developed devices. It covered more than 4 times larger area on fingerpad with denser pin arrangements. An experiment to evaluate the recognition performance with the device was conducted. Participants discriminated 10 kinds of 2D patterned alphabet shapes with only haptic information. The result showed participants could recognize the ten kinds of 2D patterned shape with 93.8% accuracy. Though our participants’ task in the experiment was more difficult, the accuracy was better than previous studies. It suggests the effect of the higher density and the larger size of the coverage.


I. INTRODUCTION
Humans are highly skilled in perceiving objects' shapes even in the absence of visual information. They can sense the geometry of the real object's surface based solely on haptic cues that are applied onto the fingertip. Blindfolded people can allegedly perceive the shape of solid objects with an accuracy of 98% within a few seconds [1]. People also appear capable of distinguishing between twelve solid copies of bell peppers just from touch, with the same accuracy as if they used vision alone [2].
Thus, humans can reasonably be expected to recognize the shape of computer-generated objects in virtual environments if the haptic stimuli are replicated accurately. Researchers have developed pin-array displays to present the distributed haptic pressure stimuli when contacting a virtual object to users. There are two types of pin-array display: grounded-type and wearable-type. Mechanically grounded pin-array displays could present robust haptic cues using The associate editor coordinating the review of this manuscript and approving it for publication was Zheng Chen . grounded forces with users [3]- [5]. However, the groundedtype display can present the object at only limited space. When users move the hand or finger to any position in the space around the users, these devices cannot display shapes outside the limited space. Thus, the workspace is constrained.
Recently, more haptic system designs have started appearing with wearability in mind, and in this context, wearable, pin-array displays have been developed [6], [7]. However, currently, the ability to recognize the shapes of virtual objects when using these wearable devices is worse than when users recognize real objects using a bare finger or hand. For example, a work by [8] evaluated shape recognition performance using a vibrotactile whole hand glove with vibrators. The correct answer ratio was approximately 70%, and the response time was close to 20 s, which was far inferior to the ones (98%) coming from the interaction between the real finger and the real objects, as obtained by [1].
To achieve a higher shape recognition performance, one of the promising approaches is considered to implement a high contact point density to provide richer haptic cues [9]. On the other hand, large contact areas are essential for providing VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ : We adopted pneumatic drive because a pneumatic actuator, or air cylinder, has a simple and slim structure and thus, the pin size instead of the actuator defines the contact point density.
richer haptic cues. Here, we introduce the approaches of previous studies [7], [10]- [14] that have addressed the increase in the contact point density and contact area on wearable pinarrays. Benko et al. [12] used 16 linear servo motor to realize 4 × 4 pin arrays on wearable displays. Their display realized 4.3 mm pin pitch and covered 13 × 13 mm 2 on fingerpad. We refer to the interval between centers of two adjacent holes as pin pitch. Jang et al. [10] utilized piezoelectric actuators instead of the linear servo motor to increase the pin density. Their pin pitch was 2.5 mm and 40 pins were arrayed in 1 dimension. For robust shape recognition, the 2-dimensional array should be required. As compared to these studies, Kim et al. [7] placed the pins along with the curvature of the fingerpad to increase the pin density. Though their display achieved a pin pitch of 1.5 mm, the number of pins remain 32 (4 × 8). It covered 105 × 45 mm 2 area which was only a part of the fingerpad. With these approaches, it is difficult to enlarge the area size of the presentation while keeping the high contact point density. It is because the pins were integrated with the actuators such as motors, and the size of the actuator was a constraint (shown in Fig.1 (a)). On the other hand, recent haptic displays [15]- [17] are increasingly focusing on pneumatics actuation due to its softness and compactness. For example, Wu and Culbertson [16] proposed a sleeve-type device where balloons inside sleeves inflated by pneumatic actuation. Miyakami et al. [17] proposed a balloon-based kinesthetic feedback device. At the timing of gripping virtual objects with index and thumb fingers, the balloons positioned on index finger and thumb inflated and presents users with kinesthetic feedback. Following these studies, there can be an idea of arranging balloons that inflates in an array to provide distributed pressure [15]. However, if the balloon is simply made into an array, it is difficult to increase the contact point density because of the volume of the balloon.
Thus, in contrast to these previous work, this study adopted this pneumatics to drive an air cylinder, which can be a simple and slim structure. Because of this design, the pin size defines the contact point density (shown in Fig. 1 (b)). Our developed the finger-mounted pin-array display had higher contact point density and larger coverage area than any other wearable display (summarized in Table. 1). We evaluated the recognition performance with the developed device. Note that the interest of this study is the realization of the pin-array display with higher density and larger coverage area in a laboratory environment. It is not intended for portable use, and it is clear that it is difficult by our implementation.

II. RELATED WORK A. HUMAN SHAPE RECOGNITION
Humans can recognize the shape of a real object based solely on haptic cues, which comprise kinesthetic and cutaneous cues [1], [2]. Various attempts have been made to disentangle the contributions of the two cues in haptic shape recognition [19]. Kinesthetic mechanoreceptors encode information on the state of muscles, tendons, and joints. Cutaneous mechanoreceptors respond to the deformation of the skin. Both kinesthetic and cutaneous cues are known to be important for shape recognition [20]- [23]. The work in [20] showed that the addition of cutaneous cues to kinesthetic cues significantly improved the recognition of the orientation of a surface. Similarly, recent studies have shown the importance of cutaneous stimuli in addition to kinesthetic stimuli in discerning curvature [24], [25].

B. WEARABLE PIN-ARRAY DISPLAY
To improve the shape recognition performance using a wearable pin-array haptic display, we expected that increasing the contact density and area is important.
On the other hand, regarding wearable pin-array display, there is a problem that the finger can penetrate into virtual objects, which can be never solved, unfortunately. In our previous study [9], we prototyped testbeds that could simulate lower contact point density and finger penetration into the objects in the real world. We investigated the influence of both contact point density and penetration on the shape recognition performance. The result showed that larger contact point density made up for the finger penetration into the object and lead to reduction of recognition time.
There have been some attempts to present dense force distribution with wearable pin-array displays [7], [10], [26]- [28]. Kim et al. [7] developed a wearable display composed of a 4 × 8 pin-array on the fingertip. The diameter of the pins was 0.5 mm, and the pins were arranged in a 1.5 mm interval. The pins moved normally against the skin, and normal indentation was achieved. In these studies, custom displays were built with an increasing contact density. However, we assume that it is challenging to realize increasing the stimulation area and contact point density since the pins were integrated with the actuators, and the size of the actuator created constraints.
On the other hand, the present study paid attention to the pneumatic system, with which we can make a smaller actuator or air cylinder. Thus, we can create a system with a higher density on large areas on the fingertip with a pneumatic system. For example, the work by [18] developed a 5 × 5 array of air jets placed in direct contact with the fingertip and five additional air nozzles that are in direct contact with each side of the finger to produce the lateral force. However, the interval between air jets was 3.9 mm and it was larger than the two-point threshold of the finger.

III. PIN-ARRAY DISPLAY SYSTEM
The pin-array display system developed in this study is composed of finger-mounted pin-array display and air pressure controller.

A. FINGER-MOUNTED PIN-ARRAY DISPLAY
The pin-array display is worn by users' index finger. Pins are driven by air sent from the air pressure controller, and they press the fingertip skin to create a tactile sensation. Fig.3(a) shows the specification of the display that shows the pin arrangements. Users put their index finger at the semicircular dip in the 3D printed resin piece (Clear Resin RS-F2-GPCL-04) where the pins are arranged (shown in Fig.3(b)). It has 128 holes for setting pins, which are arranged in a staggered arrangement of 12 × 10 + 8 at the surface of the dip. Stainless steel pins with a diameter of 0.95 mm were set in the holes. The pitch was set to 1.4 mm, which was determined by the printing accuracy limitations of the 3D printer (Form2, Formlabs). An air tube for sending air was connected to each hole and connected to an air pressure control valve (SMC, VY1B00). The individual air tubes and valves were used for each hole, and thus, the total number of them were 128.
Because the air tube of the device was heavy and total weight was 615 g, the device was attached with a rubber cord from the ceiling so as not to burden the user. The elastic cord was about 3 meters long and the tension was negligible for a few centimeters of expansion and contraction. The users placed their index fingers at the position where the rubber became a natural length and operates the virtual finger.

B. AIR PRESSURE CONTROLLER
The components of the air pressure controller were an air source, a regulator, and a microcontroller (shown in Fig.2).
Whether the air pressure to each pin should be enabled or disabled was determined by a microcomputer. When the pin should be pressured, the air pressure regulator controlled the valves. Since the maximum output of the air pressure controller was 0.6 MPa, each pin could output a maximum force of 0.4 N. In order to quickly compress the air required for the system, two air compressors (RYOBI, ACP-50 and ACP-60) were used as the air sources for the regulator. The update rate and response time of the system was 50Hz and 75ms.
The specification of the system is summarized in Table. 2.

IV. EXPERIMENT
In order to evaluate the effect of enlarged coverage area of the pin-array display, we conducted an experiment on the user's VOLUME 8, 2020 recognition performance on 2D pattered shape. Since prior knowledge about the shapes of the letters of the alphabet is considered to be uniform for all participants, we made the participants recognize the 2D shapes of letters.
A. PARTICIPANTS 10 right-handed participants were recruited (2 females, 8 males), aged from 21 to 25 (mean = 23.3), and participated in this experiment. The University of Electro-Communications Ethics committee approved the data acquisition in this paper, and written informed consent was obtained from all participants.

B. EXPERIMENTAL SETUP
Participants wore a device and a magnetic sensor (POLHEMUS 3SPACE FASTRACK) on the fingertip of the index finger of their dominant hands and sat on a chair ( Fig.4 (a)). The magnetic sensor was used to obtain the users' fingertip positions and orientation information. The pin-array display was fixed to user's finger using velcro ( Fig.4 (b)).
In order to reduce the burden on the participants due to the weight of the device, the device was supported from above with string from the ceiling. The participants' right arms and wrists were supported by the armrest placed on the right side of the chair. Given that the direction of the ground was the z  axis and the other axes were the x and y axes, the participant could easily move his finger horizontally in the xy plane. The ten letters from ''A'' to ''J'' were used as the shape patterns that were recognized by participants. Fig.5 shows the shapes and sizes of the letters used in the experiment. The font used in this experiment was ''Acumic Variable Concept(91)''. The font model was imported and existed in the simulator environment. The simulator checked if the finger's node overlapped the font model along x and y axes and does not care about the z axis.
Fingertip position and orientation information obtained by a magnetic sensor updated at 120Hz, and they were sent to a PC via serial communication from the sensing system. The physical contact interaction between the user's virtual index finger and the virtual shape was simulated. The virtual finger had 128 nodes on the surface of the fingertip. The 128 nodes corresponded to the 128 pins of the pin-array device (Fig.6).
When the virtual finger's surface node overlapped the black area of the character, 0.2MPa air pressure was output to the corresponding pin.
Two different conditions were prepared, and the performances were compared in order to evaluate whether the presentation area of tactile stimuli contributes to the shape recognition performance. One of the conditions is to present stimuli using all of the 128 pins. We call this condition the ''All-pin condition.'' The other condition is to present stimuli using half of all pins. We call this condition the ''Half-pin condition.'' Under the half-pin condition, only the pins arranged at the 6 center rows were actuated, and thus the presentation area was approximately halved. The number of actuated pins was 66( = 6 × 11) under half-pin condition. Fig.7 shows the pins driven under 2 conditions.

C. PROCEDURE
The experiment consisted of a practice phase and a test phase.
In the practice phase, participants touched the letters from ''A'' to ''J'' consecutively, in order to familiarize themselves with the operation of the device. Participants touched each letter for 30 seconds. At this phase, not only the haptic information but also the visual information of the character was presented by the display monitor. The monitor was placed about 1.0 m from the participant. Thus, the total time for touching the alphabet at the practice phase was 300 seconds ( = 30 × 10). At the end of the practice phase, the visual information displayed to participants stopped.
In the test phase, participants did not received visual information and received only haptic information. They wore noise-canceling headphones and the white noise was presented in order to completely shut off the other noise around participants. Letters from ''A'' to ''J'' were presented randomly without visual information. The participants recognized the shapes by resorting only to haptic stimuli from the pin-array display. The participants answered the character after they recognized it. The participants were not informed whether their answer was correct. The time limit for answering was 10 seconds; counting started when the virtual finger hit the virtual character shape.
The number of trials was 6 for each character, and thus the total number of trials per participant was 60. In order to reduce the order effect, the presentation order was determined using the Latin square method. The practice phase and the evaluation phase were performed for each participant under 2 conditions: half-pin and all-pin. The order of the all-pin condition and the half-pin condition was changed across participants and counterbalanced. After all trials were completed, the participants wrote down free-form comments.

D. RESULTS
Firstly, we show the overall results. The left table in Fig.8 shows the total number of correct and incorrect answers for all participants in two different conditions. The right bar graph in Fig.8 shows the proportion of correct answers for two conditions. The proportion of correct answers was 0.938 ( = 563/600) under the all-pin condition and 0.873 ( = 524/600) under the half-pin condition. To determine whether there is a difference in the population proportions of the two conditions, we performed a two-proportion z-test.  The null hypothesis is that the two population proportions are the same. A significant difference was found between two conditions (z = 0.99, p < 0.01).
Next, we show the results for each presented character. Fig.9 shows the proportion of correct answers for two conditions for each presented character. Except for the character of ''E,'' the recognition performance was equal or better under the all-pin condition than under the half-pin condition. In the all-pin condition, participants completely recognized the characters of ''A,'' ''C,'' ''H,'' and ''J.'' On the other hand, the character of ''G'' was the most difficult to recognize in both conditions. Next, we show how the participants made the wrong choices for each presented character. Fig.10 shows the confusion matrix of the all-pin condition. Fig.11 shows the confusion matrix of the half-pin condition. Each row represents the character that was presented to the participants, while each column represents the character that the participants answered. In other words, the value C r,c at a certain row r and column c shows the total number of times that participants answered the character c when they were presented character r. Thus, the cases in the diagonals are those that are classified correctly, while the cases outside the diagonals show the number of times that each possible error occurred. According to these Figures, when ''G'' was presented, it was often mistaken for ''B'' and ''D.'' FIGURE 10. Confusion matrix of the all-pin condition. It represents the number of times participants answered each character compared to the presented character. Each row represents the character that was presented to the participants, while each column represents the character that the participants answered.

E. DISCUSSION
In the all-pin condition, the correct answer rate was 93.8%, even under strict conditions with a time limit of 10 seconds. To compare the results with existing wearable pin-arrays, we extracted the studies that conducted the same type of user-study that identify the shapes of our experiments and summarized them into Table. 3. It should be noted that the characteristics of the shapes to be identified in the task and the number of shapes differed between studies. Among the studies, our study planned one of the most difficult tasks that required a maximum number of distinctions on the shapes. For example, Kim et al. made users discriminate 6 lines which had different width and it was an easier task than ours in terms of the number of comparison targets. It turned out that our task was one of the most difficult, yet the result with our display left the best recognition performance. The result clearly demonstrated the effectiveness of our pin-array design and we consider the reason for the improvement in the performance was the effect of pin pitch and coverage.
According to the result of the two-proportion z-test, we could confirm that the performance was significantly better under the all-pin condition than that under the half-pin condition. It suggested that the coverage area of the fingertip influenced recognition performance. Some participants said that it is easy to discriminate alphabet characters when the larger area was stimulated. Another comment was that it is difficult to imagine a particular character by integrating fragmentary haptic information in the brain when the smaller area was stimulated. These results proved our expectation that a large coverage area with high contact point density is essential.
Next, regarding recognition performance for each character, it became clear that recognition of the character ''G'' had the highest difficulty among all letters since the correct recognition proportion was 58% with the half-pin condition and 75% with the all-pin condition. We consider that the participants misrecognized the shape of ''G'' as ''D'' or ''B'' because it was difficult to judge whether the character was in a closed shape. Still, the double-sized coverage area improved the correct recognition proportion by 16%.
Similarly, in the case of the character ''H,'' the correct recognition proportion was 85% for the half-pin condition, but it was improved to 100% for the all-pin condition. In this way, the recognition of the characters ''A,'' ''C,'' ''H,'' and ''J'' were perfect (100%) under the configuration of this experiment. Thus if we conduct experiments with similar tasks and evaluate the influence of area on performance in the future, we should increase the number of candidate characters or shorten the time limit.

V. LIMITATION AND FUTURE WORK
There remain several limitations of this study. Since the pin can only present pressure in one direction, it cannot express the force in the lateral direction, such as friction. In order to reproduce the tactile sensation closer to reality, it is necessary to present forces in various directions.
In addition, the weight of the air tube could be a burden on the user. In the experiment we conducted, the device was supported by a string from the ceiling, which may constrain free exploration. More free movement of the finger is required when touching the three-dimensional shapes. Therefore, weight reduction of the device is a future issue.
Also, it should be noted that there is a safety issue. In our design and implementation, there is nothing other than the user's index finger to prevent the pins from being pushed out by the air and popping out. If this design is implemented in other places in the future, it is better to design the pins so that they do not pop out. For example, making the bottom of the pin thicker so that it will get caught in the device is one of the solutions.

VI. CONCLUSION
To the best of our knowledge, this study developed the multi-point haptic display that had the highest contact point density and covered the largest area on the fingertip. We conducted an experiment to evaluate the recognition performance with the device. The result showed users could recognize the ten kinds of 2D patterned shape with 93.8% accuracy. One of the future works is to present forces in various directions to represent the force in the lateral direction. KOICHI HIROTA received the B.S. and Ph.D. degrees from the University of Tokyo, Japan, in 1988 and 1994, respectively. He was then an Assistant Professor with the Toyohashi University of Technology, in 1995. In 2000, he was an Associate Professor with the University of Tokyo. He is currently a Professor with the Department of Human Media Systems, Graduate School of Information Systems, The University of Electro-Communications. His research interests include haptic rendering and human interfaces. He received several academic awards.