Research on Conscious Interactive Angle of Pen in 3D Contactless Air-Drawing and Writing

In this paper, a three-dimensional pen interaction technique based on the pen roll angle in normal writing or drawing is proposed. Two experiments are conducted to present the methods. In experiment 1 the range of the pen’s pitch angle, yaw angle and roll angle due to normal writing and drawing unconsciously in 3D space are explored to show that the roll angle is used only small part of the possible range so it could be used in interactive control. In experiment 2, the range of the roll angle, accurate and efficient under conscious interaction are further investigated. Two independent variables, angular width and angular distance are introduced to evaluate the performance of the system. The experiment results show that the available angle range [−135°, −20°] and [20°, 135°] together with an angle resolution W = 10 degrees are the preferred parameter combination. The method in this paper would provide an approach to implement interactive instruction through pen roll and meanwhile do not interfere with normal writing and drawing so as to promote the interaction performance of the system.


I. INTRODUCTION
Pen-based interaction is becoming more and more important as a classic way of interaction with the development of virtual reality and human-computer interaction. There are contact pen interaction and contactless pen interaction, in contact pen interaction the pen is writing on a hard surface just like an electronic pen writing on the electronic whiteboard, while in contactless pen interaction the pen is manipulated in the three-dimensional (3D) air space without contact with any hard surface.
Lyu et al. [1] showed that adding auxiliary channels to input devices could significantly improve the input performance. In the contact pen interaction, the research on the parameters such as tilt angle, rolling angle and azimuth angle of the pen has been quite mature. Tian et al. [2] realized menu selection by using the tilting angle of pen, proved that the probability of wrong selection of 4 to 8 target areas in tilting menu was less than 12 target areas, and concluded that the user's response time and error rate were affected by the size and visual feedback of tilting menu. Xin et al. [3] The associate editor coordinating the review of this manuscript and approving it for publication was Eunil Park . studied the pen azimuth for the first time and verified that in the mode switch design, the error trigger rate of using both inclination angle and azimuth angle to achieve the pen mode switch was lower than that of azimuth angle alone. Bi et al. [4] designed painting and writing tasks to distinguish conscious pen rolling from unconscious pen rolling, and determined the usable angle range of pen rolling, namely, the angle range of the pen that users could consciously and easily roll. The results showed that the range of rolling angle [−10 • , 10 • ] and rolling speed [−30 • /s, 30 • /s] was the behavior of automatic rolling pen.
At present, in the 3D pen interaction with the pen as a data input tool [5], the research mainly focuses on 3D tracking, handwriting recognition, prediction model verification and other aspects and the research on pitch angle, yaw angle, roll angle and other input parameters brought by the pen are inadequate.
This paper studied the effect of the pen's yaw angle, roll angle and pitch angle in the normal writing and drawing process of the pen in 3D space, and studied whether these three attitude angles could be chosen as additional input parameters to improve the interaction performance of the pen in 3D space. We conducted two experiments about the rotation of the 3D pen on the X axis, Y axis and Z axis. Experiment 1 measured the range of use of the pen's pitch angle, yaw angle, and roll angle during traditional drawing and writing to determine the threshold for distinguishing these events from the interactive pen used. Experiment 2 quantified the accurate and timely parameter range of conscious pen rolling. The contributions of this paper included the following: 1) Through the free drawing and writing experiment three unconscious attitude angles were measured, the result showed that the unconscious range of the pitch angle was approximately [−165 • , 165 • ]; the unconscious range of the yaw angle was approximately [−100 • , 100 • ]. Both of them exceeded half of the entire rotation range and unsuitable to be a control parameter in an interactive operation. Only the roll angle, which unconscious range was approximately [−20 • , 20 • ], was suitable to be an conscious input parameter to promote the interaction performance in writing and drawing tasks.
2) The conscious roll angle experiment was conducted to measure and analysis three parameters: the task time, error rate and number of missed target. The results showed that within the angle distance of −135 • < D < −20 • or 20 • < D < 135 • and the target width(resolution) of W > 10 • , the participant could effectively control conscious scrolling and feels ''comfortable'' in an roll interactive task.

II. RELATED WORK
As always, our work was built on the work of others. It mainly included various pen input parameters and the application of 6 DOFs in 3D space. A lot of literature focuses on.

A. 2D CONTACT PEN INTERACTION
A lot of literatures focused on the tilt angle, azimuth angleand rolling angle in 2D contact pen interaction have been widely investigated [1], [6]- [9]. Xin et al. [6] compared pen pressure, pen tilt angle and pen azimuth angle through trajectory stroke task from the perspective of parameter accuracy, and concluded that the effect of pen pressure on interaction performance was better than that of pen tilt angle and azimuth angle. Bi et al. [4] from the university of Toronto, Canada designed painting and writing tasks to distinguish conscious pen scrolling from unconscious pen scrolling, and determined the usable scrolling range of pen that users can roll consciously and easily. The results showed that the range of the rolling angle and the rolling speed was the behavior of the automatic rolling pen. In addition, they showed that the user was comfortable when scrolling the pen in above range for purposeful interactive tasks. Mizobuchi et al. [10] studied the properties of force-based input on a handheld device.Statistical comparisons indicated that subjects differentiated roughly seven input levels within the set of ten force ranges actually used. The results are discussed in terms of the design of appropriate feedback for force input. Miura and Kunifuji [11] used electronic pens to interact with handheld devices. At the same time, a new technology named RodDirect was proposed and applied to different practical programs such as map viewer, dispatcher and game. Meanwhile, it was proved that the rolling electronic pen could be used as a rotary knob to achieve functions such as scaling and rolling in different places.

B. 3D CONTACTLESS PEN INTERACTION
In the 3D contactless pen interaction, some research on the pen's interaction performance had also been studied. Liu et al. used a head-tracking stereoscopic display and a 6 DOFs input stylus to perform 3D tracking tasks and established an interaction model [12].
Certain input devices which were designed for specific tasks that required simultaneous control of multiple degrees of freedom could provide a parallel stream of input. Zhai [13] investigated the positioning and orienting objects in 3D space with a 6 DOFs input device.
Hancock et al. [14] proposed the first method to provide full 3D interaction, which had the advantage of force-based interaction, Sticky tools -complete 6 DOFs operationsallowed person to pick up objects, placed them in other objects, and used these virtual objects as tools. The use of virtual objects as a tool opened up the possibility of actual operation, thus providing more complete functionality to the desktop interface. Suzuki et al. [15] described a technology that used a stylus to enrich the interaction between a user and a computer. The technology allowed the stylus to be manipulated in the air with an accelerometer attached to the stylus. The stylus could be controlled by new operations such as rolling and shaking, or by traditional operations such as tapping. Pfeufer et al. [16] studied the interaction between thumb and pen. The approach engaged the thumb of the device-holding hand, such that the thumb interacted with the touch screen in an indirect manner, thereby complementing the direct input provided by the preferred hand.
Although degrees of freedom of a pen were more or less related, there was evidence that control of them could be separated. Human movement research showed that holding a pen in a regular writing grip ( Fig.1) could be classified as a type of precision grasp, which allowed users to perform delicate tasks [17], [18]. Bi et al. [4] provided some data to the parameters that govern the human ability to control rolling. However, in 3D contact interactive space, the rolling, yaw and pitching information of the pen had rarely been studied, which provided motivation for our current research and explored the 3D pen-based rolling interaction technology. 3D pen attitude angle model was presented in Fig.2.

III. EXPERIMENTAL OVERALL DESIGN
To understand the method of this paper better, the concept of conscious interactive angle is introduced. The main function of the electrical pen is writing words or drawing pictures, the pitch, yaw and roll angles of the pen are inevitably changed when normal writing or drawing happens, here the change of angels is not purposeful so it is called unconscious angle. However, when the pen is used to give interactive instructions just like a menu being selected when the mouse's  key being stroked, here the manipulation of the pen is purposeful and the change of angel is called conscious angle. So the aim of this paper is to explore whether or not the pitch, yaw and roll angles of the pen could be used as the control variables to implement the interactive instruction and have no interference to normal writing or drawing in 3D space.
The overall design and experimental conditions were shown in Fig. 3. Experiment 1 was conducted to estimate the range of the pitch, yaw and roll angle under free or normal writing and drawing in 3D space. Each participant would complete 12 trails (6 writing pages and 6 drawing pictures) as quickly and accurately as possible. To reduce the effect on experiment result by the participants a 10 minutes pilot experiment was conducted to be familiarity with the apparatus before every task of writing and drawing. For every angle among pitch, yaw and roll, the possible range of angle were all 360 degrees, if any one of the three unconscious angles in normal writing and drawing was used only small part of the possible range then it could be used in interactive control. Experiment 2 was conducted to further investigate the range of the roll angle, accurate and efficient under conscious interaction. Because the time interval was about a month between Experiment 1 and Experiment 2, and the participants did not almost use the apparatus so a preliminary experiment was added for the participants to be familiarity with apparatus again. Two independent variables, angular width (W) and angular distance (D) were introduced to evaluate the performance of the system in different width and different distance. Task time, error rate and missed number of target area were used to describe the performance of the system. There were 30 combinations of 10 distance values and 3 width values and 20 trials in every combination, so 600 trails for every participant were needed. At the end of the experiment there was a questionnaire, participants were asked to present the user experience by feeling ''comfortable'' or ''uncomfortable'' for different width and different distance.

IV. EXPERIMENT 1: UNCONSCIOUS PITCHING, YAWING AND ROLLING
A 3D pen generally had an unconscious pen rotation when a participant was drawing and writing. In contrast, pen-based pitch, yaw, and rolling interaction techniques required the participant user to rotate the pen consciously. Suzuki et al. [15] performed a study on avoiding the interference between incidental rolling and conscious rolling. In their work, a 10 • rotation angle in 100 ms was used to distinguish different incidental rolling scenarios from conscious rolling. Bi et al. [4] showed that occasional pen rolling could be reliably identified, which enabled pen rolling to improve pen-based interactions in a reasonable way. In addition, this work could identify a rolling speed between −30 • /s and 30 • /s and a rolling angle between −10 • and 10 • . Here we first researched the range of angles of the pen used for the pitch, yaw, and roll angles of the pen when the pen was rotated unconsciously to avoid interference between conscious pen rotation and unconscious pen rotation.

A. APPARATUS AND ENVIRONMENT
During the experiment, the display was set on a 23.5-inch LCD monitor with a resolution of 1920 × 1080. This experimental software written in C# was run on a 3.6-GHz PC with a Windows 10 system. The experimental apparatus was a 6 DOFs input 3D pen as shown in Fig.4, which was based on the electronic pen [19], incoporated the MPU6515 module in Fig.5 to measure the 6 DOFs of the pen. There was a marker on the pen which could be seen as a reference point and its position could be perceived by the apparatus through IMU information and shown in the screen. The sample rate of the 3D pen was 100Hz. We called the point when the data was uploaded every 10ms a sample point. This attitude acquisition module provided precise attitude detection accuracy. There were two cylindrical buttons on the 3D pen, the first buttton should be hold down for writing and drawing, and the second button would be set to mode switch. Fig.5 shows a circuit  diagram of a transmitting unit and a receiving unit of the electronic pen, in which DK2535 was a microprocessor with wireless function.
To avoid the influence of the distance on the experimental results the participants were asked to sit at a distance of 1m from the computer screen during the procedure of the experiment, the hand was placed above a table with a height of 0.8m. The square activity zoom was limited to 0.5m from the screen and the side length of the square was 0.2 meters (Fig.6).

B. PARTICIPANTS
Eight volunteers (6 females, 2 males), 23-27 years or age participated in the experiments. Four were left-handed, and the others were right-handed. One right-handed participant used a 3D device approximately once a week, and the others used it approximately once a month. They were not told what we were investigating.

C. TASKS
The experiment consisted of two tasks: 1) Free Drawing: The participants drew simple pictures of those being displayed in the LCD screen in the contactless 3D air space.
2) Writing: The participants wrote specific simple sentences of those being displayed in the LCD screen in the contactless 3D air space. Fig.7 showed the scene of experiment 1. A participant, who was seated before screen with a electrical pen in hand, was writing in air according to content of the page on the screen in the 3D space. The upper of the screen showed the content of the page and the lower of the screen showed the handwriting of the participant.

D. DESIGN
In the experiment, the participants completed both of the two tasks, whose operation sequence was balanced using the Latin square. For each task, participants should complete six trails (i.e. six pages of writing and six pages of drawing) and a total of 12 trials for the two tasks. During the tasks, the display page of the pictures was random. Before each task, the participants were given two pages of pictures to familiarize themselves with the task, and the participants were requested to be quick and accurate in completing the task. The experiment lasted 1 hour for each participant.

E. MEASURES
A stroke was defined as the distance that the 3D pen pressed the first button until the first button was bounced. For each sample point the pitch, yaw and roll angle were measured. Additionally, we defined a pitch, yaw and roll angle that rotates clockwise to be positive. For example, a 15-degrees pitch indicated that the 3D pen rotated 15 degrees clockwise on the X-axis, and a yaw 15-degrees indicated that the 3D pen rotated 15-degrees clockwise on the Y-axis; a rolling 15-degrees angle indicated that the 3D pen rotated 15-degrees clockwise on the Z-axis. To estimate the measurement accuracy of the system, the standard deviation of the roll angle was measured in a static state, which was ±0.2 degrees.

F. RESULTS OF EXPERIMENT 1
As shown in Fig.8 and Fig.9, when the participant performed normal writing and drawing, the range of the pitch angle covered almost the entire angle domain, and the range was [−165 • , 165 • ] (Fig.8 (a), Fig.9 (a)). Although the range of the yawing angle did not cover the entire angle domain, the angle range was [−100 • , 100 • ] and exceeded half of It could be seen that in the process of drawing and writing, the unconsciously range for the pitch angle and yaw angle of the pen had occupied most of the possible range and was more likely to cause misoperation when consciously using the these two angles to control the interaction; however the unconscious roll angle of l was relatively small and was possible to study these aspects of the techniques in writing and drawing.

G. DISCUSSION OF EXPERIMENT 1
According to the experimental results, we could describe the characteristics of the pitch, yaw and roll angles of the 3D pens in writing and drawing. The unconscious pitch and yaw were too large to be used as a control parameter in interaction. While the unconscious roll was just [−20 • , 20 • ] and there was 320 • left to be used as the conscious control range of interaction when drawing and writing. The next step was to study the rolling characteristics of the pen.

V. EXPERIMENT 2: CONSCIOUS ROLLING
In the two-dimensional contact interaction, Ren et al. had studied the interaction performance of the scroll angle and determined the angle range that could be used for purposeful interaction. However, in the 3D contactless interaction, the range of the scroll angle was still uncertain. Next, we would study the range of available angles when the scroll angle interacted in 3Dspace.

A. APPARATUS AND PARTICIPANTS
The experimental facilities were the same as in Experiment 1. The eight volunteers who participated in Experiment 1 (6 females, 2 males), 23-27 years of age, participated in Experiment 2.

B. TASK AND PROCEDURE
In this section, on the basis of experiment 1 the sector target selection task was designed to determine the interactive range of the rolling angle by analyzing the time spent in the task and the error rate of operation, and combining with the subjective evaluation.
Sector target selection task: when the pen was in dynamic state the pen was difficult to maintain a fixed angle due to the hand jitter and the inherent error of the system, usually the angle deviation was [1 • , 2 • ]. Each task was set to start from a fixed area, which was called the starting area. Considering the angle deviation, the width of the starting area was set as [−2 • , 2 • ]. The scene of experiment 2 was shown in Fig.10 including a screen and a pen. The projection of pen from tail of pen to nib of pen was displayed in the LCD screen in Fig.11(a) where the gray round part was projection of the tail of the electronic pen, and the light green part was the marker of the pen. When the marker was rolled to the start area, the start area changed from blue to VOLUME 8, 2020 green, which indicated the start of the task. At the same time, the target area appeared and was shown in orange, as shown in Fig.11(b). When the participant's goal was to rotate the marker to the target sector the color of the target area turned black to provide visual feedback to the user, and at the same time, the participant could press the second button of the pen to confirm the selection, as shown in Fig.11(c).The angular width (W) and angular distance (D) of the target were considered as independent variables in the experiment. When the pen started to rotate, the pen's scroll angle would be mapped to an equal angle on the screen. For example, clockwise rotation 15 • corresponded to 15 • on the screen, while counterclockwise rotation 15 • corresponded to −15 • or 345 • on the screen. In the interactive scene, the attitude angle of the electronic pen was independent to the depth of the pen (namely distance from the pen to screen) so that it could be considered that mode switching under different depth would not affect the switching of attitude angle. Experimental methods and procedures: the participants were the same as in experiment 1. Participants were required to complete a 20-minute preliminary experiment in which the participant will complete the selection as accurately and quickly as possible in a way near to reality before the formal experiment. There was a three-minute break between the two groups, which lasted about 120 minutes for each participant. At the end of the experiment, a short questionnaire survey was conducted to collect the subjective opinions of the participants. In the questionnaire, participants are asked to mark each D and W as ''comfortable'' or ''uncomfortable'' based on their feelings in the experiment.
At the end of the experiment, the results of 4800 trails of 8 participants were elaborated from three aspects: the time to complete the selection task, the error rate of operation, and the number of times the target area was missed.

1) TASK TIME
The time taken by the participant to control the marker movement from the starting area to the target area. The average task time of 20 trials of each participant in each combination was taken, then the overall average task time of 8 participants in each combination is calculated, further the average task time of all participants in all combinations was obtained.

2) OPERATIONAL ERROR RATE
The probability that the target was wrongly selected before the marker reached the target region. The average error probability of each participant in 20 trails in each combination was calculated, then the average error probability of 8 participants in each combination was calculated, and the average error probability in all combinations was obtained.
Number of missed target area: the number of missed targets due to failure to confirm the target selection after the marker reached the target area. Counted the number of missed the target area each participant in 20 trails in each combination, then calculated the total average number of missed the target area 8 participants in each combination, further obtained the average number of missed the target area all the participants in all combinations.
As shown in Fig.12, significance analysis was performed to show that there was a significant difference between W and the task time (F(2, 14) = 11.757, p = 0.001), and there was a significant difference between W = 15 • and The post experiment questionnaire survey results was presented in Fig.13. All of the eight participants marked W = 10 • , W = 15 • ''comfort'', but W = 5 • ''uncomfortable.'' Here ''comfortable'' means the participants could control the pen rotate to the target position which could be labeled ''comfortable'' relatively easily because the manipulation only required tiny movements of the thumb and index The participants reported that it was easy for them to rolling to the angles labeled ''comfortable'' because it required only tiny movements of the thumb and index finger. However, they were uncomfortable to roll to the angles labeled as ''uncomfortable''. At 10 degrees and −10 degrees, the participants easily missed the target area. These subjective ratings were consistent with the previously estimated identifiable rotation and available range.

D. EASILY DISCRIMINABLE ROTATION
Anova test showed that D had significant impact on the task time (F(9, 63) = 218.466, p < 0.001), the number of targets missed (F(9, 63) = 88.799, p < 0.001) and the error rate (F(9, 63) = 44.921, p < 0.001). As could be seen from the Fig.13, the task time was approximately symmetric in both clockwise and counterclockwise directions, and there was no significant difference between corresponding angles (i.e., p values were all greater than 0.05). When D = −10 • and D = 10 • , the task time was the shortest, it seemed D = −10 • and D = 10 • was a good choice, but due to the short angular distance, the number of missed target areas and the operation error rate greatly increased, which leads to the reduction of participant's comfort. When the angle distance was greater than 50 • , the task time increased with the increase of angle distance (D). When the angle distance D = 180 • and D = −180 • , the task time was the longest, the error rate was the highest, and the participant's comfort was the worst. Therefore D = −180 • and D = 180 • was not suitable for the participant to use as the angle range for mode switching. Considering that the unconscious roll angle range of pen in writing and drawing given in experiment 1 was [−20 • , 20 • ], the interaction performance between D = −20 • and D = 20 • would be further studied next. The experimental results were shown in Fig.14. The number of targets missed was significantly reduced, and the task time and error rate were basically consistent with the situation when the angular distance was D = −10 • and D = 10 • . Based on these results, a reasonable conclusion was drawn: −135 • < D < −20 • and used 20 • < D < 135 • were the available scope.

VI. CONCLUSION AND FUTURE WORK
In this paper, two experiments were conducted to study the use of the pen's pitch angle, yaw angle and roll angle when performing normal drawing and writing tasks and to study the ability for the participant to control the pen rolling with the hand. Experiment 1 indicated that when we used the pen for drawing and writing, we unconsciously used the pen's pitch, yaw, and roll angles. It could be seen from the results that only the rolling angle was used in a small range [−20 • , 20 • ] so that it could be used for conscious pen interaction technology. Experiment 2 showed that the available roll angle for interaction control was −135 • < D < −20 • and 20 • < D < 135 • with an angle resolution W = 10 degrees.
Although these roll-based interaction techniques are promising, they are still in the early stages of design. They obviously require fine-tuning and formal assessment. We require further research and improvement. HAIYUN WANG received the M.Sc. degree in communication and information system from Jilin University, in 2019. Her main research interest includes ultrasonic localization technology. VOLUME 8, 2020