Perceived Depth and Roughness of Virtual Buttons with Touchscreens

— With the rapidly increasing penetration of touchscreens in various application sectors, more sophisticated and conﬁgurable haptic effects can be rendered on touchscreens (e.g., buttons). In this paper, we presented a design process to instantiate a wide range of vibrotactile stimuli for rendering various virtual buttons on touchscreens. We study the perceived depth and roughness of rendered virtual buttons. There are two stages: the design of the drive signals and the main study. We generated and screened drive signals to render vibrotactile stimuli for virtual buttons through varying envelope shapes, superposition methods, compound waveform composition (CWC) types, durations, and frequencies. The results show that the perceived depth of virtual buttons can be very deep, and the perceived roughness can be very rough around the resonant frequency. Perceived depth and roughness decrease when the frequency increases or decreases from the resonant frequency. A longer duration of vibrotactile stimuli and adding pulse numbers could increase the perceived depth and roughness. Perceived depth and roughness have a similar trend with varying frequencies at a ﬁxed duration.


I. INTRODUCTION
R ECENTLY, touchscreens have been widely used in mobile devices.The simple mechanical structure, attractive look, and rich potentials for visual rendering make touchscreens popular [1].Recent technologies allow touchscreens to replace the physical keypads in mobile devices [1].However, lacking the haptic feedback from the machine would make users not confident in interacting with the touchscreens, which might increase the input delay and error [1].Haptic feedback could solve the problem in touchscreen-based human-machine interactions.
With the advancement of the transducer and haptic technologies, haptic feedback combined with touchscreens is gaining momentum in various sectors.Most smartphones have embedded advanced haptic actuators [2] to provide haptic feedback.For example, the Taptic Engine in recent generations of iPhones (since iPhone 7) can provide haptic feedback to emulate various physical effects [3].The Taptic Engine is a typical wideband Linear Resonant Actuator (LRA), which not only has high energy conversion efficiency at the resonant frequency but also can render a relatively wider bandwidth when compared to traditional LRAs (such as the 0832 LRAs used in Samsung/LG/Huawei phones and 0815 LRAs used in HTC and Google phone) that have large Q values (although the frequency response at a higher frequency is still much lower than f0).The industry confirms that this is the direction to go, given the richer capability to render more complex haptic effects.
Virtual buttons play a crucial role in touchscreen-based systems, as the buttons are one of the essential mechanical elements that have been long used for human-machine interaction.Haptic feedback make the users' input and interactive experience more efficient [2].Touchscreens with haptic feedback can bring even better user experiences than traditional mechanical buttons.The key reason is that, although touchscreens usually have a nondeformable (nearly) rigid surface, haptic feedback can potentially render a wide range of configurable sensations for various scenarios.In comparison, traditional mechanical buttons can only render a fixed and predefined sensation.Hence, it is crucial to understand how various sensations can be rendered.
For mimicking physical buttons on touchscreens of mobile devices with haptic stimuli, Liu et al. [4] mentioned that it was possible to emulate the feel of a button or key with 1-2 mm travel.Tan et al. [5] estimated human fingertip position resolution to be 2.2 mm during active free movements from a series of finger joint-angle discrimination thresholds.When the index finger presses on a solid surface, the fingertip tissues can yield up to about 2 mm [5].Since human cannot sense such a position change at the fingertip during active moments in free space, it might be possible to create the illusion of a virtual key yielding 1-2 mm under the fingertip instead of the fingertip being compressed by the same amount [4].
According to this perceptual cue, some researchers also tried to mimic the real effect of physical buttons on touchscreens.Kim and Lee [6] and Sadia et al. [7] designed haptic stimuli for virtual buttons on touchscreens.The research results showed that the designed virtual buttons with haptic stimuli were realistic and distinctive.Participants were able to associate different virtual buttons with their physical counterparts.These researches demonstrate that providing a realistic response of a physical button on the touchscreen with haptic feedback is possible.To be more specific, like physical buttons, users can feel an illusion of vertical travel when pressing a virtual button with specific haptic feedback on touchscreens.
In this study, we present virtual buttons on touchscreens with visual information and vibrotactile stimuli together.
Digital Object Identifier 10.1109/TOH.2021.3126609 When users press the virtual buttons on the touchscreen, vision and touch both provide information for estimating the virtual buttons [8].Visual information is helpful when judging size, shape, or position [8].In this study, the gray squares (default settings of buttons from Android Studio) on the touchscreen help show a button-like image that could present an illusion of a button at first sight.We amplify the high-frequency tactile sensation using vibratory actuators to make the perception clearly affected by haptics.
The dimensions of virtual buttons with haptic feedback on touchscreens are not discussed yet.We try to mimic physical buttons on touchscreens, so we consider the dimensions of the physical buttons.With a systematic user study, Liu et al. [4] have found, 'shallow -deep' and 'rough -smooth' are two perceptual dimensions of manual button clicks when focusing the tactile sensation associated with the vertical travel of the buttons and keys.Sadia et al. [7] also revealed that 'roughsmooth' was a key perceptual dimension when participants rated the virtual push button with vibrotactile stimuli similar to its physical counterpart.
This study focuses on exploring the perceived depth and roughness of virtual buttons with vibrotactile stimuli rendered on touchscreens.We use a smartphone that contained a typical embedded wideband LRA (type 1040, available from many different vendors such as MPlus, Jahwa, Jinlong, etc.).A wideband LRA can convert wideband drive signals into vibrotactile stimuli on touchscreens.It can also present audio stimuli at the same time.
The study is split into two stages: (1) the design of the drive signals and (2) the main study.We generated drive signals for virtual buttons through varying envelope shapes, superposition methods, and CWC types.By comparing, analyzing, and screening the vibrotactile stimuli, we selected the physical parameters and determined their values that significantly impact the perceived differences of virtual buttons.
In the main study, we applied varying frequencies, durations, and CWC types to enrich the haptic effects of virtual buttons.We conducted two tasks to explore how the virtual buttons can be perceived with a different depth and roughness through manipulating the parameters of drive signals generation described in a mathematically formulated general framework.

A. Application Context of Virtual Buttons on Touchscreen With Vibrotactile Stimuli
One typical application sector is the mobile device.Users' performance when interacting with a touchscreen with haptic feedback has been explored for mobile devices.For the text entry task [9], [10], [11], and the task path-finding task [2], users' input accuracy has improved with tactile feedback on the touchscreen.Meanwhile, the tactile feedback makes the sensation of virtual buttons close to that of a real physical button [9].For users' preference evaluation, Part et al. [1] and Pakkanen et al. [12] evaluated the users' preference of virtual buttons with haptic stimuli.They revealed design guidelines for realistic and favorable virtual buttons with tactile feedback.

B. Design Methods of Haptic Stimuli for Virtual Buttons on Touchscreens
A common way to generate haptic stimuli for virtual buttons is to set physical parameters of haptic stimuli according to a "feels like" sensation.Researchers [10] and [13] directly modulated frequency, duration, and envelope shapes of haptic signals for a touchdown event as a feeling of pressing a button.Others [9] modulated frequency and envelope shapes for a button's pushing and releasing stage separately to create a "click" sensation on the touchscreen similar to a physical button.
Another way is to record objective signals when pressing a physical button and then modulate haptic signals according to the recorded signals to make users have a similar sensation when pressing a virtual button.Kim and Lee [6] provided a designed virtual button with haptic feedback based on the force-displacement curves of a physical button.Researchers recorded the Jump, Slope, and Bottom-out sections of a displacement curve.They mapped the displacement with vibrotactile stimuli by implementing a similar friction grain model.Park et al. [1] measured accelerations of real button clicks to collect the time and vibration magnitude of the pushing and releasing stage.And they got an intrinsic frequency through Fast Fourier Transform, which transformed the time-domain accelerations into the frequency domain [1].The vibrotactile stimuli of virtual buttons were rendered on touchscreens according to the recorded signals of real button clicks.Sadia et al. [7] recorded and analyzed the force, acceleration, and voltage data of three physical buttons: latch, toggle, and pushbuttons.Vibrotactile stimuli were generated for each button based on the recorded data.
Researchers generate haptic stimuli with varying physical parameters of signals for a large set of virtual buttons.In [14], the rise time and the displacement amplitude were two main parameters of the vibrotactile stimuli, controlled by the driving voltage and the current of the piezo actuator.Nishino et al. [15] applied three main parameters for desired tactile effects: vibration strength, activation time length, and vibration type.Park et al. [1] used three main parameters for desired tactile effects: vibration strength, activation time length, and vibration type.Pakkanen et al. [12] generated haptic stimuli with various pulse shapes and displacement amplitudes to explore virtual buttons' most pleasant haptic stimuli.Besides, the haptics library of Texas instrument [16] presents a list of haptic signals with different amplitude modulation methods and different CWC types, which can be used for virtual buttons.

C. Research Gaps
In summary, for the application context, most virtual buttons with haptic feedback were designed for enhancing users' performance in a specific context (e.g., in a smartphone) for a specific task.Also, researchers tried to generate virtual buttons with haptic stimuli which have a similar sensation as pressing a real physical button and tried to explore the most pleasant haptic stimuli for a virtual button on touchscreens.
However, there is a gap in the exploration of generating a larger set of virtual buttons with different perceived depth and roughness.Different effects with various sensations for virtual buttons are essential because they can be applied in a different context for different applications.These contexts and applications refer to various scenarios when a button needs to be used with different haptic effects.For example, safety crucial virtual buttons in automotive applications should give strong/ deep/rough haptic feedback to the user.While for dial-pad for mobile phone use cases, buttons should give soft/gentle/shallow haptic feedback to the user.
For generation methods, most researchers tried to mimic the haptic effects of physical buttons for touchscreens, so they set specific values of standard physical parameters like frequency and duration for haptic stimuli.Besides, the haptics library of Texas instrument [16] presents a list of haptic effects with different amplitude modulation methods and different CWC types, which can be used for buttons.
However, there is no design guidance process provided.It lacks the discussion of specific sensations as perceived depth and roughness.We design the drive signals aiming at reaching different perceived depth and roughness of virtual buttons on touchscreens.
Hence, there is a gap in a general design process and the experiment-driven understanding of various parameters and perceived sensations.In our study, we will discuss what sensations can be reached with different haptic stimuli.As Liu et al. [4] also suggested, it was a trend to explore the physical parameters corresponding to the perceptual dimensions and design virtual buttons using the relevant parameters.
In this study, we use a smartphone that has an intrinsic mechanical structure embedding a wideband LRA.We aim to generate a large set of drive signals with varying physical parameters and convert them into vibrotactile stimuli for virtual buttons with different perceived depth and roughness.

A. Prototype
The experiment device is a special version of the LG V30 smartphone.The phone contains a typical wide-band LRA motor (MPlus 1040) which can convert drive signals into vibrotactile stimuli.The rework of the special LG V30 phone had the following key elements: the original 1040 LRA was replaced by a wider band 1040 LRA with a lower Q value.The original driver circuit was replaced by a new driver based on a class-D audio amplifier (similar to that in iPhone).The haptic signal was then generated by the audio amplifier and its associated software.When the tested Android APP needs to trigger a haptic effect, an audio-like waveform is sent to the audio amplifier to drive the aforementioned LRA in the phone.Low-level hardware and software have been modified to enable this study: Although the standard Android operating system supported simple haptic interface APIs that could only render limited effects, we have modified the underlying software so that audio-like haptics signals could be used to render any vibrotactile stimuli.
A boosted (from 3.6V battery voltage to 10V) class-D audio amplifier was used (similar to the latest iPhone generations from iPhone 7) for the reproduction of the above audio-like haptics drive signal so that the wideband LRA was capable of reproducing the intended amplitudes of the intended frequencies in the study.The "Boosted" audio amplifier could boost the battery output voltage to 10V maximum and then drive the LRA.So, although the battery was 3.84V, 3300mAh, when measuring the voltage amplified and applied on the LRA, up to 10V could be measured (in fact, only for peak signals).In this way, the LRA could be driven with much higher peak power to render sharp effects without problems.iPhone Taptic Engine follows the same principle as well.ESI U24XL was used in the recording path for the vibration signal, but not the reproduction path.We strictly followed a realistic scenario of practical usage for haptic rendering, so the phone's internal battery was used.For the recording of vibration signals, we used the power supplied by the USB ports of the testing PC.According to [6], we used an accelerometer (TI DRV-ACC16-EVM, with three axes) to measure the frequency response of the actuator (Fig. 1).The device resonates at 160Hz.
Drive signals were generated through MATLAB R2018a with a set of physical parameters listed in Table I.We developed an interface presenting all trials of virtual buttons through Android Studio (API: 25) (Fig. 2).To control the effects from the visual information, we set all the virtual buttons in the same shape, the same size, and the same color -grey.For animations, we applied the default setting of buttons in Android Studio when pressing and releasing virtual buttons.

B. The Recording Method of the Acceleration on Touchscreens
An accelerometer (TI DRV-ACC16-EVM, with three axes) and an audio input device (U24XL) were employed to record the acceleration of virtual buttons on the touchscreen (Fig. 3).We did not use the internal accelerometer since its location is unknown and different for every phone.Applying the internal accelerometer would make experiments less repeatable.And its maximum sampling frequency was too low (around 200Hz with the fastest mode), while we were sampling at a much higher frequency.
TI DRV-ACC16-EVM tool converted the physical acceleration to analog signals.The analog signal was then amplified and converted to a digital signal by ESI U24 XL.The digital signal was transmitted via USB cable to the PC and then accessed by MATLAB via the driver for ESI U24 XL.
Displaying and processing of the acquired signal were done by MATLAB.The smartphone was placed on a foam pat.This was a typical way in the industry to measure.The material property of the foam was optimized so that low-frequency vibration was minimally damped.
The small accelerometer board was attached to the back of the smartphone with transparent adhesive tape to make it stable when recording the acceleration (Fig. 3(a)).The vibration measurement was done on one single point on the phone: location of the home key.This location was the most interesting one because it was where the thumb and palm feel the vibration.When measuring the vibration, only the sensor part of the TI DRV-ACC16-EVM tool was attached to the phone using Blu Tack, which did not show differences compared to the glue-based attachment.
We connected the audio input device as an input to the Zaxis test points on the measurement board.The audio input device was connected to Audacity on a computer to display and record the acceleration (Fig. 3(b)).
The average acceleration on Z-axis was recorded as audio with a sampling frequency of f ¼ 44100Hz.Accelerations of virtual buttons were recorded several times, and we selected the most stable sample for further analysis.

IV. DESIGN OF THE DRIVE SIGNALS
In this part, we first provide a design process and related functions of drive signals for virtual buttons on touchscreens.Then, we apply the design process and choose parameters for the main study.

A. Design Process
This part provides a design process of drive signals for virtual buttons on touchscreens (Fig. 4).

TABLE I PHYSICAL PARAMETERS OF DRIVE SIGNALS
Duration (t 0 ) refers to the duration of an elementary pulse (Fig. 5).Envelope shapes are for the elementary pulse.Parameters Ã we considered in the design of the drive signals.
Parameters ÃÃ we considered in the main study.
the frequency and the duration and modulated the elementary pulse with different envelope shapes.This is a general amplitude modulation method for the signal design like [17] and [18] did before.
The second step (S2) is to design audio effects.The LRA applied in this study could convert the drive signal into haptic and audio stimuli simultaneously, the minor changes of the waveform show a significant difference in the physical effects on the sound level.Though we only test stimuli on a haptic level in this study, putting this step in our design process is practical for future research.
We consider the superposition of signals in this step (S2).The superposition of different signals present expected vibrotactile and audio effects simultaneously.The hearing sensitivity of humans is growing from 20Hz to 2000Hz [19] and [20], while fingertips are sensitive to a frequency from about 50Hz to 1000 Hz for the vibration/acceleration [21].Landstr€ om et al. [22] show that the levels of frequencies below 250 Hz often do not reach the hearing perception threshold.Thus, people are more sensitive to a higher frequency on a sound level.
The third step (S3) is to composite the compound waveforms to reach expected physical effects.
For drive signals design, we firstly need and elementary pulse that provides elementary effects.We propose the following general expression for the elementary pulse: AðtÞ is a general function for generating an elementary pulse.Multiple frequencies can be used at the same time to render complex effects.f k is the possible frequency of drive signals in Hz, t 0 is the duration of an elementary pulse, n is the number of drive signals for superposition without considering the noise signal.uðtÞ is a step function.O k ðtÞ is the envelope function that is used to modulate drive signals.c Á RðtÞ is a function that can generate random noise, and c is a coefficient that determines the amplitude of the noise generated by RðtÞ.The noise signal is also optional to render more sophisticated sensations associated with real-life buttons, especially mimicking the sound of physical buttons.

B. Parameters
In this section, we apply the design process and choose related parameters.
Although we only test stimuli on a haptic level in the main study, we still consider designing audio effects -the second step (S2) here.The reasons are as follows: 1) The design process and AðtÞ are proposed based on multimodal virtual buttons.We will apply the design process and explain how each step works and choose the parameters we need.2) As the LRA applied in this study could convert the drive signal into vibrotactile and audio stimuli simultaneously, we need to make sure the audio stimuli that the drive signals provided are as a physical button.3) We want to show whether the second step of the design process proposed in this study is efficient by studying if the superposition methods significantly affect the expected vibrotactile stimuli based on our system.
We chose the sinewave as the waveform since Dabic et al. [23] revealed that there was no role in the waveform for perceived differences of vibrotactile feedback on touchscreens.
We considered the frequency and the duration in the later main study.Dabic et al. [23] have demonstrated the frequency is the most salient parameter for the perceived differences, and the duration also impacts vibrotactile signals perception when vibrotactile feedbacks are short.So, we only considered other unclear parameters in this section.For all stimuli in this section, the baseband frequency was fixed at 150Hz, and the duration of the elementary pulse was fixed at 0.01s (t 0 ¼ 0:01s).
Envelope shapes, superpositions, and CWC types are abstract parameters.In this section, we chose and discussed three envelope shapes, three superposition methods, and three CWC types (Table I).Nine typical drive signals are in Table II.Envelope shapes and superpositions are applied for generating the elementary pulse (Fig. 5).Expected physical effects (perceived depth and roughness) are reached by CWC types (Fig. 5).We explain how to choose parameter for the main study as follows: 1) Envelope Shapes: We chose envelope shapes according to the characteristics of the vibration of buttons and the advantages of the LRA applied in this study.We considered three types of envelope shapes (Table I).Sadia et al. [7] recorded the actual acceleration of pressing a physical push button.There had time before and after the peak point.So, we applied the triangle as one type of envelope shape to mimic the actual acceleration.The LRA could start vibrating and reach the peak point in a very short time, so we applied a ramp-down envelope shape to make use of the actuator.The square shape was applied to compare with the other two envelope shapes.
The functions for envelope shapes are as follows: is the envelope function used to modulate the elementary pulse.It is described in (2) when the envelope shape is ramp down, described in (3) when it is a triangle, described in (4) when it is a square (see envelope shapes in Table I).
2) Superposition Methods: Superposition methods in the design process are mainly for the audio of the virtual button (as mentioned in section IV -A).We would not test the effect of audio in this study, so we applied superposition methods according to researchers' experiences to illustrate the second step of the design process is efficient based on our system.For the superposition of signals, we applied three methods in this study.
In the first method (A), we applied a simple and elementary effect like other studies did ( [17] and [18]).We did not conduct additivity or homogeneity in the superposition.
In the second method (B), we applied a high frequency to the drive signal to show a different effect on the audio level.We would not test the effect of high frequency on an audio level in this study, and we just need a high frequency to illustrate this superposition type.So, we applied a sinusoidal signal with a high frequency (400Hz and 2200Hz, the amplitude was set as 0.3 to make a less audio cut for the drive signal) according to researchers' experiences and a sinusoidal signal for low frequency (150Hz).So, users would feel the vibration of 150Hz and hear out the sounds of 400Hz and 2200Hz.
In the third method (C), a sinusoidal signal and a white noise were applied for the superposition, as we wanted to apply noise to mimic the sound of physical buttons.Detailed values of each superposition method are in Table II.Fig. 5 shows the three superposition methods.
3) CWC Types: We designed CWC types for expected physical effects (perceived depth and roughness in this study).Different CWC types of drive signals in this study are based on the combination of the elementary pulse.
Besides the elementary pulse, we provided two other CWC types to influence virtual buttons' perceived depth and roughness.
We assumed that adding the duration of the stimulus would increase the perceived depth of virtual buttons, as a longer duration provides a longer response.Meanwhile, a longer response of vibration leads to a different perception when the frequency is fixed [23].So, we assumed that this could increase perceived roughness at the same time.We added a continuous decreasing signal to the elementary pulse to mimic the elastic effect of physical buttons.
We assumed that adding pulse numbers would enhance the perceived roughness of virtual buttons, as more than one pulse could provide an uneven and grainy effect.We added another pulse signal here.Meanwhile, adding another pulse adds the duration of the signal.We assumed that this could increase the perceived depth at the same time.Together with the elementary pulse, we have three CWC types in this study.
In Fig. 5, the drive signal in Type I (T 1 ) contains one elementary pulse; Type II (T 2 ) contains an elementary pulse with a fading out signal; Type III (T 3 ) contains two elementary pulses.Using mathematical language, T 1 , T 2 and T 3 for CWC types are determined by "CWC types" are compound waveform composition types.Recorded accelerations are in Fig. 6.
where MðtÞ is the elementary pulse, it could be AðtÞ with different values.T is the period before the second pulse is activated (Type III in Fig. 5).We do not test the preferable effects of T in this study.We just need stimuli to mimic an uneven and grainy effect.We set T ¼ 0:04s by iterative selection [12] according to the researcher's experiences.Hale and Stanney [24] showed that for interstimulus interval, 5.5ms could be used to perceive separate stimuli.The output interstimulus interval was more than 5.5ms when T ¼ 0:04s.The separate stimuli could be applied to mimic an uneven and grainy effect.In this study, EðtÞ is a part of an exponential function of the envelope shape to modulate stimuli, which is determined by where (a,b) is the range of input argument.The function represents the envelope of the modulated signal.ðb À aÞ is the extra duration adding to the elementary pulse.We do not test the preferable effects of a, b, or k in this study.We just need an envelope shape with a low amplitude with a trend of fading out to mimic the elastic effect of physical buttons.So we set a ¼ À2:82, b ¼ À2:92, k ¼ 20 by iterative selection [12].
Other detailed values are in Table I, Table II, and Table III.
For presenting virtual buttons on touchscreens, Pakkanen et al. [12] showed that users preferred Simple design which mean the same stimuli should be used whether moving towards or away from the virtual button.So, the drive signals of pressing and releasing stage were the same in this study.
The accelerations of typical vibrotactile stimuli converted from drive signals were recorded and presented in Fig. 6.

C. Screening Parameters for the Main Study
This section aims to decrease parameters for the main study since all combinations (3 Ã 3 Ã 3 Ã 5 Ã 2 ¼ 270) of the five parameters are too many to test (Table I).Meanwhile, different combinations may reach a similar effect.So, we compare, analyze, and screen parameters for the main study.
As frequency and duration would be tested in the main study (mentioned in IV-B), we only discussed the other three parameters in this section.
1) Envelope Shapes: Fig. 6 shows that recorded accelerations in Type I and Type III are similar, respectively, no matter which envelope shape is applied for the drive signals.As the duration of one pulse was too short, the fine envelope shape could not be displayed.
In Type II, the different envelope shapes were applied on the short pulse.The envelopes of the adding fading out signals were the same (Fig. 5).So, the envelope of the recorded accelerations in Type II are also similar.
We only need one envelope shape when the duration is such short.Maybe the envelope shape would lead to different accelerations when the duration was longer.We would not discuss that situation in this study.
The square envelope shape was excluded because of an amplitude cut of the drive signals when applying different superposition methods.Finally, we choose a decaying ramp envelope for the elementary pulse (see Type I in Fig. 5) for the main study.
2) Superposition Methods: Fig. 6 shows that in Type I and Type III, the recorded accelerations are similar and are not affected by superposition methods significantly when regarding haptic effects as a single interaction modality.In Type II, the small differences of three recorded accelerations would not significantly affect the perceived differences of vibrotactile stimuli.So, users could feel the vibration of low frequency and hear out the sound of high frequency or white noise.The second step in the design process is efficient based on our system.
On the audio level, we excluded superposition methods B and C. The stimuli were not like physical buttons on the audio level when the CWC type was Type II, so we chose superposition method A for the elementary pulse (see Type I (A) in Fig. 5).
3) CWC Types: Fig. 6 show the recorded accelerations of three CWC types are different.We take all three CWC types into the main study for further analysis

V. MAIN STUDY
The research aim of the main study is to explore the perceived depth and roughness of virtual buttons with vibrotactile stimuli on the touchscreen.

A. Research Questions
There are two tasks in this part.In the first task, frequency and the duration of the elementary pulse are the two test factors.The research questions are if the perceived depth and roughness of virtual buttons on touchscreens increased when the frequency is closer to the resonant value of the actuator.And if a longer duration of the elementary pulse increases the perceived depth and roughness of virtual buttons on touchscreens.In the second task, the research questions are if adding the duration of the stimulus could increase the perceived depth and roughness of virtual buttons on touchscreens.And if adding pulse numbers could increase perceived depth and roughness of virtual button on touchscreens.
For frequency, the actuator resonates at 160Hz (Fig. 1).To protect the actuator, we chose 150Hz rather than the resonant frequency to present a strong vibration.We regarded the resonant frequency as the center.Besides 150Hz, we needed some frequencies away from the resonant frequency.Hatzfeld and Kern [21] have demonstrated that the frequency of good sensitivity on fingertips for vibration starts around 50Hz.According to Fig. 1, we chose 60Hz, 90Hz, 300Hz, and 450Hz to present weaker vibration than 150Hz.
For the duration, Dabic et al. [23] have shown that the duration impacts perceived differences of short vibrotactile feedback.The difference between the long and short output duration is from 20ms to 70ms in [23].In our study, we applied 0.01s and 0.03s as two values of the drive signal.The difference in output duration (Fig. 7) is more than 20ms at each frequency.So, these two durations are distinguishable and can be applied as [23] did.
There were three groups of virtual buttons based on three CWC types.In Table III and Fig. 7, virtual buttons in Type I include button M1 to button M10.Virtual buttons in Type II include button M11 to button M20, while button M21 to button M30 are in Type III.The recorded accelerations of virtual buttons are in Fig. 7.

C. Methods
1) Participants: Twenty participants (ten males and ten females) aged from 23 to 35 participated in this study.All participants have no constraints of sensing touch, according to their report.Participants wear noise-canceling headphones playing white noise, which has no pitch or rhythm to block out the sound effects of vibrotactile stimuli (Fig. 8).
2) Questionnaires: A questionnaire was created to collect data from participants.Two parts were aiming at two tasks in the questionnaire.In the first part, the questionnaire was set according to Liu et al. [4].Questions about perceived depth and roughness were constructed with a 10-point Likert scale that ranged from 1 (very shallow) to 10 (very deep) and from 1 (very smooth) to 10 (very rough).In the second part, questions about perceived depth and roughness were constructed with a ranking way, using ">" to show the relations of perceived depth and roughness among specific virtual buttons.The questionnaire was displayed on the computer.
3) Procedures: There were two tasks in this study.The first task was to compare and evaluate the perceived depth and roughness of three types of virtual buttons (Table III and Fig. 7: Type I, Type II, Type III).Three types of virtual buttons did not affect each other, and participants tested three types in a randomized order.In each type, the order of 10 stimuli was randomized at first, which meant button 1 did not refer to M1.In each type of virtual button, the task was to press and compare virtual buttons within each group.It was considered that randomizing the order of stimuli in each type for each participant was not needed [12].Thus, the stimuli in each group were in a randomized order but the same for all participants.
Participants firstly tried each virtual button in one type for a training session to feel possible perceived depth and roughness since perceived depth and roughness were relative instead of absolute.The order of sensing the perceived depth and roughness was counterbalanced for each participant, which meant one participant tested the perceived depth first while the next participant tested the perceived roughness first.II) through an LRA.The frequency of drive signals is 150Hz, the duration of the elementary pulse is 0.01s.Above are the accelerations of the pressing stage of each virtual button.
The second task was a ranking task where participants were asked to rank the perceived depth and roughness of virtual buttons.There were three virtual buttons in each group, with ten groups of virtual buttons in total in this task (Table III: group 1 to group 10).One group had three CWC types.The frequency and duration of the elementary pulse in each group were fixed.The task was to press and compare virtual buttons within each group, so it was considered that randomizing the order in each group was not needed [12].All the randomized orders in this study were delivered to each participant before testing on the questionnaire.Participants followed the order and the number of virtual buttons they got and felt the virtual buttons.The order was obtained by the random function in Python.

D. Results
In the first task, we used SPSS 23.0 to conduct a two-way repeated-measures ANOVA firstly.Then, we applied Bonferroni analysis for the post hoc test to explore the trend of the perceived depth and roughness with varying frequencies.Finally, we conducted a Spearman correlation analysis to check the correlation between perceived depth and roughness.Two fixed durations of the elementary pulse (0.01s and 0.03s) were tested separately.
Pairwise comparisons were conducted between constructive frequencies.We only considered two neighboring frequencies as a pair, resulting in four frequency pairs (pair 1: 60Hz-90Hz, pair 2: 90Hz-150Hz, pair 3: 150Hz-300Hz, pair 4: 300Hz-450Hz).The results are shown in Fig. 9. Interaction effects were only observed between frequency and duration of the elementary pulse in Type I, F(2.957,112.354)¼ 3.744, p<0.05.The duration of the elementary pulse had significant effects on perceived depth and roughness only in Type I, F(1,38) ¼ 11.170, p<0.05 for perceived depth, F(1,38) ¼ 9.313, p<0.05 for perceived roughness.The frequency had significant effects on perceived depth and roughness in all pairs of three types (p<0.05).
For the post hoc test, results showed that there were no significant differences for perceived depth and roughness between 60Hz and 90Hz at 0.03s (p>0.05) in Type I, between 300Hz and 450Hz at both 0.01s and 0.03s (p>0.05) in Type II, between 60Hz and 90Hz at both 0.01s and 0.03s (p>0.05), between 300Hz and 450Hz at 0.03s (p>0.05) in Type III;.No significant effects were found between 60Hz and 90Hz at 0.01s for perceived roughness in three types.Significant differences for perceived depth and roughness could be observed in the rest pairs (p<0.05).
In the second task, Friedman's rank tests and Wilcoxon signed-ranks tests were performed on the data in the ranking task.A Friedman test indicated a significant main effect of three CWC types in all groups (p<0.05).The detailed results for the perceived depth and roughness were in Table IV.
We conducted a Spearman correlation analysis to check the correlation between perceived depth and roughness.The detailed results were in Table V.The correlation between perceived depth and roughness was not always significant, but it was significant at most frequencies and durations of the elementary pulse.

E. Discussion
The two-way repeated-measures ANOVA indicates that frequency plays a dominant role in perceiving the perceived depth and roughness of virtual buttons on the touchscreen.Perceived depth is the deepest, and perceived roughness is the roughest when the frequency is at 150Hz in this study as the recorded acceleration amplitude is significantly higher than those at other frequencies (Fig. 7) when the frequency is almost at the resonant frequency.The curves of equal perceived intensity at the fingertip [21] show that the perceived intensity at 150Hz is not weaker than the rest frequencies applied in this study when the acceleration amplitude is fixed.The acceleration amplitude at 150Hz was actually significantly higher than the rest frequencies applied in this study.This result suggested the perceived depth and roughness of virtual buttons on touchscreens positively correlated with perceived intensity of vibrotactile stimuli.The strong perceived intensity made users feel that the perceived depth and roughness were very deep and rough.So, there might be just one dimension of perception for virtual buttons on touchscreens.Fig. 7 shows the recorded acceleration amplitudes are similar when the frequencies are at 60Hz, 90Hz, 300Hz, and 450Hz.So, the frequency is the main factor affecting users' perception when the frequency is away from the resonant frequency in this study.We regarded the resonant frequency as the center.In each CWC type, the perceived depth and roughness at 90Hz were not shallower or smoother than those at 60Hz.The perceived depth and roughness at 300Hz were not shallower or smoother than those at 450Hz.So, Perceived depth and roughness decrease when the frequency increases or decreases from the resonant frequency in this study.
Fig. 9 indicates that the trend of perceived depth and roughness are similar when the frequency and the duration of the elementary pulse change.Table V shows a positive correlation between perceived depth and roughness at most frequencies and durations.This result also indicates that there may be just one dimension of perception for virtual buttons on touchscreens.
A longer duration increases the perceived depth and roughness of pulse signals at a fixed frequency.Those longer signals make users feel a longer response of haptic feedback during the vertical travel when pressing a button.This behavior gives them a feeling of deeper and rougher virtual buttons.Thus, ranking tasks indicate that Type II and Type III make a deeper and rougher sensation of virtual buttons on touchscreens than Type I.However, for We used "¼" to represent the relations between two virtual buttons when there were no significant differences between them; We used ">" to represent the former virtual button ranked higher than the latter one."60D-60R" means explore the correlation between perceived depth and roughness when the frequency is 60Hz.N¼20; Ã p<0.01, two-tailed test; ÃÃ p<0.05, one-tailed test.ÃÃÃ No significant correlation between perceived depth and roughness.
Type II and Type III, the duration of the elementary pulse does not affect the perceived depth and roughness.The whole duration of converted vibrotactile stimuli in these two types is much longer than those in Type I (Fig. 7).When comparing the effects between Type II and Type III, we can see the whole duration of vibrotactile stimuli in Type III is longer than those in Type II at a fixed frequency at 60Hz.The whole duration of vibrotactile stimuli in Type II is longer than those in Type III at the rest frequencies in this study (Fig. 7).There is no regularity of perceived differences with the change of the whole duration.
In the main study, the changes in duration and frequency caused a changing perceived depth and roughness of virtual buttons on the touchscreens.In the future design, we could set an elementary drive signal firstly.And try to reach an expected perceived depth and roughness based on the results in this study.We could increase the perceived depth and roughness of virtual buttons on touchscreens by adding duration or adding pulse to the vibrotactile stimuli or setting the frequency near the resonant value.We could decrease the perceived depth and roughness by making the frequency away from the resonant frequency of the vibration actuators or make the output duration of vibrotactile stimuli shorter.
We provided the design process and chose parameters based on our system.In other systems, maybe parameters would be different, but a similar perception could be reached.The research questions in this study focused on the physical effects of vibrotactile stimuli and users' perceptions rather than the waveform.The duration and pulse numbers of vibrotactile stimuli are not limited to our system.Other systems can also reach a perception of a longer or a shorter duration, and with different pulse numbers.So, the ways about increasing or decreasing the perceived depth and roughness could also be practical in other systems.
Suppose the possible calibration is needed in other systems.The calibration could be conducted based on the design process (Fig. 4) and the physical effects discussed in this study.For example, in this study, we regarded the resonant frequency as the center.It leads to a very deep and rough virtual button because it presents a much stronger perceived intensity.In other systems, the resonant frequency may be 130Hz or 180Hz.Then, researchers could choose 130Hz or 180Hz as the center to calibrate.Meanwhile, the frequency dependency [21] at fingertips should be considered when conducting the possible calibration.

VI. CONCLUSION
This paper presents a design process to generate a large set of virtual buttons on touchscreens to mimic the sensations of different physical effects.We applied a modified smartphone containing an embedded wideband LRA motor that can convert wideband drive signals into vibrotactile stimuli on the touchscreen.We designed drive signals with a set of varying physical parameters, and we conducted two phases of studies to: Show the design process of drive signals for virtual buttons on touchscreens.Explore the perceived depth and roughness of virtual buttons on touchscreens.
In addition to the design process, we also have the following findings that can benefit the future design of virtual buttons: 1) The perceived depth of virtual buttons can be very deep, and the perceived roughness can be very rough when the frequency is around the resonant frequency.Perceived depth and roughness will decrease when the frequency increases or decreases from the resonant frequency.2) A longer duration of stimuli helps to increase the perceived depth and roughness of a virtual button on touchscreens.3) Adding pulse numbers helps to increase the perceived depth and roughness of a virtual button on touchscreens.4) Perceived depth and roughness have a very similar trend with varying frequencies at a fixed duration.The correlation between perceived depth and roughness is not always significant, but it is significant at most frequencies and durations.The above findings can guide virtual button design using the general process proposed in this paper.Although we only applied several parameters in this study, parameters are not just limited to those applied in this study.Researchers or designers could apply the design process, or the general functions proposed in this study and choose parameters meeting their expected effects.

VII. LIMITATIONS AND FUTURE WORK
In this paper, we chose frequencies based on the characteristics of our systems.We took advantage of the LRA applied in this study.However, the recorded acceleration amplitudes of vibrotactile stimuli were not controlled uniformly around the resonant frequency.We discussed results from two aspects: around the resonant frequency and away from the resonant frequency.The application of the resonant frequency is more related to the intensity of the vibration actuator.In this situation, the results about general frequency may not always be efficient in other systems.
Future work should control the acceleration amplitudes of vibrotactile stimuli at all chosen frequencies.A general frequency guideline for the perceived depth and roughness should be explored.Then, the research results would be more efficient for different systems and different vibration actuators.
We only considered the haptic effects of virtual buttons on touchscreens as a single interaction modality.In addition, only a few values were used when applying different amplitude modulations and superpositions.Hence, it seems that the amplitude modulation and superposition methods of drive signals have little effect on perceived differences.
However, when considering the audio and haptic effects of drive signals together as cross-modality interfacing, we found that the effects of virtual buttons become more sophisticated while modifying the parameters of amplitude modulation and superposition.
Different haptic sensations from different CWC types could present rich perceived depth and roughness of virtual buttons.We need to provide more CWC types for different scenarios and different demands of haptic sensation.
We are currently studying multimodal stimuli for virtual widgets on the touchscreen.Multimodal stimuli could present richer effects compared to vibrotactile stimuli only.The design process proposed in this study will optimize the user experience further.Also, we will take other parameters (such as rhythm) into consideration to get more CWC types to enrich the effects of virtual widgets on touchscreens.

Fig. 1 .
Fig.1.The frequency response of the system using a mobile phone mockup.
The dashed box on the left side describes the main procedure to design drive signals.The upper row in the left dashed box shows the design choices of physical effects.The bottom row in the left dashed box shows the function generation to reach expected physical effects.The dashed box on the right side describes the main procedure to convert the drive signals to vibrotactile stimuli.There are three steps of drive signals design.The first step (S1) is to design the elementary pulse for a basic physical effect.We set

Fig. 2 .
Fig. 2. The interface of the main study with 30 virtual buttons.

Fig. 3 .
Fig. 3. (a) The small accelerometer board was attached to the back of the smartphone with transparent adhesive tape.(b) The recording installation.

Fig. 4 .
Fig. 4. Overview of the design process of the stimuli.There are three steps to design drive signals.After determining the elementary pulse, audio effects should be chosen.Then, three CWC types are presented to get final drive signals.The final vibrotactile stimuli would be converted and presented by the LRA.

Fig. 5 .
Fig. 5. Introduction of three superposition methods and three CWC types of drive signals.Type I contained one elementary pulse, Type II contained one elementary pulse with a gradually fading out signal, Type III contained two elementary pulses (T¼0.04s in this study).The envelope shape, duration of the elementary pulse (t 0 ), and frequency of three examples of types here are a decaying ramp, 0.01s, and 150Hz.The superposition methods from up to bottom are A, B, and C.

Fig. 6 .
Fig. 6.Above are recorded accelerations of typical vibrotactile stimuli of virtual buttons converted by 9 drive signals (TableII) through an LRA.The frequency of drive signals is 150Hz, the duration of the elementary pulse is 0.01s.Above are the accelerations of the pressing stage of each virtual button.

Fig. 7 .
Fig. 7. Above are recorded accelerations of 30 vibrotactile stimuli of virtual buttons converted by 30 drive signals through an LRA.Above are the acceleration of the pressing stage of each virtual button.Type I: the drive signals in this group are one elementary pulse.Type II: the drive signals in this group are one elementary pulse with a gradually fading out signal.Type III: the drive signals in this group are two elementary pulses.

Fig. 9 .
Fig. 9. Perceived depth and roughness (averaged scores of all participants).Type I: the drive signals in this group are one elementary pulse.Type II: the drive signals in this group are one elementary pulse with a gradually fading out signal.Type III: the drive signals in this group are two elementary pulses.Duration (t 0 ) is the duration of the elementary pulse.

TABLE II PROPERTIES
OF TYPICAL DRIVE SIGNALSSpecific values of each superposition methods Ã are as follows:

TABLE III PROPERTIES
OF 30 DRIVE SIGNALS IN THE MAIN STUDY Group 1 to group 10 are for the ranking task in the main task.Duration (t 0 ) is the duration of the elementary pulse; "CWC types" are compound waveform composition types.Type I: the drive signals in this group are one elementary pulse, including M1 to M10.Type II: the drive signals in this group are one elementary pulse with a gradually fading out signal, including M11 to M20.Type III: the drive signals in this group are two elementary pulses, including M21 to M30, T¼0.04s.

TABLE IV PERCEIVED
DEPTH AND ROUGHNESS OF VIRTUAL BUTTONS IN TASK 2 Type I: the drive signals in this group are one elementary pulse.Type II: the drive signals in this group are one elementary pulse with a gradually fading out signal.Type III: the drive signals in this group are two elementary pulses.Duration (t 0 ) is the duration of the elementary pulse.

TABLE V THE
SPEARMAN CORRELATION BETWEEN PERCEIVED DEPTH AND ROUGHNESS OF VIRTUAL BUTTONS IN TASK 2 Type I: the drive signals in this group are one elementary pulse.Type II: the drive signals in this group are one elementary pulse with a gradually fading out signal.Type III: the drive signals in this group are two elementary pulses.D ¼ Perceived depth; R ¼ Perceived roughness.