Back to Finger-Writing: Fingertip Writing Technology Based on Pressure Sensing

Handwriting was since the start of the history, a higher expression of human skills, and was used for documentation of experiences, and for communication. Existing writing technology require a writing tool, like a pen, and a dedicated writing surface, like paper, or more recently an electronic tablet. These accessories of writing, of writing tool and service, are not available in many daily life situations. Furthermore, the writing accessories, are not natural, in many cases are not ergonomic, and thus can cause fatigue, and in extreme cases contribute to muscular and neurological diseases. In this work, we suggest to step back in history and step forward in technology, and to create, for the first time, an alternate writing solution without any accessories, using one own finger as writing tool, and write on almost any surface. For this, we used directional pressure sensors attached to the fingernail. Changes in the pressure induced on the fingertip in different directions while writing, are projected to the fingernail, and then assessed as a voltage pattern by the sensor. Decoding the pattern, can reveal symbols like letters, punctuations, and writing commands. In this paper, we describe the new pressure sensing modality and tailor processing methods. We tested the new technology on two subjects having different writing patterns while writing alphabet and sentences on different surfaces. We reached letter detection of over 80% while writing on a table, and the word detection rate, was near 70%, after applying the correction algorithm include language priors. The results of this work can revolutionize the way people write and communicate using more convenient, and more approachable, finger-tip writing.


I. INTRODUCTION
Handwriting is an higher expression of human communication [1]. In beginning of human history, human started using their hands for with painting on caves' walls [2]. Along time, writing tools like pencil, or pen were utilized to write over different surfaces like paper, board, and recently electrical tablet [3]. The existing handwriting techniques requires accessories of writing tool and dedicated writing surface, suffer from hand fatigue, and can contribute to the severity of muscular and neurological based diseases [4], and in extreme cases for an injury. This motivated recent attempts to find technologies to improve the writing experience.
An optimal writing technique should be: ergonomic; can be used continuously for long time without fatigue and causing The associate editor coordinating the review of this manuscript and approving it for publication was Wenge Rong . injury; in any environment; and with minimal writing tools. There are two main ways to minimize the use of the writing accessories. One group of techniques built without dedicated writing surfaces, like paper or tablet, another one without writing tools like pens or pencils.
To exclude the need from a dedicated writing surface, devices with sensory modules that can track the movement and transmit it wirelessly to a terminal can be used. An electronic wireless pen was suggested in [5], [6]. It is based on pressure measurements and/or motion of the pen, which is used as features for handwriting recognition in the computation terminal. Pen with accelerometers was suggested [7]. Another surface-less solution, is a fingerless glove that can implement a virtual keyboard for handwriting and was suggested in [8], and [9]. The angle at which the user's finger bends at the proximal interphalangeal joint is used to decode a row of the keyboard. Discrimination between columns VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see http://creativecommons.org/licenses/by/4.0/ operated by the same finger is achieved through an abduction/adduction sensor. To write without writing tool, some technologies use an electronic surface that is sensitive to finger touch. A Touch sensing display screen, can be on mobile devices like [10]. Writing letters on specific points on touch sensitive gloves was presented in [11]. The paper in [12], suggested to use a surface on the nail sensor to write different numbers. Both touch sensitive techniques, might be not comfortable writing in such device, not accessible, its performance are not always sufficient [13].
A tool-less and surface less writing techniques, can be based on assessing gestures that represent letters and other symbols. One way, is to attach to the finger magnetic sensors [14], or inertial sensors [15]. A ring, that include multiple sensor modalities was shown to implement a virtual standard QWERTY keyboard [16]. Sign gesture recognition using advanced temporal video image processing, and to decode sign gesture recognition using genetic algorithm, was developed in [17], [18]. All these techniques, require additional sensor modalities, are not fully seamlessly, need extensive training sessions as they do not exploit the natural way of writing. In this work, we suggest and demonstrate using a new sensor technology based on sensor pressure measurement of the index fingernail for finger-writing. The sensor technology fundamentals derived in IBM research labs [19]. This sensor enables using the natural intuitive hand movement seamlessly and thus is extremely ergonomic, uses the natural human way of writing from back in history, and can work on almost any available surface, in any environment. The pressure sensor can be implemented by a Strain-Gauge sensor (SG sensor) [20], or by Photoplethysmograph Fingernail Sensors [21]. The sensor is placed on the fingernail of the index finger. In the work in [22], a pressure sensor based on strain-gauge technology placed on the fingernail was introduced. It was shown, how the sensor is capable of measuring pressure point in real-time, can be used for assessing parameters like grip strength vital for medical diagnosis, and with machine learning tools, can also be used for coarse gesture recognition that can be used for enabling enhanced humanmachine interface. In the work in [23], we showed first feasibility of using the sensor for writing and recognizing basic shapes using only two pressure points. This paper extends the work in [22], and [23], and focus on using any pressure based sensor in task of writing full sentences in real-time in challenging conditions like different surfaces.
The paper has the following contributions: 1) development of a paradigm for continuous finger-writing recognition using a fingernail pressure sensor. The paradigm enables using natural hand and finger movement and thus is extremely ergonomic, and emit the need from having writing accessories of writing tool like pen, and writing dedicated surface like paper. 2) Development of analytical tools that enable using the pressure sensor for writing and decoding of single symbol, word, and sentence in real-time. The methods are tailored to the unique characteristics of pressure FIGURE 1. A fingernail is used for writing (1.a) over different writing surfaces (1.b). The writing process over the surface induces pressure over the fingertip that is projected to the nail. A Pressure Sensor (PS) unit is attached to the finger nail to capture the pressure pattern. It is implemented by embedding few strain-gauge in different directions (1.c). The raw data is then transmitted wirelessly to a storage and processing unit (1.d).
measurements, and differ from other methods like [16], where the analysis pipeline for accelerometers and gyroscopes was reported. 3) Showing first feasibility of the technology and comparing to accelerometer based only sensors. 4) Provide first validation of the new finger writing technology in different challenging every daily-life condition, like writing over different writing surfaces.
The paper outline is as follows: in section II the sensor system and the sensor technology is described; in section III, the methods tailored to the sensor for real-time writing finger writing are derived; section IV, describe the experiment setup; the results are shown and discussed in section V; and in section VI a summary of the work, and future directions are given.

II. SENSOR DESCRIPTION
The system is based on sensing the pressure projected into the fingernail while writing on a surface. The sensor fundamental technology was derived in IBM research labs [19]. The pressure sensor can be implemented by a Strain-Gauge sensor (SG sensor) [20]. This sensor enables using the natural movement freely and thus is extremely ergonomic. The sensor is placed on the fingernail of the index finger, and doesn't disturb the subject to perform his/her natural activities. When the subject is writing using his index fingertip, a pressure is induced on the nail, which results in deformation of the SG sensor. The deformation is translated to voltage-change data stream, which is transmitted wirelessly to a computation for decoding.
The suggest system is composed of the following parts: a finger that is used for writing (a), a writing surface or medium (b), Pressure Sensor (PS) unit (c), and storage and processing unit (d). Figure 1 describes the system.
The PS unit is attached to the fingernail (Fig 1.a). The surface for writing (Fig 1.b) can be any surface that induce changes in pressure on the nail, when writing with the fingertip, e.g. 2-D surface like table, but can also be non-rigid 35456 VOLUME 8, 2020 FIGURE 2. Processing data flow in finger-writing recognition. The processing consist of the flowing stages: a) pre-processing of the PS raw-data, which include excluding of sensor bias, filtering, scaling, and interpolatio; b) detection of start and end boundaries of each written shapes; c) shape feature extraction; d) symbol recognition based on the feature shape, that include also multi-shape symbol, and is based on individual and public trained symbol data bases; and e) word and sentence detection using Natural Language Processing (NLP) exploiting language priors, dictionary. surface like cloth, or even the air (movement of the body parts in the air, can induce changes in the pressure of the nail). The sensing unit is shown in Fig. 1.c. The PS unit can be implemented by different technologies that can assess pressure in different directions. In this work, we choose to implement the PS by Strain-gauge technology (Fig 1.c1); additional sensor like accelerometer can be aggregated (Fig 1.c2); the physical pressure measurements, are digitized, and pre-processed in the pre-processing unit (Fig 1.c3), and then sent wirelessly for further processing to decode the finger writing and store the results ( Fig. 1.d).
The sensor is placed on the nail. When the fingertip interacts with surfaces, a unique pattern of pressure is induced on fingertip and the pressure pattern is projected to the fingernail ( Fig. 1.a). The figure demonstrate the pressure map, as was obtained by thermal mapping on the nail, when the pressure is induced on the center. This pressure directionality pattern can be used similar to [24], for evaluating the direction and intensity of movement of the finger over time and enable extracting informative features to decode the finger writing.

III. METHODS
The methods derived in this work, form the baseline for Finger-Writing (FW) recognition using directional pressure sensor. The subject writes and induces pressure on the surface, which translates to deformation of the nailin different directions. When the subject stop writing, the pressure is relaxed, and go to steady state values. The pressure pattern induced while writing a shape on the surface is correlated to the shapepattern. The shapes are than aggregated and recognized as symbols of letters, or numbers, or punctuation, or commands, or breaks between words. The symbols and their probabilities are used to higher level meaning of words and the complete sentence.
The FW method is composed of the flowing stages: a) pre-processing of the PS raw-data; b) detection start and of writing actions of writing the shapes; c) shape feature extraction; d) symbol recognition; and e) sentence detection. Figure 2 describes the processing data flow.

A. PROBLEM FORMULATION
A symbol is defined as the minimal information unit that convey meaning, and can be a letter, punctuation, mathematical expression, general writing commands, or a keyboard like commands like break between words, or deletion of a letter [25]. Let define the k'th symbol, as g k , where g k ∈ {g k } k=K s k=1 . A sentence is then composed from a series of J symbols: The criterion for FW recognition to find from the sensor's raw data the sentence that was written is: where ε t , ε t+1 , ε t+T 0 , are the PS's inputs over period of time [t, t + T 0 ], corresponding to different location and finger VOLUME 8, 2020 orientation, and s [t,t+T 0 ],i , is the i'th hypothesis of a possible sentence from almost infinite possibilities of symbol permutations.
In case there are more sensor modalities, the recognition depends also on the other sensor data, which is assumed here, without loss of generality, as accelerometer data, denoted by, a t , a t+1 . . . a t+T 0 . The recognition criterion than becomes: To solve the criterion, we divide the solution to four stages: 1) detect from the raw data, shape start and end time for each pressure/relaxation session; 2) extract shape features; 3) recognize symbol's set of shapes and then recognize the symbol, like letters, punctuation, commands and breaks between words; and 4) decode the higher level meaning of words and the complete sentence.

B. PREPROCESSING
The N SG sensor (SGS) outputs, denoted by: ε t 1 , ε t 2 , . . . ε t N , reflect the change of the voltage due to the nail deformation from the finger writing operation. These outputs can be estimated as follows [20]: where i moves between 1 to N , D t i is the deformation, E t i is the transfer constant of pressure to voltage, and K is a normalization constant that is a function of the temperature. The deformation of the nail is spatial, and varies with the nail surface, and over time. Normalization of the pressure measurements can then be seen as a measure to the nail deformation up to the constant (KE t i ). The pre-processing process consist of the following stages: exclusion of the changing voltage bias, filtering out of artifacts, mitigate for missing samples, scaling the pressure signal, rotation of axes, and dimensionality reduction in case of multiple pressure points and when interpretable directions are needed.
To remove the DC effect, a high pass filtering of typically 0.1 Hz is used to exclude very slow movements and biases that are not likely to be part of the writing. To mitigate over missing samples, the missing samples are interpolated. Distorted samples are then filtered out by using low-pass filtering. The nail sensor can suffer from changing bias, due to temperature change, from undesired induction in the circuit [26], or from battery aging. This bias can be removed estimating continuously the bias, and excluding it using Kalman filtering similar to [27]. Scaling of the input is applied to standardize the features and improve their tolerance to different subjects' finger pressure profile, different pressure surfaces, and by this to improve the overall classification results [28]. Dimensionality reduction can be also used when the number of SGS is high.
The signal after pre-processing, is a signed temporal vector for each SGs. It can be characterized by its amplitude, and polarity. The polarity, reflects the direction of the induced voltage, and can be estimated in calibration phase. The SG's amplitude can be used to estimate the intensity of the pressure, while the sensor polarity, can be used to estimate the direction of writing. The sensor polarity, is unique to the subject, and is related to SGs location, nail topology and sensor properties.
For the additional inputs (like accelerometers), a similar pre-processing, based on characteristics of the signals, should be obtained.

C. DETECT SHAPE BOUNDRIES
A precise detection of symbol start and end times is required in FW recognition, similar to handwriting recognition [29]. Unlike visual based writing recognition methods, the task for shape start and end of writing times, can't be detected by spatial time discontinuity, but need to be derived by temporal measures or learning [7].
When the subject writes with his fingernail, a pressure is induced on the surface, which is then projected to its fingernail and the measured pressure move from steady state level to active pressure values. Then when he finishes inducing pressure on the surface after writing the shape, the pressure goes back to its stead state condition, which is not necessarily the one with minimal pressure change of the sensor. This steady state pressure level can be considered as the reference point, or the ''off'' state, denoted as OFF. Deviation from this state, are defined as ''on'' state, denoted as ON. These states, are often referred to as loading and unloading states [30], and can help to detect start and end writing times [31]. The ways to measure the deviation from the reference state can be performed by edge detection algorithms, e.g. by thresholding the signal over the stead state level, and in case the PS have deformation together, by using the envelope, composed of average sum of the PS signal or on the PS components. The PS envelope can be defined as: where ε t , is the vector of PS measurements at instance time t, . Acommon way to detect shape start and end, is by thresholding the envelope, when compared to the reference OFF state, r t ref .
More advanced methods, can use statistical features of the envelope, like the Sobel edge detection algorithm [32]. We denote the start and end of the k'th shape as t k s , t k e .

D. EXTRACT SHAPE FEATURES
Each shape, in the writing interval, has a typical pattern. Based on the pattern, features are extracted and fed continuously to a symbol classifier. This pattern can be used as a feature, or to extract physical interpretable features like pressure intensity, and shape duration. Spectral features like peak and median frequencies in [33] can be used. Features can be derived automatically by exploiting neural network [34].
Since FW is unique in the sense it is not limited to use with memory-based surfaces like a tablet or paper, it lacks spatial information. Thus, the FW recognition is more complex, and it needs to be based more on temporal features, like start and end of a new shape, shape duration, delay between consecutive shapes, and the temporal pattern to try to compensate over the missing spatial information. Additional sensors can be used to provide spatial information that can be aggregated to the PS's based features, and together provide spatial-temporal information. We denote the k'th feature set as: where k moves between 1, to K 0 , and indicate the shapes' index in the sentence writing time, [t, t + T 0 ].

E. SYMBOL RECOGNITION
In many cases, symbols like capital letters are combination of a few shapes, and then aggregated to a symbol. The aggregation uses statistical relationships between the shapes and learning on prior-knowledge related to the shapes [23]. A shape aggregation criterion for finding the start and end of the j'th symbol is: where j moves between 1, and the number of possible symbols, N sym . This criterion, can be solved by deploying clustering techniques like the one used in Sonar and Radar systems [35]. The symbol start and end time, t j s , t j e can be found using statistical methods that combine the best decoded hypothesis from a pool of recognizers used in speech recognition [34], and in mathematical expression recognition [7]. The hypothesisĵ, for the shapes related to the symbols, can be used to form the symbol feature set F j sy . After start and end symbol detection, the symbol classification recognition is reduced to simpler pattern matching problem estimation: where e . Without any loss of generality, we choose for this study, set of features relate to the writing pattern, and are complementary set that emphasize non-pattern characteristics like writing duration and intensity. The following features were examined for each SG separately for the non-pattern features: Letter writing duration, τ , maximal pressure, A max statistical characteristics of pressure: mean, standard deviation, skewness, kurtosis (A mean , τ A , σ A , S, K ); pattern properties features: Number of positive and negative peaks, and their relative location (N Pp , N Np , P Pp , P Np ); and Spectral features of peak and median frequencies, energy content at low, medium, and high frequencies ( In case where a symbol is composed of one shape, K j,s = K j,e , and there is no need for shape aggregation. For more accurate estimation, the criterion in (8) can be conditioned on prior knowledge available on fingertip physiological movement constraints, like typical values, maximal and minimal features values similar to [36]. The recognition of the symbol in (8) is a classification problem and machine learning techniques like handwriting can be used [6]. A training phase can incorporate the prior-knowledge related to the symbol. The classifier output is a vector that consists of the confidence probability of each symbol that can be fed to the word/sentence decoding algorithm. The j'th decoded symbols in a sentence, is denoted by gˆj, and its confidence vector is p j sy . In cases where recognizing the writing of one subject, a relatively small change in the pattern between consecutive writings of the letters (symbols) is expected, then methods like Dynamic Time Wrapping (DTW), can be utilized like in speech recognition [37]. The symbols pattern, with different velocities' hypothesis, can be then correlated to a dictionary of patterns of different symbols patterns of the subject, {g k } k=K s k=1 . When the pattern vary among different subjects, more advanced classifiers like deep learning, should be applied in future.
Design of commands in FP, is challenging task. In general, symbols like letters and punctuations emerge from perceptual and motor models [38], and thus are assumed to have low spatial correlation. But in some punctuation symbols that involve low and short pressure, like dot, or comma, they are less separated, and are more likely to be erroneous. For commands, like space, special FP commands can be designed that either have spatial difference, or induce change in temporal pattern, like repeating short localized pressure, resembling a point, few times, in different speeds. To choose commands with low distance tools from space-time coding like in [39] can be used.
Space is a special command, since it can be evaluated, by the delay between the writing of different words, which is natural, since the time between words in handwriting is also longer than between words, as the pen needs to move a longer distance. The distribution of the delay between words can be assumed to have a Gamma distribution [40]. Therefore, the likelihood of having a space between two consecutive symbols is: The parameters of the distributions can be found empirically, and are subject dependent. To find the end of one word, or the start of new (keyboard space) a boundary between the distributions should be found, that minimizes false detection. Usually the subject is expected to induce longer delay after the last letter in a word, following the natural way of writing, but even if not, through training and visual feedback the false detection can be minimized. VOLUME 8, 2020

F. SENTENCE DECODING
The symbols' confidence values for each symbol over time, together with language prior knowledge, as described in Fig. 2, are fed to classifier to decode a full sentence. The sentence decoding criterion, based on the raw sensor data in (3), is reduced to the following criterion: where P L , is language prior, trained on data set. The language priors, that is commonly used in handwriting and speech recording is sometimes refer to as priors of Natural Language Processing (NLP) [41]. It includes the likelihood of letter sequence, which is related to human vocal properties and limitations, word dictionary, relationship between the word, as can be determined from higher level processing and context. This relationship can be extracted by training the model over a large amount of data with many examples and large computation resources using advanced machine learning techniques, such as deep learning.
A sub-optimal approach that requires reduced level of trained data, and less cumbersome computations, is to use the traditional approach of first detecting the words using punctuations symbols, and subsequently use for each word, a Hidden Markov Model (HMM) [42], that include the letter probability and their relationship over time. Dictionary correction can be applied as post-processing stage. The accuracy of this algorithm depends on the symbol detection accuracy, the accuracy of transition between letters, and on the accuracy of the break probabilities. Thus, for each word, the HMM, solve in Maximum Likelihood (ML) optimal manner the decoding of each word, conditioned on the features of the letters of the word. Upper layers corrections, can be applied after the stage, using language based using query spelling suggestions [43].

IV. EXPERIMENT SETUP A. EXPERIMENT SETUP DESCRIPTION
The experiment setup is composed of a PS sensor equipped with wireless capability, a computer to process the data and display the results, as described in section II, and is shown in Fig 3.a. The sensor unit is composed of three SG sensors and 3-D accelerometers, amplifiers, analog to digital converter (ADC), micro-controller for initial processing, a wireless data transmission and a battery. The sampling frequency was set to 50 Hz. The A/D output signal range is 0 to 2047, where the maximum voltage of ADC is 1.87V. An offline power control was set that the maximal voltage range will fit the strongest deformation of the SG. A 3-D acceleration system on the nail was used as a reference system to the pressure one. Before the experiment, the sensor was attached to the index fingernail, using a nail glue. The directionality of the sensor was then tested by rolling the finger on a surface, to verify the coupling of the sensor to the nail and to assess its directionality pattern. The driver and the processing methods were implemented using MATLAB (Matlab.com, 2016b).  (Fig. 3a). Figure 3b, shows the three writing surfaces used in the experiment, table, cloth, and the subject's hand.
The experiment for evaluation of the method was performed with two subjects, which English is their second language, and thus their writing style is different. They performed writing on a table, cloth, and on their hand, as demonstrated in figure 3.b.

B. EXPERIMENT SETS
The aim of the experiment sets was to characterize the PS sensor signals, provide feasibility for the methods for decoding the FW, and produce first order performance evaluation of the new technology. For this, we designed an experiment set with two subjects that write sentences, in natural way as possible, on the different surfaces. We examined the inter-subject variability in a session, and the intra-subject variability. FW, similar to handwriting, has different characteristics pattern that changes from subject to subject and over time, and writing surface.
There were two main sets. One was built to examine the performance of writing on different surfaces (table, cloth, and hand), when trained on others (table). The other one was to test the suggested encoding of full sentences, when trained on only two repetition of the alphabet (one shout training), which is short enough to be used in daily life system after the deployment of the sensor. For both sets, the training was writing the English alphabet with two repetitions on the table. The first test set was writing on three surfaces (table, cloth, and hand) twice, the entire alphabet. In the second set, each subject wrote same 9 sentences (that were randomly chosen from the start of the book ''Moby Dick'') on the table, with varying word length from 2 to 9 letters, in total of 33 words.

A. SIGNAL PRE-PROCESSING
Before the start of the experiment, the sensor was tested to verify that the sensor was attached properly, and to assess the specific fingernail topology, similar to what is used in other sensing technologies that require high level coupling like Electroencephalography (EEG) sensor modality [44]. The absolute directionality response of the sensor, and its polarity, that can be used to estimate writing intensity and direction, are obtained by rolling the finger from right to left (roll) with 35460 VOLUME 8, 2020 different orientation. Figure 4, shows the amplitude and polarity (sign) of the rolling test, performed by the second subject. The separation between the SGs sensors, which govern the sensor directionality, is significant in finger rolling angles of over 20 degrees. The correlation coefficient between left, and center SGs is 0.5, right and center is negative 0.483, and right and left, of negative 0.339. This indicated a small coupling between left and center SGs, and more dominant negative polarity of the right SG, which is related to the fingernail morphology at the SG's location.
The SG signal is pre-processed to mitigate over missing samples, exclude artifacts, and compensate for bias. Figure 5 shows the signal in writing small letters alphabet. The miss sample rate was less than 0.5 percent, and the missing samples were distributed along the entire training session, and thus can be mitigated by interpolation similar to [45]. From the figure, it is seen that the SGs captures around half of the voltage dynamic range in regular writing operation, which will still support double force induced on the surface. The voltage reference points changed between each SG by less than 10 percent of the full dynamic range. The signal as seen in Fig 5.a, suffers from slow frequency drift over time, where there is a change of the OFF reference point from around in the beginning of the session to decreases by around 0.03 V in the end of the writing session after around 50 second. A small-scale variable random change can induce changing bias. For the removal of the slow bias, a high-pass filter of 0.1 Hz was applied. For the continuous drift, a Kalman filter that continuously estimate and subtract the bias by using other sensor measurements similar to [27] can be applied. The data is then normalized according to the strongest pressure value in the session. The data before, and after the filtering, is shown in (Fig. 5.b).

B. SYMBOL BOUNDARIES DETECTION
Accurate shape and symbol boundary detection affects the accuracy estimation of the FP writing. In the case where a symbol is a multi-shape symbol, like in mathematical expression, or in case of writing capital letters not in cursive writing, the shape boundaries should be decoded, prior to symbol's shape aggregation and calculation of symbol start and stop. In case of one-shape symbols, or a symbol that is composed from multiple shapes, but its writing is performed continuously where the finger does is in OFF stage for only fraction of second, like in cursive writing, the symbol start/end time can be estimated directly.
Since in FP writing process tries to be with minimal limitation, and mimic the natural way of handwriting, the detection algorithm for shape or symbols, can exploit prior knowledge in handwriting, like typical delays between the symbols, symbol duration, and the typical pressure level induced in FIGURE 5. Signal of writing alphabet in small letters, before and after pre-processing. The pre-processing includes interpolation to mitigate over possible missing samples, high pass filter to exclude bias, and normalization of the signal.  the FP writing. This is used in setting the envelope detection thresholds. The ON state was determined, when the letter envelop in (5), went down from the peak to 30 percent above the OFF reference level. Figure 6.a, and 6.b, shows an example of writing punctuations and commands symbols. The punctuations were of ''?'', and ''!''. For other punctuations like dot, and other writing commands like space, or delete, we suggest using dedicated gestures, as shown in Fig. 6.b. The commands are performed twice, continuously, bottom up, up bottom, left right, right left (comma, period, delete, space). Figure 6.c, shows the entire alphabet writing that was used for training, and its start and end are detected automatically At the bottom of Fig. 6.c, the signal envelope (3) is shown, and the limits between OFF and ON states.

C. SIGNAL REPRESENTATION/ FEATURE EXTRACTION
Each shape and symbol is characterized by a typical pattern, and a set of temporal, spectral, and kinematic features. To characterize the signals pattern, we normalize their power, and wrap them into a 256 length vector. The normalization, helps to emphasize FP's writing pattern differences that are invariant of writing speed, and is similar to what is used in machine learning based techniques [28]. Features matching algorithms can then be applied on the pattern waveform. Neural network based pattern matching techniques required massive data sets, and are less interruptible [46]. In addition, since the main focus of this initial study is to characterize the signal in relation to the physical sensor features, we used as features the coarse signal pattern, in addition to fundamental common features commonly used in handwriting recognition and have interpretable physical meaning [29]. Figure 7 shows the normalized pattern of the three SGs while writing English alphabet on the table during training for the first, and second subjects. The left, central, and right SGs patterns are in blue, red, and green colors, and the inter-session variability (2 repetitions) is characterized by the area around the line. The letters start and end were found according to (3). The direction of the FP writing over time can be decoded by the directionality pattern. The polarity of the central and left SGs is positive, while the right SG, has dual polarity, which can be related to the relation between the nail morphology and the sensor location. It can be seen from the figure, that the inter-session variability is much lower than the inter-subject variability. It is explained by subjects' different writing style, that include writing same letters sometimes in different directionality (like writing the letter l, bottom up, or up bottom), different pressure (individual strength, or different nail shape that project different pressures), and different timing. This implies that subject specific patterns, or massive data from many diverse subjects should be used to reflect these kind of variations for training in future.
Writing of the letters c, and m, at different writing surfaces (table, cloth, and hand), for the two subjects is shown in Fig. 8. The letters c is one of the simplest letters that usually has lower subject variability (writing half circle counter clock wise) in its writing compared to other letters [23]; therefore, it can be used as a reference letter. From the patterns, it seems that the letter is written in similar way in the different experiment conditions (surfaces) for the different users, but with higher inter-subject variability due to different directionality pattern between the two subjects. The letter m is more complex in writing, and has higher variability in its writing, as it contains three lines, and 2 half circles. For the two subjects, FIGURE 7. Normalized writing patterns of alphabet for writing in the table for the first and second subjects. The inter-session variability (2 repetitions), is characterized by the area around the line. The inter session variability is much lower for most letters, than the inter-subject variability, due to each subject writing style, that include different letters directionality (writing from left to right, vs. writing right to left), pressure (individual strength, or different nail shape that project different pressures) and timing. Thus, subject specific patterns, or massive data from many diverse subjects, should be used for training.

FIGURE 8.
The pattern change between subjects and writing surfaces for the letters c, and m. The differences between the experiment conditions (surfaces) are much lower than between the subjects. This indicates that for accurate results, subject specific training is preferred, and that it is possible for each subject to recognize letter, when trained in other surface.
at least two peaks (positive, or negative) can be recognized, which suits the nature of writing similar shapes. The directionality of the PS can be used to assess the direction of the FP movement. For instance, the pressure on the fingertip while going down, and releasing the pressure while going up, can be seen on the central SG, and in some of the SGs that are coupled. The direction of movement can be assessed by the magnitude of the un-normalized SG amplitudes in the calibration as shown in Fig. 1. For instance, the right SG is more dominant than the left one in most letters, due to the nature of writing from right to left. This value of magnitude is one of the features that are used in addition to the pattern in the symbol recognition.
Writing the letter on a cloth, is characterized by higher inter-session variability, and less smoothness compared to the smoother surfaces of the hand and the table. Writing over the hand, is characterized by the highest inter-session variability, which can be explained by the different curves on the hand, and different start writing points. For the two letters, the intersubject variability is higher than the inter-session variability, which indicate on the need to tailor calibration for each subject separately, or use massive data side, that capture all the variations in different letters writing.
The average inter-subject letter variability is function of the subject, his/her technique of writing, and of the writing surfaces. Table 1. shows the inter-subject variability (between experiment repetitions) in different surfaces, calculated as the mean distance in the feature space. The table inter-subject variability is the lowest, which can be explained by the relatively smooth and more certain surface of the table.
The other two surfaces tend to be more challenging for writing, with more varying surface in one hand, which result  in higher inter-subject variability. The inter-subject variability (variability between the subject, intra-subject variability) is much higher (>50% in average) than the inter-session variability of same subject, which can be explained by the difference in the sensor placement and the differences between the subjects' nail morphology. Figure 9 shows the normalized mean and standard deviation of the features values in training data, averaged on all subjects. It seems that there is no consistency among subjects on the deviation of the features among the surfaces. The letter duration seems to deviate between the two subjects, may be due to the writing limitation of the experiment. The pressure power seems to be the highest on the table and on writing with on the cloth, for subject one, and highest on table, and lower in cloth and hand surfaces. This can be explained by the smoothness of the surface, and by the subject's individual motor control. For example, when the surface is smoother (like the table) to have effective writing with efficient effect on the nail pressure, the subject needs less writing power. The maximal frequency of writing seems to be the lowest on the table and can again be explained by the smoothness of the writing surface.
Applying F-test, to the features, shows that the four most significant features to recognize the letters in the training sequence in decreasing order, are: µ A (the average pressure when writing the letter), E fm (the average frequency content of the signal between 1 to 3 Hz), τ A (the duration of writing the symbol), σ A , the standard deviation of the signal, which indicate on the smoothness of the writing.

D. SYMBOL RECOGNITION
Symbols can be letters, which are composed of shapes (in case of capital letters), single shape letters (small letters), punctuations, or writing commands like the one implemented in keyboard. In this work, we focus, without loss of generality, in recognizing single shape letters. Extension of the work to the other symbols can be achieved using methods like in [23].
For the symbol recognition, we used two classifiers that are traditionally used in handwriting recognition, Dynamical Time Wrapping (DTW) [47], and Discriminative Function Analysis (DFA) classifier [48]. The DTW works on the pattern while the other classifiers can work on the features, pattern, and combination of both. To reduce overfitting in the training phase, we added noise according to statistics of the measurements similar to [27], and performed on a features Principle Component Analysis (PCA) and kept the components that preserved 98% of the explained variance.
For the training set we used for the DTW, the alphabet FP on the table, which due to the smoothness is ''cleaner'' pattern of the signal, and thus is natural choice to preserve the reference pattern. For the DFA classifier, the alphabet writing at different surfaces was used, to reflect the spread changes of the pattern in the different surfaces. The training error for the DTW, and DFA, was 9.9, and 3.8, and 9.9, and 3.8 percent, for the first, and second subjects, respectively.
The letter detection rate for the multiple experiment sets is shown in Fig. 10. It shows the results of DTW, based on the pattern only applied on the PS's data; DFA classifier applied on the PS's data, based on pattern after dimensionality reduction using PCA to 20, in addition to the 7 significant features from each PS point (left, center, and right), and DFA applied similarly on the reference acceleration data. The accuracy of the training set was high for all classifiers, and for both PS and acceleration measurements. The accuracy of the DTW was higher for the PS's data, which indicate that the information in the pattern is critical for high recognition rate. The test set using the table as reference, was the highest, as expected, since the classifier was trained on the table, and the table is the smoothest surface, and therefore, is likely to maintain FIGURE 11. Histogram of delays between consecutive writing. The separation between words and letters is determined by threshold that minimizes the word start false detection, which depends on the Gamma approximation accuracy. The threshold was found to be 0.65, and 0.85 seconds for the first and second subjects respectively. consistency with the patterns over time. The accuracy of writing on the cloth, was still high, as beside the increase friction, it doesn't have unexpected curves, like the hand, that has the lowest performance. Training the FP writing on the cloth, and the hand, might come with higher detection rate for these surfaces, but will down perform the performance on the table, which is around 80%. This shows a tradeoff between high detection performance and compatibility to different surfaces. The acceleration reference, gave performance of around 20%, which is less than third of the accuracy using the pressure sensor, but around 5 times higher than the random selection of 3.85% (shown in black line).
To detect word's start/end times, a dedicated symbol can be used, that can be for example matched to the keyboard symbol set. FP writing can exploit the delay between letters and words and impose way of writing that is close to natural writing style, but with the awareness of the subject to have longer break between words than compared to between letters. This is equivalent to the increased duration in handwriting, when moving the pen when start writing new word, compared to the one when writing small letter. Figure 11 shows the histogram of delays of between writing letters vs. the one between writing words. The curves (red and blue), show the Gamma distribution fitting for the letter and words. The separation between words can be determined by threshold that minimizes the word start false detection according to the distribution tail, which depends on the Gamma approximation accuracy. The threshold was found to be 0.65, and 0.85 seconds for the first subject, second subject, respectively. Using this threshold, with one misdetection of word start, there were 4 (4.4%) and 1 (1.1%) false detection of word start (as part of the word). The difference in the distributions, and thresholds between subjects is due to difference in writing characteristics like velocity, and the way the subject perform the OFF phase, when the subject raise his/her finger, between writing the letters. Thus, more training in FP, real-time visual feedback, and methods that know to correct the spaces between words according to the word and sentence content, are expected to improve the word start and end recognition. Figure 12 demonstrate an example for a sentence data analysis flow for FP writing of the first subject. The sentence, is ''never mind how long ago precisely'' taken from the book Moby Dick. Figure 12.a shows the pressure signals after post-processing and applying the symbol boundaries detection algorithm. Features are extracted and fed to DWT based classifier, which output the likelihood of each  letter, p j sy , in the confidence matrix in Fig. 12.b. The right column shows the probability of the break between words, p j sy,space , denoted in the figure as P b . To decode a letter, we look on the maximal probability of the classifier in a row. Then, an internal HMM model is applied, that output the two maximal hypotheses. We apply the following efficient algorithm that exploit the first, second, and the spelling corrections, used as prior knowledge of the language. If the first HMM hypotheses is a valid dictionary word, then we choose this word. If not, we apply the dictionary on the second HMM hypothesis, if it is part a valid dictionary word, then we choose this word. If not, we choose the dictionary correction of the first HMM hypothesis. If there is no correction to the word, which is likely with names that are not part of the dictionary, then we keep the first HMM hypothesis. The sentence is then fed in post processing stage, to an algorithm using query spelling suggestions like NLP in [43], to improve the results accuracy, as seen in the bottom of Fig. 12.C.

E. SENTENCE DECODING
We compared three different methods for word recognition: 1) HMM hypothesis without applying any prior letter knowledge; 2) with word spelling correction applied on first two HMM outputs, which is example of applying word prior knowledge; and 3) with applying on the HMM with prior knowledge outputs prior knowledge related to the sentence (NLP), using google suggestion tool. After first HMM hypothesis, the word detection success rate were, 49.3, and 45.5 percent (compared to around 3 percent for random selection). The detection rate of words without or with only one spelling errors based on the first HMM hypothesis, were 84.8, and 75.7 percent. After applying the suggested dictionary correction based algorithm from above, the success rate increased to 63.6, and 64.2 percent. This indicates that only part of the one letter level errors could be mitigated by using the prior knowledge about the words. After applying NLP knowledge, the relationship between words in sentence, and the content, the percentage increased to 69.7, and 66.0, for the first and second subjects receptively. Table 2 summarizes the word performance.

VI. CONCLUSION AND FUTURE WORK
This work describes for the first time, systems and methods that has the potential to enable natural writing with a finger on almost any available surfaces without need of writing accessories. In this work, the new concept was described, and processing methods were tailored to the nail sensor. The new technology was validated by two subjects with different writing style patterns, different nail morphology, on different surfaces. The accuracy of the letter detection on table was over 80%, and the word detection rate, was near 70%, after applying the full correction algorithm include language priors. Adding more pressure points, extending the data base size to reflect all the settled writing variations, and aggregating information from other sensors, can improve the results. The results of this work can also assist in detection of abnormal pattern of writing and assist in making writing more accessible to new populations throughout the world.
Next planned work, will be to extend the data sizefor each subject, extend the number of subjects, and apply more advanced classifiers like deep learning classifiers that will include automatic feature selection and can exploit big data efficiently. The proposed finger writing recognition features, like pressure and writing style, can be aggregated and used to enhance existing biometric systems using techniques like in [49]. A future challenge, is to implement this FW system in real-time, connecting it directly to the watch or the cellphone in interactive manner, to be used as human-machine interface.
GADDI BLUMROSEN was born in Jerusalem, Israel. He received the B.S. and M.S. degrees in electrical engineering from Tel Aviv University, in 2005, and the Ph.D. degree in biomedical engineering from the School of Engineering and Computer Science, Hebrew University, in 2011.
From 2012 to 2014, he held a postdoctoral position with the Computer Science Department, Tel Aviv University, in collaboration with the Zoology Department, and Tel Hashomer Hospital, producing new sensing and analysis tools in biology and medicine mainly to characterize human movement. In 2014, he was a Visiting Scholar with the Harvard Medical School, where he developed and implemented new tools to analyze the effect of electrical pulses for terminating cancer cells. In 2015, he was a Visiting Scholar with the Neuroscience Department, New York University, where he developed new tools to for non-primate facial recognition. In 2016, he joined the IBM Thomas J. Watson Research Center, to explore and quantize human gestures with new sensing and processing methods, in particular to detect scores to evaluate human and patient with Parkinson Disease performance. He is currently a Research Associate with the Faculty of Engineering, Bar Ilan University, where he works to improve methodologies in neural networks, inspired by the brain.