PTPG: A Poisson Triangular Pattern Generator for Tokens on Tangible Surfaces

We present PTPG (Poisson triangular pattern generator), a method that generates and optimizes triangular patterns for tokens on tangible surfaces. PTPG uses spatial information of markers to form triangular patterns and processes a template-matching algorithm to identify patterns. We introduce Poisson disk sampling to search patterns in the feature vector space and set sampling constraints to obtain patterns that meet the requirements of size, shape, and pattern distance. The patterns optimized by a shape constraint address the marker-loss problem caused by a common design of capacitive tokens, and therefore improve the recognition rate. We manufacture ten sets of prototype tokens based on the patterns generated by Poisson sampling and test these tokens on different devices. Experimental results show that our method provides more unique patterns than other heuristic methods, and the patterns are applicable to different tangible surfaces with high recognition rates.


I. INTRODUCTION
Tangible User Interfaces (TUIs) enable users to interact with digital content by manipulating physical objects [1], [2]. As an interactive paradigm more in line with natural human behavior than Graphical User Interfaces (GUIs), TUI has been introduced into areas such as gaming [3], [4], education [5], [6] and design [7], [8]. TUI systems rely on sensing and recognizing patterns of token objects to obtain information on identification, position, direction and behavior. Pattern design is one of the key building blocks of TUI systems. However, research on pattern design and generation remain sparse. In particular, pattern generation in nonoptical tangible systems mainly relies on heuristic methods.
Patterns are generally formed by the 'footprints' that are left on tabletops by objects or by the fiducial markers that are attached to objects [9]- [12]. Initially, optical systems mainly used high-contrast pictures composed of black-and-white rectangles as recognition patterns [13], [14]. Optical patterns require additional cameras to capture images, making optical systems generally bulky and unsuitable for mobile devices. In addition, optical pattern recognition is easily affected by recognition rate and a large number of unique IDs. PTPG uses the spatial information of three marker points to construct a triangular pattern, and introduces Poisson disk sampling to search for a maximal number of patterns in pattern space. By setting sampling constraints, we optimize the shape of patterns to improve recognition rates and stability. In conclusion, this paper makes the following contributions: (1) Present a triangular pattern design suitable for various tangible screens, and a light-weight and robust recognition algorithm. (2) Introduce a novel Poisson-disk-sampling-based method to generate a maximal number of patterns in the feature vector space, and improve the recognition rate of patterns by optimizing their shapes. (3) Perform experiments to evaluate our method in terms of the number of patterns, recognition rates on different devices, and compare with other patterns.
After reviewing related work, we describe how to construct the feature vector of a triangular pattern and the pattern recognition algorithm based on template matching. Then, we present the algorithm based on Poisson disk sampling for generating and optimizing patterns. Next, we detail the structure of the prototype token, the experimental results on different devices, and the comparison with other methods. Finally, we report the interactive experiences in three applications, summarize the limitations of our method and provide a future research plan.

II. RELATED WORK
We first give an overview of patterns used for optical tracking, magnetic tracking, and capacitive tracking technologies, and then discuss the active tokens with digital pattern IDs.

A. OPTICAL PATTERNS
Initial TUI tabletop systems used computer-vision methods to track objects and users' gestures [7], [8]. Such optical systems typically include a camera to capture images of objects and a projector to output graphical interfaces and digital information [21], [22]. The computer-vision algorithm tracks an object by analyzing the fiducial markers on the surface of the object [23] or the ''footprint'' left by the object on the desktop [24]. ARTag [25] used several planar fiducial markers to identify objects. ReacTIVision [13] used images composed of black-and-white rectangles as patterns, which can be converted into a special tree structure to provide 128 unique pattern IDs. Lumino [26] used a glass fiber bundle in each tangible block to rearrange the marker images of 3D structure blocks, and encoded 14 classes of objects by 4 marker bits. Optical patterns are mature and suitable for large tabletops. However, optical tracking relies on the resolution of cameras and is easily affected by light conditions and occlusions, thus reducing the recognition rate. Moreover, optical systems are always too bulky to use with mobile devices.

B. MAGNETIC PATTERNS
As a low-cost and reliable object sensing technology, magnetic sensing methods have been adopted by some tangible systems. Sensetable [27] uses unique resonant frequencies generated by coils to identify objects. GaussBits [16] used a thin magnetic sensor grid that is attached to the back of a display to track the bi-polar magnetic fields of tokens. GaussStones [10] combines different features of magnetic fields such as shape, intensity, and quantity to code pattern IDs. MagnID [15] spun a magnet by a motor and encoded eight tokens with the frequencies of polarity changes. Magnetic patterns can utilize different features of magnetic fields to provide hundreds of unique IDs. However, Hall-sensor grids have a short sensing distance so that they are often used for thin screens. Furthermore, there are interferences between magnetic tokens that need to be handled.

C. CAPACITIVE PATTERNS
Capacitive touchscreens have the advantages of portability, low cost, and durability. An increasing number of researchers are exploring object recognition methods using capacitive sensing [11], [12], [14], [19], [28]. General capacitive patterns use relative spatial information among several touches [17]. TouchTokens [20] used three touches to form a triangular pattern, and selected a group of 12 patterns by heuristic experiments. TriPOD [29] generated 24 triangular patterns by project points in a 3D space according to size and distance conditions. To further expand the number of capacitive patterns, ToMMI [28] and TUIC-2D [30] use three fiducial points plus one coding point to generate patterns. These four-marker patterns have more spatial features than three-marker patterns and are able to provide more unique IDs, but they do not make full use of the relative spatial information between the marker points. Capacitive touchscreens were originally designed to sense human finger contacts, so they usually have a limited number (10)(11)(12)(13)(14)(15)(16)(17)(18)(19)(20) of simultaneous touches and a low accuracy of position information. The more touches a token contains, the fewer tokens that can exist on the screen simultaneously. How to use the limited number of touches and the imprecise position information to generate patterns is a challenging problem in capacitive tangible systems.

D. PATTERNS OF ACTIVE TOKENS
To address the limitation in the number of unique patterns, some researchers have installed additional chips into active tokens to provide unlimited pattern IDs. PERCs [19] used three fiducial touches to provide location and direction information and provided object IDs via Bluetooth wireless communication. IDsense [31] and RapID [32] used RFID (radio frequency identification) tags to track objects at room-scale distances. Zanzibar [11] used NFC (near field communication) to mark the unique IDs of objects and used flexible, touchable materials to further improve the portability of the system. Active tokens can provide a nearly unlimited number of pattern IDs, but there are problems such as manufacturing difficulty, high cost, and limited battery life.

III. TRIANGULAR PATTERN
In this section, we first propose a triangular pattern design and the feature vector of the pattern. Then, a pattern recognition algorithm based on template matching is described in detail.

A. PATTERN DESIGN
We expect our pattern design to be applied to a variety of tangible systems that use different sensing technologies. The pattern design needs to meet the following requirements: 1) can express ID, location, and direction and provide as many IDs as possible.
2) use a small number of marker points.
3) apply to a variety of sensing technologies. 4) apply to active tokens and passive tokens. Drawing on the pattern designs of previous studies [12], [19], [29], we use the spatial information of three markers to construct a pattern. The triangular pattern design is based on the following considerations: the shape of a triangle can express ID, position, and direction, and we can provide unique IDs by selecting different triangles. The pattern formed by three markers is suitable for tangible devices that have limited markers such as capacitive touchscreens and infrared touchscreens. The pattern only uses spatial information of markers, so it is applicable to different sensing technologies and tokens.
To describe and compare patterns, we need to select appropriate features to construct a feature vector. Many parameters can be used to describe a triangle, such as side lengths, area, angles, vertex coordinates, local coordinates of vertices, etc. Inspired by previous work [29], we use the counterclockwise arrangement of side lengths to construct the feature vector ( Figure 1b). The advantages of the feature vector are that each feature vector can be uniquely mapped to a triangle, and other information such as angles and area can be easily calculated by side lengths.
Three sides of a triangle have three counterclockwise arrangements, corresponding to three feature vectors, such as where x 1 , x 2 , x 3 are the lengths of the triangle sides. We call f , f , f equivalent feature vectors that describe the same triangle. Assume that there are two triangles T and G, and their feature vectors are f and g, respectively. We define the distance d between the two triangles as the minimum L2 norm between the equivalent feature vectors of the two triangles as follows: d = 0 denotes that the two triangles have the same shape and size, and a larger d refers to a greater difference between the two triangles. Apart from equilateral triangles whose orientations are ambiguous, an unlimited number of triangles can be used as patterns. In practice, the selection of triangles needs to consider the conditions of token size, sensing accuracy, etc. These conditions limit the maximum and minimum lengths of triangle sides and the distance between patterns.

B. PATTERN RECOGNITION
When an object with a triangular pattern is placed on a tangible surface, the TUI system senses the markers of the object and provides marker positions to the recognition algorithm. VOLUME 8, 2020 The recognition process is shown in Figure 1. The recognition algorithm first sorts the three vertices' positions counterclockwise to obtain a sorted vertex sequence {v 1 , v 2 , v 3 } and then calculates the lengths of the triangle sides to obtain the feature vector f as follows: For a known set of pattern templates {T 1 , T 2 , . . . T n }, we used equation (1) to calculate the distance between f and each template, and then the ID of the template that has the smallest distance is obtained as the recognition result. Next, we define a reference frame on the triangle to obtain the position and the direction of the pattern. The origin of the reference frame is defined at the center of the three vertices. The x-axis is defined by the vector formed by the first point of the matching template and the origin. Finally, the angle between the x-axis of the frame and the positive x-axis of the screen is used as the rotation angle of the triangular pattern ( Figure 1d).
Our recognition algorithm is easy to implement, robust and fast. We implement the template-matching algorithm in the Unity engine using C#, which amounts to approximately one hundred lines of code. We also considered and tested several alternative approaches including linear least squares regression, k-nearest neighbor (KNN), support vector machine (SVM), and deep neural network (DNN) methods. We used both raw data and descriptive features to evaluate these alternative approaches. The raw data was preprocessed to make it independent from rotation and screen position. The descriptive features we considered included the bounding rectangle of markers, as well as various statistics for measures such as point-centroid distance, distance between any pair of points, etc. These alternative approaches achieved recognition rates ranging from 95.3%-100% using different devices and tokens, consistent with the result of the template-matching approach.

IV. PATTERN GENERATOR
In this section, we describe the details of the pattern generation algorithm based on Poisson disk sampling, and the optimizing constraint to address the marker-loss problem on capacitive screens.

A. PATTERN GENERATION
It is a challenge to choose a set of triangles for triangular patterns. The pattern set needs to consider the distance between patterns, token size, position error caused by sensing noise, etc. Manual selection relies on experimental and heuristic methods, and conflicts may occur when the number of patterns increases. PTPG introduces Poisson disk sampling to find a pattern set in the pattern space that meets different conditions. Let F be the whole feature vector space of the patterns. Every vector f i = (x 1 i , x 2 i , x 3 i ) in F satisfies the following triangle side length constraints: We expect to sample a set of feature vectors f 1 , f 2 , . . . f n in F, and each element f i of the set satisfies the following constraints: (a) Side length constraint: The maximum side length l max is determined by the physical size of the object, and the minimum side length l min is determined by the smallest distinguishable distance between two markers on the tangible surface.
(b) Distance constraint: The distance between any two feature vectors is greater than the specified distance: where r depends on the position error of the sensing technology.
(c) Shape constraint: We need to exclude equilateral triangles because they cannot uniquely determine the angle. The standard deviation of three side lengths should be larger than a specified value σ .
We record the subspace of F that satisfies the constraint (a) as S. Then, we perform Poisson disk sampling [33]- [35] to find the feature vectors that satisfy the constraints (b) and (c) in S as follows.

Algorithm 1 Pattern Generation Algorithm
Step 1. Randomly select an initial vector f 0 that satisfies constraints (b) and (c) in S. Add f 0 and its two equivalent feature vectors to an active list.
Step 2. Randomly select a vector f i in the active list, and randomly select a test vector f k at a random distance between r to 2r around f i , where r is the minimum distance between sample points.
Step 3. Check if f k satisfies constraints (b) and (c). If it satisfies the constraints, accept f k and add f k and its equivalent feature vectors to the active list. If f k does not satisfy the constraints, repeat Step 2 and reselect an f k to test again. If the number of attempts exceeds a constant value k (typically k = 30), remove f i from the active list and add it to an inactive list.
Step 4. Repeat Step 2 until the active list is empty. Remove the redundant feature vectors in the inactive list, which means only one of the three equivalent feature vectors is reserved. Finally, the sampling result is obtained. Figure 2 shows a sampling result using parameters: l min = 12 mm, l max = 55 mm, r = 10 mm, and σ = 5 mm. Ten samples are obtained in the feature vector space, corresponding to 10 triangular patterns with different shapes.

B. PATTERN OPTIMIZATION
By setting constraints in algorithm 1, we generate patterns that meet different requirements. With this feature, we can improve the recognition rates of passive tokens on capacitive screens. Tokens that can generate touches without maintaining human contact have been widely used in capacitive tangible systems [12], [19], [29]. The principle is that the token utilizes the inactive electrodes on a capacitive surface to simulate the grounding effect, so that its conductive marker pads generate touches on the capacitive screen (Figure 4b). However, if a pair of pads is coupled to the same electrode, the two conductive pads are unable to decrease the capacitance, and thus cannot generate touches on the capacitive screen. As shown in Figure 3a, marker A, B and C form a right-triangle pattern, and marker A and B are aligned with the same horizontal electrode, meanwhile marker A and C are aligned with the same vertical electrode. In this case, the capacitance of marker A will not decrease, so that marker A can not be detected by the capacitive screen, while marker B and C are still detected reliably. This marker-loss problem will lead to a decrease in the recognition rate. PERCs [19] addressed this problem by adding a light sensor to infer the position of marker A from the positions of marker B and C, but this solution requires additional sensors and power. We propose a novel method to solve this problem by optimizing the shape of the pattern.
We add a new constraint (e) to algorithm 1 as follows: As shown in Figure 3d, AD, BE, and CF are the perpendicular lines of the triangle, and l AF , l BF , l BD , l CD , l AE , l CE are the lengths of AF, BF, BD, CD, AE, CE segments, pitch is the distance between two neighboring parallel electrodes, and d m is the diameter of the marker. Patterns that satisfy the constraint (e) ensure that each marker is always coupled to different electrodes in any orientation, so that the marker-loss problem will not occur (Figure 3c). The experimental results in Section V show that the optimized patterns achieved a 100% recognition rate on tested capacitive screens, while the nonoptimized patterns only achieved a lower recognition rate of approximately 95%.

V. EXPERIMENT AND DISCUSSION
We now detail the implementation of prototype tokens. Then, the experimental setup is described. Finally, we evaluate the number of unique patterns generated by PTPG and the recognition rates on three tangible devices, and discuss the comparison with other patterns.

A. PROTOTYPE TOKEN
The implementation of the prototype token should meet the requirements of low cost, ease of manufacture, and compatibility with various tangible devices. The structure of the prototype token builds upon the prior studies [12], [29] and extends to make it applicable to capacitive devices and infrared devices. As shown in Figure 4, the prototype token consists of three layers: three cylindrical legs made of conductive silica gel, a layer of copper foil for connecting the legs, and an acrylic base. The legs are attached to the copper foil using conductive glue, so that they are conductive with each other. This design allows the legs to reduce the capacitances of the electrode intersections on a capacitive touchscreen, thus creates touches without human contact (Figure 4b). The exposed legs can also generate touches on infrared surfaces by blocking infrared light. The token can create a desired triangular pattern by adjusting the relative positions of the legs, and then the ID of the token can be identified from the pattern. The silica gel leg has a diameter of 10 mm and a height of 6 mm, the diameters of the copper foil and acrylic base are 65 mm, and the height of the acrylic base is 10 mm. The materials cost only approximately $0.50 and can be make into a token within two minutes.

B. EXPERIMENTAL SETUP
We first tested the number of patterns generated by sampling method. We combined different parameters, including VOLUME 8, 2020 maximum side length, minimum side length, sampling distance, shape optimization constraint, and used algorithm 1 to sample patterns in the feature vector space. Considering the randomness of Poisson disk sampling, we sampled 20 times with each parameter setting, and then recorded the average result. Next, we tested the recognition rate and recognition stability of the patterns. We performed ten samplings with parameters of l min = 15 mm, l max = 55 mm, r = 10 mm, and σ = 5 mm, in which five samplings enabled the shape optimization constraint and the other five samplings disabled the shape optimization constraint. For each pattern set, we randomly selected ten patterns to make tokens, and obtained a total of 100 tokens. Figure 5 shows one set of sampling patterns and the set of tokens based on them.
We recruited three participants from neighbor colleges (aged 22-35 years) to perform a large number of test cycles on three tangible devices, which included an ILITEK 23.6-inch capacitive touchscreen, a 9.7-inch iPad 5, and a CMC 32-inch infrared touchscreen. Each test cycle was detailed as followed: each participant tested a set of 10 tokens on a device at a time. The participant placed a token on the screen, released their hand, waited for 2 seconds, and then picked the token up. Each token was placed 50 times at different positions and angles on each screen. A recognition application automatically recorded the experimental results. We performed this test cycle for each device and each set of tokens, with a total of 100 × 3 × 50 = 15,000 placements. The whole experiment lasted approximately four hours.
Each of the devices was separately connected to a computer running the recognition application. Each computer has an Intel I7-6700 CPU at 3.40 GHz, an NVIDIA GeForce GTX  where l max is the maximum side length; l min is the minimum side length; d is the sample distance; n is the number of patterns; n(optimized) is the number of patterns sampled with the shape constraint; avg. and SD are the average and the standard deviation of 20 sampling results. Table 1 lists the numbers of sampling results. We observe that the sampling distance has the greatest effect on the number of patterns. With a 5-mm sample distance, we can obtain approximate 42 patterns for a token of 40 mm diameter, and 443 patterns for a token of 80 mm diameter. By setting a small sampling distance, the number of patterns greatly increases. This is because a small sampling distance means a high pattern density in the feature vector space, and therefore more patterns can be selected. The value of the sampling distance depends on the tracking accuracy of the tangible device. For devices with high sensing resolution, we could set a small sampling distance to obtain more patterns. For devices with low sensing resolution, we need to set a large sampling distance to overcome the position error and ensure the accuracy of recognition. The maximum and minimum side length parameters determine the volume of the sampling space and therefore the number of patterns. We also found that the TABLE 2. A comparison of different patterns. Our method provides more unique patterns than other three-marker capacitive patterns, and have a higher recognition rate than optical patterns (symbols '-' in the table indicate that the data is not provided by the authors). Compared with active tokens and magnetic tokens, our tokens have advantages in manufacture and cost. The sampling parameters of our data are: sample distance = 5 mm; minimum side length = 10 mm; maximum side length = 40 -80 mm.

C. RESULTS AND DISCUSSION
use of pattern optimization reduced the number of sampled patterns by 16% -59% because the shape constraint used by algorithm 1 eliminates many patterns that have approximate right-triangle shapes. Recognition rates on three devices. Token sets 1-5 contain nonoptimized patterns, and token sets 6-10 contain optimized patterns. The recognition rates of the nonoptimized patterns are lower than those of the optimized patterns because of the marker-loss problem. Figure 6 shows the recognition rates of 10 sets of tokens on the three tested devices. The token sets 6-10 with optimized patterns achieved 100% recognition rates on the two capacitive devices, and a 96.4% average recognition rate on the infrared device. The token sets 1-5 with nonoptimized patterns achieved a 94.32% average recognition rate on the 9.7-inch iPad 5, a 96.52% average recognition rate on the ILITEK 23.6-inch capacitive touchscreen, and a 91.88% average recognition rate on the infrared device. According to our observation, the token sets 1-5 contain some patterns that are close to right triangles. When these right-triangle patterns were placed on the screens at a specific angle, the marker-loss situation occurred; then, the recognition algorithm failed to identify the patterns. As we mentioned in Section IV, the optimized tokens can address the marker-loss problem using non-right-triangle shapes, so that token sets 6-10 can achieve 100% recognition rates on capacitive devices. In addition, we also observed that the recognition rate of the infrared screen was easily affected by the ambient light, which reduced the sensing accuracy of touches and caused a large error of touch position. Therefore, the infrared screen had a lower recognition rate than that of the capacitive screens. Table 2 compares our method with other patterns in terms of the number of patterns, number of markers, recognition rate, token size, etc. According to the comparison, PTPG provides the largest number of patterns among the three-marker patterns. It is because the Poisson-disk-sampling-based algorithm explores the whole pattern space and samples patterns in the space while maintaining a certain distance between them. Compared to four-marker patterns, our method provides more patterns on a small token size (40 mm). Nevertheless, as the token size increases, the four-marker patterns benefit from more expressive features and have more patterns than ours. On the other hand, because our tokens use only three touches, the number of our tokens is 33% more than the number of four-marker tokens when simultaneously placed on the capacitive screen. Supporting more tokens could be useful when applied to multi-user tabletop applications. Compared with magnetic tokens and active tokens, PTPG generates a limited number of patterns because of using only spatial information, but our tokens have advantages in manufacture, cost, durability, and volume. Among passive capacitive tokens, our optimized patterns have the highest recognition rate of 100% on capacitive devices because of the solution to the marker-loss problem. In comparison with optical patterns, our patterns achieve a higher recognition rate but support a fewer number of patterns.

VI. APPLICATIONS
We developed three prototype applications and invited six participants (aged 21-43 years) to experience tangible interactions. All the patterns of tokens were created by our pattern generator, and had different shapes and sizes for a variety of usages.  Figure 7a shows the application of human body structure introduction. Users rotate the tangible button to switch the body part and move the tangible ring to reveal the body details. The screen area inside the ring displays the blood vessels and muscle structures beneath the skin. When users are operating, the application plays audio to provide additional information.
A dual-screen application for car customization is shown in Figure 7b. The application provides two tangible buttons, one to operate the camera and the other to adjust the appearance of the car. By controlling the virtual camera, users can observe the car from multiple views, such as the top view and the inside view. Another button can adjust colors, accessories, components to customize the car. Users can acquire the most satisfactory appearance through different combinations.
Tabletop ice hockey supports single-player mode and two-player mode (Figure 7c). Players move handles to hit a puck into the opposite goal while watching for random vortices that could engulf the puck and throw it out of a random direction.
Feedbacks from the six participants demonstrate that tokens with triangular patterns have high recognition rates and low operation latency.
These prototype applications illustrate that triangular patterns generated by our method are applicable to tokens of different size and shape, and have high recognition rates and stability on different devices. We believe that our study can be introduced to a variety of areas such as spacecraft design, collaborative tabletops, and education. Our pattern generation algorithm can also help other capacitive patterns to address the marker-loss problem and improve the recognition rate.

VII. CONCLUSIONS AND FUTURE WORK
We presented PTPG, a method for triangular pattern generation and optimization. PTPG can generate more patterns than previous heuristic methods. This is achieved by introducing Poisson disk sampling to explore the feature vector space and generate a maximal number of patterns. By using sample constraints to optimize the shape of patterns, we addressed the marker-loss problem in a common design of capacitive tokens, and thus improved the recognition rates to 100% on the tested capacitive screens. Experimental results and applications proved that our triangular patterns could be reliably recognized on common tangible surfaces.
There is still potential to improve upon the number of patterns and the robustness of recognition. First, the number of unique patterns for small tokens remains limited compared with active tokens. For example, a token with a 60-mm diameter can use approximately 150 unique patterns. Second, some three-finger gestures tend to be misidentified as triangular patterns. In future research, we will consider a hybrid pattern combining spatial information and other information such as the area of touches to expand the number of patterns. Furthermore, we plan to apply machine learning methods to learn the features of finger gestures, and then we can add a gesture feature constraint to exclude gesture-like patterns in the sampling process. The gesture constraint would reduce the interference between finger gestures and tokens on tangible surfaces.
MINYONG SHI received the Ph.D. degree from the Institute of System Science, Chinese Academy of Sciences, in 1997. He is currently a Professor with the School of Computer Science and Cybersecurity, Communication University of China, Beijing. He is the author of four books and more than 40 articles. His research interests include big data science, computer animation, and computer graphics.
GUANGZHENG FEI received the Ph.D. degree in computer science and technology from the Institute of Software, Chinese Academy of Sciences. Since 2009, he has been a Professor with the School of Animation and Digital Arts, Communication University of China, Beijing. His research interests include computer graphics, visual reality, and computer animation. He joined the Future Laboratory of Tsinghua University as a Research Assistant, in July 2019. He is currently involved in the research on tangible user interface and virtual reality. VOLUME 8, 2020