On the Mathematics of Mathematical Handwriting Recognition

Accurate computer recognition of handwritten mathematics offers to provide a natural interface for mathematical computing, document creation and collaboration. Mathematical handwriting, however, provides a number of challenges beyond what is required for the recognition of handwritten natural languages. For example, it is usual to use symbols from a range of different alphabets and there are many similar-looking symbols. Many writers are unfamiliar with the symbols they must use and therefore write them incorrectly. Mathematical notation is two-dimensional and size and placement information is important. Additionally, there is no fixed vocabulary of mathematical "words" that can be used to disambiguate symbol sequences. On the other hand, there are some simplifications. For example, symbols do tend to be well-segmented. With these charactersitics, new methods of character recognition are important for accurate handwritten mathematics input. We present a geometric theory that we have found useful for recognizing mathematical symbols. Characters are represented as parametric curves approximated by certain truncated orthogonal series. This maps symbols to the low dimensional vector space of series coefficients. The Euclidean distance in this space is closely related to the variational integral between two curves and may be used to find similar symbols very efficiently. Training data sets with hundreds of classes are seen to be almost linearly separable, allowing classification by ensembles of linear SVMs. In this setting, we find it particularly effective to classify symbols by their distances under various norms to the convex hulls of nearest neighbors from known classes. By choosing the functional basis appropriately, the series coefficients can be computed in real-time, as the symbol is being written. Using truncated series for integral invariant functions, orientation- and shear-independent recognition is achieved. We have seen that the distances to the SVM separating planes or to the convex hulls of nearest neighbors provide a reliable confidence measure for classifications. This allows the combination geometric recognizers with n-gram based recognizers. To this end we can use statistical information from corpora of mathematical research papers and university engineering mathematics texts. The relative frequency of symbols depends on the mathematical domain, and can even be used to find subject classification of mathematical documents. We are currently investigating how orthogonal series representations may be used to compress ink traces in a form that may allow recognition without decompression of the database. Preliminary work on this problem is reported. We find this geometric appraoch, based on distances in a space of functional approximations, quite appealing. It gives a single, coherent view and several related techniques with remarkably high recognition rates that do not rely on peculiarities of the symbol set.