Combinatorics in AI: Principles & Algorithms An Extended Abstract

Therein we describe a few cases where combinatorics appears in machine learning. The fundamental problem of learning is intimately connected to combinatorial geometry. The geometric invariant Vapnik-Chvronenkis dimension or VC-dimension [13], plays a crucial role in the analysis of machine learning procedures especially the suitability of an algorithm to shatter a set of labeled examples. The notion that data is represented as a correlation matrix becomes practical when regarded as a graph on which graph algorithms are used to extract information about the underlying data. Algorithms which partition a graph are, in fact, stratification procedures of the underlying data. Yet another combinatorial application to statistical models involve selecting a subset of variables “variable selection” used in linearized models like the ubiquitous logistic regression, or Cox partial likelihood models. Those techniques are particularly useful in modeling with genomics data which tend to be high dimensional. Indeed, the number of genes under study is typically in the thousands while in clinical trials the number of subjects under study may be rather small, possibly in the hundreds. Having thousands of features and only a few hundreds of subjects poses difficult modeling challenges. Feature selection methods work well when there are far fewer features than examples and statistical feature selection methods do not seem to work well in such extreme cases. Combinatorial methods embedded in a workflow help alleviate the problem by eliminating redundant inputs. We demonstrate how combinatorial algorithms, which extract useful information from the matrix of pairwise correlations are used in feature selection. In fact, we show how computing special variants of what is known as “maximal independent set” could help solve intractable statistical modeling problems where classical selection methods are confounded by excessively larger feature sizes.