Regression learning on patches

Neural networks often do poorly at representing dis-continuous functions, or even just functions with rapid transitions in the response surface between closely-spaced points in feature space. However, such `edges' in the data can be a useful way to partition the feature space in order to train specialised learners for individual regions. This is particularly beneficial where these regions are relatively simple, and hence low-complexity learners can be used successfully on them. Another benefit of such an approach is that it is easily parallelisable: the specialised learners use independent partitions of the data, and so they can be trained in parallel, while output prediction is based on the output of just one network, so there is no need to combine predictions. We introduce an algorithm to partition the data that is inspired by Finite Element Tearing and Interconnecting. Using an implementation based on a decision tree with neural networks at the leaves, we demonstrate our approach for regression learning on patches of the feature space. We use both artificial and real-world datasets to show that, in some use cases, this method can outperform conventional neural networks that see the entire feature set in the original training.