A method for designing the internal representation of neural networks

A method for determining the states of the hidden units of feedforward neural networks for an arbitrary output function is proposed. The method uses properties of a vector space spanned by state vectors. The state vector used represents a set of states (input states, inner states, or output states) of a unit for many sample data. The inner state vector is the linear combination of the input state vectors and is nonlinearly transformed into the output state vector. Internal representation can be expressed by the output state vectors for the hidden units, which are the input state vectors necessary for the output unit. The problem is how to determine the appropriate internal representation in order to produce an arbitrary output function. This method, called the orthogonal complement method (OCM), is based on the orthogonality between the subspace spanned by the input state vectors and its orthogonal complement. The nonlinearity of the vector transformation is treated as a constraint with respect to the inner state vector. Unknown components of the output state vectors for the hidden units can be determined from this orthogonality, and the number of necessary hidden units can be estimated from the dimension of the subspace spanned by the input state vectors. The OCM is very useful for binary output systems, since this calculation can be simplified by using a basis of the orthogonal complement. Using a basic procedure of the OCM, a minimum network can be designed for about 70% of all four-variable Boolean functions