Grounding symbols in sensorimotor categories with neural networks

The symbol grounding problem is: how can you ground the meanings of symbols autonomously, i.e. without the mediation of external interpretation? One solution to the symbol grounding problem is to abandon symbol systems altogether. The alternative is to abandon pure symbol systems for hybrid symbolic/nonsymbolic systems in which the symbol-meaning connection is not interpretation-dependent but autonomous and direct. It is such an approach that I describe in this paper. Suppose that the features in the sensory projection that allow the system to correctly categorise the objects of which they are the projections are found by some sort of learning mechanism, e.g. a neural net that takes sensory projections as input and gives category names as output. The name would be connected to the category of objects it denotes by a physical link including the neural net and the sensory projection of the objects. That would be the grounding, and the composite shape of the arbitrary name plus the net plus the sensory projection would no longer be arbitrary. If the combinations into which the name could enter were governed not only by syntactic rules operating on the arbitrary shape of the name alone, but also by the nonarbitrary shape of its grounding, then we would have a kind of doubly-constrained hybrid system, with heritable grounding. Neural nets can do simple categorisation, but it still remains to embed them in autonomous sensorimotor systems that also have compositional capacity, to see how the two sets of constraints interact. One clue about this interaction may come from human categorical perception.