Entropy as Temporal Information Density

Besides the spatial contents from current sensory inputs, the relevant contexts from past frames are very important for temporal processing tasks (e.g., speech recognition, video analysis, and natural language processing). Our Developmental Network (DN) has demonstrated the ability to learn any emergent Turing Machine (TM), it can learn feature patterns from current and attended past natural inputs as their states. We have shown the dense actions can serve as natural sources of contexts, and the DN can autonomously generate actions as contexts when dealing with sequences. In this work, we use entropy to define and measure the information density of the temporal sequences. We also introduce the "free of labeling" property, which can help DN deal with a large number of states emerging from dense contexts. We experimented with DN for phoneme recognition as the example of auditory modality, but the principles are modality independent. Our experimental results showed the denser contexts extracted from the sequences, the better DN can perform. With the quantization of information density by entropy, we have better understanding of how to provide the contexts when training DN. This work is an important step toward enabling machines to autonomously abstract concrete concepts from contexts through life-long development.