Automatic Spoken Language Acquisition Based on Observation and Dialogue

Human babies are born without knowledge of any specific language. They acquire language directly from observation and dialogue without being limited by the availability of labeled data. We propose spoken language acquisition agents that simulate the process. Such an ability requires multiple types of learning, including 1) word discovery, 2) symbol grounding, 3) message generation, and 4) pronunciation generation. Several studies have targeted one or combined learning types to elucidate human intelligence and aimed to equip spoken dialogue systems with human-like flexible language learning ability. However, their language ability was partially lacking some of the components. Our agents are the first to integrate them all. Our key concept is to design an architecture to integrate unsupervised, self-supervised, and reinforcement learning to utilize clues naturally existing in raw sensory signals and drive the learning based on the agent’s intrinsic motivation. Experimental results show agents successfully acquire spoken language from scratch by interacting with an environment to act by speaking. Our proposed focusing mechanism significantly improves learning efficiency. We also demonstrate that our agents can learn neural vocoder and the concept of logical negation as a part of language acquisition.