This paper shows the possible development of particle swarm optimization (PSO)-based fuzzy-neural networks (FNNs) that can be employed as an important building block in real robot systems, controlled by voice-based commands. The PSO is employed to train the FNNs that can accurately output the crisp control signals for the robot systems, based on fuzzy linguistic spoken language commands, issued by a user. The FNN is also trained to capture the user-spoken directive in the context of the present performance of the robot system. Hidden Markov model (HMM)-based automatic speech recognizers (ASRs) are developed, as part of the entire system, so that the system can identify important user directives from the running utterances. The system has been successfully employed in two real-life situations, namely: 1) for navigation of a mobile robot; and 2) for motion control of a redundant manipulator.