EAI-SIM: An Open-Source Embodied AI Simulation Framework with Large Language Models

In the era of embodied intelligence, developing and testing various intelligent algorithms for robots, including unmanned aerial vehicles (UAVs) and manipulators, poses significant challenges. Researchers are increasingly depends on simulation technologies to evaluate the performance of new al-gorithms across various conditions and applications. The advent of large language model (LLM) technology creates new demand for simulations of robotics tasks with the help LLMs. This study introduces EAI_SIM, a simulation system capable of controlling unmanned aerial vehicles (UAVs) and manipulators based on user prompts. The modular framework acts as an extension of NVIDIA Isaac Sim, facilitating real-time simulation of UAVs and manipulators in photo-realistic environments. Thanks to the modular implementation and user-friendly Graphical User Interface (GUI), it can be seamlessly integrated with the widely used PX4-Autopilot and ROS2 system. To showcase its performance, a variety of control Application Programming Interfaces (APIs) are implemented within EAI_SIM. A range of tasks can be achieved by combining different APIs. Supplementary material, including code and documentation for this framework, is provided to benefit the community. The source code can be found at https://github.com/PengICS/eai_sim, and a demo video can be found at https://youtu.be/YcNkwo_PNnQ.