SortingBot: Leveraging Large Language Models and 3D Vision for Multi-Category Material Sorting

This paper introduces an LLM-centered control framework for industrial robots, designed to address key manufacturing tasks such as task formulation, resource selection, environment initialization, path planning, and operations. The framework employs agents powered by large language models (LLMs) to interpret human instructions, plan and execute tasks, and interact with humans and their environment in a goal-oriented manner. We demonstrate the practicality of our approach through simulations in various manufacturing scenarios. Our contributions include a novel LLM-centric framework that enhances adaptability and efficiency in small-batch and customized production, multimodal robots capable of processing visual and natural language inputs, and detailed validation of singleagent and multi-agent work modes. The components of our framework include voice, perception, vision, communication, robot control, and safety modules, with the Industrial Large Language Model acting as the core decision node. This framework significantly enhances the adaptability and efficiency of industrial robots, paving the way for more intelligent, flexible, and context-aware robotic systems in manufacturing environments.