Determination of Atomic Actions of a Human Operator in a Collaborative Robotic Environment Using LLM

The article presents a hybrid approach to recognizing human operator actions in industrial collaborative environments, combining computer vision methods and the semantic capabilities of large language models. The scientific novelty of the work lies in the development of a hybrid method for determining the actions of a human operator in a collaborative robotic environment using large language models. The paper presents a method for fine-tuning the immersion model in a subject area based on the developed software adapted for analyzing key points of body POSITION, which formalizes the rules for classifying atomic actions (tilt, turn, lift) considering the dynamics of changes in coordinates and angles of articulation. This allows language models to generate descriptions of movements even with partially overlapping points, eliminating the dependence on large data markup. To assess the accuracy of forecasts, a modified weighted semantic action accuracy indicator is proposed, combining semantic similarity based on implementations and expert rules from a predefined matching matrix. The experiments were conducted using a multi-camera system equipped with three cameras. The comparative analysis was carried out using three language models. The Llama3.2-Vision model demonstrated the best results. The results obtained confirm that the proposed approach makes it possible to interpret the actions of a human operator, ensuring safe and effective interaction between a human and a collaborative robot. The developed methods serve as the basis for creating adaptive collaborative systems capable of analyzing both individual actions and their technological sequences.