FbBot: A Cross-Lingual Social Automation Framework Based on Multimodal Perception

The rapid UI updates, multilingual demands, and complex interaction scenarios in social platforms pose significant challenges to automation tools. Existing approaches primarily rely on API/UI tree-based methods or emerging general-purpose vision solutions enabled by recent advances in LLMs. However, these methods struggle to balance dynamic adaptability and semantic understanding, resulting in insufficient robustness of automation tools in real-world scenarios. To address this, we present FbBot, a vision-semantics dual-driven social automation framework for Facebook’s mobile platform. Building upon the hierarchical multiagent architecture of the general-purpose vision-based approach-we integrate an LLM-powered foundation with an icon detector and a multilingual interaction engine. By constructing a dynamic prompt library, our framework reduces reliance on fixed visual features and enhances image-text recognition accuracy in linguistically mixed interfaces. Additionally, we introduce a self-learning module to enable longterm performance optimization. Experimental results on Facebook’s English and Traditional Chinese interfaces demonstrate that FbBot significantly outperforms traditional solutions in both task completion success rate and operational efficiency across complex social interaction scenarios.