Small Language Model Agent for the Operations of Continuously Updating ICT Systems

The management of ICT systems requires the implementation of appropriate operational procedures that are aligned with the specific system state and adhere to the latest updates and revisions of manuals and policies. The implementation of autonomous system operation has the potential to reduce the time and burden of human endeavors. Large Language Model (LLM) agents are expected to interact with ICT systems and operate them autonomously. However, existing LLM agents presume a sophisticated reasoning capability of API-based proprietary LLMs, which gives rise to concerns regarding operational costs and confidentiality. Therefore, we propose a novel framework for the Small Language Model (SLM) agent that is designed to adapt to a continuously updating environment. The proposed method addresses the limitation of the reasoning performance of the SLM agent using nested thoughts and prompt reconfiguration. In particular, we empirically identify the shortcut reasoning of the SLM agent and put forward an exemplar selection method to address this issue. We extensively evaluate the proposed method using both synthetic and real-world benchmarks and demonstrate that it significantly enhances the performance of the SLM agent with some computational overhead in comparison with the baselines.