Predicting Token Length for Large Language Models with Multi-Branch Headers

Large Language Models (LLMs) are revolutionizing cloud robotics by enabling sophisticated capabilities such as task comprehension, collaborative planning, and decision-making. However, real-time applications require efficient communication, where token length directly impacts computation, bandwidth, and latency. This study introduces a novel lightweight framework for real-time token length prediction in LLMs, designed to work concurrently with token generation. The approach includes several key innovations: Multi-Branch Headers (MBH), where our Estimated Steps to Terminate (EST) header predicts token length concurrently with the language model header, Query as Trainable Parameters (QaT), which treats the query in the transformer block as trainable parameters, and Token Length Bins (TLB), converting token length prediction into a multi-class classification problem. Additionally, we propose Token-level Data Segmentation (ToDS), which reduces bias and improves the reuse of semantic features from the base LLM. Experimental results demonstrate significant advancements, highlighting the high performance of our approach in token length estimation, particularly in terms of both accuracy and zero-shot ability. Code is available at token-length-prediction.