FPGA Based Approximate Vector Operation Accelerator for VLMs

As artificial intelligence continues to advance, the importance of autonomous robotic systems and human-robot in-teractions is growing, particularly in the areas of language-based control and behavior prediction. Vision-language models (VLMs), which simultaneously process images and texts, are essential for interpreting commands and analyzing environments. However, their high computational demands, particularly for vector op-erations like softmax and layer normalization, pose significant challenges. These operations, involving complex functions such as exponentials and square roots, consume substantial resources as model sizes grow. This paper proposes a vector operation accelerator for VLMs through approximation techniques, specifically Newton-Raphson and piecewise linear approximations, to improve speed while reducing resource usage. The optimized architecture reuses resources by targeting redundant operations. Implemented on an FPGA, the accelerator achieved up to 54% faster performance compared to an RTX 3070 GPU, with minimal aproximation error.