A Comprehensive Review of Low-Rank Decomposition Techniques for Parameter Efficient Fine-Tuning

Recently, low-rank decomposition methods have brought about powerful methods for tuning deep learning models, reducing training parameters by orders of magnitude and maintaining performance. These methods, including LoRA, Delta LoRA, DyLoRA, WARP, KronA, as well as Intrinsic SAID, have seen widespread usage in many application domains, ranging from natural language processing (NLP) to computer vision, to multimodal tasks. Despite their success in reducing computational cost, these methods also suffer from some limitations including the risk of execution loss, especially for complex task, and the undesirable bipolarity of the model parameters. In addition, before most have addressed the problem, no pre-trained models optimised for low rank techniques have been made available and necessitate resource intensive fine tuning. This paper review the strengths and limitations of each method and point out the tradeoff between efficiency and performance. In addition, it presents possible improvements, including hybrid models, and developments in adaptability and scalability for use in operations of real-world applications like healthcare and finance. Future directions include: increasing efficiency; generalizing the use across many different architectures relevant to AI; and tackling the current limitations to make low rank decomposition techniques more widely usable in AI software.