Dynamic LoRA Rank Selection for Parameter-Efficient Fine-Tuning Under Memory-Constrained Environments

• Author(s): Siddhartha Goola; C. Akhila Krishnan
• Paper ID: 1718503
• Page: 4999-5007
• Published Date: 01-06-2026
• Published In: Iconic Research And Engineering Journals
• Publisher: IRE Journals
• e-ISSN: 2456-8880
• Volume/Issue: Volume 9 Issue 11 May-2026

Abstract

Large Language Models (LLMs) have achieved remarkable performance across various Natural Language Processing (NLP) tasks; however, fine-tuning these models requires significant computational resources and memory. Parameter-Efficient Fine-Tuning (PEFT) techniques such as Low-Rank Adaptation (LoRA) reduce training costs by updating only a small number of parameters. Traditional LoRA approaches generally use a fixed rank value throughout training, which may lead to inefficient memory utilization and suboptimal model performance in memory-constrained environments. This paper proposes a Dynamic LoRA Rank Selection approach that adaptively adjusts the rank during fine-tuning based on memory availability, model complexity, and task requirements. The proposed method aims to improve training efficiency while maintaining model accuracy and reducing computational overhead. Experimental analysis demonstrates that dynamic rank adaptation can achieve better resource utilization and comparable performance when compared to static-rank LoRA methods. The proposed approach is especially beneficial for edge devices, low-resource systems, and environments with limited GPU memory, enabling efficient deployment of large-scale AI models with reduced hardware requirements.

Keywords

Dynamic Rank Selection, Large Language Models, Low-Rank Adaptation (LoRA), Memory-Constrained Environments, Parameter-Efficient Fine-Tuning (PEFT)

Citations

IRE Journals:
Siddhartha Goola, C. Akhila Krishnan "Dynamic LoRA Rank Selection for Parameter-Efficient Fine-Tuning Under Memory-Constrained Environments" Iconic Research And Engineering Journals Volume 9 Issue 11 2026 Page 4999-5007 https://doi.org/10.64388/IREV9I11-1718503

IEEE:
Siddhartha Goola, C. Akhila Krishnan "Dynamic LoRA Rank Selection for Parameter-Efficient Fine-Tuning Under Memory-Constrained Environments" Iconic Research And Engineering Journals, 9(11) https://doi.org/10.64388/IREV9I11-1718503