Home / Current Issue / Paper 1718503
Dynamic LoRA Rank Selection for Parameter-Efficient Fine-Tuning Under Memory-Constrained Environments
Subject area: Science,Engineering and Technology · Area of research: Artificial Intelligence
DOI: https://doi.org/10.64388/IREV9I11-1718503
Abstract
Large Language Models (LLMs) have achieved remarkable performance across various Natural Language Processing (NLP) tasks; however, fine-tuning these models requires significant computational resources and memory. Parameter-Efficient Fine-Tuning (PEFT) techniques such as Low-Rank Adaptation (LoRA) reduce training costs by updating only a small number of parameters. Traditional LoRA approaches generally use a fixed rank value throughout training, which may lead to inefficient memory utilization and suboptimal model performance in memory-constrained environments. This paper proposes a Dynamic LoRA Rank Selection approach that adaptively adjusts the rank during fine-tuning based on memory availability, model complexity, and task requirements. The proposed method aims to improve training efficiency while maintaining model accuracy and reducing computational overhead. Experimental analysis demonstrates that dynamic rank adaptation can achieve better resource utilization and comparable performance when compared to static-rank LoRA methods. The proposed approach is especially beneficial for edge devices, low-resource systems, and environments with limited GPU memory, enabling efficient deployment of large-scale AI models with reduced hardware requirements.
Keywords
Dynamic Rank Selection, Large Language Models, Low-Rank Adaptation (LoRA), Memory-Constrained Environments, Parameter-Efficient Fine-Tuning (PEFT)
How to cite this paper
@article{1718503,
author = {Siddhartha Goola, C. Akhila Krishnan},
title = {Dynamic LoRA Rank Selection for Parameter-Efficient Fine-Tuning Under Memory-Constrained Environments},
journal = {Iconic Research And Engineering Journals},
year = {2026},
volume = {9},
number = {11},
pages = {4999-5007},
issn = {2456-8880},
url = {https://www.irejournals.com/formatedpaper/1718503.pdf},
abstract = {Large Language Models (LLMs) have achieved remarkable performance across various Natural Language Processing (NLP) tasks; however, fine-tuning these models requires significant computational resources and memory. Parameter-Efficient Fine-Tuning (PEFT) techniques such as Low-Rank Adaptation (LoRA) reduce training costs by updating only a small number of parameters. Traditional LoRA approaches generally use a fixed rank value throughout training, which may lead to inefficient memory utilization and suboptimal model performance in memory-constrained environments. This paper proposes a Dynamic LoRA Rank Selection approach that adaptively adjusts the rank during fine-tuning based on memory availability, model complexity, and task requirements. The proposed method aims to improve training efficiency while maintaining model accuracy and reducing computational overhead. Experimental analysis demonstrates that dynamic rank adaptation can achieve better resource utilization and comparable performance when compared to static-rank LoRA methods. The proposed approach is especially beneficial for edge devices, low-resource systems, and environments with limited GPU memory, enabling efficient deployment of large-scale AI models with reduced hardware requirements.},
keywords = {Dynamic Rank Selection, Large Language Models, Low-Rank Adaptation (LoRA), Memory-Constrained Environments, Parameter-Efficient Fine-Tuning (PEFT)},
month = {May},
doi = {https://doi.org/10.64388/IREV9I11-1718503}
}