LVL1
0 Total XP0 / 100 XP to Lvl 2
1 Day Streak
Parameter-Efficient Fine-Tuning
LoRA & PEFT: Low-Rank Adaptation Lab
Understand how Low-Rank Adaptation (LoRA) decomposes large dense weight updates into tiny low-rank matrices A and B, allowing 70B parameter models to be fine-tuned on consumer GPUs with zero inference latency.
LORA WEIGHT DECOMPOSITION ARCHITECTURE
FROZEN (Read-Only)
Base Weights ($W_0$)
Dense projection matrix: [4096 × 4096]
16.78M parameters
Zero gradients computed during training+Scale 2.00×
TRAINABLE ADAPTER
Low-Rank ($B \cdot A$)
Matrix B [4096 × 8] • Matrix A [8 × 4096]
65.5k parameters
99.61% fewer trainable parameters!LORA RANK (r):r = 8
r=2 (Ultra-fast)r=8 (Standard)r=64 (High expressivity)
LORA ALPHA (α):α = 16
Scaling multiplierα / r = 2.00
HIDDEN DIMENSION:4096
Why LoRA Introduces Zero Extra Latency at Deployment
During training, keeping $W_0$ frozen and learning $B \cdot A$ saves massive GPU VRAM because gradients and optimizer states only exist for the tiny $A$ and $B$ matrices.
When training is complete, the product $\frac{\alpha}{r}(B \cdot A)$ can be mathematically added directly into the base matrix:
W_deployed = W_0 + (alpha / r) * (B @ A)
The deployed model runs standard single matrix multiplications without any auxiliary adapter layers!