How does LoRA work and why is it memory-efficient?

WHAT THIS TESTS: This question probes whether you understand the mechanics of parameter-efficient fine-tuning beyond buzzwords. The interviewer wants to see that you grasp low-rank matrix decomposition, know exactly where memory savings come from during training, and understand why inference latency does not increase. They also care if you can connect the math to practical deployment constraints like serving multiple task-specific adapters.

A GOOD ANSWER COVERS: A strong response walks through four pieces in order. First, state that LoRA freezes the original pretrained weight matrix W and instead learns a low-rank update delta-W defined as the product of two smaller matrices B and A, where B has shape d by r and A has shape r by k with rank r much smaller than the minimum of d and k. Second, explain that the effective weight during the forward pass is W plus BA, and because W is frozen, gradients flow only through A and B. Third, quantify the efficiency gain: for GPT-3 175B, LoRA reduces trainable parameters by 10,000 times and GPU memory requirements by 3 times compared to full fine-tuning with Adam, because optimizer states and gradients do not need to be stored for the frozen weights. Fourth, note that at inference the BA term can be merged into W by simple addition, so there is zero additional latency unlike adapter layers.

COMMON WRONG ANSWERS: Watch out for three red flags. One is confusing LoRA with quantization or distillation and claiming it reduces the total model size on disk; LoRA only reduces the number of trainable parameters and the memory footprint during training, not the base model size. Another is saying LoRA adds inference latency; the BA matrices can be absorbed into W after training, so inference is identical to a standard linear layer. A third is vague hand-waving about low-rank matrices without explaining the additive update W plus BA or without mentioning that W stays frozen.

LIKELY FOLLOW-UPS: An interviewer might ask how you choose the rank r and whether higher rank always helps. They might ask where LoRA should be applied in a Transformer, such as only the attention weights versus all linear layers. They could also ask how LoRA compares to adapters or prefix tuning, or how you would serve thousands of LoRA adapters on top of a single base model.

ONE CONCRETE EXAMPLE: Suppose you are fine-tuning a 175B parameter GPT-3 model. Full fine-tuning with Adam stores two optimizer states per parameter, so training requires hundreds of gigabytes of GPU memory just for the optimizer states and gradients. With LoRA applied to the query and value projection matrices using rank 4, you might train only 35 million parameters instead of 175 billion. That drops the optimizer state memory by roughly threefold, lets you fit the job on fewer GPUs, and after training you merge the small BA products into the original W matrices so serving remains unchanged.