Full fine-tuning or LoRA on a tight compute budget?

WHAT THIS TESTS: The interviewer is checking whether you understand the practical economics of adapting large models to many tasks. They want to see you compare full fine-tuning against parameter-efficient methods like LoRA under resource constraints. Key concepts include trainable parameter count, storage overhead, compute cost, and performance parity.

A GOOD ANSWER COVERS: First, recommend LoRA or another PEFT method given a limited compute budget and multiple tasks. Second, explain that PEFT methods only fine-tune a small number of extra model parameters, which significantly decreases computational and storage costs. Third, note that these methods yield performance comparable to a fully fine-tuned model, making training accessible on consumer hardware. Fourth, contrast this with full fine-tuning, which updates all parameters and requires storing a separate complete copy of weights for each task, causing costs to scale linearly with task count. Fifth, mention inference implications: with LoRA you keep one base model and swap lightweight adapters, whereas full fine-tuning forces you to load entirely separate models or dedicate large memory to each checkpoint.

COMMON WRONG ANSWERS: A major red flag is claiming that full fine-tuning always outperforms PEFT and is therefore worth the cost. Another mistake is ignoring storage overhead and only discussing training FLOPs; senior candidates should address the full lifecycle including checkpoint storage and inference serving. A third error is suggesting you can average full fine-tuned weights across tasks without mentioning catastrophic forgetting or interference, which reveals shallow understanding of multi-task dynamics.

LIKELY FOLLOW-UPS: The interviewer may ask how you would serve hundreds of task-specific LoRA adapters efficiently at inference time. They might probe on rank selection and how it affects model capacity versus compute. They could ask when full fine-tuning is actually justified, such as when the pretraining distribution is very different from the target domain or when maximum capacity is needed and budget is not constrained. Another follow-up is adapter merging or composition strategies for multi-task inference.

ONE CONCRETE EXAMPLE: Imagine you have a large base LLM and twenty downstream tasks. Full fine-tuning would require training and storing twenty complete checkpoints the same size as the base model, which is prohibitively expensive in GPU memory and disk space. Using LoRA, you train only a small number of extra parameters per task, store twenty small adapter checkpoints, and keep a single shared base model. This sharply reduces training cost and makes deployment feasible on consumer hardware, while the PEFT documentation notes that such methods achieve performance comparable to fully fine-tuned models.

Source: huggingface.co/docs/peft