Pre-training versus fine-tuning an LLM
grasp of the two-stage LLM training lifecycle.
pre-training is broad self-supervised next-token prediction on huge corpora at massive cost; fine-tuning adapts on small labeled data cheaply.
What's really being asked
The interviewer wants to confirm you understand that an LLM is built in stages with very different data, cost, and goals, rather than one monolithic training run.
The full answer
Pre-training is self-supervised: the model predicts the next token over trillions of tokens of unlabeled web text, books, and code. The objective is generic language modeling, so no human labels are required, but the compute is enormous, often thousands of GPUs over weeks and a large fraction of total cost. Fine-tuning starts from those frozen-then-updated weights and adapts them on a much smaller, curated dataset, frequently labeled or instruction-formatted. The objective is narrow: follow instructions, match a domain style, or solve a specific task. Cost is orders of magnitude lower, and techniques like LoRA reduce it further by updating only a few parameters.
The mistakes people make
Saying pre-training uses labeled data, conflating the two objectives, or claiming fine-tuning injects large amounts of new factual knowledge. Fine-tuning mostly steers and surfaces capabilities already learned during pre-training; it rarely teaches genuinely new facts reliably.
What usually comes next
Expect questions on parameter-efficient fine-tuning such as LoRA, on instruction tuning versus continued pre-training, on catastrophic forgetting, and on when retrieval augmentation is preferable to fine-tuning for adding knowledge.
A concrete example
A base model trained on web-scale text learns grammar, facts, and reasoning patterns through next-token prediction. A startup then fine-tunes it on five thousand customer-support conversations so it answers in the company tone and follows the support playbook. The base run might cost millions of dollars; the fine-tune runs in hours on a single machine, yet it does not add the company's internal pricing tables, which are better supplied via retrieval at inference time.
Interview question
A team wants their LLM to reliably cite this week's internal sales figures. Why is fine-tuning the base model a poor primary tool for this requirement?
- a.Fine-tuning can only be done on unlabeled data, so facts cannot be added
- b.Fine-tuning requires more compute than the original pre-training run
- c.Fine-tuning mainly steers existing capabilities and is unreliable for injecting fresh, specific factsCorrect
- d.Fine-tuning always erases the model's grammar and language ability
Why? this is the answer
Fine-tuning adapts and steers learned behaviors rather than dependably storing new facts, so retrieval is better for fresh data. It does not cost more than pre-training, nor does it wipe out language ability.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #pre-training
- #fine-tuning
- #self-supervised
- #llm-training
- #transfer-learning
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles