Describe supervised fine-tuning for a pre-trained language model

WHAT THIS TESTS: This question checks whether you understand supervised fine-tuning as the critical post-training alignment step that turns a general pre-trained base model into an instruction-following assistant. Interviewers want to see that you know SFT requires curated demonstration data, preserves the causal language modeling objective, and differs fundamentally from pre-training on raw corpora.

A GOOD ANSWER COVERS: First, define SFT as continued training on a smaller, high-quality dataset of prompt-response pairs where the model learns to emulate desired outputs. Second, describe the dataset types: standard prompt-completion pairs or conversational message lists with user and assistant roles. Third, state the primary objective remains next-token prediction, but loss is typically computed only on the completion or assistant tokens while prompt tokens are masked. Fourth, note practical details such as using a small learning rate, training for only a few epochs, and optionally applying chat templates to format conversations correctly.

COMMON WRONG ANSWERS: Do not say SFT uses raw unlabeled internet text like pre-training; it relies on curated demonstrations. Do not claim the objective switches to classification, regression, or reward maximization; it stays causal LM. Avoid stating that both prompts and completions contribute equally to the loss; best practice masks the prompt to avoid wasting gradient updates on inputs. Do not suggest SFT automatically adds new parameters; standard full fine-tuning updates existing weights, while parameter-efficient methods like LoRA are separate choices.

LIKELY FOLLOW-UPS: Expect questions on how SFT differs from RLHF or preference optimization methods like DPO. Interviewers may ask why prompt masking matters, how to choose between full fine-tuning and LoRA, what hyperparameters prevent catastrophic forgetting, or how multi-turn conversations are formatted and truncated.

ONE CONCRETE EXAMPLE: Using Hugging Face TRL, you might load a pre-trained Qwen 0.6B model and a conversational dataset such as Capybara. You format each example as a messages list with user and assistant roles. The SFTTrainer automatically applies the model's chat template and computes next-token prediction loss only on the assistant content. You train with a small learning rate for one to three epochs, producing a model that follows instructions in the style demonstrated by the dataset.

Source: Hugging Face TRL documentation