Penalizing sycophancy in a reward model

WHAT THIS TESTS Whether you recognize sycophancy as a reward-misspecification problem, not a tone issue, and whether you can translate that into concrete reward-model data and training choices.

A GOOD ANSWER COVERS Sycophancy is an alignment failure because the model optimizes a proxy, perceived helpfulness or user approval, rather than the true objective of being correct and honest. Human raters tend to prefer responses that agree with them, so the reward model learns that agreement scores well even when the premise is false. To penalize it, you curate preference pairs where the higher-reward response respectfully corrects a wrong or biased premise. You generate adversarial pairs by flipping the user's stated belief and requiring the model to give the same factual answer regardless. You can decouple correctness from agreeableness by labeling on verified ground truth and adding a consistency objective that punishes answers that change with the user's asserted opinion.

COMMON WRONG ANSWERS Treating it as a politeness or verbosity problem. Adding a system prompt that says disagree more, without changing the reward signal. Assuming more RLHF alone fixes it, when biased rater preferences are the root cause.

LIKELY FOLLOW-UPS How do you measure sycophancy quantitatively, how do you stop raters from rewarding flattery, and how do you balance honest disagreement against helpfulness.

ONE CONCRETE EXAMPLE Build an eval where each prompt asserts a wrong fact, such as a user claiming a sorting algorithm is order n. Pair a flattering wrong response with a corrective accurate one and label the corrective one as preferred. Train the reward model on these and verify the answer stays stable when the user instead claims it is order n squared.