Alignment
25 bites tagged Alignment — interview questions with model answers, and 60-second explainers.
Components of a project technical roadmap
State goals, phased milestones, dependencies and risks, and success metrics; use it to align the team and report upward. structuring and communicating a roadmap. confusing a roadmap with a rigid dated Gantt chart.
Scalable oversight of superhuman models
Humans cannot judge outputs beyond their expertise, so feedback degrades; techniques like AI debate or recursive reward modeling decompose judgment. supervising models you cannot fully evaluate.
Penalizing sycophancy in a reward model
Sycophancy is reward proxy gaming where agreeableness substitutes for correctness; counter it with truth-anchored labels, perturbed-premise pairs, and consistency checks. reward hacking and truthfulness.
Constitutional AI versus standard RLHF
A written principle set guides self-critique and revision, plus AI feedback (RLAIF) replaces human preference labels. alignment methods beyond human feedback. calling it just RLHF with extra steps or human-only labeling.
Diagnosing sycophancy from RLHF
Annotators reward agreeable, inoffensive answers so the reward model learns to favor them, fix by diversifying labelers, rewarding factual correctness over agreeableness, and… understanding how reward modeling induces sycophancy.
What RLHF is and the safety problem it solves
Collect human preference rankings, train a reward model, fine-tune the policy with PPO; it aligns outputs with human intent the loss function cannot specify. grasp of alignment via RLHF.
Setting up an LLM-as-a-judge evaluation
Define rubric, prefer pairwise comparison, randomize order, calibrate against humans; control position, verbosity, and self-preference bias. rigorous LLM-as-judge design.
Aligning text and image representations
Contrastive learning like CLIP pulls matched image-text pairs together and pushes mismatches apart; alternatively projection layers map one modality into a frozen model's space. cross-modal representation alignment.
Direct Preference Optimization explained
DPO reparameterizes the RLHF reward in terms of the policy itself, turning alignment into a simple classification loss on preference pairs with no separate reward model or PPO. understanding of DPO versus RLHF.
Reward models in RLHF and PPO
It learns from human preference comparisons to score responses, then supplies the reward signal that PPO maximizes while a KL penalty keeps the policy near the reference. understanding of the reward model in RLHF.
Center a div with Flexbox on the parent
Set display flex on parent, justify-content center for the main axis, align-items center for the cross axis. fluency with Flexbox alignment axes.
Value Learning
Value learning is the AI-safety approach of having a system infer what humans actually value, rather than optimizing a hand-coded proxy, so that capable agents pursue goals aligned with human intent even in novel situations.
Alignment: Invisible Lines That Structure Interfaces
Alignment is invisible scaffolding: eyes travel along shared edges instead of hunting anchors. In Figma, left-aligning labels and inputs turns chaos into readable flow. Center-aligning long text is the footgun; ragged invisible edges increase reading friction.
What are the risks of a roadmap without product strategy?
Risks include building wrong things fast, orphaned features, and tradeoff paralysis. Whether you see roadmaps without strategy misalign execution. Blaming PMs rather than showing engineering can surface strategic gaps.
Vision Storytelling: Turn Roadmaps Into Shared Futures
Vision storytelling makes strategy tangible by narrating the future your team is building. Use it when roadmaps explain what but not why. The footgun is confusing slogans for stories; without a user, conflict, and resolution, you get branding, not alignment.
What does the KL-divergence penalty do in RLHF PPO, and if zeroed?
It tests RLHF reward hacking awareness. The KL penalty anchors PPO to the reference model to stop mode collapse; zeroing it causes over-optimization against the proxy reward model, yielding incoherent outputs.
Walk through RLHF's three stages, outputs, and purposes.
Tests your grasp of the RLHF pipeline end-to-end. A strong answer lists: pretrain an instruction-following LM, train a reward model outputting a scalar preference score, then fine-tune the LM via RL.
How does struct field ordering affect memory layout in Go and Rust?
It tests alignment, padding, and compiler layout knowledge. A strong answer explains that alignment inserts padding, Go and Rust keep declared order, and reordering by size can shrink size. Red flag: saying order is irrelevant or that compiler auto-packs.
mainAxisAlignment vs crossAxisAlignment in Column, and textDirection in Row
Tests Flutter flex axis awareness. In a Column, mainAxisAlignment is vertical and crossAxisAlignment is horizontal. In a Row, textDirection governs the main axis, not crossAxisAlignment; verticalDirection governs the cross axis.
Strategic Narrative: Your Company's Source Code for Story
A strategic narrative is the source code for your company's story, defining its unique value. It's the blueprint for marketing, sales, and internal decisions, ensuring everyone tells the same story.
Visiontype: Prototyping Your 3-5 Year Product Future
A visiontype is an interactive prototype of your product's 3-5 year future, making abstract goals tangible. It aligns teams on a long-term direction, breaking the cycle of purely incremental updates. The biggest mistake is creating it in a silo.
Orthogonality Thesis: An AI's Intelligence and Goals Are Unrelated
The Orthogonality Thesis states an AI's intelligence and its ultimate goals are independent. A superintelligence could pursue any objective, from beneficial to catastrophic, with equal capability.
Explain Supervised Fine-Tuning, RLHF, and DPO
This tests your understanding of modern LLM alignment techniques. A strong answer explains that Supervised Fine-Tuning (SFT) teaches the model a task via imitation, while RLHF and DPO align it with human preferences. RLHF uses a reward model and reinforcement learning, whereas DPO is a simpler, direct optimization method. The key red flag is conflating these distinct stages or failing to explain the 'reward model' step in RLHF.
Constitutional AI: Teaching an AI Right from Wrong
Constitutional AI teaches a model to be harmless by making it follow a set of principles—a constitution—instead of relying on human-labeled examples of bad behavior. This self-correction process, called Reinforcement Learning from AI Feedback (RLAIF), is used to align powerful models, enabling them to refuse harmful requests while explaining their reasoning. The entire system's safety, however, hinges on the quality and completeness of the initial human-written constitution.
Get Alignment bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.