Alignment
25 bites tagged Alignment: interview questions with model answers, and 60-second explainers.
Components of a project technical roadmap
State goals, phased milestones, dependencies and risks, and success metrics; use it to align the team and report upward.
Scalable oversight of superhuman models
Humans cannot judge outputs beyond their expertise, so feedback degrades; techniques like AI debate or recursive reward modeling decompose judgment.
Penalizing sycophancy in a reward model
Sycophancy is reward proxy gaming where agreeableness substitutes for correctness; counter it with truth-anchored labels, perturbed-premise pairs, and consistency checks.
Constitutional AI versus standard RLHF
A written principle set guides self-critique and revision, plus AI feedback (RLAIF) replaces human preference labels.
Diagnosing sycophancy from RLHF
Annotators reward agreeable, inoffensive answers so the reward model learns to favor them, fix by diversifying labelers, rewarding factual correctness over agreeableness, and…
What RLHF is and the safety problem it solves
Collect human preference rankings, train a reward model, fine-tune the policy with PPO; it aligns outputs with human intent the loss function cannot specify.
Setting up an LLM-as-a-judge evaluation
Define rubric, prefer pairwise comparison, randomize order, calibrate against humans; control position, verbosity, and self-preference bias.
Aligning text and image representations
Contrastive learning like CLIP pulls matched image-text pairs together and pushes mismatches apart; alternatively projection layers map one modality into a frozen model's space.
Direct Preference Optimization explained
DPO reparameterizes the RLHF reward in terms of the policy itself, turning alignment into a simple classification loss on preference pairs with no separate reward model or PPO.
Reward models in RLHF and PPO
It learns from human preference comparisons to score responses, then supplies the reward signal that PPO maximizes while a KL penalty keeps the policy near the reference.
Center a div with Flexbox on the parent
Set display flex on parent, justify-content center for the main axis, align-items center for the cross axis.
Value Learning
Value learning is the AI-safety approach of having a system infer what humans actually value, rather than optimizing a hand-coded proxy, so that capable agents pursue goals aligned with human intent even in novel situations.
Alignment: Invisible Lines That Structure Interfaces
Alignment is invisible scaffolding: eyes travel along shared edges instead of hunting anchors. In Figma, left-aligning labels and inputs turns chaos into readable flow. Center-aligning long text is the footgun; ragged invisible edges increase reading friction.
What are the risks of a roadmap without product strategy?
Risks include building wrong things fast, orphaned features, and tradeoff paralysis.
Vision Storytelling: Turn Roadmaps Into Shared Futures
Vision storytelling makes strategy tangible by narrating the future your team is building. Use it when roadmaps explain what but not why. The footgun is confusing slogans for stories; without a user, conflict, and resolution, you get branding, not alignment.
What does the KL-divergence penalty do in RLHF PPO, and if zeroed?
It tests RLHF reward hacking awareness. The KL penalty anchors PPO to the reference model to stop mode collapse; zeroing it causes over-optimization against the proxy reward model, yielding incoherent outputs.
Walk through RLHF's three stages, outputs, and purposes.
Tests your grasp of the RLHF pipeline end-to-end. A strong answer lists: pretrain an instruction-following LM, train a reward model outputting a scalar preference score, then fine-tune the LM via RL.
How does struct field ordering affect memory layout in Go and Rust?
It tests alignment, padding, and compiler layout knowledge. A strong answer explains that alignment inserts padding, Go and Rust keep declared order, and reordering by size can shrink size. Red flag: saying order is irrelevant or that compiler auto-packs.
mainAxisAlignment vs crossAxisAlignment in Column, and textDirection in Row
Tests Flutter flex axis awareness. In a Column, mainAxisAlignment is vertical and crossAxisAlignment is horizontal. In a Row, textDirection governs the main axis, not crossAxisAlignment; verticalDirection governs the cross axis.
Strategic Narrative: Your Company's Source Code for Story
A strategic narrative is the source code for your company's story, defining its unique value. It's the blueprint for marketing, sales, and internal decisions, ensuring everyone tells the same story.
Visiontype: Prototyping Your 3-5 Year Product Future
A visiontype is an interactive prototype of your product's 3-5 year future, making abstract goals tangible. It aligns teams on a long-term direction, breaking the cycle of purely incremental updates. The biggest mistake is creating it in a silo.
Orthogonality Thesis: An AI's Intelligence and Goals Are Unrelated
The Orthogonality Thesis states an AI's intelligence and its ultimate goals are independent. A superintelligence could pursue any objective, from beneficial to catastrophic, with equal capability.
Explain Supervised Fine-Tuning, RLHF, and DPO
This tests your understanding of modern LLM alignment techniques. A strong answer explains that Supervised Fine-Tuning (SFT) teaches the model a task via imitation, while RLHF and DPO align it with human preferences. RLHF uses a reward model and reinforcement learning, whereas DPO is a simpler, direct optimization method. The key red flag is conflating these distinct stages or failing to explain the 'reward model' step in RLHF.
Constitutional AI: Teaching an AI Right from Wrong
Constitutional AI teaches a model to be harmless by making it follow a set of principles—a constitution—instead of relying on human-labeled examples of bad behavior. This self-correction process, called Reinforcement Learning from AI Feedback (RLAIF), is used to align powerful models, enabling them to refuse harmful requests while explaining their reasoning. The entire system's safety, however, hinges on the quality and completeness of the initial human-written constitution.
Get Alignment bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.