Llms
63 bites tagged Llms — interview questions with model answers, and 60-second explainers.
Self-Attention: The Transformer's Core Idea
Self-attention lets a model weigh the importance of different words in a sequence to understand context. This core mechanism of the transformer architecture powers LLMs for translation and generation.
LSTMs: Giving Neural Networks a Longer Memory
LSTMs give neural networks a longer memory, letting them connect events across long sequences. They excel at tasks like language translation or time-series analysis where distant context is key.
Loss Function: Quantifying 'How Wrong' a Model Is
A loss function is a score that tells a machine learning model how wrong its predictions are. The lower the score, the better. It's the engine of training, guiding the model to adjust its parameters to get closer to the correct answers.
Anthropic's $5B/yr deal with SpaceXai boosts Claude capacity
Anthropic is spending an estimated $5B annually to take over SpaceXai's Colossus I cluster, immediately doubling Claude Code rate limits for most users. This massive compute deal addresses severe capacity bottlenecks that throttled developers after unexpected usage growth. The partnership positions Elon Musk's xAI as a new "neocloud" provider, directly competing with AWS and GCP for large-scale AI workloads. Expect improved Claude performance and reliability.
AI Replicates 16k-Line Go App From CLI Alone
Claude Opus 4.6 successfully reverse-engineered `gotree`, a 16,000-line Go toolkit, using only its command-line interface in the new MirrorCode benchmark. This demonstrates AI can autonomously replicate complex, multi-command programs—a task estimated to take a human engineer weeks. This leap in capability suggests AI is ready for long-horizon coding challenges, moving beyond simple function generation to full system cloning.
AI May Automate AI R&D by EOY 2028
Claude Mythos Preview now solves 93.9% of real-world GitHub issues on SWE-Bench, a massive leap from Claude 2's 2% in late 2023. This near-saturation of coding benchmarks is a key indicator that AI can automate its own engineering. Based on this trend, Anthropic's Jack Clark predicts a 60%+ chance of no-human-involved AI R&D by EOY 2028. This shifts the focus from AI-assisted coding to fully automated AI development.
What is the vanishing gradient problem and how do transformers avoid it?
This tests your understanding of core deep learning training issues and the transformer's specific architectural solutions. A great answer defines vanishing gradients in sequential models, then explains how the transformer's parallel attention mechanism creates direct, short paths for gradients between any two tokens, regardless of distance. A red flag is vaguely mentioning 'attention' without explaining why its parallel nature is the key to solving the problem for long sequences.
Constitutional AI: Teaching an AI Right from Wrong
Constitutional AI teaches a model to be harmless by making it follow a set of principles—a constitution—instead of relying on human-labeled examples of bad behavior. This self-correction process, called Reinforcement Learning from AI Feedback (RLAIF), is used to align powerful models, enabling them to refuse harmful requests while explaining their reasoning. The entire system's safety, however, hinges on the quality and completeness of the initial human-written constitution.
Knowledge Distillation: Shrinking Models, Keeping Smarts
Knowledge distillation trains a small 'student' model to mimic a large 'teacher' model, capturing its expertise in a much smaller package. This is used to deploy powerful but slow models onto resource-constrained hardware like smartphones for real-time inference. The footgun is assuming the student perfectly matches the teacher; you're trading a small amount of accuracy for a massive gain in efficiency and lower computational cost.
Tool Use: Giving LLMs Access to External Systems
Tool use lets an LLM call external functions, like a brain accessing a calculator or the internet. This is the core mechanism behind AI agents that can search the web, run code, or query a database to answer questions. The biggest footgun is assuming the model will always generate a valid function call; without enforcing a strict schema to match your function's expected input, your agent can fail unpredictably.
Perplexity: Measuring a Model's Uncertainty
Perplexity frames a model's uncertainty as the effective number of choices it's considering. For a fair die with six outcomes, the perplexity is 6, reflecting perfect confusion among six options. When evaluating language models, a lower perplexity score indicates a better ability to predict a sequence of text. The footgun is judging the score in a vacuum; a 'good' perplexity is always relative to the task's inherent randomness.
BLEU Score: Judging Translation by Human Overlap
The BLEU score judges a machine translation by how closely its text matches a professional human translation. It's a popular, automated, and inexpensive way to benchmark translation systems, like comparing different versions of a model. The main footgun is that a high score indicates high textual overlap, not necessarily better fluency or meaning, as it's just a proxy for human judgment.
Softmax Function: Turning Scores into Probabilities
The softmax function turns a list of raw scores from a model into a clean probability distribution where all values sum to 1. It's most often the final step in a neural network for multi-class classification, like deciding if an image is a 'cat', 'dog', or 'bird'. The main footgun is mistaking a high softmax probability for high model confidence; it only reflects the score's strength relative to the other scores, not its absolute certainty.
Cosine Similarity: Measuring Direction, Not Distance
Cosine similarity measures the angle between two vectors, not their distance, to gauge similarity. It asks, "Do these point in the same direction?" This is fundamental in AI for comparing text embeddings, where a vector's direction represents its meaning. The main footgun is confusing it with Euclidean distance; cosine similarity ignores vector magnitude, so two vectors can be far apart in space but still be considered nearly identical if their orientation is the same.
Word Embeddings: Turning Words into Vectors
Word embeddings turn words into numerical vectors, like coordinates on a map of meaning. Words with similar meanings, like "king" and "queen," are placed close together in this vector space. This is fundamental for text analysis in machine learning, allowing models to grasp semantic relationships instead of just matching text. The footgun is assuming the vector's individual numbers are human-interpretable; they are abstract features learned from data.
Get Llms bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.