Concepts in AI & ML, page 27
Spark DataFrame API: SQL Smarts on Distributed Data
The DataFrame API is like giving Spark a schema for your distributed data, letting its Catalyst optimizer plan queries like a database would. Use it for structured data processing with column-based operations.
Exposure Fusion: HDR Looks Without the HDR File
Exposure fusion blends the best-lit pixels from several bracketed shots into one image. This lets you capture high-contrast scenes, like a bright window in a dark room, without creating a full HDR file.
Centralized vs Decentralized ML Platforms
A centralized ML platform trades team autonomy for standardization, while decentralized platforms embed ML tooling inside product teams. Centralized suites drown in ticket queues; decentralized ones duplicate cost and security holes without strong governance.
Function Calling: LLMs Using Tools
Function calling turns an LLM into an API translator: it reads input and emits JSON telling your code which tool to run. Use it when the model needs live data it cannot store in weights. The model never executes the call and can hallucinate arguments.
Data Partitioning: Spreading Data for Scalability
Partitioning splits a huge dataset across many machines, like assigning phonebook sections to different librarians. This allows systems to scale beyond a single server.
Multi-frame Noise Reduction: Finding Signal in the Noise
Multi-frame noise reduction averages multiple shots to isolate the true signal. The underlying image is consistent, while random noise cancels out. It's key for low-light phone photos and video.
Model Risk Management: The Immune System for Production Models
Model Risk Management treats every deployed model as a liability that can silently decay. Banks use it to stop bad predictions from becoming bad decisions. The footgun is treating validation as a one-time checkbox instead of continuous governance.
Tool Definition Schema: Contracts for LLM Actions
A tool schema is JSON that tells an LLM what actions it can take. Use it when you want the model to call APIs instead of just chatting. The model only emits arguments; it never runs the tool, and vague descriptions cause silent failures.
Dashboard Design: Guide, Don't Overwhelm
A good dashboard guides users to an insight, not just displays charts. Place your key takeaway in the top-left and limit views to 2-3 to maintain focus. The biggest mistake is including too many views, which clutters the message and slows down the dashboard.
Synthetic Aperture: Faking a Big Lens with Software
Synthetic aperture uses depth data to computationally fake the shallow depth-of-field of a large lens. It's the magic behind smartphone 'Portrait Mode,' blurring the background to make a subject pop.
ROUGE Score: Recall Overlap for Generation
ROUGE measures text generation recall by counting overlapping words and phrases against a reference. It is the default metric for summarization benchmarks. Perfect paraphrases score poorly while keyword-stuffed nonsense can score high.
Data Sonification: Hearing Your Data's Story
Data sonification is data visualization for your ears, mapping data points to sound properties like pitch or volume. It helps find patterns in complex datasets, like network traffic, where visuals fail.
Super-Resolution: Creating Detail from Less
Super-resolution creates a high-res image from low-res sources by inferring missing details. It powers smartphone digital zoom and video upscaling. The footgun: generated details are plausible hallucinations, not ground truth, making it risky for scientific…
Hallucination Detection in LLMs
Hallucination detection is the set of techniques for flagging when a language model states something fluent but false or unsupported, using signals like self-consistency, model uncertainty, and grounding against retrieved evidence to catch fabrications before…
How a SQL SELECT Query Actually Runs
A SQL SELECT query runs in a different order than you write it. It first builds the dataset with FROM/JOINs and filters it with WHERE, only then computing the final columns in SELECT. This is crucial for debugging.
Seam Carving: Resizing Images Without Distortion
Seam carving resizes images by removing or adding pixel "seams" of low importance, not by squashing or cropping the whole frame. It's used in content-aware tools to change aspect ratios without distorting key subjects like faces or buildings.
HHH: The Three-Way Tug-of-War in LLMs
HHH frames LLM alignment as a three-way tug-of-war: helpful, harmless, honest. It governs RLHF reward models and safety filters, where maximizing one axis weakens the others. The footgun is optimizing helpfulness, producing sycophants or dangerous compliance.
The Big Idea: Your Presentation's Single-Sentence Core
The "Big Idea" is a single sentence distilling your presentation's core message. It must state your point of view and what's at stake, telling your audience what to know and what to do.
Photometric Stereo: Reconstructing Shape from Light
Photometric stereo deduces an object's 3D shape by watching how its surface reflects light from different directions. It’s used in industrial inspection and computer graphics to capture detailed surface geometry. The footgun is assuming it works on any object.
Python Virtual Environments
A virtual environment is an isolated Python installation with its own packages, so each project gets the exact dependency versions it needs without conflicting with other projects or the system Python.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles