Infrastructure
110 bites tagged Infrastructure — interview questions with model answers, and 60-second explainers.
PaaS: The Managed Platform for Building Applications
PaaS gives you a ready-to-use development environment, handling the OS and middleware so you can just code. It's used to accelerate app development for web, IoT, or ML. The main footgun is vendor lock-in, making future platform migrations difficult.
Pipeline Step Caching: Don't Recompute What You Don't Have To
Pipeline step caching is memoization for your ML infrastructure, saving time and money by reusing previous results. It's used in MLOps pipelines when inputs and code haven't changed. The footgun: the cache is scoped to one pipeline and a timeout, not globally.
Inference Throughput: How Many Predictions Per Second?
Inference throughput measures how many predictions your system can make per second, not how fast a single one is. It's the system's total capacity, critical for high-volume tasks like recommendation engines. The footgun is confusing it with latency.
Multi-Model Serving: Packing More Models into Less RAM
Multi-model serving is a carpool for your ML models. Instead of one server per model, you pack many into a single process to share resources and cut costs. It's ideal for serving many models with intermittent traffic.
Autoscaling ML Inference Endpoints
Autoscaling matches your ML model's compute to real-time demand, like an elastic container for your inference service. It handles spiky traffic for online endpoints, scaling up for peaks and down to save costs.
Inference Batching: Grouping Requests for Throughput
Think of inference batching as a carpool for your ML model. Instead of sending each request in its own car, you wait a few microseconds to fill a bus, dramatically improving GPU efficiency.
TorchServe: Serving PyTorch Models in Production
TorchServe is a web server for your PyTorch models, turning them into production-ready API endpoints. It's used to expose trained models over a network via REST or gRPC for inference, handling batching and multi-model serving.
Load Balancing for Model Serving
A load balancer is a traffic cop for your AI model's API, directing requests to multiple model copies to prevent overload. It's essential for production systems to ensure high availability. The footgun is forgetting health checks, causing failed requests.
Edge AI: Running Models Where the Data Is
Edge AI runs machine learning models directly on devices, not in a distant cloud. This enables real-time, offline applications like smart cameras or voice assistants. The footgun is underestimating device hardware limits; models must be small and efficient.
Online Inference: Predictions on Demand
Online inference is a vending machine for predictions: you make one request and get one result back immediately. It powers real-time features like fraud detection or content moderation.
GitOps for MLOps: Your ML System as Code
GitOps for MLOps treats your entire machine learning pipeline—data, code, and models—as declarative configuration in Git. It automates ML workflows, ensuring reproducibility by making every change a reviewable commit.
Slurm: The Job Scheduler for Supercomputers
Slurm is the reservation system for a shared supercomputer, queuing up jobs and assigning them to available nodes. It's the backbone of high-performance computing clusters in science and ML.
Ray AI Runtime (AIR): A Unified ML Toolkit
Ray AIR is a unified toolbox for the ML lifecycle, bundling libraries for data, training, tuning, and serving. It's for scaling end-to-end ML workflows on one distributed platform.
Horovod: Scale ML Training Across Many GPUs
Horovod scales a single-GPU training script to hundreds of GPUs with minimal code changes, slashing training time. It's used when models are too big for one machine.
Parameter Servers for Distributed ML Training
A parameter server splits the work in distributed training: central servers hold the model's parameters, while worker nodes pull parameters, compute gradients on data subsets, and push updates back. This enables training models too large for one machine.
Secrets Management: Beyond Environment Variables
Treat secrets like cattle, not pets: they should be temporary and replaceable. Use a central vault to dynamically inject credentials into apps at runtime, especially in CI/CD and containerized environments.
Multi-stage Docker Builds: Lean Images, Fast Deploys
Build your app in one container stage and run it in another, separate one. This keeps your final Docker image lean by shipping only the compiled artifact, not the entire build environment, compilers, and source code.
Hydra: Composable Configuration for Complex Apps
Hydra treats configuration like LEGOs. Instead of one monolithic file, you compose small, reusable config pieces for each run. It's ideal for ML experiments where you override settings from the command line.
Data Lake: A Single Repository for Raw Data
A data lake is a central repository that stores vast amounts of raw data in its native format. It acts as a single source for analytics and machine learning, but without proper management it can become a useless "data swamp".
Docker Compose Profiles: Activate Service Groups
Docker Compose profiles let you toggle groups of services on or off within a single `compose.yaml` file. Use it to separate your core app from debugging utilities or to define a "local dev" setup versus a "CI" setup.
IaC for MLOps: Your ML Factory's Blueprint
Treat ML infrastructure—training clusters, model registries, serving endpoints—as code in version-controlled files. This ensures reproducible experiments and consistent deployments across environments.
Kubernetes Node: The Cluster's Worker Machine
A Kubernetes Node is a worker machine that runs your applications. Think of it as an employee receiving tasks (Pods) from the control plane manager. The common footgun is confusing the Node with the Pod; a Node is the server, while a Pod is the.
Cloud Direct Connect: A Private Lane to the Cloud
Think of Direct Connect as a private fiber-optic highway from your datacenter to the cloud, bypassing the public internet. It's for stable, high-bandwidth needs like large data transfers where public internet performance is too unpredictable or insecure.
Cloud Subnets: Your Virtual Network's Neighborhoods
Think of a subnet as a neighborhood in your virtual network (VPC), giving a block of IP addresses to a group of resources. Use them to isolate web servers from databases or apply specific firewall rules.
Get Infrastructure bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.