GPU
27 bites tagged GPU — interview questions with model answers, and 60-second explainers.
Diagnosing poor distributed training scaling
Communication overhead (gradient all-reduce, interconnect), data-loading starvation, load imbalance, and small per-GPU batches; profile with the PyTorch profiler and Nsight. distributed training bottlenecks.
CPU versus GPU serving: cost, latency, throughput
GPUs win on throughput for batched parallel work but cost more; CPUs suit low-volume or small models. inference hardware tradeoffs. claiming GPU is always faster or ignoring batching and utilization.
CPU vs GPU vs Edge TPU for inference.
CPU is flexible but slow, GPU offers massive parallelism at high power, Edge TPU gives efficient low-power int8 inference but is constrained; choose by latency, power, cost, and model fit. inference hardware trade-offs.
How would you speed up slow single-GPU training?
Vertical scaling to bigger or multi-GPU instances, then data-parallel or model-parallel distributed training across nodes. knowledge of scaling training.
FlashAttention and IO-Aware Attention
FlashAttention is IO-aware, tiling and fusing attention in fast SRAM to avoid materializing the n-by-n matrix in slow HBM. hardware-aware optimization of attention. claiming it changes the math or lowers asymptotic compute.
Explain the roles of vertex and fragment shaders in WebGL
Vertex shaders write gl_Position per vertex; fragment shaders write gl_FragColor per pixel. GPU pipeline separation between geometry transform and pixel color. swapping their outputs or claiming CPU/DOM access.
How do you configure Docker for host GPU access and CUDA libraries?
This tests GPU passthrough via the NVIDIA Container Toolkit. Strong answers use nvidia/cuda base images matching the host driver, pass GPUs with --gpus all, and avoid installing drivers inside the container.
How would you systematically debug an inference API latency breach?
This tests structured debugging across the full inference stack. A strong answer traces the request path from ingress to GPU, splits TTFT from token-generation latency, inspects queuing and batching, then applies targeted fixes.
Design multi-tenant GPU cluster scheduling and preemption policies
Tests ability to design fair GPU scheduling preventing starvation and noisy-neighbor issues. Answer: Kueue for fair-share, namespace quotas with MIG, priority classes with backoff.
How do you containerize a Python training script for GPU cloud VMs?
This tests reproducible GPU containerization. A strong answer uses an NVIDIA CUDA base image, installs Python dependencies at build time, copies the training script, and runs with --gpus.
Inference Health Checks: Traffic Gates, Not Heartbeats
An inference server's health check is a traffic gate, not a heartbeat. Kubernetes uses it to route requests only after the model is loaded. The footgun is probing the root path, which stays green even when the model has crashed or the GPU is wedged.
Right-Size Inference and Stop Paying for Idle GPUs
Instance right-sizing matches inference to the smallest hardware that serves it without choking. It matters when GPU endpoints idle at 10% utilization. The footgun is copying your training spec into production; inference rarely needs that memory or multi-GPU.
Why GPUs Dominate Neural Network Training
A GPU is a freight train, a CPU a race car: deep learning moves identical math across huge batches. GPUs win on transformers and CNNs. The footgun is using them for tiny models, where data transfer overhead eats the gains.
Explain data, tensor, and pipeline parallelism and hybrid training strategy
Tests communication and memory tradeoffs of core distributed training strategies. Strong answers contrast data parallelism (shard batch), tensor parallelism (shard layers, all-reduce), and pipeline parallelism (shard stages, p2p), then propose a 3D hybrid…
What is overdraw in Android UI and how do you reduce it?
This tests GPU fill awareness. A strong answer defines overdraw as redrawing pixels repeatedly, names Debug GPU Overdraw's color overlay, and offers fixes: remove unnecessary backgrounds and flatten hierarchies. Red flag: confusing this with CPU layout issues.
WebGL Textures: From Image File to GPU Pixels
WebGL textures are images uploaded to the GPU for fast access when "painting" 3D models. They're used to apply detailed surfaces like brick patterns. The footgun: images must be CORS-approved, and non-power-of-two dimensions break mipmapping in WebGL1.
WebGL Shaders: Your Direct Line to the GPU
WebGL shaders are small programs written in GLSL that run directly on the GPU, bypassing the CPU for massively parallel graphics tasks. They are essential for all WebGL rendering, positioning vertices and coloring pixels.
GPU Utilization: Are You Wasting Your Most Expensive Resource?
GPU utilization isn't just a percentage; it's a measure of your return on investment. It tells you if your expensive hardware is computing or just waiting for data. Use it to diagnose slow training jobs and right-size cloud instances for ML workloads.
Microsoft DeepSpeed: Training Massive Models Across GPUs
DeepSpeed trains models too big for one GPU by partitioning model states across many devices. It's essential for training foundation models like BLOOM, but its complexity is overkill for smaller models and misconfiguration can harm performance.
NVIDIA CUDA: General-Purpose GPU Computing
CUDA unlocks a GPU's thousands of cores for general-purpose computing, not just graphics. It's key for accelerating tasks like machine learning by running the same operation on massive datasets in parallel.
Dynamic Batching: Balancing LLM Throughput and Latency
Dynamic batching groups LLM requests like a bus that leaves on a schedule or when full, whichever comes first. This improves throughput in inference servers by avoiding long waits. The footgun: all requests in a batch are still held hostage by the slowest one.
vLLM: Faster LLM Inference with PagedAttention
vLLM is a serving engine that speeds up LLM inference by treating GPU memory like virtual memory. It's used to serve models with higher throughput by batching requests without wasting memory on padding.
FlashAttention: Faster, Memory-Efficient Exact Attention
FlashAttention is an IO-aware algorithm that computes exact attention faster and with less memory. It avoids slow GPU memory transfers, making it a key optimization for training and serving large models on modern GPUs.
Tensor Parallelism: Split Layers, Not Just Models
Tensor Parallelism splits a single large model layer, like a weight matrix, across multiple GPUs to run in parallel. This is crucial for inference with models whose layers exceed a single GPU's VRAM.
Get GPU bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.