Inference performance bottlenecks on Lambda

WHAT THIS TESTS This probes whether you understand how the serverless execution model affects ML inference and can apply targeted mitigations within its constraints.

A GOOD ANSWER COVERS The dominant bottleneck is the cold start. When Lambda spins up a new execution environment it must initialize the runtime and, for ML, load potentially large model weights and frameworks, adding seconds of latency to those requests. Closely related is repeated model loading: if you load the model inside the handler, you pay that cost on every invocation, so load it once in the global initialization scope outside the handler so warm invocations reuse it. Lambda also has bounded resources: memory is capped and CPU is allocated proportionally to memory, there is no native GPU, and total deployment package and layer size is limited, which constrains big models and frameworks. Mitigations include enabling provisioned concurrency to keep a pool of initialized instances warm and eliminate cold starts for steady traffic, increasing the memory setting to get more CPU for faster inference, using smaller, distilled, or quantized models and lighter runtimes like ONNX Runtime, and storing large weights in a Lambda layer, container image, or mounted EFS rather than inflating the package. If the model genuinely needs a GPU or very low tail latency at scale, recognize that a dedicated endpoint or container service may be a better fit than Lambda.

COMMON WRONG ANSWERS Loading the model inside the handler on every call. Ignoring cold starts entirely. Assuming Lambda offers GPUs. Adding more memory without realizing it also raises CPU, or conversely ignoring that lever. Trying to force an oversized model into Lambda instead of choosing a managed endpoint.

LIKELY FOLLOW-UPS When is provisioned concurrency worth its cost. How do container images change the size limits. When should you abandon Lambda for SageMaker.

ONE CONCRETE EXAMPLE An image classifier on Lambda showed multi-second tail latency from cold starts reloading the model each call. Moving model load to the init phase, quantizing the model, raising memory for more CPU, and enabling provisioned concurrency cut warm latency sharply and removed cold-start spikes for steady traffic.