Inference performance bottlenecks on Lambda
serverless ML serving limits.
cold starts loading the model, memory and CPU limits, no GPU, and package size dominate; mitigate with provisioned concurrency, loading the model once outside the handler, smaller models, and right-sized…
What's really being asked
This probes whether you understand how the serverless execution model affects ML inference and can apply targeted mitigations within its constraints.
The full answer
The dominant bottleneck is the cold start. When Lambda spins up a new execution environment it must initialize the runtime and, for ML, load potentially large model weights and frameworks, adding seconds of latency to those requests. Closely related is repeated model loading: if you load the model inside the handler, you pay that cost on every invocation, so load it once in the global initialization scope outside the handler so warm invocations reuse it. Lambda also has bounded resources: memory is capped and CPU is allocated proportionally to memory, there is no native GPU, and total deployment package and layer size is limited, which constrains big models and frameworks. Mitigations include enabling provisioned concurrency to keep a pool of initialized instances warm and eliminate cold starts for steady traffic, increasing the memory setting to get more CPU for faster inference, using smaller, distilled, or quantized models and lighter runtimes like ONNX Runtime, and storing large weights in a Lambda layer, container image, or mounted EFS rather than inflating the package. If the model genuinely needs a GPU or very low tail latency at scale, recognize that a dedicated endpoint or container service may be a better fit than Lambda.
The mistakes people make
Loading the model inside the handler on every call. Ignoring cold starts entirely. Assuming Lambda offers GPUs. Adding more memory without realizing it also raises CPU, or conversely ignoring that lever. Trying to force an oversized model into Lambda instead of choosing a managed endpoint.
What usually comes next
When is provisioned concurrency worth its cost. How do container images change the size limits. When should you abandon Lambda for SageMaker.
A concrete example
An image classifier on Lambda showed multi-second tail latency from cold starts reloading the model each call. Moving model load to the init phase, quantizing the model, raising memory for more CPU, and enabling provisioned concurrency cut warm latency sharply and removed cold-start spikes for steady traffic.
Interview question
Why load the model in Lambda's global init scope rather than inside the handler function?
- a.It gives the function access to a GPU
- b.Warm invocations reuse the already-loaded model instead of reloading it each callCorrect
- c.It removes the deployment package size limit
- d.It bypasses Lambda's memory limit
Why? this is the answer
Loading at init lets warm invocations reuse the in-memory model, avoiding per-request load latency. It does not grant a GPU or lift Lambda's memory or package size limits.
Just read this? Test yourself on what you have been reading.
Read the original → docs.aws.amazon.com
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on cloud — each one lists the topics its interview covers.
See open roles