Strategies to reduce a 5GB ML Docker image size

WHAT THIS TESTS: This question evaluates whether you understand Docker layer caching, multi-stage builds, and ML-specific image bloat vectors. Interviewers want to see that you can distinguish between build-time dependencies and runtime artifacts, that you know how CUDA and Python packaging inflate images, and that you prioritize build-time hygiene over post-hoc fixes.

A GOOD ANSWER COVERS: First, adopt a multi-stage build where a builder stage installs compilers, CUDA development headers, and build tools, then compiles Python wheels or exports a frozen requirements set, and a final runtime stage copies only the installed packages and application code. Second, switch from nvidia/cuda devel images to runtime or base images, and prefer distroless or slim variants such as python:3.11-slim, because full Ubuntu plus CUDA development libraries often exceed 4 GB alone. Third, collapse cleanup into the same RUN layer that performs installation, for example by chaining pip install and rm -rf /root/.cache/pip in a single command, since separate RUN steps permanently commit intermediate layers. Fourth, use .dockerignore aggressively to exclude training data, large model checkpoints, notebooks, and local virtualenv directories so they never enter the build context. Fifth, consider BuildKit mount caches for apt or pip to avoid embedding cache directories at all, and evaluate tools like pip-compile or poetry export to minimize transitive dependency bloat.

COMMON WRONG ANSWERS: A red flag is suggesting docker squash or experimental layer flattening as the primary strategy, because that masks poor build hygiene rather than fixing it. Another red flag is proposing rm -rf in a separate RUN instruction, which leaves the deleted files in previous layers and does not reduce image size. Candidates who recommend simply switching to Alpine without checking CUDA or manylinux wheel compatibility for PyTorch and TensorFlow reveal shallow MLOps experience. Similarly, suggesting manual tarball extraction without understanding layer caching shows a lack of operational rigor.

LIKELY FOLLOW-UPS: The interviewer may ask how you would verify which layers consume the most space, so you should mention docker history and dive. They might probe whether you would unbundle the model weights from the image entirely, leading to a discussion of volume mounts or S3-backed sidecars at pod startup. They could also ask how you would maintain reproducibility while using slim images, which opens the door to pinning digests and SBOM generation.

ONE CONCRETE EXAMPLE: Suppose a PyTorch serving image based on nvidia/cuda:11.8.0-devel-ubuntu22.04 with torch and transformers is 6.2 GB. You refactor to a builder stage using the devel image to pip install and compile any custom C++ extensions, then copy the resulting site-packages into a final stage based on nvidia/cuda:11.8.0-runtime-ubuntu22.04 plus python:3.11-slim. You merge pip install, apt-get clean, and rm -rf /var/lib/apt/lists into one RUN. You add .dockerignore lines for *.pt, checkpoints/, and data/. The resulting image drops to roughly 1.8 GB, and pull times on a 1 Gbps link fall from about 50 seconds to 15 seconds.