How do you containerize a Python training script for GPU cloud VMs?

What's really being asked

This question checks if you understand the difference between a generic Docker container and a GPU-enabled reproducible ML environment. The interviewer wants to see that you know Docker images encapsulate the application, libraries, and environment variables, but that GPU access requires specific host driver compatibility and runtime flags. It also reveals whether you treat containers as immutable artifacts versus mutable servers.

The full answer

A good answer hits four things in order. First, choose an NVIDIA base image such as an NGC PyTorch or TensorFlow container, or an NVIDIA CUDA runtime image, because these already contain the correct CUDA, cuDNN, and NCCL libraries aligned to host driver versions. Second, install Python dependencies inside the Dockerfile using a requirements file and pin versions so the environment is reproducible. Third, copy the training script and any local data or configuration into the image during the build stage rather than mounting them at runtime, which ensures the image is self-contained. Fourth, run the container with the Docker --gpus all flag so the NVIDIA Container Toolkit exposes the host GPU devices into the container without needing the legacy nvidia-docker2 runtime.

The mistakes people make

Red flags include suggesting you SSH into the VM and install CUDA manually inside the container, because containers share the host kernel and should not bundle kernel drivers. Another red flag is using a generic python base image and trying to compile CUDA from source, which creates massive images and version skew. Candidates also err by omitting the --gpus flag and then wondering why torch.cuda.is_available returns False, or by building the image on a laptop with an ARM chip and expecting it to run on a cloud x86 VM without multi-platform builds.

What usually comes next

The interviewer may ask how you handle large datasets that do not fit in the image, and the correct pattern is to mount a volume at runtime for data while keeping the code in the image. They may also ask about multi-stage builds to reduce image size, or how you verify that the host NVIDIA driver is compatible with the container CUDA version. Another common follow-up is how you would orchestrate this on Kubernetes, which requires requesting NVIDIA GPUs via resource limits and ensuring the device plugin is installed.

A concrete example

Suppose you have train.py and requirements.txt. Your Dockerfile starts with FROM nvcr.io/nvidia/pytorch:23.10-py3, then runs pip install --no-cache-dir -r requirements.txt, copies the script with COPY train.py /workspace/train.py, and sets CMD python /workspace/train.py. On the cloud VM, which already has the NVIDIA driver and Docker 19.03 or later, you build with docker build -t trainer . and run with docker run --rm --gpus all trainer. This works because Docker natively supports NVIDIA GPUs as devices since Docker 19.03, so the special nvidia-docker2 runtime is no longer necessary.