How do you configure Docker for host GPU access and CUDA libraries?

What's really being asked

This question probes whether you understand the boundary between the host kernel driver and user space libraries in a containerized GPU workflow. The interviewer wants to see that you know the NVIDIA driver lives on the host, that CUDA toolkit and ML frameworks belong inside the container, and that the NVIDIA Container Toolkit is the bridge that exposes the host devices without polluting the image. It also checks if you understand CUDA compatibility matrices and base image selection.

The full answer

A good answer hits four things in order. First, it states that the host must have a sufficiently recent NVIDIA driver installed because the container never includes the driver itself; the host driver must support the CUDA version required by the training script. Second, it names the NVIDIA Container Toolkit as the required host software and mentions the docker run --gpus all flag or the nvidia runtime as the mechanism to pass GPU devices and driver libraries into the container. Third, it specifies using an official nvidia/cuda base image in the Dockerfile, choosing a tag that matches the required CUDA version and includes cuDNN when needed, such as nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu22.04, rather than starting from a generic python or ubuntu image and trying to install CUDA manually. Fourth, it notes that only user space libraries like cuDNN, NCCL, and the ML framework should be added in the Dockerfile, while the driver and container toolkit remain host level concerns.

The mistakes people make

The biggest red flag is suggesting that the Dockerfile should install the NVIDIA driver via apt install nvidia-driver or runfile, which reveals a fundamental misunderstanding of how device drivers interact with the host kernel. Another weak pattern is proposing a standard ubuntu base image and manually downloading CUDA toolkit runfiles inside the container, which creates brittle, oversized images and ignores the officially maintained nvidia/cuda repositories. Candidates who omit the --gpus flag or suggest privileged mode as a substitute also signal that they do not understand the purpose of the NVIDIA Container Toolkit. Finally, ignoring the host driver version and assuming any CUDA version will work on any GPU host is a serious gap.

What usually comes next

Interviewers often push deeper by asking how you would handle a host driver that is too old for the required CUDA version, which should trigger an upgrade of the host driver rather than a workaround in the container. They may ask about the difference between runtime and devel CUDA image variants, where devel includes headers and compilers needed for building custom extensions while runtime suffices for prebuilt wheels. Another common thread is multi GPU training, which leads to questions about NCCL versions, infiniband or NVLink topology visibility inside containers, and whether you would use docker compose with deploy resources reservations or move to Kubernetes with device plugins.

A concrete example

Suppose your training script needs PyTorch compiled against CUDA 11.8 and you are running on an Ubuntu host with NVIDIA driver 535. You would verify driver compatibility, then write a Dockerfile starting from nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu22.04, install Python and pip, then pip install torch==2.0.1+cu118. You would avoid any RUN apt-get install nvidia-driver step. At runtime you would execute docker run --gpus all --rm -v $(pwd):/workspace my-ml-image python train.py. This setup keeps the driver on the host, delivers the correct CUDA user space libraries through the base image, and exposes all GPUs via the container toolkit.