NVIDIA Container Toolkit TOCTOU Escape: CVE-2024-0132 Fix and CDI Switch

TL;DR

---

In mid-September 2024, NVIDIA published Security Bulletin 5582, patching a critical container escape vulnerability (CVE-2024-0132) in the NVIDIA Container Toolkit and NVIDIA GPU Operator. Reported by Shir Tamari’s research team at Wiz, the flaw carries a CVSS v3.1 base score of 9.0 (Critical).

When mounting host GPU libraries into a container to enable GPU access, a Time-of-check Time-of-use (TOCTOU) race condition occurs. Running a maliciously crafted container image tricks the toolkit into mounting the entire host root filesystem (/) into the container. This leads to arbitrary code execution on the host, privilege escalation, and full host compromise—a complete container escape.

Affected products and fixed versions

The affected products and patched versions are summarized below.

Product	Affected versions	Fixed version
NVIDIA Container Toolkit	<= 1.16.1	1.16.2 or later
NVIDIA GPU Operator	<= 24.6.1	24.6.2 or later

The NVIDIA Container Toolkit is the core component that gives container runtimes—including Docker, Podman, and containerd—access to host NVIDIA GPUs. In Kubernetes environments, it is typically deployed across clusters via the NVIDIA GPU Operator, which automates GPU driver and runtime configuration.

This flaw is especially dangerous in environments that accept external container images. In multi-tenant AI/ML platforms, CI/CD pipelines that build and run arbitrary submitted container images, and cloud-hosted dev environments, an attacker only needs to run a crafted container to compromise both the host node and every other container running on it.

How the TOCTOU race works during library mounting

By default, the NVIDIA Container Toolkit uses a legacy mechanism: running a pre-start hook (nvidia-container-cli) during container startup to inject host GPU files into the container. During this initialization sequence, it mounts CUDA compatibility libraries under /usr/local/cuda/compat along with various GPU driver libraries.

A TOCTOU race condition (classified as CWE-367) occurs when a time gap exists between checking a resource (Check) and actually using it (Use). The toolkit validates the destination mount path inside the container image beforehand, but the check and the subsequent mount operation are not atomic.

An attacker builds a malicious container image containing carefully crafted symlinks at the paths targeted for library mounts. When the toolkit verifies that the target path is safe, and then resolves symlinks at the exact moment of the mount, the host root filesystem (/) or its parent directory gets pulled in as the mount destination.

Container escape execution flow

The sequence from running a crafted image to complete host takeover looks like this:

flowchart TD
  A["Run request for malicious container image<br/>(contains crafted symlinks)"] --> B["nvidia-container-cli launches<br/>(hook initialization)"]
  B --> C["Inspect target mount path for safety<br/>(Check phase)"]
  C --> D["Symlink evaluated in the gap between check and mount<br/>(TOCTOU race)"]
  D --> E["Entire host root filesystem mounted inside container<br/>(Use phase misdirection)"]
  E --> F["Access host root filesystem from within container<br/>(Read/write access acquired)"]
  F --> G["Tamper with host configs or run code via control sockets<br/>(Arbitrary code execution & privilege escalation)"]
  G --> H["Full container escape achieved<br/>(Compromise host and all co-located containers)"]

From root filesystem exposure to full host takeover

Once an attacker gains read and write access to the host root filesystem (/) from inside the container, multiple paths lead to host takeover.

With access to the host filesystem, an attacker can modify /etc/shadow, append SSH public keys to administrative accounts, or rewrite cron jobs and systemd service units. They can also directly access container engine control sockets on the host (such as /var/run/docker.sock or containerd.sock) to spawn a privileged container, establishing remote code execution (RCE) on the host.

This completely collapses the container isolation boundary, giving the attacker root privileges on the underlying host. Every other tenant’s container processes, in-memory data, secrets, and API keys running on the same GPU node become exposed to tampering and data exfiltration.

Package updates and migrating to the Container Device Interface

To eliminate the vulnerability, update to the patched packages provided for your distribution immediately. On APT-based distributions (Ubuntu, Debian), update the repository index and upgrade the package:

sudo apt-get update
sudo apt-get install --only-upgrade nvidia-container-toolkit

On RPM-based distributions (RHEL, CentOS, Rocky Linux), apply the update with dnf:

sudo dnf upgrade nvidia-container-toolkit

For Kubernetes clusters running the NVIDIA GPU Operator, upgrade to version 24.6.2 or later via Helm:

helm repo update
helm upgrade --reuse-values -n gpu-operator gpu-operator nvidia/gpu-operator --version v24.6.2

Beyond updating packages, switching to the Container Device Interface (CDI) is an effective architectural defense. CDI is a standard specification designed to let container runtimes inject devices—such as GPUs—into containers safely and declaratively.

CDI does not rely on the dynamic pre-start hooks or risky library discovery mechanisms that made the legacy NVIDIA Container Toolkit vulnerable. Instead, the container runtime directly reads a static JSON/YAML specification file generated according to the CDI spec and attaches the devices. Because the configuration is static, there is no window for a TOCTOU race.

To generate the CDI specification file, run the toolkit’s CLI utility:

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

After verifying the generated specification, enable CDI mode in your Docker daemon configuration (daemon.json) or containerd configuration (config.toml). This completely decouples your GPU containers from the legacy mount mechanism, ensuring a safer operational baseline.