[Bug]: create-symlinks and update-ldcache hooks fail container create when the container rootfs is read-only #2131

Activity

1. sahanasreenath commented on Oct 10, 2026

   

   sahanasreenath

   on Oct 10, 2026

   Author

   More actions

   Additional data: a reproduction that needs no GPU, no container runtime and no CDI spec, which isolates the hooks from the runtime.

   Create a bundle whose root is a genuine read-only mount and invoke the hook directly with the OCI state on stdin:

   B=/var/tmp/hooktest/bundle
   mkdir -p $B/rootfs/{usr/lib64,etc,sbin}
   cp /sbin/ldconfig $B/rootfs/sbin/ldconfig
   echo '{"ociVersion":"1.0.2","root":{"path":"'$B'/rootfs"},"process":{"args":["/bin/true"]}}' > $B/config.json
   mount --bind $B/rootfs $B/rootfs && mount -o remount,ro,bind $B/rootfs
   
   STATE='{"ociVersion":"1.0.2","id":"t","status":"created","pid":1,"bundle":"'$B'"}'
   echo "$STATE" | nvidia-cdi-hook create-symlinks --link libcuda.so.1::/usr/lib64/libcuda.so
   echo "$STATE" | nvidia-cdi-hook update-ldcache --folder /usr/lib64

   Result is identical on v1.17.8 and v1.19.0, so this is not a regression introduced by the 1.18 update-ldcache change:

   hook	writable root	read-only root
   create-symlinks	exit 0	exit 1
   update-ldcache	exit 0	exit 1

   # create-symlinks, v1.19.0 and v1.17.8
   failed to create link [libcuda.so.1 /usr/lib64/libcuda.so]: failed to create symlink:
   symlink libcuda.so.1 .../rootfs/usr/lib64/libcuda.so: read-only file system
   
   # update-ldcache, v1.19.0
   Error updating ldcache: error mounting /proc: mkdirat proc: read-only file system
   
   # update-ldcache, v1.17.8
   Error updating ldcache: error mounting /proc: error creating directory: mkdir .../rootfs/proc: read-only file system
   

   Note the update-ldcache failures above are from the released binaries, which create proc in the container root. main no longer does that (maskPseudoFilesystems skips a missing proc), so on main the first read-only write is ensureLdsoconfFile/createLdsoconfdFile under /etc. Both are Go-level writes wrapped with %w, which is what #2132 keys off.

   Why this is not reported more widely

   Whether the hooks ever see a read-only rootfs depends on the runtime, not on the workload. On a stock OpenShift node the createContainer hooks run before the read-only remount is applied, so the hooks write into the still-writable upperdir and a readOnlyRootFilesystem: true GPU pod starts normally.

   Measured on OCP 4.19.26 / CRI-O 1.31.13 / crun 1.26 / driver 580.82.07, toolkit v1.17.8, CDI forced to mode = "cdi". A single pod shows both facts at once: /usr/lib64/libcuda.so exists (the hook created it) while /proc/mounts reports overlay / overlay ro and touch is denied on /, /etc and /usr/lib64.

   Where the rootfs is read-only at hook time, container creation aborts, and the hooks are then the only thing standing between a hardened workload and a working GPU. That asymmetry is the argument for tolerating EROFS rather than requiring every affected cluster to drop the hardening.
2. sahanasreenath commented on Oct 10, 2026

   

   sahanasreenath

   on Oct 10, 2026

   Author

   More actions

   Root cause update, superseding my earlier analysis above.

   The hooks are not at fault. crun before 1.26 remounted the rootfs read-only before running createContainer hooks, while runc does it after; NVIDIA filed this as containers/crun#1924 and it was fixed in containers/crun#1925 (crun 1.26, 2025-12-22). On a runtime that applies the remount after the hooks, a readOnlyRootFilesystem: true GPU pod starts fine with CDI hooks running, confirmed on a crun 1.26 node.

   Two corrections to what I wrote earlier:

   • "The libraries remain reachable" is wrong. create-symlinks creates the SONAME -> library links for the injected driver libraries, so on a read-only root nothing can resolve libcuda.so.1 regardless of image layout. Tolerating EROFS would start a container whose GPU is unusable and move the failure to the first CUDA call.
   • A hook-side fallback (bind-mounting a generated ld.so.cache) does not help either: ldconfig records dir/soname and relies on the symlink it creates, so with a read-only library directory every entry dangles.

   #2132 is therefore reduced to diagnostics: the hooks keep failing container creation, but the error now names the read-only rootfs and the runtime requirement (runc or crun >= 1.26) instead of a bare exit code 1.

   For reference, OpenShift ships crun 1.23.1 through 4.20.16 and crun 1.26 from 4.20.17.