Support CVMFS

[vpenso]

Problem

Currently simple mounts within the container are not possible?

[root@submitter ~]# mount -t tmpfs tmpfs /mnt
mount: /mnt: permission denied.

[root@submitter ~]# capsh --print | grep cap_sys_admin
Current IAB: !cap_dac_read_search,!cap_linux_immutable,!cap_net_broadcast,!cap_net_admin,!cap_ipc_lock,!cap_ipc_owner,!cap_sys_module,!cap_sys_rawio,!cap_sys_ptrace,!cap_sys_pacct,!cap_sys_admin,!cap_sys_boot,!cap_sys_nice,!cap_sys_resource,!cap_sys_time,!cap_sys_tty_config,!cap_lease,!cap_audit_control,!cap_mac_override,!cap_mac_admin,!cap_syslog,!cap_wake_alarm,!cap_block_suspend,!cap_audit_read,!cap_perfmon,!cap_bpf,!cap_checkpoint_restore

Question

Is there a way to configure --cap-add=SYS_ADMIN for the Docker containers to enable mount privileges?

My particular use-case is to mount CernVM-FS¹ in submitter & worker containers, to enable some more elaborate test-cases.

Footnotes

1. CernVM-FS Documentation
   https://cvmfs.readthedocs.io ↩

[dennisklein]

I hoped to get around SYS_ADMIN caps and otherwise privileged containers. Bind-mounting from the host is not an option for your use case?

[dennisklein]

I will also investigate https://github.com/cvmfs/cvmfsexec, at first glance, this is the facility developed by the Singularity/Apptainer folks to mount cvmfs in unprivileged containers.

[dennisklein]

I took the liberty to repurpose this issue more specifically to the cvmfs use case. At the moment, I am hesitant to enable privileged containers in general.

[dennisklein]

I investigated several approaches for CVMFS support in sind. They serve different use cases:

Option A: Host bind-mount (consuming CVMFS)

If CVMFS is already mounted on the host, sind bind-mounts it read-only into the containers. No container privilege changes required.

User perspective

kind: Cluster
name: test

storage:
  cvmfs:
    - software.cern.ch

nodes:
  - role: controller
  - role: submitter
  - role: worker
    count: 3

$ sind create cluster -c cluster.yaml
$ sind enter test
[user@submitter data]$ ls /cvmfs/software.cern.ch/
bin  etc  lib  share

Under the hood, sind adds -v /cvmfs/<repo>:/cvmfs/<repo>:ro per repository.

Does this work with the current sind config? No. sind only supports one bind mount today (storage.dataStorage). Supporting this requires extending Storage with a CVMFS section, adding the -v flags in BuildRunArgs(), and supporting read-only mounts. The implementation delta is small.

In CI without host CVMFS, one can use cvmfsexec's mountrepo to FUSE-mount repositories on the runner via fusermount, then point sind at that path.

Option B: Docker volume plugin (consuming CVMFS)

A CVMFS Docker volume plugin would run as a privileged daemon on the host, handling the FUSE mount. Containers stay fully unprivileged.

User perspective

# One-time setup
docker plugin install ghcr.io/gsi-hpc/docker-volume-cvmfs --grant-all-permissions

The cluster config would look the same as Option A. Under the hood, sind would pass --mount type=volume,volume-driver=cvmfs,source=<repo>,target=/cvmfs/<repo>,readonly instead of a host bind-mount.

Such a plugin does not exist yet. The CVMFS CSI driver (cvmfs-contrib/cvmfs-csi, Go) has its mount logic in internal/ packages tightly coupled to CSI/gRPC — not reusable as a library. But the core operation is just exec.Command("cvmfs2", repo, mountpoint, "-o", "config="+path), so a new plugin following the vieux/docker-volume-sshfs pattern would be ~300 lines of Go.

Comparison A vs B

Aspect	Host bind-mount	Docker volume plugin
Host CVMFS required	Yes (or cvmfsexec mountrepo)	No
Container privileges	None	None
sind implementation	Small (~config + -v flags)	Same config, different --mount flag
External component	None (or cvmfsexec for CI)	Plugin daemon (~300 LOC, needs packaging)

Both result in the same user-facing config. sind could support both backends — use the volume driver if installed, fall back to host bind-mount.

Option C: Native CVMFS inside containers (testing config management)

Options A and B hand the container a pre-mounted filesystem. They are not useful for testing the config management logic (Puppet, Ansible, Salt, ...) that provisions CVMFS on bare-metal nodes. That stack involves package installation, /etc/cvmfs/ configuration, autofs/automount setup, and the FUSE mount lifecycle — all of which is bypassed when CVMFS arrives as a bind-mount.

Since sind containers run full systemd, they can in principle run the complete CVMFS service stack: autofs or systemd .automount/.mount units triggering cvmfs2 FUSE mounts on demand — exactly like a real node. This would allow testing config management end-to-end.

What's needed

There is no way around some privilege escalation here — the mount syscall requires it:

• --device=/dev/fuse — expose the FUSE device
• --cap-add SYS_ADMIN — allow the mount syscall (minimum capability for mounting)
• Unmask sys-fs-fuse-connections.mount — currently masked in the sind-node Dockerfile

This is much less than --privileged (which grants ALL ~40 capabilities, disables all LSMs, and exposes all devices). SYS_ADMIN alone still operates within the container's mount namespace, cgroup namespace, and seccomp policy.

User perspective

kind: Cluster
name: test

nodes:
  - role: controller
  - role: submitter
  - role: worker
    count: 3
    security:
      capAdd: [SYS_ADMIN]
      devices: [/dev/fuse]

Then config management provisions CVMFS as it would on a real node:

$ sind enter test
[root@worker-0 ~]# yum install -y cvmfs cvmfs-config-default
[root@worker-0 ~]# echo 'CVMFS_HTTP_PROXY=DIRECT' > /etc/cvmfs/default.local
[root@worker-0 ~]# cvmfs_config setup   # sets up autofs
[root@worker-0 ~]# systemctl start autofs
[root@worker-0 ~]# ls /cvmfs/software.cern.ch/
bin  etc  lib  share

Alternatively, with systemd automount units (no autofs needed):

# /etc/systemd/system/cvmfs-software.cern.ch.mount
[Mount]
What=software.cern.ch
Where=/cvmfs/software.cern.ch
Type=fuse.cvmfs2

# /etc/systemd/system/cvmfs-software.cern.ch.automount
[Automount]
Where=/cvmfs/software.cern.ch

[Install]
WantedBy=multi-user.target

Both approaches work because sind containers already run systemd as PID 1 — the full service lifecycle (start, stop, restart, automount triggers) behaves like a real node.

Implementation in sind

1. Extend the Node config struct with optional security fields (capAdd, devices)
2. Extend BuildRunArgs() to emit --cap-add and --device flags when configured
3. Provide a sind-node image variant (or runtime unmask) that doesn't mask sys-fs-fuse-connections.mount
4. Keep the defaults unchanged — unprivileged by default, opt-in per node

Summary

	A: bind-mount	B: volume plugin	C: native
Use case	Consume CVMFS	Consume CVMFS	Test CVMFS provisioning
Tests config mgmt	No	No	Yes
Container privileges	None	None	SYS_ADMIN + /dev/fuse
sind implementation	Small	Small + plugin	Node.security fields

[dennisklein]

@vpenso Do I assume correctly use case C is the relevant one to you?