Unable to run container with userns in auto mode

[abpetrov]

I am trying to run a container with the following userns config, to create a particular mapping between a user on the host and the user in the container:

      driver = "podman"
      config {
        image = "docker.io/library/cassandra:4.1.10"
        ports = ["cassandra"]
        userns = "auto:uidmapping=999:20999:1,gidmapping=999:20999:1"
      }

However, looking at podman inspect, it seems that the podman driver ignores the config

          "HostConfig": {
...
               "UsernsMode": "",

Is there something I miss, I followed https://docs.podman.io/en/latest/markdown/podman-run.1.html#userns-mode to construct the userns option.

[tgross]

Hi @abpetrov! I've been able to reproduce this...

tl;dr we're missing the IDMappings struct from our API and will need to fix that.

	// IDMappings are UID and GID mappings that will be used by user
	// namespaces.
	// Required if UserNS is private.
	IDMappings *types.IDMappingOptions `json:"idmappings,omitempty"`

I'm close to the end of my day here but I should be able to take a crack at it tomorrow. (Internal ref https://hashicorp.atlassian.net/browse/NMD-1094)

---

For my investigation, I used the following jobspec:

job "example" {

  group "web" {

    task "http" {

      driver = "podman"

      config {
        image = "docker.io/library/busybox:latest"
        command = "httpd"
        args    = ["-v", "-f", "-p", "8001", "-h", "/var/www"]
        userns  = "auto:uidmapping=33:1000:1,gidmapping=33:1001:1"
      }

      resources {
        cpu    = 100
        memory = 128
      }
    }
  }
}

And as you've said, there's no ID mappings.

$ podman inspect 847 | jq '.[0].HostConfig.IDMappings'
null

To check that the problem isn't the flag value itself, if I run this same --userns flag in the Podman CLI, I get the expected mapping:

$ podman run -it --userns='auto:uidmapping=33:1000:1,gidmapping=33:1001:1'  --rm docker.io/library/busybox:latest`

$ podman inspect 291 | jq '.[0].HostConfig.IDMappings'
{
  "UidMap": [
    "0:1:33",
    "34:34:966",
    "1000:1001:64535",
    "33:1000:1"
  ],
  "GidMap": [
    "0:1:33",
    "34:34:967",
    "1001:1002:64534",
    "33:1001:1"
  ]
}

I also double-checked to make sure this wasn't an issue of not having the appropriate users in /etc/subuid and /etc/subgid, which was the case when I ran podman run --userns as root, which is what my Nomad agent would be doing. But that doesn't seem to improve the situation.

I've attached a debugger to the podman driver process to make sure we're configuring the object and I see it there before we hand it off to the API:

(dlv) print createOpts.ContainerSecurityConfig.UserNS
github.com/hashicorp/nomad-driver-podman/api.Namespace {
        NSMode: "auto",
        Value: "uidmapping=33:1000:1,gidmapping=33:1001:1",}

I've even verified it's getting serialized correctly in the request body to Podman by dumping that in the debugger to "userns":{"nsmode":"auto","value":"uidmapping=33:1000:1,gidmapping=33:1001:1"}. This matches the field values of the ContainerCreateLibpod API.

I ran podman system service -t 5000 tcp://127.0.0.1:7078 to see if I could figure out the difference of behavior and found that the CLI has an extra idmappings block that's not in the API request that the driver is making.

podman CLI

{
  "env_host": false,
  "httpproxy": true,
  "terminal": true,
  "stdin": true,
  "annotations": {
    "io.podman.annotations.userns": "auto:uidmapping=33:1000:1,gidmapping=33:1001:1"
  },
  "stop_timeout": 10,
  "log_configuration": {},
  "systemd": "true",
  "sdnotifyMode": "container",
  "pidns": {},
  "utsns": {},
  "remove": true,
  "containerCreateCommand": [
    "podman",
    "--url",
    "tcp://127.0.0.1:7078",
    "run",
    "-it",
    "--rm",
    "--userns=auto:uidmapping=33:1000:1,gidmapping=33:1001:1",
    "docker.io/busybox"
  ],
  "init_container_type": "",
  "unsetenvall": false,
  "manage_password": true,
  "image": "docker.io/busybox",
  "raw_image_name": "docker.io/busybox",
  "image_volume_mode": "anonymous",
  "init": false,
  "ipcns": {},
  "volatile": true,
  "privileged": false,
  "seccomp_policy": "default",
  "userns": {
    "nsmode": "auto",
    "value": "uidmapping=33:1000:1,gidmapping=33:1001:1"
  },
  "idmappings": {
    "HostUIDMapping": false,
    "HostGIDMapping": false,
    "UIDMap": null,
    "GIDMap": null,
    "AutoUserNs": true,
    "AutoUserNsOpts": {
      "Size": 0,
      "InitialSize": 0,
      "PasswdFile": "",
      "GroupFile": "",
      "AdditionalUIDMappings": [
        {
          "container_id": 33,
          "host_id": 1000,
          "size": 1
        }
      ],
      "AdditionalGIDMappings": [
        {
          "container_id": 33,
          "host_id": 1001,
          "size": 1
        }
      ]
    }
  },
  "read_only_filesystem": false,
  "read_write_tmpfs": false,
  "umask": "0022",
  "cgroupns": {},
  "netns": {},
  "publish_image_ports": false,
  "Networks": null,
  "use_image_resolve_conf": false,
  "use_image_hostname": false,
  "use_image_hosts": false,
  "healthconfig": {},
  "healthLogDestination": "local",
  "healthMaxLogCount": 5,
  "healthMaxLogSize": 500
}

nomad-driver-podman

{
  "name": "http-af27ccfb-7ac4-6c49-595a-c2828e80f381",
  "command": [
    "httpd",
    "-v",
    "-f",
    "-p",
    "8001",
    "-h",
    "/var/www"
  ],
  "env": {
    "NOMAD_ALLOC_DIR": "/var/nomad/dev/data/alloc/af27ccfb-7ac4-6c49-595a-c2828e80f381/alloc",
    "NOMAD_ALLOC_ID": "af27ccfb-7ac4-6c49-595a-c2828e80f381",
    "NOMAD_ALLOC_INDEX": "0",
    "NOMAD_ALLOC_NAME": "example.web[0]",
    "NOMAD_CPU_CORES": " ",
    "NOMAD_CPU_LIMIT": "100",
    "NOMAD_DC": "dc1",
    "NOMAD_GROUP_NAME": "web",
    "NOMAD_JOB_ID": "example",
    "NOMAD_JOB_NAME": "example",
    "NOMAD_MEMORY_LIMIT": "128",
    "NOMAD_NAMESPACE": "default",
    "NOMAD_PARENT_CGROUP": "nomad.slice",
    "NOMAD_REGION": "global",
    "NOMAD_SECRETS_DIR": "/var/nomad/dev/data/alloc/af27ccfb-7ac4-6c49-595a-c2828e80f381/http/secrets",
    "NOMAD_SHORT_ALLOC_ID": "af27ccfb",
    "NOMAD_TASK_DIR": "/var/nomad/dev/data/alloc/af27ccfb-7ac4-6c49-595a-c2828e80f381/http/local",
    "NOMAD_TASK_NAME": "http"
  },
  "log_configuration": {
    "driver": "k8s-file",
    "path": "/var/nomad/dev/data/alloc/af27ccfb-7ac4-6c49-595a-c2828e80f381/alloc/logs/.http.stdout.fifo"
  },
  "pidns": {},
  "utsns": {},
  "image": "08ef35a1c3f050afbbd64194ffd1b8d5878659f5491567f26d1c814513ae9649",
  "mounts": [
    {
      "destination": "/var/nomad/dev/data/alloc/af27ccfb-7ac4-6c49-595a-c2828e80f381/alloc",
      "type": "bind",
      "source": "/var/nomad/dev/data/alloc/af27ccfb-7ac4-6c49-595a-c2828e80f381/alloc",
      "options": [
        "z"
      ]
    },
    {
      "destination": "/var/nomad/dev/data/alloc/af27ccfb-7ac4-6c49-595a-c2828e80f381/http/local",
      "type": "bind",
      "source": "/var/nomad/dev/data/alloc/af27ccfb-7ac4-6c49-595a-c2828e80f381/http/local",
      "options": [
        "z"
      ]
    },
    {
      "destination": "/var/nomad/dev/data/alloc/af27ccfb-7ac4-6c49-595a-c2828e80f381/http/secrets",
      "type": "bind",
      "source": "/var/nomad/dev/data/alloc/af27ccfb-7ac4-6c49-595a-c2828e80f381/http/secrets",
      "options": [
        "z"
      ]
    }
  ],
  "ipcns": {},
  "netns": {
    "nsmode": "slirp4netns"
  },
  "userns": {
    "nsmode": "auto",
    "value": "auto:uidmapping=33:1000:1,gidmapping=33:1001:1"
  },
  "resource_limits": {
    "memory": {
      "limit": 134217728
    },
    "cpu": {}
  },
  "oom_score_adj": 0,
  "cgroupns": {},
  "cgroup_parent": "nomad.slice"
}

So the driver is definitely handing our userns block to the API, but we're missing this idmapping field which I suspected is the underlying problem. That made me check the code and see the comment that requires this.

[tgross]

PR is up here #481