Tenant label applied by a kyverno mutating cluster policy is rejected by capsule

[martijnhybrit]

Bug description

We are using a single ArgoCD to roll out resources. Tenants get their own RBAC on ArgoCD so they can roll out only within their own tenant namespaces and see only their tenant resources in Argo. However, ArgoCD rolls out resource using it's own service account. To still make sure the namespaces created by ArgoCD still are attached to the appropriate tenant we've written a ClusterPolicy in kyverno that checks the name of the namespace against the known tenants and automatically set the appropriate label on the namespace. The problem is that the label is mutated onto the namespace resource, we get the following error:

Error from server: error when creating "tests/kyverno/apply-tenant-label/example-test.yml": admission webhook "namespaces.projectcapsule.dev" denied the request: namespace label "example" does not match owner reference ""

When we predefine the label onto the namespace the namespace resource is excepted by capsule and attached to the appropriate tenant.

How to reproduce

Steps to reproduce the behavior:

1. Configure ArgoCD (and cluster-admin for testing purposes) as tenant administrator (using helm deployment)

manager:
  options:
    forceTenantPrefix: true
    administrators:
    - kind: ServiceAccount
      name: system:serviceaccounts:argocd:argocd
    - kind: Group
      name: cluster-admins

2. Define a ClusterPolicy in kyverno that mutates a namespace automatically to be part of a tenant if it starts with a tenant prefix.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: apply-tenant-label
  annotations:
    policies.kyverno.io/title: Apply Capsule Tenant Label
    policies.kyverno.io/category: Multi-Tenancy
    policies.kyverno.io/minVersion: 1.6.0
    policies.kyverno.io/description: |
      This policy applies the capsule.clastix.io/tenant label to namespaces
      that start with a tenant prefix. It fetches all existing tenants from the cluster
      and enforces that namespaces with tenant prefixes are labeled appropriately.
spec:
  background: false
  rules:
  - name: apply-tenant-label
    match:
      any:
      - resources:
          kinds:
          - Namespace
          operations:
          - CREATE
          - UPDATE
    context:
    - name: tenants
      apiCall:
        urlPath: "/apis/capsule.clastix.io/v1beta2/tenants"
        jmesPath: items[].metadata.name
    preconditions:
      all:
      - key: "{{ request.object.metadata.name | split(@, '-') | [0] }}"
        operator: In
        value: "{{ tenants }}"
    mutate:
      patchStrategicMerge:
        metadata:
          labels:
            capsule.clastix.io/tenant: "{{ request.object.metadata.name | split(@, '-') | [0] }}"

(this is verifies by running kyverno test with a kyverno test and on a cluster where the administrators are ignored by capsule, the label indeed get mutated on the resource)

3. Example Tenant:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  labels:
    kubernetes.io/metadata.name: example
    managed-by: terraform-tenancy
spec:
  cordoned: false
  ingressOptions:
    hostnameCollisionScope: Disabled
  limitRanges: {}
  networkPolicies: {}
  owners:
  - clusterRoles:
    - capsule-namespace-provisioner
    kind: Group
    name: example-namespace-creator
  - clusterRoles:
    - capsule-namespace-deleter
    kind: Group
    name: example-namespace-creator
  - clusterRoles:
    - capsule-openshift-developer
    - capsule-owner
    kind: Group
    name: example-openshift-developer
  - clusterRoles:
    - capsule-openshift-viewer
    kind: Group
    name: example-openshift-viewer
  preventDeletion: false
  resourceQuotas:
    scope: Tenant

4. Apply (as a cluster admin) a namespace that starts with the example prefix without the tenant label.
   (Simulating what happens in ArgoCD)

apiVersion: v1
kind: Namespace
metadata:
  name: example-test

this will result in

Error from server: error when creating "tests/kyverno/apply-tenant-label/example-test.yml": admission webhook "namespaces.projectcapsule.dev" denied the request: namespace label "example" does not match owner reference ""

5. Apply the same namespace with the label pre-applied before the admission control takes place:

apiVersion: v1
kind: Namespace
metadata:
  name: example-test
  labels:
    capsule.clastix.io/tenant: example

This will be accepted and attached to the example tenant

Expected behavior

We expect the namespace mutated by kyverno to be accepted by capsule and attached to the appropriate tenant.

Logs

Logs of the kyverno controller after applying the namespace without a label:

2026-02-02T11:13:12Z TRC github.com/kyverno/kyverno/pkg/webhooks/resource/mutation/mutation.go:136 > mutation rules from policy applied successfully URLParams= clusterroles=["basic-user","calico-tiered-policy-passthrough","capsule-namespace-deleter","cluster-admin","cluster-status","console-extensions-reader","helm-chartrepos-viewer","self-access-reviewer","self-provisioner","system:basic-user","system:build-strategy-docker","system:build-strategy-jenkinspipeline","system:build-strategy-source","system:discovery","system:oauth-token-deleter","system:openshift:discovery","system:openshift:public-info-viewer","system:openshift:scc:restricted-v2","system:openshift:useroauthaccesstoken-manager","system:public-info-viewer","system:scope-impersonation","system:webhook"] gvk="/v1, Kind=Namespace" gvr={"group":"","resource":"namespaces","version":"v1"} kind=Namespace logger=webhooks/resource/mutate name=example-test6 namespace=example-test6 operation=CREATE policy=apply-tenant-label resource.gvk="/v1, Kind=Namespace" roles=["kube-system:extension-apiserver-authentication-reader","openshift-config-managed:console-public","openshift-config-managed:openshift-network-public-role","openshift-config-managed:system:openshift:oauth-servercert-trust","openshift-console-user-settings:user-settings-59fec644-9284-412a-be59-4dd12f3cc928-role","openshift:copied-csv-viewer","openshift:shared-resource-viewer"] rules=["apply-tenant-label"] uid=fd4b07e9-c01d-4e5b-a611-a9ba808f3b3a user={"extra":{"scopes.authorization.openshift.io":["user:full"]},"groups":["argocd-admins","cluster-admins","harbor-admins","openbao-admins","tenant-owners","system:authenticated:oauth","system:authenticated"],"uid":"59fec644-9284-412a-be59-4dd12f3cc928","username":"REDACTED"} v=2

The capsule controller log never seem to specify anything other that reconciliation of tenant resources and not admission. (Do I need to configure something for this)

Additional context

• Helm Chart version: 0.12.3
• Kyverno version: 0.16.2
• ArgoCD version: v3.2.3+2b6251d
• Openshift version: v1.32.8

[martijnhybrit]

Forgot to specify, but I have given the kyverno admission controller access to get tenants:

# Admission controller configuration
admissionController:
  rbac:
    clusterRole:
      # -- Allow to view capsule tenants
      - apiGroups:
        - capsule.clastix.io
        resources:
        - tenants
        verbs:
        - get
        - list
        - watch

[oliverbaehler]

Ah okay, can you try setting this:

webhooks:
  hooks:
    namespaces:
      reinvocationPolicy: IfNeeded

For more logs:

manager:
  options:
    logLevel: "5"

[martijnhybrit]

This works! For my own context, is it because assigning the owner block also happens during the mutate phase of the admission control?

[martijnhybrit]

Maybe a bit too fast, It works now for new namespaces but when editing existing namespaces I still get the error

[oliverbaehler]

Yeah i would have been suprised if that would have solved it. It's a bug with the admin system. Currently has little priority for me. I think you could work around like this:

---
apiVersion: capsule.clastix.io/v1beta2
kind: TenantOwner
metadata:
  labels:
    team: platform
  name: cluster-admins
spec:
  kind: Group
  name: "cluster-admins"
---
apiVersion: capsule.clastix.io/v1beta2
kind: TenantOwner
metadata:
  labels:
    team: platform
  name: argocd-sa
spec:
  kind: ServiceAccount
  name: "system:serviceaccounts:argocd:argocd"

Then for all your tenants

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: example
spec:
  permissions:
    matchOwners:
    - matchLabels:
        team: platform

Configuration (remove the Administrators).

manager:
  options:
    forceTenantPrefix: true
    users:
    - kind: ServiceAccount
      name: system:serviceaccounts:argocd:argocd
    - kind: Group
      name: cluster-admins

[oliverbaehler]

After it was created, does it have Ownerreferences?

[martijnhybrit]

I will try to do the work around.
After it was created it did get an owner reference:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"name":"example-test"}}
    openshift.io/sa.scc.mcs: s0:c33,c12
    openshift.io/sa.scc.supplemental-groups: 1001080000/10000
    openshift.io/sa.scc.uid-range: 1001080000/10000
    security.openshift.io/MinimallySufficientPodSecurityStandard: restricted
  creationTimestamp: "2026-02-03T13:52:02Z"
  labels:
    capsule.clastix.io/tenant: example
    kubernetes.io/metadata.name: example-test
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: latest
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest
  name: example-test
  ownerReferences:
  - apiVersion: capsule.clastix.io/v1beta2
    kind: Tenant
    name: example
    uid: a53caa42-7b5f-4da7-90f5-a7bd1eba85c9
  resourceVersion: "137509715"
  uid: 08026936-8193-4cb7-b492-8e48339f3df8
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

[oliverbaehler]

Okay yeah it's a bug with the administrators. It should be somewhat fixed in the next release. However administrators have the power to remove namespace from tenants as of now, that's also why this is working. I assume you are not regarding changes from mutatingwebhooks in argocd?

[martijnhybrit]

You assume correctly, It's the namespace that argocd itself creates when you specify the create namespace sync option. Due to a quirk in argocd itself that created namespace is actually not being tracked as a argocd managed resource.