assisted service fails to start with Multi-Attach error

[DanielFroehlich]

Pod of assisted service deployment fail to start.
Error is

Multi-Attach error for volume "pvc-7b06ae1c-b8b0-421b-bf64-3b0684922924" Volume is already exclusively attached to one node and can't be attached to another

PVC is on RWO netapp NAS. netapp trident pods have been restarted as part of troubleshootiung.
Its unclear to me which pod / node has the PV attached.

[DanielFroehlich]

@rbo or @rguske , can you PTAL?

[rbo]

% oc get pv pvc-7b06ae1c-b8b0-421b-bf64-3b0684922924 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: csi.trident.netapp.io
    volume.kubernetes.io/provisioner-deletion-secret-name: ""
    volume.kubernetes.io/provisioner-deletion-secret-namespace: ""
  creationTimestamp: "2023-12-05T09:20:25Z"
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/csi-trident-netapp-io
  - external-provisioner.volume.kubernetes.io/finalizer
  name: pvc-7b06ae1c-b8b0-421b-bf64-3b0684922924
  resourceVersion: "1737326403"
  uid: 869de627-5eef-43d7-b171-e884b56c6fbb
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 100Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: assisted-service
    namespace: multicluster-engine
    resourceVersion: "50467789"
    uid: 7b06ae1c-b8b0-421b-bf64-3b0684922924
  csi:
    driver: csi.trident.netapp.io
    volumeAttributes:
      backendUUID: ce186e90-370e-4c3b-bafb-6ee391aa5881
      internalName: isar_pvc_7b06ae1c_b8b0_421b_bf64_3b0684922924
      name: pvc-7b06ae1c-b8b0-421b-bf64-3b0684922924
      protocol: file
      storage.kubernetes.io/csiProvisionerIdentity: 1701593930666-8081-csi.trident.netapp.io
    volumeHandle: pvc-7b06ae1c-b8b0-421b-bf64-3b0684922924
  persistentVolumeReclaimPolicy: Delete
  storageClassName: coe-netapp-nas
  volumeMode: Filesystem
status:
  phase: Bound

Let's check the volume at the NetApp isar_pvc_7b06ae1c_b8b0_421b_bf64_3b0684922924.

Looks like active connection from ucs57

[rbo]

Let's try to train & reboot the node.

[rbo]

FYI:

% oc get volumeattachments | grep pvc-7a5146b6-4b7d-46fe-b53f-e337e5e51ab9
csi-c473f657e282469885698270abc76c283f06b059f4ada8977db7e218c5c1bfc7   csi.trident.netapp.io                   pvc-7a5146b6-4b7d-46fe-b53f-e337e5e51ab9   ucs57    true       28d
% oc get volumeattachments csi-c473f657e282469885698270abc76c283f06b059f4ada8977db7e218c5c1bfc7 -o yaml
apiVersion: storage.k8s.io/v1
kind: VolumeAttachment
metadata:
  annotations:
    csi.alpha.kubernetes.io/node-id: ucs57
  creationTimestamp: "2025-11-04T08:31:29Z"
  finalizers:
  - external-attacher/csi-trident-netapp-io
  name: csi-c473f657e282469885698270abc76c283f06b059f4ada8977db7e218c5c1bfc7
  resourceVersion: "3269295363"
  uid: 775f7a35-7406-49bb-8204-0d812d280d0d
spec:
  attacher: csi.trident.netapp.io
  nodeName: ucs57
  source:
    persistentVolumeName: pvc-7a5146b6-4b7d-46fe-b53f-e337e5e51ab9
status:
  attached: true
  attachmentMetadata:
    filesystemType: nfs
    formatOptions: ""
    mountOptions: ""
    nfsPath: /isar_pvc_7a5146b6_4b7d_46fe_b53f_e337e5e51ab9
    nfsServerIp: 10.32.97.8
    protocol: file

% oc debug node/ucs57
Starting pod/ucs57-debug-h4mm8 ...
To use host binaries, run `chroot /host`
Pod IP: 10.32.96.57
If you don't see a command prompt, try pressing enter.
sh-5.1# chroot /host
sh-5.1# mount | grep e337e5e51ab9
10.32.97.8:/isar_pvc_7a5146b6_4b7d_46fe_b53f_e337e5e51ab9 on /var/lib/kubelet/pods/24af3fe7-91e6-44e5-a587-6951f3633bdb/volumes/kubernetes.io~csi/pvc-7a5146b6-4b7d-46fe-b53f-e337e5e51ab9/mount type nfs4 (rw,relatime,vers=4.2,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.32.96.57,local_lock=none,addr=10.32.97.8)

[rbo]

Node drained, reboot... ☕ 1TB memory

[rbo]

Pod is running, wait until Node is back online.

[rbo]

Node is back online