Feature availability: Dynamic MIG slicing based on k8s pod requests

[Pavel-Okruhlica-SZN]

Hello,

on Kubecon I heard about this feature to enable/disable MIG and create slices of it based on the requests of a pod. something similar to gpu-test4 and I believe DRA should be exactly for this.
What I'm expecting from this is that by default, MIG on node would be disabled, but as soon as a pod with request for a MIG device via resourceClaims comes, for example mig-1g.10gb. Kubernetes scheduler should then find a MIG capable node, enable MIG and create the slice needed. Is this dynamic slicing something that is already released? I'm trying to implement this but running into some issues. However before I share any deeper details, I first wanted to know if it's even something already doable and if yes, is there some documentation for it?

So far, I was able to statically create MIG slices via GPU-operator's ConfigMap and assign those slices with ResourceClaimTemplate and ResourceClaim, but if i wanted a different slicing of GPU I would need to change the ConfigMap manually, before pods can use different MIG slices, when I would like the operator to slice up GPU based on pods requests automatically.

GPU used: NVIDIA-H100-PCIe
kube-apiserver version: v1.35.0

[klueska]

Please see the "Dynamic MIG" section of the release notes for the most recent release:
https://github.com/NVIDIA/k8s-dra-driver-gpu/releases/tag/v25.12.0

[Pavel-Okruhlica-SZN]

Ah,the issue I had was that MIG was enabled without any slices by gpu-operator

  config:
    name: custom-mig-parted-configs
    create: true
    data:
      config.yaml: |-
        version: v1
        mig-configs:
          custom-mig-parted-configs:
            - devices: all
              mig-enabled: true
              mig-devices: {}

when it should have just been the default all-disabled config.

there is one strange thing I also had with the ResourceClaimTemplate definition. like this:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: mig-devices-template
spec:
  spec:
    devices:
      requests:
        - name: mig-1g-10gb
          exactly:
            deviceClassName: mig.nvidia.com
            selectors:
              - cel:
                  expression: "device.attributes['gpu.nvidia.com'].profile == '1g.10gb'"
        - name: mig-1g-10gb-me
          exactly:
            deviceClassName: mig.nvidia.com
            selectors:
              - cel:
                  expression: "device.attributes['gpu.nvidia.com'].profile == '1g.10gb+me'"
        - name: mig-1g-20gb
          exactly:
            deviceClassName: mig.nvidia.com
            selectors:
              - cel:
                  expression: "device.attributes['gpu.nvidia.com'].profile == '1g.20gb'"
        - name: mig-2g-20gb
          exactly:
            deviceClassName: mig.nvidia.com
            selectors:
              - cel:
                  expression: "device.attributes['gpu.nvidia.com'].profile == '2g.20gb'"
        - name: mig-3g-40gb
          exactly:
            deviceClassName: mig.nvidia.com
            selectors:
              - cel:
                  expression: "device.attributes['gpu.nvidia.com'].profile == '3g.40gb'"
        - name: mig-4g-40gb
          exactly:
            deviceClassName: mig.nvidia.com
            selectors:
              - cel:
                  expression: "device.attributes['gpu.nvidia.com'].profile == '4g.40gb'"
        - name: mig-7g-80gb
          exactly:
            deviceClassName: mig.nvidia.com
            selectors:
              - cel:
                  expression: "device.attributes['gpu.nvidia.com'].profile == '7g.80gb'"

      constraints:
      - requests: []
        matchAttribute: "gpu.nvidia.com/parentUUID"

when you made a pod requesting 2g.20gb:

apiVersion: v1
kind: Pod
metadata:
  name: mig-claiming-pod
spec:
  containers:
  - name: mig-claiming-pod
    image: docker.ops.iszn.cz/ftxt-gpu/cuda:13.1.1-runtime-ubuntu24.04
    command: ["sleep", "6000"]
    resources:
      claims:
      - name: mig-devices-template
        request: mig-2g-20gb
  resourceClaims:
    - name: mig-devices-template
      resourceClaimTemplateName: mig-devices-template
  tolerations:
    - key: "nvidia.com/gpu"
      operator: "Exists"
      effect: "NoSchedule"

it created the MIG device on node, but with profile 1g.10gb. However by just isolating the template definition:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: mig-2g-20gb-template
spec:
  spec:
    devices:
      requests:
        - name: mig-2g-20gb
          exactly:
            deviceClassName: mig.nvidia.com
            selectors:
              - cel:
                  expression: "device.attributes['gpu.nvidia.com'].profile == '2g.20gb'"

      constraints:
      - requests: []
        matchAttribute: "gpu.nvidia.com/parentUUID"
---
apiVersion: v1
kind: Pod
metadata:
  name: mig-claiming-pod-2
spec:
  containers:
  - name: mig-claiming-pod-2
    image: docker.ops.iszn.cz/ftxt-gpu/cuda:13.1.1-runtime-ubuntu24.04
    command: ["sleep", "6000"]
    resources:
      claims:
      - name: mig-2g-20gb-template
        request: mig-2g-20gb
  resourceClaims:
    - name: mig-2g-20gb-template
      resourceClaimTemplateName: mig-2g-20gb-template
  tolerations:
    - key: "nvidia.com/gpu"
      operator: "Exists"
      effect: "NoSchedule"

it works as expected.
But that is probably more of a Kubernetes issue rather than k8s-dra-driver one. Feel free to close if you have no comments and thank you for pointing me the right way.

[klueska]

I'm confused how the first template does anything at all. I would expect a pod referencing it to stay pending forever.

It's basically saying:

• find a node where all of those profiles can be instantiated on the same GPU with the same gpu.nvidia.com/parentUUID

With such constraints, a pod referencing it should never have been schedulable because it is impossible to allocate all of those MIG devices at the same time from the same underlying GPU.

The DRA driver doesn't care that your pod only asks for one or two of the requests out of it. It will instantiate the entire claim (and all of the devices it knows about in the claim) but only selectively inject the ones you asked for in your pods claim stanza.

[jgehrcke]

And by the way, instead of asking for a specific profile via name, if you want, you can keep things a little more dynamic (and maybe future-proof, depending on the use case). Just for inspiration:

k8s-dra-driver-gpu/tests/bats/specs/gpu-simple-mig.yaml

Line 18 in f40f9d9

expression: |

      requests:
      - name: mig1g
        exactly:
          deviceClassName: mig.nvidia.com
          selectors:
          - cel:
              # Define a lower bound of memory and compute. Deliberately do not
              # ask for a specific profile by name.
              expression: |
                device.capacity['gpu.nvidia.com'].multiprocessors.isGreaterThan(quantity("10"))
                &&
                device.capacity['gpu.nvidia.com'].memory.isGreaterThan(quantity("10Gi"))

[Pavel-Okruhlica-SZN]

thank you, so now if i understand this constrain:gpu.nvidia.com/parentUUID it shouldn't matter in this case right?

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: mig-4-2-1-template
spec:
  spec:
    devices:
      requests:
        - name: mig-1g-10gb
          exactly:
            deviceClassName: mig.nvidia.com
            selectors:
              - cel:
                  expression: "device.attributes['gpu.nvidia.com'].profile == '1g.10gb'"
        - name: mig-2g-20gb
          exactly:
            deviceClassName: mig.nvidia.com
            selectors:
              - cel:
                  expression: "device.attributes['gpu.nvidia.com'].profile == '2g.20gb'"
        - name: mig-4g-40gb
          exactly:
            deviceClassName: mig.nvidia.com
            selectors:
              - cel:
                  expression: "device.attributes['gpu.nvidia.com'].profile == '4g.40gb'"

      constraints:
      - requests: []
        matchAttribute: "gpu.nvidia.com/parentUUID"

because it should be a valid MIG profile for H100 based on documentation.
However, when I try to use it with this pod config:

apiVersion: v1
kind: Pod
metadata:
  name: mig-claiming-pod-3
spec:
  containers:
  - name: mig-claiming-pod-3
    image: docker.ops.iszn.cz/ftxt-gpu/cuda:13.1.1-runtime-ubuntu24.04
    command: ["sleep", "6000"]
    resources:
      claims:
      - name: mig-4-2-1-template
        request: mig-4g-40gb
  resourceClaims:
    - name: mig-4-2-1-template
      resourceClaimTemplateName: mig-4-2-1-template
  tolerations:
    - key: "nvidia.com/gpu"
      operator: "Exists"
      effect: "NoSchedule"
    - key: "gpu"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

the pod doesn't want to start and is in state: ContainerCreating
It is the only pod that is trying to claim MIG from this node and describing that pod gives:

Events:
  Type     Reason                         Age   From               Message
  ----     ------                         ----  ----               -------
  Normal   Scheduled                      19s   default-scheduler  Successfully assigned nvidia-dra-driver-gpu/mig-claiming-pod-3 to gpu-h100-row-j-a
  Warning  FailedPrepareDynamicResources  18s   kubelet            Failed to prepare dynamic resources: prepare dynamic resources: NodePrepareResources failed for ResourceClaim mig-claiming-pod-3-mig-4-2-1-template-9c2rl: error preparing devices for claim nvidia-dra-driver-gpu/mig-claiming-pod-3-mig-4-2-1-template-9c2rl:ccca0e09-8438-4b99-9e48-775847faf08b: prepare devices failed: error creating MIG device: error creating GPU instance for 'gpu-0-mig-2g20gb-14-0': Invalid Argument

but when I check logs of pod/nvidia-dra-driver-gpu-kubelet-plugin-wsx5k. they talk about this:

I0302 12:03:40.603634       1 device_state.go:509] Device gpu-0-mig-1g10gb-19-0 is a MIG device, DynamicMIG mode: deleteMigDevIfExistsAndNotUsedByCompletedClaim()
I0302 12:03:40.603638       1 device_state.go:1167] Device 'gpu-0-mig-1g10gb-19-0' is not in use by any completely prepared claim, find corresponding actual MIG device
I0302 12:03:40.613716       1 nvlib.go:1255] FindMigDevBySpec: match found for profile ID 19 and placement start 0
I0302 12:03:40.613732       1 nvlib.go:1284] FindMigDevBySpec result: {ParentMinor:0 GIID:11 CIID:0 MigUUID:MIG-7bfd1ff3-36c9-5943-801e-ee320b69b080 ParentUUID:GPU-c5dc08af-14bd-8444-4b8c-807a2b927bfc}
I0302 12:03:40.613740       1 device_state.go:1178] MIG device corresponding to name gpu-0-mig-1g10gb-19-0 found with UUID MIG-7bfd1ff3-36c9-5943-801e-ee320b69b080 -- attempt to tear down
I0302 12:03:40.862828       1 nvlib.go:1122] Called nvml.SetMigMode(nvml.DEVICE_MIG_DISABLE) for device gpu-0-GPU-c5dc08af-14bd-8444-4b8c-807a2b927bfc, got activationStatus: Success
I0302 12:03:40.862877       1 device_state.go:509] Device gpu-0-mig-2g20gb-14-0 is a MIG device, DynamicMIG mode: deleteMigDevIfExistsAndNotUsedByCompletedClaim()
I0302 12:03:40.862883       1 device_state.go:1167] Device 'gpu-0-mig-2g20gb-14-0' is not in use by any completely prepared claim, find corresponding actual MIG device
I0302 12:03:40.863273       1 nvlib.go:1288] Iterated through all potential MIG devs -- no candidate found
I0302 12:03:40.863281       1 device_state.go:1174] No live MIG device corresponding to name gpu-0-mig-2g20gb-14-0 currently exists (nothing to clean up)
I0302 12:03:40.863285       1 device_state.go:509] Device gpu-0-mig-4g40gb-5-0 is a MIG device, DynamicMIG mode: deleteMigDevIfExistsAndNotUsedByCompletedClaim()
I0302 12:03:40.863288       1 device_state.go:1167] Device 'gpu-0-mig-4g40gb-5-0' is not in use by any completely prepared claim, find corresponding actual MIG device
I0302 12:03:40.863684       1 nvlib.go:1288] Iterated through all potential MIG devs -- no candidate found

and from what I'm reading, the devices should be there and no more are allocable on the same GPU, but only the 1g10gb one was created on the node itself, when it should have been only 4g40gb

# nvidia-smi -L
GPU 0: NVIDIA H100 PCIe (UUID: GPU-c5dc08af-14bd-8444-4b8c-807a2b927bfc)
  MIG 1g.10gb     Device  0: (UUID: MIG-7bfd1ff3-36c9-5943-801e-ee320b69b080)
GPU 1: NVIDIA H100 PCIe (UUID: GPU-26feb2ed-98d7-c1b0-85cc-9742bde90813)
GPU 2: NVIDIA H100 PCIe (UUID: GPU-406c8304-24fb-4e0f-ac82-8cc39e5deabe)
GPU 3: NVIDIA H100 PCIe (UUID: GPU-75578d92-6462-98be-dc5e-9d90ec4d5fed)

I tried even a lot of smaller slices that should have been generated fine. but only some of them were generated

GPU 0: NVIDIA H100 PCIe (UUID: GPU-c5dc08af-14bd-8444-4b8c-807a2b927bfc)
  MIG 1g.10gb     Device  0: (UUID: MIG-d6dd23ea-937b-554a-be5e-b27f421a4d52)
  MIG 1g.10gb     Device  1: (UUID: MIG-84fbccd4-fb2b-52b4-b5d2-3ed4822df8e4)
GPU 1: NVIDIA H100 PCIe (UUID: GPU-26feb2ed-98d7-c1b0-85cc-9742bde90813)
  MIG 2g.20gb     Device  0: (UUID: MIG-bfb81274-011f-50c8-bf82-1e29624941af)
  MIG 2g.20gb     Device  1: (UUID: MIG-9f7c82d2-e620-5cec-a44e-72dabc3de0d5)
  MIG 1g.10gb     Device  2: (UUID: MIG-c095354a-a7cc-5ec3-95ca-29488cf7e1f5)
  MIG 1g.10gb     Device  3: (UUID: MIG-9d902795-0544-50e7-99d0-7824f0dab405)
GPU 2: NVIDIA H100 PCIe (UUID: GPU-406c8304-24fb-4e0f-ac82-8cc39e5deabe)
GPU 3: NVIDIA H100 PCIe (UUID: GPU-75578d92-6462-98be-dc5e-9d90ec4d5fed)

it should have been 4 times, each profile, but not every pod got started

NAME                                             READY   STATUS              RESTARTS   AGE
pod/mig-claiming-pod-1g-0                        1/1     Running             0          109s
pod/mig-claiming-pod-1g-1                        1/1     Running             0          109s
pod/mig-claiming-pod-1g-2                        1/1     Running             0          109s
pod/mig-claiming-pod-1g-3                        1/1     Running             0          109s
pod/mig-claiming-pod-2g-0                        0/1     ContainerCreating   0          109s
pod/mig-claiming-pod-2g-1                        1/1     Running             0          109s
pod/mig-claiming-pod-2g-2                        1/1     Running             0          108s
pod/mig-claiming-pod-2g-3                        0/1     ContainerCreating   0          108s

describe was showing the same Invalid Argument error as all the other pods stuck in ContainerCreating
these ResourceClaimTemplates were without the constraints

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: mig-1g-10gb-template
spec:
  spec:
    devices:
      requests:
        - name: mig-1g-10gb
          exactly:
            deviceClassName: mig.nvidia.com
            selectors:
              - cel:
                  expression: "device.attributes['gpu.nvidia.com'].profile == '1g.10gb'"

similar for the 2g20gb

I take into account what @jgehrcke said and we will move to more abstract way of telling what we need but with these profiles, we want to see if we can fully use this feature.

Another experiment I tried is to take a few MIG slices from GPU-0 and GPU-1. then allocate 3 full GPUs (nvidia.com/gpu: 3) with a pod and that failed, as expected.
When I try the other way: allocate 3 full GPUs and try to generate the MIG slices after. it let's me slice up the already allocated GPUs, when I think it shouldn't. Slicing happens even when you allocate all the GPUs to a pod.

[klueska]

If you have this as your template, then all three of those should be created on some GPU on the node, with the addition of matchAttribute: "gpu.nvidia.com/parentUUID" they should all be created on the same underlying GPU.

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: mig-4-2-1-template
spec:
  spec:
    devices:
      requests:
        - name: mig-1g-10gb
          exactly:
            deviceClassName: mig.nvidia.com
            selectors:
              - cel:
                  expression: "device.attributes['gpu.nvidia.com'].profile == '1g.10gb'"
        - name: mig-2g-20gb
          exactly:
            deviceClassName: mig.nvidia.com
            selectors:
              - cel:
                  expression: "device.attributes['gpu.nvidia.com'].profile == '2g.20gb'"
        - name: mig-4g-40gb
          exactly:
            deviceClassName: mig.nvidia.com
            selectors:
              - cel:
                  expression: "device.attributes['gpu.nvidia.com'].profile == '4g.40gb'"

      constraints:
      - requests: []
        matchAttribute: "gpu.nvidia.com/parentUUID"

As I mentioned previously, it doesn't matter if you only reference one of these requests in your pod spec -- the whole claim will be prepared, including all of the requests. Only refereincing one of the requests in your pod spec will simply limit which ones are seen by that pod, not which ones get pepared, in general.

The fact that only the 1g.10gb device is actually getting created is clearly problematic, but I just want to make clear, that it most definitely should be being created (along with the other two devices listed) and that that creation should be limited to the same underlying GPU (because of the matchAttribute: "gpu.nvidia.com/parentUUID" constraint).

[jgehrcke]

@Pavel-Okruhlica-SZN when you reproduce the error

error creating GPU instance for 'gpu-0-mig-2g20gb-14-0': Invalid Argument

please share the full kubelet plugin log (not just an excerpt). That would be appreciated. Ideally, open a new issue. We will probably consider this a bug. Thanks for your feedback (it's very valuable)!

As a side note: the 4-2-1 packing you're trying to achieve on one particular H100 parent may currently not work if the 4g device isn't created first. For this to work, we need to reliably perform right-to-left placement (which nvidia-smi would do as I had described #361 (comment)). This isn't guaranteed as of today and needs to be built with proper behavioral specification and test coverage).

Edit: as a workaround, maybe try defining mig-4g-40gb first in the list of requests -- this may do the trick.

[Pavel-Okruhlica-SZN]

New issue created
Sadly change of order didn't help.

One last thing I would like to ask is that, if for example all GPUs are sliced into some MIG profiles, or have at least a single profile on them. they shouldn't be able to be allocated by pods requesting the classic full GPU nvidia.com/gpu in my opinion, but they can.

eg.
GPUs split into multiple MIG devices

# nvidia-smi -L
GPU 0: NVIDIA H100 PCIe (UUID: GPU-c5dc08af-14bd-8444-4b8c-807a2b927bfc)
  MIG 1g.10gb     Device  0: (UUID: MIG-bf4b8502-5ceb-56bc-ab32-1987fd3ca1bb)
  MIG 1g.10gb     Device  1: (UUID: MIG-a82e2eef-2ac9-5a39-b162-7b6b1e4e4867)
  MIG 1g.10gb     Device  2: (UUID: MIG-1b20042e-7d74-59fd-a462-73d1b2289033)
  MIG 1g.10gb     Device  3: (UUID: MIG-a8db903e-fd3b-591a-8f35-69a647251a06)
  MIG 1g.10gb     Device  4: (UUID: MIG-1fe59b54-110f-5203-97e9-9cce65688629)
  MIG 1g.10gb     Device  5: (UUID: MIG-fb7af32b-52af-504d-90be-cf1678b93c58)
  MIG 1g.10gb     Device  6: (UUID: MIG-574e22ed-77b2-5250-acd9-9058fcbc352b)
GPU 1: NVIDIA H100 PCIe (UUID: GPU-26feb2ed-98d7-c1b0-85cc-9742bde90813)
  MIG 1g.10gb     Device  0: (UUID: MIG-b583ead6-28d0-5cbe-87ce-1211e8b6552f)
  MIG 1g.10gb     Device  1: (UUID: MIG-fa9c7920-44a8-5756-86c0-a3edb9dc7f84)
  MIG 1g.10gb     Device  2: (UUID: MIG-6646ad00-a73b-566c-a279-3adc685e393b)
  MIG 1g.10gb     Device  3: (UUID: MIG-d399ffee-4326-54e5-bacd-4cf516498afd)
  MIG 1g.10gb     Device  4: (UUID: MIG-e283dad6-25dc-53bf-b983-1309f74b654e)
  MIG 1g.10gb     Device  5: (UUID: MIG-96faf007-f074-562c-b621-1e3ad3be4e59)
GPU 2: NVIDIA H100 PCIe (UUID: GPU-406c8304-24fb-4e0f-ac82-8cc39e5deabe)
  MIG 1g.10gb     Device  0: (UUID: MIG-730666d0-1265-5c40-be58-61877685180e)
  MIG 1g.10gb     Device  1: (UUID: MIG-37a09053-02cf-5515-b3d8-095fc8ba5287)
  MIG 1g.10gb     Device  2: (UUID: MIG-2064b458-05c8-5043-88b4-2adb659d1925)
  MIG 1g.10gb     Device  3: (UUID: MIG-be302ce9-ae56-5385-9d0d-98d75e263d19)
  MIG 1g.10gb     Device  4: (UUID: MIG-a3035767-3c49-51f8-98dd-605d52b6e39a)
GPU 3: NVIDIA H100 PCIe (UUID: GPU-75578d92-6462-98be-dc5e-9d90ec4d5fed)
  MIG 1g.10gb     Device  0: (UUID: MIG-92da8bc8-8bb6-531e-a142-5de25ebc85b6)
  MIG 1g.10gb     Device  1: (UUID: MIG-06a57a81-a39d-58ba-8eed-bc2af26a672a)
  MIG 1g.10gb     Device  2: (UUID: MIG-a07d0f52-cc1b-572e-b1bf-ec9683c396d2)
  MIG 1g.10gb     Device  3: (UUID: MIG-48b089b9-9512-596b-a52a-bc5725fd0db8)
  MIG 1g.10gb     Device  4: (UUID: MIG-b2fdbda8-9fa1-5523-af2c-7d1644795e2d)
  MIG 1g.10gb     Device  5: (UUID: MIG-ee403b67-27db-54e1-86f9-ecb11e62940b)

and after that a pod that will claim full GPU

apiVersion: v1
kind: Pod
metadata:
  name: full-gpu-pod
spec:
  nodeName: gpu-h100-row-j-a
  containers:
  - name: full-gpu-pod
    image: docker.ops.iszn.cz/ftxt-gpu/cuda:13.1.1-runtime-ubuntu24.04
    command: ["sleep", "6000"]
    resources:
      requests: 
        nvidia.com/gpu: 4
      limits:
        nvidia.com/gpu: 4

is getting started no problem. Earlier I said it wasn't possible but that was some fluke on my side and it is possible, sorry.

[jgehrcke]

I would like to ask is that, if for example all GPUs are sliced into some MIG profiles, or have at least a single profile on them. they shouldn't be able to be allocated by pods requesting the classic full GPU nvidia.com/gpu in my opinion, but they can.

The classic device plugin and the DRA driver are not aware of each other. Either of them claims full ownership over all GPU resources in a node. Mutual exclusivity must be ensured by an operator as of today; otherwise conflicting allocation as you observe is the expected outcome.

In other words, every node in a Kubernetes cluster must decide on just one of these two GPU allocation methods:

• NVIDIA device plugin
• NVIDIA DRA Driver for GPUs

If you want to use the DRA driver for GPU allocation on a certain node, you have to disable the device plugin on that node -- and make sure that nvidia.com/gpu isn't even announced anymore via the device plugin strategy. We can probably document all that better :-).