DRA: find a way to have larger numbers of resources per slice OR another way to do the accounting.

[catblade]

Enhancement Description

DRA currently has a restriction of the number of devices per slice, which we will hit pretty quickly with CPU assignment with many of the larger servers. Also will be a limit that we don't need for memory et cetera. Need this removed or lifted. How is TBD.

cc: @johnbelamaric @pravk03 @ffromani @dchen1107

• One-line enhancement description (can be used as a release note):
• Kubernetes Enhancement Proposal:
• Discussion Link:
• PRs by stage and milestone:
  • Alpha - v1.xx
    • KEP (k/enhancements) update PR(s):
    • Code (k/k) update PR(s):
    • Docs (k/website) update PR(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[johnbelamaric]

I think we're mostly OK here - you can split devices for the same pool/node across many slices. The one issue we had was that with partitionable devices, the counter sets were on a per-slice basis, but that was fixed in 1.35.

/wg device-management

[pohly]

I think we're mostly OK here - you can split devices for the same pool/node across many slices.

We might not have an API limit anymore, but how well the scheduler works when some DRA driver really starts publishing huge number of devices is a different story. It simply might not work well enough. This is currently at the discretion of the DRA driver, which is a trusted system component.

The one issue we had was that with partitionable devices, the counter sets were on a per-slice basis, but that was fixed in 1.35.

There's one small winkle that we haven't addressed yet: with all devices in the same slice, a DRA driver can list "preferred" devices first and optimize for the current "depth first search". We haven't officially documented that and it could change, but it works at the moment. When devices are in different slices, the order of those slices is random - @mortent ran into that in a unit test which became non-deterministic in kubernetes/kubernetes#133678.

We might have to add an explicit "priority" field to a ResourceSlice. I am not sure into which KEP that fits. Maybe binding conditions because it introduced sorting of slices by some less explicit criteria (having vs. not having binding conditions)?

cc @ttsuuubasa

[johnbelamaric]

Thanks Patrick. On the first part, I think there's a lot we could do algorithmically to improve how we find a match. That's probably our best bet. But it's hard and tricky work.

For the second one ... oof. I wonder if we can just always pick the "smallest device that fits". Not quite scoring, but sort of. That would require use though to understand what "small" means, and to be able to somehow try "smallest" devices first. Maybe there's a sort attribute or function that the driver could publish? Rather than a per-device priority.

[mortent]

Letting the driver control the ordering by letting it specify the order/rank of ResourceSlices within a resource pool seems like the simplest way to do this. But I'm not sure if that is sufficient. It still wouldn't give users any control the order in which devices from across resource pools are attempted. I don't think we want to do per-device priority.

The "select smallest device that fits" seems like a more flexible solution, but also much more complex. Would the "smallest" be per driver, per resource pool or across all available devices? It can also dramatically increase the time required to find a device, since we are going from "first fit" to "best fit", so (at least with a naive algorithm) we need to check all devices before we know which one is the smallest one.

And yeah, when we are talking about potentially hundreds or thousands of devices per node (if CPU, accelerators and NICs are all managed through DRA), and maybe tens of devices needs to be allocated, the current algorithm will not work. The number of permutations/combinations are far too large for us to try them all. So we need a way to discard infeasible devices/solutions earlier and/or be able to use heuristics that means we are able to allocate a good (if not the best) set of devices without checking them all.

[johnbelamaric]

The "select smallest device that fits" seems like a more flexible solution, but also much more complex. Would the "smallest" be per driver, per resource pool or across all available devices? It can also dramatically increase the time required to find a device, since we are going from "first fit" to "best fit", so (at least with a naive algorithm) we need to check all devices before we know which one is the smallest one.

One simple thing we can do is allow the driver to specify a "sort attribute" (or maybe more than one). We would sort the devices in a pool by this attribute, and take the first fit in that order. It's not super flexible, but it would solve the problem without going to a "best fit".

[mortent]

One simple thing we can do is allow the driver to specify a "sort attribute" (or maybe more than one). We would sort the devices in a pool by this attribute, and take the first fit in that order. It's not super flexible, but it would solve the problem without going to a "best fit".

Since devices are already ordered within a ResourceSlice, we can allow a driver to specify a single ordering of devices within resource pool by just adding a rank/priority for each ResourceSlice.