feat: add Topology Aware Scheduling proposal

[julienmancuso]

No description provided.

[galrevach]

Added a few comments throughout the file - would appreciate your feedback

[galrevach]

As far as I know, Grove had a default behavior of setting preferred topology constraint on all its components (PCS, PCSG, and PodClique). Does having this "enabled" flag means this default behavior in Grove will be done only when set to true?

[julienmancuso]

Grove does not apply any default topology constraints. The PCS defaulting webhook has no topology-related logic, and the validating webhook actually rejects any PCS that has topology constraints when TAS is disabled at the operator level. If a PCS is created without TopologyConstraint on any level (PCS/PCSG/PodClique), none are injected, the workload simply doesn't get topology-aware scheduling.
The enabled flag in this proposal controls whether Dynamo injects topology constraints into the PCS it generates. When enabled: true, Dynamo applies smart defaults (PCS=block, PCSG=rack, etc.) and merges any user overrides before producing the PCS. When enabled: false (the default), Dynamo generates the PCS with no topology constraints at all, exactly as it does today.

So to clarify:

• Grove: Topology constraints are opt-in at the PCS spec level. No defaults are ever applied by Grove.
• Dynamo (this proposal): The enabled flag controls whether Dynamo populates those PCS fields. Smart defaults are a Dynamo-layer concept, not a Grove one.

[galrevach]

As far as I understand Grove doesn't plan to add option for Preferred constraints, and will have the option to auto-apply preferred constraints at all levels. See the discussion here: ai-dynamo/grove#368
Can we align with Grove on this?

[nvrohanv]

I think Grove will keep topology packing as opt-in. However it is an open question whether that topology packing should by default result in the scheduler trying to pack everything as close as possible, or if it should just obey specified requirements, or if that in itself should be a switch. This is something we should discuss and close out soon @sanjaychatterjee

[galrevach]

Was a decision made here?

[galrevach]

Applying these by default means these are hard/required constraints, correct? aren't we concerned it will fail the workload? I think it should be considered if Grove's default behavior of adding "preferred" constraint is enough

[julienmancuso]

let me add the following clarifications to this doc:
1 - These will be configurable at dynamo operator level (let me add
2 - as soon as Grove will support "preferred" then yes we could think of setting these default as preferred.

Also users are the one choosing whether or not they want default applied and if they don't want to, they can just disable the feature

[julienmancuso]

updated in 8f68ff0

[tmonty12]

I have a similar concern, it essentially requires that:

1. Grove TAS is enabled with label to domain mapping defined
2. That the block and rack domains exist in the Grove ClusterTopology

Also block and rack domains varies from cluster to cluster and optimal workload topology placement will vary from workload to workload. This feels like something difficult to have a good default for.

Enabling cluster wide re-mapping of the defaults does help circumvent 2) above but my concern still remains. Would like to hear @sanjaychatterjee @nvrohanv thoughts.

[nvrohanv]

The more I think about it the more I lean towards not setting any defaults for now. For an opt-in feature, defaults feel more like a nice to have than a requirement so I think we should err on the side of making sure our defaults don't mess anything up. Once Grove has support for auto-packing everything I think we could make that the default.

[julienmancuso]

ok let me remove it from proposal

[galrevach]

how are these values defined? does it depend on Grove's cluster topology to be defined with these exact levels?

[julienmancuso]

correct, dynamo provides users to set these well-known topology domain, but under the hoow there must exist a topology in the cluster which exists with these domains.

[galrevach]

are these the values in Grove's cluster topology? doesn't it mean it is not agnostic to any topology that is being defined?

[julienmancuso]

Good point, the hardcoded domain list is indeed Grove's vocabulary, which contradicts our framework agnosticism goal.
To address this, the allowed domain list will be configurable at the Dynamo operator level (via Helm values), not hardcoded in the webhook. For Grove, the Helm chart ships with Grove's standard domains as defaults.
For a different framework (e.g., Kueue), the admin overrides allowedDomains to match that framework's supported domain names.
This gives us early validation at the DGD webhook level (catching typos before they hit the framework) while keeping Dynamo decoupled from any specific framework's domain vocabulary. Updated the proposal accordingly.

[julienmancuso]

updated in 22335b8

[galrevach]

I couldn't find in the design a reference to supporting multiple topologies.
In Run:ai, we support multiple topologies for multiple node pools in the same cluster (e.g. H100 vs. GB200). therefore, once a Dynamo workload is submitted through Run:ai, we want to provide the required topology definition based on the node pool on which it Runs.
The issue for Grove: ai-dynamo/grove#369

[julienmancuso]

updated in 8f68ff0

[julienmancuso]

actually I removed the multi-topology req for now, will be introduced later

[galrevach]

so the plan is to wait for Grove to complete the implementation and then add it here?

[julienmancuso]

correct

[tmonty12]

Instead of having a top-level serviceOverrides field, did we consider having topologyConstraint at the service level? This is similar to how Grove formatting and I think more consistent with our field conventions:

spec:
  topologyConstraints:
    enabled: true
    topologyName: aws-p5   # optional; omit for framework default
    packDomain: zone
  services:
    - name: VllmWorker
      multinode: { nodeCount: 4 }
      replicas: 2
      topologyConstraint:
        packDomain: block
    - name: Frontend
      replicas: 1
      topologyConstraint:
        packDomain: zone

[julienmancuso]

updated in 22335b8

[tmonty12]

This might be difficult to achieve. In the current proposal we are tightly coupled to Grove's topologyDomain list.

This is an example of TAS enablement for LWS with Kueue where the required topology is set as labels - there is no intermediate CR to convert from domains to labels. I think the current proposal is fine - but just wanted to point this out.

[galrevach]

To add to this, in Run:ai we allow the admin to configure the network topology which is a list of labels. then users can request the constraints based on the topology labels defined for this cluster

[julienmancuso]

Good observation. You're right that the domain names (rack, block, etc.) come from Grove's vocabulary today. However, abstract domain names seem to be the right user-facing API, users should say "rack" not topology.kubernetes.io/rack.
If Dynamo later supports a different backend (e.g., Kueue/LWS), the translation from abstract domains to framework-specific values (node labels for Kueue, domain names for Grove) can be done via operator configuration (configmap, another CRD, ...) with domain-to-label mappings per framework. The DGD API would stay the same and only the translation layer would change.
For now, since Dynamo only generates Grove PCS, this works as-is. We can add a note in the open questions/future work section about generalizing the domain vocabulary for non-Grove backends.

wdyt ?

[julienmancuso]

updated in 22335b8

[nvrohanv]

Ya I think this makes sense @julienmancuso, for now we cant really avoid some coupling but in the future we can do translation at the operator level like you said.

[galrevach]

I want to add that for KAI/Run:ai the topology CRD is also without domain names, as in Kueue.

If using these specific domain names I can't see how this fits the option for the user to provide the topology name.
The topology name is what defines which topology levels can be selected as constraints. if the list of topology levels if a closed list of domain names (rack, block, etc.), how do we plan this to work?

[julienmancuso]

the translation from dynamo domain to framework-specific values can be done via a domain-to-label mappings per framework.

i added that in 453f2af

[galrevach]

I think using the abstract vocabulary makes it specific because only Grove knows how to translate from these domain names to the relevant node labels. the more generic approach is using the labels. I do understand why we want to abstract the labels for the users, but not sure how this solution makes this generic

[julienmancuso]

Using labels directly would actually make the API less generic: labels are cluster-specific (e.g., cloud.provider.com/topology-rack vs topology.kubernetes.io/rack), so DGDs wouldn't be portable across clusters. It also wouldn't work with Grove, since Grove's PackDomain takes domain names, not labels, we'd need to reverse-lookup labels to domain names, which is fragile.

The abstract domains (rack, block, zone) are Dynamo's own vocabulary, they're generic at the user-facing API level. Users express intent in natural terms without needing to know cluster-specific labels. For Grove, these map 1:1 with no translation. For future backends like Kueue that use labels, the operator can translate internally via a domainToLabelMapping config; the DGD API stays the same. The genericness is at the API layer; the framework-specific translation is an internal operator concern.

[tmonty12]

Might want to break this out into CREATE vs. UPDATE. My understanding is that topologyConstraint at the Grove level is immutable and therefore we would also constrain it as immutable.

[julienmancuso]

updated in 22335b8

[tmonty12]

I have a similar concern, it essentially requires that:

1. Grove TAS is enabled with label to domain mapping defined
2. That the block and rack domains exist in the Grove ClusterTopology

Also block and rack domains varies from cluster to cluster and optimal workload topology placement will vary from workload to workload. This feels like something difficult to have a good default for.

Enabling cluster wide re-mapping of the defaults does help circumvent 2) above but my concern still remains. Would like to hear @sanjaychatterjee @nvrohanv thoughts.

[nvrohanv]

Overall looks good I think we just need to settle on the defaults/if we are going to have defaults.

[nvrohanv]

I'm worried if large deployments will exceed a block and if we should instead have something like zone.

[julienmancuso]

defaults are configurable, users can either change the default or override the constraint in their dgd directly

[nvrohanv]

I think Grove will keep topology packing as opt-in. However it is an open question whether that topology packing should by default result in the scheduler trying to pack everything as close as possible, or if it should just obey specified requirements, or if that in itself should be a switch. This is something we should discuss and close out soon @sanjaychatterjee

[nvrohanv]

The more I think about it the more I lean towards not setting any defaults for now. For an opt-in feature, defaults feel more like a nice to have than a requirement so I think we should err on the side of making sure our defaults don't mess anything up. Once Grove has support for auto-packing everything I think we could make that the default.

[nvrohanv]

Ya I think this makes sense @julienmancuso, for now we cant really avoid some coupling but in the future we can do translation at the operator level like you said.

[galrevach]

what does this mean exactly? that the Dynamo admin will also need to define the levels of topology (e.g. rack, block...)? I think this can be confusing for admins to define the topology structure (CRD) on Grove, Dynamo, and Run:ai (in cases it is being used).
This is why we suggested to allow the user to add the topology name on the workload spec. this name will let Grove and the Scheduler to know which topology CRD to use, and the constraints requested will be based on this specific topology CRD

[galrevach]

I want to add that for KAI/Run:ai the topology CRD is also without domain names, as in Kueue.

If using these specific domain names I can't see how this fits the option for the user to provide the topology name.
The topology name is what defines which topology levels can be selected as constraints. if the list of topology levels if a closed list of domain names (rack, block, etc.), how do we plan this to work?

[Ronkahn21]

Since this is already in a deployment context, should we rename this field to topologyConstraints for better clarity?

[julienmancuso]

fixed

[Ronkahn21]

Is the Enabled flag necessary? Couldn't we just infer that TAS is enabled if the constraints are provided?

[julienmancuso]

fixed

[Ronkahn21]

Currently, we allow changes to the Cluster Topology CR even if it's in use. If a user requests a constraint that no longer exists, enforcement stops. Shouldn't Dynamo propagate this scenario in the status so the user knows why the constraints are no longer being enforced?

[julienmancuso]

Good point, this is a real scenario. However, looks like this is a grove-level responsibility: if the ClusterTopology is mutated in a way that breaks active PCS constraints, Grove should report that on the PCS status (e.g., a degraded condition). Dynamo already propagates PCS status/conditions to DGD status, so any issue Grove surfaces on the PCS will be visible at the DGD level. Having Dynamo independently watch the ClusterTopology CR for drift would tightly couple the controller to Grove internals and duplicate responsibility. If Grove doesn't currently surface this, it might be worth raising as a Grove enhancement.

[Ronkahn21]

Dynamo doesn't need to watch ClusterTopology for drift. Grove handles this by reporting a TopologyLevelsUnavailable condition on the PCS whenever a required topology level is removed

[julienmancuso]

ok yes that makes sense, i updated the doc

[Ronkahn21]

If user set constrains only at the service level, does it mean the feature is enabled
or it always required to set the spec.topologyConstraint to enable the feature

[julienmancuso]

full setup should be there (deployment + all services) otherwise we would reject it

[Ronkahn21]

Could you clarify the rationale behind making this a strict requirement?

[julienmancuso]

which i realized is not correct, I'm going to relax the constraint, the feature will be enabled only when 1 of the deployment or either service has topology enabled