RFC: Adding Routable Taints, Tiering and Tainting Workers

[michaelfeil]

A model like DeepSeek can be deployed two ways:

• high throughput (DP, MTP off)
• low latency (TP, MTP on)

You might have two customers or use-cases:

• one cares about low TTFT and ITL
• one cares about total throughput

It would be great if both use-cases could run on the same endpoint, and not be backed by multiple deployments.
In an complex world, it would be helpful to dynamically switch between deployments.

It would be great to have metadata, taints associated with a deployment:

worker-xxx:
 - metadata:  "TAINTS": "LOW_LATENCY"
 worker-xxx
 - metadata:  "TAINTS": HIGH_THOUGHPUT  

Then a client to dynamo could sent a http request to dynamo via:

For this to happen, the user could e.g. send a X-Allowed-Taints: "High-Throughput,Low-Latency" or X-Preferred-Taints: "Low-Latency" header.
As result , the taint / preference is forwarded to the router. Request could be routed to a LOW_LATENCY tainted worker, using shared router/frontend components. In case the LOW_LATENCY workers show higher load (aka lower performance) than HIGH_THOUGHPUT, we could additionally route them to HIGH_THOUGHPUT workers.