[Feat] In-context classification with LLM

[yuezhu1]

Describe the feature

We are introducing in-class classification capabilities to the vLLM Semantic Router, enabling fine-grained classification within broader categories without requiring model training or fine-tuning. By leveraging few-shot learning through in-context examples injected into vLLM prompts, the router can distinguish between subcategories (e.g., "finance - stock market" vs "finance - marketing") or specific intents within a category, providing more precise routing decisions.

Why do you need this feature?

Key Benefits:

• No Training Required: Uses in-context learning with few-shot examples
• Bring Your Own Model: Works with any vLLM-supported generative model
• Backward Compatible: Opt-in feature, existing configurations continue to work
• Flexible Configuration: Users define subcategories and provide examples

Additional context

Coming soon! Stay tuned!