Question: How much memory does the model need?

[carlory]

copid from https://github.com/vllm-project/semantic-router/blob/main/website/docs/tutorials/intelligent-route/embedding-routing.md#performance-characteristics

Model	Dimension	Latency	Accuracy	Memory
Qwen3	1024	30-50ms	Highest	600MB
Gemma	768	10-20ms	High	300MB
Gemma	512	8-15ms	Medium	300MB
Gemma	256	5-10ms	Lower	300MB

The Memory column seems to be incorrect. 300 and 600 are model sizes, not memory usage. And dtype is f32, so it should be at least 4 times the model size.

semantic-router/candle-binding/src/model_architectures/model_factory.rs

Line 180 in 912fe2a

candle_core::DType::F32,

[rootfs]

@carlory thanks, can you share a PR?