Latest Compose Spec

[vivek7405]

Is the latest compose spec that introduced the "models" specification for running AI models supported in "uc deploy"?

If not, are there any plans of supporting it?

[psviderski]

AI models in Compose are not supported by uncloud. Tbh, I didn't even know about that before. I'll need to learn more about how this works in docker.

In the meantime, I'd appreciate if you can describe your use case and how you'd like to use models in an uncloud cluster.

[airtonix]

Docker Model Runner only works on macs, since it requires Docker Desktop.

Even though that page seems to indicate that it supports linux, it's actually seemingly impossible to use it because the required binaries to support are only made available if you use Docker Desktop on Mac.

[vivek7405]

Sorry, I'm not arguing, but I just fact checked with Claude and it seems to disagree. Though, it may be incorrect 😅

https://claude.ai/share/efddf6cc-5cd8-43e4-a8d0-dd979a860919

[vivek7405]

Few links for reference:

https://www.docker.com/products/model-runner/

https://docs.docker.com/ai/model-runner/?_gl=1*qfydf4*_gcl_au*MzAxMjQ3NzAyLjE3NTgyNzk4NjI.*_ga*MTYwOTA2NDI0Mi4xNzUwNDUxMDQx*_ga_XJWPQMJYHQ*czE3NTgyNzk4NjIkbzE4JGcxJHQxNzU4MjgwMDMxJGozNSRsMCRoMA..#requirements

https://www.docker.com/blog/build-ai-agents-with-docker-compose/

[tonyo]

(copying from Discord)

Docker Model Runner should work on Linux (tested it on a linux server with "ai/smollm2" model but without GPUs)
Integrating it with Uncloud should be pretty doable; DMR basically starts a given model on an internal http://model-runner.docker.internal/ URL, so as long as we do the same on every uncloud machine where the ai-enabled container runs -- we should be good. We could optimize it further by only running the model on a single machine and then exposing an uncloud-wide endpoint via Caddy, if we decide to optimize it a bit further.

On the Compose side, what happens is basically every service that has the "models" attribute configured receives (automatically injected) env variables that contain URLs pointing at the specified model (https://docs.docker.com/ai/compose/models-and-compose/#model-configuration-options)
Also something that we'll have to handle on the Uncloud side, but doesn't seem to be a lot of work