ImportError: attempted relative import with no known parent package

[BenDes21]

Hi, Im trying to use this model https://huggingface.co/dphn/Dolphin-Mistral-24B-Venice-Edition-FP8, so my env variables look like this :

When I try to send a curl request Im getting this :

INFO 09-07 03:11:46 [init.py:244] Automatically detected platform cuda.
Traceback (most recent call last):
File "/src/handler.py", line 13, in
from utils import format_chat_prompt, create_error_response
File "/src/utils.py", line 3, in
from .models import ChatMessage, ErrorResponse
ImportError: attempted relative import with no known parent package

Any ideas how to fix it @pandyamarut ? Thanks you

edit : its not working even with no fp8 quantization with the model dphn/Dolphin-Mistral-24B-Venice-Edition

[VanshRana12]

Just remove the '.' from '.models' in utils.py and handler.py import statements

[BenDes21]

Just remove the '.' from '.models' in utils.py and handler.py import statements

Hi thanks, another very quick question : is it ok to fork the repo and modify the version of vllm ( to the latest ) then use the repo with the github deployment feature ?

[VanshRana12]

Just remove the '.' from '.models' in utils.py and handler.py import statements

Hi thanks, another very quick question : is it ok to fork the repo and modify the version of vllm ( to the latest ) then use the repo with the github deployment feature ?

Haven't tried that myself, but you could give it a go.
Tho i am sure they must have set all the packages' versions in requirements.txt and the format of their vllm engine in handler.py according to the vllm version they have frozen in dockerfile, so issues will definitely arise.

[BenDes21]

Just remove the '.' from '.models' in utils.py and handler.py import statements

Hi thanks, another very quick question : is it ok to fork the repo and modify the version of vllm ( to the latest ) then use the repo with the github deployment feature ?

Haven't tried that myself, but you could give it a go. Tho i am sure they must have set all the packages' versions in requirements.txt and the format of their vllm engine in handler.py according to the vllm version they have frozen in dockerfile, so issues will definitely arise.

Ok thanks, If you have time it's would be great to update it from time to time ( like only the major update )

[BenDes21]

Just remove the '.' from '.models' in utils.py and handler.py import statements

Hi thanks, another very quick question : is it ok to fork the repo and modify the version of vllm ( to the latest ) then use the repo with the github deployment feature ?

Haven't tried that myself, but you could give it a go. Tho i am sure they must have set all the packages' versions in requirements.txt and the format of their vllm engine in handler.py according to the vllm version they have frozen in dockerfile, so issues will definitely arise.

seems update to 10.1.1 broke the repo, when I fork, update the vllm version and then deploy it's initializing all the workers infinitively

[BenDes21]

Hi @pandyamarut , I forked the repo and remove only the "." in the 2 files you mentioned but the repo just dont want to build when Im using the github deployment feature ( all workers are in INIT state for an infinite time ). I didnt modify the dockerfile or any other files.

Whats can be the solution ? Thanks

[Solidsoldier12]

https://github.com/Solidsoldier12/vllm-loadbalancer-ep
Maybe try this?
I applied the fixes here and used it, built, ran fine on runpod too.

[BenDes21]

https://github.com/Solidsoldier12/vllm-loadbalancer-ep Maybe try this? I applied the fixes here and used it, built, ran fine on runpod too.

Hi there, thanks you ! How long it's take you to build the image from github deployment feature ? After 30 minutes its still building and all workers still initializing

[Solidsoldier12]

https://github.com/Solidsoldier12/vllm-loadbalancer-ep Maybe try this? I applied the fixes here and used it, built, ran fine on runpod too.

Hi there, thanks you ! How long it's take you to build the image from github deployment feature ? After 30 minutes its still building and all workers still initializing

Hi

I didn't try building the image via the github deployment feature, just did It locally manually, and it took some time but worked fine.
You should monitor the docker build logs to see if it's getting stuck or throwing some errors.

[BenDes21]

https://github.com/Solidsoldier12/vllm-loadbalancer-ep Maybe try this? I applied the fixes here and used it, built, ran fine on runpod too.

Hi there, thanks you ! How long it's take you to build the image from github deployment feature ? After 30 minutes its still building and all workers still initializing

Hi

I didn't try building the image via the github deployment feature, just did It locally manually, and it took some time but worked fine. You should monitor the docker build logs to see if it's getting stuck or throwing some errors.

ok I will build the image with docker instead of using the github deployment