Question regarding alpaca_eval sampling parameter

[lancerts]

Although we have set temperature parameter in here, due to this

    if do_sample:
        sampling_params.use_beam_search = True

1. Does it result in using beam_search of beam size 1 and ignoring the temperature parameter in vllm generation?
2. In fact, alpaca_eval 0.6.2 with recent vllm does not work with do_sample = True, related issue here.
3. However, we tested successfully with the old VLM version of 0.4.1 and confirmed that the temperature was neglected and that generation is always the same.

[ZeguanXiao]

I'm also confused about this, have you found the answer?

[visionxyz]

For many new versions of vllm, we can not even run alpaca eval based on the submitted configs.

[xiamengzhou]

Hi, we have pointed out the issue here in our ReadME. You would have to use alpaca_eval==0.6.2 and vllm=0.5.3 to fully reproduce the results. Hope it helps!