Evaluation on speech output tasks

[Ashbajawed]

for all the datasets mentioned in supported datasets
none have audio/speech as output, how the llama model is going to compare the reference from generated audios?

[UltraEval]

for all the datasets mentioned in supported datasets none have audio/speech as output, how the llama model is going to compare the reference from generated audios?

you can use https://github.com/OpenBMB/UltraEval-Audio, An easy-to-use, fast, and easily integrable tool for evaluating audio LLM understanding and geeration

[BinWang28]

Hi @Ashbajawed, We currently do not support audio generation (evaluation) in AudioBench.

The evaluation is focused on audio/speech understanding tasks. We may plan this in the future.

[BinWang28]

@UltraEval Very nice work.

I've added to our collection: https://github.com/AudioLLMs/Awesome-Audio-LLM