Dataset seems incomplete yet

[Luffy966]

Hi, this work is outstanding and I'm very insterested in this work!
But when I'm trying reproducing the experiment based on run_libero_eval.py, I fail to found the following json file in both OmniActionand OmniAction-LIBERO currently.

with open(f"./data/libero_{cfg.speech}.json", "r", encoding="utf-8") as f:
    data = json.load(f)

BTW, it seems that the datasets provided in huggingface are incomplete yet. There are six speech type ['overlap', 'identity', 'sentiment', 'events', 'two', 'three']. But currently there are only *_identity and *_sentiment subfolders in OmniAction-LIBERO.

Could it be solved soon? I would really appreciate that!

[sixingdeguo]

I encountered the same issue regarding the data questions and the incomplete code. I’m not sure how to resolve it, and the README does not seem to provide much help. I had to manually incorporate code from OpenVLA to make it.

[Luffy966]

I encountered the same issue.
After coping the file libero_utils.py and roboomni_utils.py from OpenVLA, I'm now solving the dependencies.

[sixingdeguo]

Yes, I solved the problem in the same way.

[Luffy966]

when I use tfds to load OmniAction-LIBERO/libero_speech10_identity as ds and print:

for item in next(iter(ds.take(1)))['steps']:
    speech_conv = item["speech_conv"].numpy().decode('utf-8')
    conv = item["conversation"].numpy().decode('utf-8')
    print(speech_conv, "\n", conv)
    break

It output:

./three/dialogue_124/user_turn_0.wav[UNK]./three/dialogue_124/user_turn_1.wav 
 [USER][S1]Dad, my brother said he was going to take the alpabet soup and cream cheese out, and I'm guessing he will need to put them in the basket. [S3]Oh no, my little one, I was thinking of using the cream cheese to make a snack for us. [S1]Hmm, but wouldn't it be better if we just got everything ready? [S2]Hey, hey, just get my stuff ready first, please! I need to go out right now![ASSISTANT][Robot]Hey there, do you need me to put the alphabet soup and cream cheese in the basket for you?[UNK][USER][S1]Um, yeah, that would be awesome![ASSISTANT][Robot]Alright. I will put both the alphabet soup and the cream cheese box in the basket. [ACT]

It's strange that the "three" audio type appears in "identity" subset. And I cannot find any audio files in OmniAction-LIBERO dataset currently.

[Luffy966]

@sinwang20 Do we have any plans on open-sourcing the audio files to support reproduction and further researches?

[sixingdeguo]

Yes, yes, the author lacked audio data, so I generated the audio myself.

I encountered the same problem.
I iterated through the data he provided, and there were only about 100 different audio paths, which is confusing and doesn't match the dataset name.

I'll try creating one "libero_{cfg.speech}.json" tomorrow😨.

[Holmes-GU]

when I use tfds to load OmniAction-LIBERO/libero_speech10_identity as ds and print:

for item in next(iter(ds.take(1)))['steps']:
    speech_conv = item["speech_conv"].numpy().decode('utf-8')
    conv = item["conversation"].numpy().decode('utf-8')
    print(speech_conv, "\n", conv)
    break

It output:

./three/dialogue_124/user_turn_0.wav[UNK]./three/dialogue_124/user_turn_1.wav 
 [USER][S1]Dad, my brother said he was going to take the alpabet soup and cream cheese out, and I'm guessing he will need to put them in the basket. [S3]Oh no, my little one, I was thinking of using the cream cheese to make a snack for us. [S1]Hmm, but wouldn't it be better if we just got everything ready? [S2]Hey, hey, just get my stuff ready first, please! I need to go out right now![ASSISTANT][Robot]Hey there, do you need me to put the alphabet soup and cream cheese in the basket for you?[UNK][USER][S1]Um, yeah, that would be awesome![ASSISTANT][Robot]Alright. I will put both the alphabet soup and the cream cheese box in the basket. [ACT]

It's strange that the "three" audio type appears in "identity" subset. And I cannot find any audio files in OmniAction-LIBERO dataset currently.

Hi, have you solved the problem and could we have a further discussion?