Visualization and training error with Real world Dataset in huggingface

[LaFeuilleMorte]

Hi, I followed your guidance in REALBOT_PIPELINE_GUIDE.md, and downloaded the arrow format dataset. But when I setup everything and run the following scripts:

python3 scripts/data_visualize.py --config configs/config_holobrain_qwen_common.py

it gave me the following error:

  File "/home/wqh/Project/RoboOrchardLab/robo_orchard_lab/dataset/horizon_manipulation/agilex_ro_dataset.py", line 159, in __call__
    data.update(self.get_instruction(data))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wqh/Project/RoboOrchardLab/robo_orchard_lab/dataset/horizon_manipulation/agilex_ro_dataset.py", line 77, in get_instruction
    text = data["instruction"].json_content["description"]
           ~~~~^^^^^^^^^^^^^^^
KeyError: 'instruction'

The same happened when I launched the training script. After checking the dataset_info.json of the arrow format dataset. I found it had no instruction field but instruction_index. The dataset_info.json was here: dataset_info.json

[crayonsea]

To fix it, please manually update the following code:

RoboOrchardLab/robo_orchard_lab/dataset/horizon_manipulation/agilex_ro_dataset.py

Line 217 in 1a2e483

ROMultiRowDataset(dataset_path=p, row_sampler=row_sampler)

Change it to:

ROMultiRowDataset(dataset_path=p, row_sampler=row_sampler, meta_index2meta=True)

Adding meta_index2meta=True ensures that the instruction field is properly loaded from the metadata.

This issue is caused by a recent change where the default value of meta_index2meta was switched from True to False. Because of that, the dataset no longer automatically resolves instruction_index into the actual instruction field, which leads to the KeyError.

We’ll push a fix soon to restore the correct default behavior. For now, this manual change should resolve the problem.

[LaFeuilleMorte]

To fix it, please manually update the following code:

RoboOrchardLab/robo_orchard_lab/dataset/horizon_manipulation/agilex_ro_dataset.py

Line 217 in 1a2e483

ROMultiRowDataset(dataset_path=p, row_sampler=row_sampler)
Change it to:

ROMultiRowDataset(dataset_path=p, row_sampler=row_sampler, meta_index2meta=True)
Adding meta_index2meta=True ensures that the instruction field is properly loaded from the metadata.

This issue is caused by a recent change where the default value of meta_index2meta was switched from True to False. Because of that, the dataset no longer automatically resolves instruction_index into the actual instruction field, which leads to the KeyError.

We’ll push a fix soon to restore the correct default behavior. For now, this manual change should resolve the problem.

Thank you for the reply, I used your fix and both visualization and training worked.