Egoschema results iconsistency

[ssantos97]

Hi, I've ran your code "demo_mistral.ipynb" with the exact same library and python versions yet my accuracy is around 0.5 as opposed to 63.6 from your paper. I'm using mistral, stage3 and the exact same folders that I downloaded of config_mistral.json. What am I doing wrong?

[yinanhe]

Most of the cases may be caused by inconsistent versions of peft, which should be 0.5.0. in specific cases, you can attach the runtime file logjudge

[ssantos97]

Hi, I leave the config and my requirements (the code is exactly the same), could you check what I'm doing wrong? I think it's weird because the difference is -13.6% which is a lot for library mismatches. Also, could you do pip freeze so I can use exactly the same versions?
requirements_mine.txt
config_mistral.json

[ssantos97]

Solved. Thank you for the help!

[ssantos97]

Just one more thing, for next qa, are you using the validation or test split for evaluation?