TULU3 MATH eval

[ypwang61]

Hi, thanks for your great work, but when I try to evaluate 'models--allenai--Llama-3.1-Tulu-3-8B' on MATH dataset using this codebase, the accuracy is just all zero. Is there some format mismatch on the evaluation? (Using the same model, I can obtain ~88% GSM8k exact match rate)

I guess the problem is caused by the stop_strings:

if not args.use_chat_format:
   stop_strings += ["\n"]

comment it can obtain ~45% result

[vwxyzjn]

Hi thanks for the issue. We use https://github.com/allenai/olmes to do evals. Would you like to check it out?