lighteval with llama3.2 [RuntimeError: No executable batch size found, reached zero.]

[Nevermetyou65]

Hi

I tried to use lighteval with meta-llama/Llama-3.2-1B.
I want to run the evaluation for Thai finetasks. But I got this error instead RuntimeError: No executable batch size found, reached zero.
Any ideas? I have a full log with me if you want.
Here is my command

echo "Running lighteval for model: $model"
    lighteval accelerate \
    "pretrained=${model},dtype=bfloat16" \
    "examples/tasks/fine_tasks/mcf/th.txt" \
    --custom-tasks "src/lighteval/tasks/multilingual/tasks.py" \
    --no-use-chat-template

I am using lighteval 0.6.0.dev0 and torch 2.2.2+cu121
I didn't use pip install install lighteval but clone the repo instead and pip install -e . [dev]

[clefourrier]

Hi!
Yes, the full log would be useful :)

[Nevermetyou65]

log_llama_3_2_03.txt

Please scroll down to the bottom of this file

[Nevermetyou65]

Hi, any update, sir?