[Bug] windows下数据集的位置以及检测结果为0.0

[Dbgsaoge]

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• The bug has not been fixed in the latest version.

Type

I'm evaluating with the officially supported tasks/models/datasets.

Environment

{'CUDA available': True,
'CUDA_HOME': 'D:\CUDA\CUDA12.1\NVIDIA GPU Computing Toolkit\CUDA\v12.1',
'GCC': 'n/a',
'GPU 0': 'NVIDIA GeForce RTX 4070 Ti',
'MMEngine': '0.10.5',
'MUSA available': False,
'NVCC': 'Cuda compilation tools, release 12.1, V12.1.105',
'OpenCV': '4.10.0',
'PyTorch': '2.0.0+cu118',
'PyTorch compiling details': 'PyTorch built with:\n'
' - C++ Version: 199711\n'
' - MSVC 193431937\n'
' - Intel(R) Math Kernel Library Version '
'2020.0.2 Product Build 20200624 for Intel(R) 64 '
'architecture applications\n'
' - Intel(R) MKL-DNN v2.7.3 (Git Hash '
'6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n'
' - OpenMP 2019\n'
' - LAPACK is enabled (usually provided by '
'MKL)\n'
' - CPU capability usage: AVX2\n'
' - CUDA Runtime 11.8\n'
' - NVCC architecture flags: '
'-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_37,code=compute_37\n'
' - CuDNN 8.7\n'
' - Magma 2.5.4\n'
' - Build settings: BLAS_INFO=mkl, '
'BUILD_TYPE=Release, CUDA_VERSION=11.8, '
'CUDNN_VERSION=8.7.0, '
'CXX_COMPILER=C:/actions-runner/_work/pytorch/pytorch/builder/windows/tmp_bin/sccache-cl.exe, '
'CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w '
'/bigobj /FS -DUSE_PTHREADPOOL -DNDEBUG '
'-DUSE_KINETO -DLIBKINETO_NOCUPTI '
'-DLIBKINETO_NOROCTRACER -DUSE_FBGEMM '
'-DUSE_XNNPACK '
'-DSYMBOLICATE_MOBILE_DEBUG_HANDLE, '
'LAPACK_INFO=mkl, PERF_WITH_AVX=1, '
'PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, '
'TORCH_DISABLE_GPU_ASSERTS=OFF, '
'TORCH_VERSION=2.0.0, USE_CUDA=ON, USE_CUDNN=ON, '
'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
'USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, '
'USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, '
'USE_OPENMP=ON, USE_ROCM=OFF, \n',
'Python': '3.10.15 | packaged by Anaconda, Inc. | (main, Oct 3 2024, '
'07:22:19) [MSC v.1929 64 bit (AMD64)]',
'TorchVision': '0.15.1+cu118',
'lmdeploy': "not installed:No module named 'lmdeploy'",
'numpy_random_seed': 2147483648,
'opencompass': '0.3.6+',
'sys.platform': 'win32',
'transformers': '4.46.3'}

Reproduces the problem - code/configuration sample

For mymodel_cfg.py

from opencompass.models import HuggingFaceCausalLM
from opencompass.models import HuggingFaceBaseModel

models = [

dict(

type=HuggingFaceCausalLM,

path='meta-llama/Llama-3.2-1B',

model_kwargs=dict(device_map='auto'),

tokenizer_path='meta-llama/Llama-3.2-1B',

tokenizer_kwargs=dict(padding_side='left', truncation_side='left'),

max_seq_len=2048,

max_out_len=50,

run_cfg=dict(num_gpus=1, num_procs=1),

)

]

models = [
dict(
type=HuggingFaceBaseModel,
abbr='Llama-3.2-1B',
path='meta-llama/Llama-3.2-1B',
max_out_len=1024,
batch_size=8,
run_cfg=dict(num_gpus=1),
)
]

For myeval_demo.py

from mmengine.config import read_base

with read_base():
from .mymodel_cfg import models # mymodel_cfg.py 中的 models 被继承到本配置文件
from opencompass.configs.datasets.demo.demo_gsm8k_base_gen import gsm8k_datasets
from opencompass.configs.datasets.demo.demo_math_base_gen import math_datasets

datasets = []
datasets += gsm8k_datasets
datasets += math_datasets

Reproduces the problem - command or script

python run.py eval_demo.py --debug

Reproduces the problem - error message

F:\Env\LLM_env\lib\site-packages\opencompass_init_.py:17: UserWarning: Starting from v0.4.0, all AMOTIC configuration files currently located in ./configs/datasets, ./configs/models, and ./configs/summarizers will be migrated to the opencompass/configs/ package. Please update your configuration file paths accordingly.
warn_about_config_migration()
11/29 01:36:09 - OpenCompass - INFO - Task [Llama-3.2-1B/demo_gsm8k,Llama-3.2-1B/demo_math]
F:\Env\LLM_env\lib\site-packages\torchvision\datapoints_init.py:12: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: pytorch/vision#6753, and you can also check out pytorch/vision#7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
warnings.warn(BETA_TRANSFORMS_WARNING)
F:\Env\LLM_env\lib\site-packages\torchvision\transforms\v2_init.py:54: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: pytorch/vision#6753, and you can also check out pytorch/vision#7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
warnings.warn(_BETA_TRANSFORMS_WARNING)
11/29 01:36:12 - OpenCompass - WARNING - pad_token_id is not set for the tokenizer.
11/29 01:36:12 - OpenCompass - WARNING - Using eos_token_id 128001 as pad_token_id.
signal.SIGALRM is not available on this platform
signal.SIGALRM is not available on this platform
11/29 01:36:15 - OpenCompass - INFO - Try to load the data from C:\Users\Administrator.cache/opencompass/./data/gsm8k/
11/29 01:36:15 - OpenCompass - INFO - Start inferencing [Llama-3.2-1B/demo_gsm8k]
[2024-11-29 01:36:15,688] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2024-11-29 01:36:15,688] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...

0%| | 0/8 [00:00<?, ?it/s]F:\Env\LLM_env\lib\site-packages\transformers\generation\configuration_utils.py:590: UserWarning: do_sample is set to False. However, temperature is set to 0.6 -- this flag is only used in sample-based generation modes. You should set do_sample=True or unset temperature.
warnings.warn(
F:\Env\LLM_env\lib\site-packages\transformers\generation\configuration_utils.py:595: UserWarning: do_sample is set to False. However, top_p is set to 0.9 -- this flag is only used in sample-based generation modes. You should set do_sample=True or unset top_p.
warnings.warn(

12%| | 1/8 [00:05<00:37, 5.33s/it]
25%| | 2/8 [00:19<01:02, 10.41s/it]
38%| | 3/8 [00:36<01:06, 13.39s/it]
50%| | 4/8 [00:52<00:58, 14.68s/it]
62%| | 5/8 [01:09<00:45, 15.33s/it]
75%| | 6/8 [01:28<00:33, 16.69s/it]
88%| | 7/8 [01:47<00:17, 17.35s/it]
100%| | 8/8 [02:06<00:00, 17.88s/it]
100%| | 8/8 [02:06<00:00, 15.80s/it]
11/29 01:38:22 - OpenCompass - INFO - Try to load the data from C:\Users\Administrator.cache/opencompass/./data/math/
11/29 01:38:22 - OpenCompass - INFO - Start inferencing [Llama-3.2-1B/demo_math]
[2024-11-29 01:38:22,195] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2024-11-29 01:38:22,195] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...

0%| | 0/8 [00:00<?, ?it/s]
12%| | 1/8 [00:34<03:59, 34.22s/it]
25%| | 2/8 [01:08<03:24, 34.03s/it]
38%| | 3/8 [01:43<02:54, 34.82s/it]
50%| | 4/8 [02:17<02:17, 34.43s/it]
62%| | 5/8 [02:51<01:42, 34.17s/it]
75%| | 6/8 [03:24<01:07, 33.95s/it]
88%| | 7/8 [03:58<00:33, 33.78s/it]
100%| | 8/8 [04:32<00:00, 33.75s/it]
100%| | 8/8 [04:32<00:00, 34.01s/it]
11/29 01:42:54 - OpenCompass - INFO - time elapsed: 404.59s
F:\Env\LLM_env\lib\site-packages\opencompass_init_.py:17: UserWarning: Starting from v0.4.0, all AMOTIC configuration files currently located in ./configs/datasets, ./configs/models, and ./configs/summarizers will be migrated to the opencompass/configs/ package. Please update your configuration file paths accordingly.
warn_about_config_migration()
F:\Env\LLM_env\lib\site-packages\torchvision\datapoints_init.py:12: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: pytorch/vision#6753, and you can also check out pytorch/vision#7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
warnings.warn(BETA_TRANSFORMS_WARNING)
F:\Env\LLM_env\lib\site-packages\torchvision\transforms\v2_init.py:54: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: pytorch/vision#6753, and you can also check out pytorch/vision#7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
warnings.warn(_BETA_TRANSFORMS_WARNING)
signal.SIGALRM is not available on this platform
signal.SIGALRM is not available on this platform
11/29 01:42:59 - OpenCompass - INFO - Try to load the data from C:\Users\Administrator.cache/opencompass/./data/gsm8k/
Parameter 'function'=<function OpenICLEvalTask._score..postprocess at 0x0000013618439510> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.

Map: 0%| | 0/64 [00:00<?, ? examples/s]
Map: 100%| | 64/64 [00:00<00:00, 21402.92 examples/s]
11/29 01:42:59 - OpenCompass - INFO - Task [Llama-3.2-1B/demo_gsm8k]: {'accuracy': 4.6875}
11/29 01:42:59 - OpenCompass - INFO - time elapsed: 2.10s
F:\Env\LLM_env\lib\site-packages\opencompass_init_.py:17: UserWarning: Starting from v0.4.0, all AMOTIC configuration files currently located in ./configs/datasets, ./configs/models, and ./configs/summarizers will be migrated to the opencompass/configs/ package. Please update your configuration file paths accordingly.
warn_about_config_migration()
F:\Env\LLM_env\lib\site-packages\torchvision\datapoints_init.py:12: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: pytorch/vision#6753, and you can also check out pytorch/vision#7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
warnings.warn(BETA_TRANSFORMS_WARNING)
F:\Env\LLM_env\lib\site-packages\torchvision\transforms\v2_init.py:54: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: pytorch/vision#6753, and you can also check out pytorch/vision#7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
warnings.warn(_BETA_TRANSFORMS_WARNING)
signal.SIGALRM is not available on this platform
signal.SIGALRM is not available on this platform
11/29 01:43:04 - OpenCompass - INFO - Try to load the data from C:\Users\Administrator.cache/opencompass/./data/math/
11/29 01:43:04 - OpenCompass - INFO - Task [Llama-3.2-1B/demo_math]: {'accuracy': 0.0}
11/29 01:43:04 - OpenCompass - INFO - time elapsed: 2.08s

Other information

（1）我已经下载了离线数据集，但是似乎在运行中opencompass仍然重新下载并缓存在了C:\Users\Administrator.cache\opencompass\data，我应如何修改离散数据集的路径，或者修改opencompass读取数据的路径？
（2）我使用的Llama3.2-1B模型在math_demo数据集上的acc为0，这是正常情况还是其他bug？