SCAI Eval 2024 Metric Validator

This repository contains code for validating metric labels submitted to SCAI Eval 2024

Local Usage

python3 scai-eval24-metric-validator.py ground-truth-directory run-directory/run.json /dev/zero

Command in TIRA:

python3 /app/scai-eval24-metric-validator.py $inputDir $inputRun/* $outputDir/evaluation.prototext

docker build -t webis/scai-eval24-metric-validator:1.0.0 .
docker push webis/scai-eval24-metric-validator:1.0.0

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
.dockerignore		.dockerignore
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
scai-eval24-metric-validator.py		scai-eval24-metric-validator.py