-
Notifications
You must be signed in to change notification settings - Fork 332
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add versioning to MTEB benchmarks #2024
Add versioning to MTEB benchmarks #2024
Conversation
- Following suggestion made in #2001 I added version to MTEB benchmarks - changed the name of the MTEB(Chinese) to MTEB(cmn, v1). Though we could go for MTEB(Chinese, v1) assuming it is a group (also covering chinese other than mandarin) - change the name of FaMTEB(fas, beta) to MTEB(fas) - did a minor formatting of imports due to a circular import error - moved the Benchmark object out of the file with the benchmarks - this is still >1000 lines so we could split it up to "external_benchmarks", "monolingual_benchmarks", "domain_specific_benchmarks", "multilingual_benchmarks" Fixes #2001
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! To avoid breaking backward compat would it make sense to temporarily allow the old names by just mapping to the new benchmark carrying that name and raising a warning that the old name will be deprecated soon?
Good catch, will add this tomorrow |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with Niklas, Also make sure this doesn't break the leaderboard (I think it probably will since the multilingual benchmark is referred to by name)
I have added changed for backward compatibility. I have also updated the leaderboard code to refer to the |
Co-authored-by: Roman Solomatin <[email protected]>
Co-authored-by: Roman Solomatin <[email protected]>
Fixes #2001
Code Quality
make lint
to maintain consistent style.Documentation
Updated Documentation: Add or update documentation to reflect the changes introduced in this PR.Testing
make test-with-coverage
.make test
ormake test-with-coverage
to ensure no existing functionality is broken.