Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add versioning to MTEB benchmarks #2024

Merged

Conversation

KennethEnevoldsen
Copy link
Contributor

  • Following suggestion made in Rename MTEB(eng) & MTEB(eng,classic) #2001 I added version to MTEB benchmarks
  • changed the name of the MTEB(Chinese) to MTEB(cmn, v1). Though we could go for MTEB(Chinese, v1) assuming it is a group (also covering chinese other than mandarin)
  • change the name of FaMTEB(fas, beta) to MTEB(fas)
  • did a minor formatting of imports due to a circular import error
  • moved the Benchmark object out of the file with the benchmarks
    • this is still >1000 lines so we could split it up to "external_benchmarks", "monolingual_benchmarks", "domain_specific_benchmarks", "multilingual_benchmarks"

Fixes #2001

Code Quality

  • Code Formatted: Format the code using make lint to maintain consistent style.

Documentation

  • Updated Documentation: Add or update documentation to reflect the changes introduced in this PR.

Testing

  • New Tests Added: Write tests to cover new functionality. Validate with make test-with-coverage.
  • Tests Passed: Run tests locally using make test or make test-with-coverage to ensure no existing functionality is broken.

- Following suggestion made in #2001 I added version to MTEB benchmarks
- changed the name of the MTEB(Chinese) to MTEB(cmn, v1). Though we could go for MTEB(Chinese, v1) assuming it is a group (also covering chinese other than mandarin)
- change the name of FaMTEB(fas, beta) to MTEB(fas)
- did a minor formatting of imports due to a circular import error
- moved the Benchmark object out of the file with the benchmarks
  - this is still >1000 lines so we could split it up to "external_benchmarks", "monolingual_benchmarks", "domain_specific_benchmarks", "multilingual_benchmarks"

Fixes #2001
Copy link
Contributor

@Muennighoff Muennighoff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! To avoid breaking backward compat would it make sense to temporarily allow the old names by just mapping to the new benchmark carrying that name and raising a warning that the old name will be deprecated soon?

@KennethEnevoldsen
Copy link
Contributor Author

Nice! To avoid breaking backward compat would it make sense to temporarily allow the old names by just mapping to the new benchmark carrying that name and raising a warning that the old name will be deprecated soon?

Good catch, will add this tomorrow

Copy link
Collaborator

@x-tabdeveloping x-tabdeveloping left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with Niklas, Also make sure this doesn't break the leaderboard (I think it probably will since the multilingual benchmark is referred to by name)

@KennethEnevoldsen
Copy link
Contributor Author

I have added changed for backward compatibility. I have also updated the leaderboard code to refer to the MTEB_multlingual.name. This should prevent inconsistencies.

@KennethEnevoldsen KennethEnevoldsen merged commit 65f3407 into main Feb 12, 2025
8 checks passed
@KennethEnevoldsen KennethEnevoldsen deleted the KennethEnevoldsen/issue-Rename-MTEBeng-MTEBengclassic branch February 12, 2025 10:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Rename MTEB(eng) & MTEB(eng,classic)
4 participants