Skip to content

Issues: huggingface/lighteval

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
70 Open 122 Closed
70 Open 122 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

More variety in prompt support feature request New feature/request
#8 opened Jan 31, 2024 by clefourrier
@clefourrier
1
Allow passing a post normalisation function feature request New feature/request
#17 opened Feb 6, 2024 by clefourrier
--no_multichoice_continuations_start_space should also cover startof word token bug Something isn't working
#45 opened Feb 20, 2024 by clefourrier
1
Append revision to filepath in --output_dir? feature request New feature/request good first issue Good for newcomers
#56 opened Feb 26, 2024 by lewtun
4
issues when testing wikitext bug Something isn't working
#57 opened Feb 26, 2024 by yangfuwei
1
9
Add single mmlu config for lighteval suite feature request New feature/request
#61 opened Feb 27, 2024 by lewtun
1
Add HumanEval and HumanEval+ new task
#63 opened Feb 27, 2024 by lewtun
1
Need to reupload TruthfulQA bug Something isn't working
#69 opened Feb 27, 2024 by clefourrier
1
Align GPQA zero-shot / few-shot prompts with paper? feature request New feature/request
#70 opened Feb 27, 2024 by lewtun
@lewtun
3
To remember for version upgrades to keep
#97 opened Mar 6, 2024 by clefourrier
Add EQ Bench new task
#114 opened Mar 18, 2024 by lewtun
1
[New Task] Add AlpacaEval LC feature request New feature/request new task
#139 opened Apr 2, 2024 by YannDubs
9
Add a nanotron model to the test suite
#144 opened Apr 4, 2024 by clefourrier
1
Add Code-Centric Interface to LightEval for Enhanced Usability
#148 opened Apr 7, 2024 by adithya-s-k
1
4
Add Sympy equivalence for MATH / GSM8K?
#170 opened Apr 23, 2024 by lewtun
2
Performance compared to lm-evaluation-harness
#179 opened Apr 29, 2024 by geoalgo
6
apply_target_perplexity_metric pops only the first response
#192 opened May 20, 2024 by sadra-barikbin
Expose stop_sequence at command line
#196 opened May 27, 2024 by lewtun
@clefourrier
1
Handling last token in tokenized_continuation
#203 opened Jun 15, 2024 by simon900314
3
Add evaluations for function calling
#210 opened Jul 3, 2024 by clefourrier
Dataset loading issue for german_rag_evals on Windows
#211 opened Jul 4, 2024 by Pommel4711
@NathanHB
33
Extend #228 with few shot samples
#230 opened Jul 17, 2024 by clefourrier
[FT] Caching feature request New feature/request
#239 opened Jul 26, 2024 by clefourrier
3
[FT] Add nucleus sampling feature request New feature/request
#246 opened Jul 31, 2024 by clefourrier
[DOC] Document the custom model config files
#247 opened Jul 31, 2024 by clefourrier
1
2
ProTip! What’s not been updated in a month: updated:<2025-01-06.