11 Pull requests merged by 8 people

• Fix TGI (Text Generation Inference) Endpoint Inference and TGI JSON Grammar Generation

  #502 merged Aug 25, 2025

• Parametrizing the sampling evals from the CLI

  #926 merged Aug 25, 2025

• Save reasoning_content from litellm as details

  #929 merged Aug 22, 2025

• Add Filipino Benchmark Community Tasks

  #852 merged Aug 22, 2025

• Test for the xet hub team

  #928 merged Aug 22, 2025

• fix tasks list

  #906 merged Aug 22, 2025

• Fix #878: load tasks before models

  #931 merged Aug 22, 2025

• Trackio when available

  #930 merged Aug 20, 2025

• Upgrade to datasets 4.0

  #924 merged Aug 20, 2025

• fix context size check in sglang model

  #787 merged Aug 20, 2025

• Update translation_literals.py

  #923 merged Aug 20, 2025

3 Pull requests opened by 2 people

• reduces cli args redundancy

  #932 opened Aug 21, 2025

• Update docstring where needed

  #935 opened Aug 22, 2025

• Add IFBench

  #937 opened Aug 26, 2025

10 Issues closed by 3 people

• [BUG] Python API Documentation Error

  #861 closed Aug 26, 2025

• [FT] Allow k in pass@k and maj@k to be customisable at the command line

  #870 closed Aug 25, 2025

• [BUG] `tasks list` is not a complete list

  #903 closed Aug 23, 2025

• [FT] Save reasoning_content of responses as details

  #927 closed Aug 22, 2025

• [FT] Fail early when a task is not defined

  #878 closed Aug 22, 2025

• [FT] Relax upper bound on `datasets` dependency

  #910 closed Aug 21, 2025

• Expose `stop_sequence` at command line

  #196 closed Aug 20, 2025

• [FT] Numpy 2.0 support

  #416 closed Aug 20, 2025

• [FT] Store system prompt in results

  #879 closed Aug 20, 2025

• [FT] Cache partial outputs to enable re-run from interruption

  #880 closed Aug 20, 2025

1 Issue opened by 1 person

• [BUG] VLLM Code Generate hangs

  #933 opened Aug 22, 2025

12 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• Add CodeElo

  #575 commented on Aug 26, 2025 • 2 new comments

• add additional_messages parameters to forward OpenAi style messages to inference providers

  #859 commented on Aug 22, 2025 • 1 new comment

• MMLU Redux

  #883 commented on Aug 25, 2025 • 1 new comment

• Add a way to easily enable/disable thinking mode for reasoning models

  #855 commented on Aug 22, 2025 • 0 new comments

• [EVAL] Add IFBench

  #908 commented on Aug 25, 2025 • 0 new comments

• [BUG] `override_batch_size` disappeared from `PipelineParameters`, but there is no deprecation notice & it is still in the doc

  #857 commented on Aug 26, 2025 • 0 new comments

• feat: add JGLUE tasks

  #469 commented on Aug 25, 2025 • 0 new comments

• Fix attribute and parameter names in loggers

  #476 commented on Aug 26, 2025 • 0 new comments

• added vllm lora support.

  #611 commented on Aug 22, 2025 • 0 new comments

• fix: add offline dataset support via LOCAL_DATASET_PATH

  #807 commented on Aug 22, 2025 • 0 new comments

• Added fallback for incomplete configs for vlm models launched as llms

  #828 commented on Aug 26, 2025 • 0 new comments

• Save and output number of samples of each task

  #851 commented on Aug 22, 2025 • 0 new comments