Skip to content

Commit

Permalink
Merge pull request mlcommons#517 from sgpyc/patch-1
Browse files Browse the repository at this point in the history
[LLM] Add exception for using different language in eval dataset
  • Loading branch information
erichan1 authored Apr 13, 2023
2 parents bd2287b + e6ee49f commit 909c70c
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions training_rules.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -442,6 +442,8 @@ CLOSED: The same quality measure as the reference implementation must be used. T

OPEN: An arbitrary stopping criteria may be used, including but not limited to the closed quality measure, a different quality measure, the number of epochs, or a fixed time. However, the reported results must include the geometric mean of the final quality as measured by the closed quality measure.

Exceptions for GPT3 OPEN: the open submissions are allowed to choose a language version that’s not English for the C4 dataset. When doing so, the submitter needs to make it clear that the dataset and convergence measures are different from the close division submissions.

Check points can be created at the discretion of submitter. No check points are required to be produced or retained.

=== Equivalence exceptions
Expand Down

0 comments on commit 909c70c

Please sign in to comment.