Basic implementation of partial scoring. #3047

meisterT · 2025-08-03T12:19:37Z

This follows the legacy spec in:
https://icpc.io/problem-package-format/spec/legacy.html#graders

We might decided at a later point whether we want to support this or just the new (currently draft) spec, but basic constructs of the change should be the same.

Not yet implemented:

any form of testing
anything on the team interface
shadowing

Part of #2518

This follows the legacy spec in: https://icpc.io/problem-package-format/spec/legacy.html#graders We might decided at a later point whether we want to support this or just the new (currently draft) spec, but basic constructs of the change should be the same. Not yet implemented: - any form of testing - anything on the team interface - shadowing Part of DOMjudge#2518

meisterT · 2025-08-05T05:55:44Z

Some notes:

there's a to-do already, but we need to talk about internal precision and precision in presentation
we should add to the jury submission page some info that allows following the result calculation with test case groups (should be easy to do with the existing function)
the number of tries on a scoring scoreboard should really be the number of total tries (not until the first correct attempt like for pass-fail)
it seems we are not handling the accept_if_any_accepted case always correctly, needs more testing
while we can ingest results from a primary CCS now, we don't compare the scores yet for shadow differences
the layout of the shadow differences table needs to be fixed
optimization problems are by their very nature less deterministic as they incentivize teams to do stuff like try this approach for that amount of time, then this other approach for that amount of time and generally the use of RNG is much more prevalent

nickygerritsen

Initial round of review done. Of course there are the todos and comments from PHPStan / PHPCS

nickygerritsen · 2025-08-17T08:19:50Z

webapp/src/Entity/Problem.php

@@ -211,6 +211,11 @@ class Problem extends BaseApiEntity implements
    #[Serializer\Exclude]
    private Collection $languages;

+    #[ORM\ManyToOne(inversedBy: 'problems')]


I'm not seeing this reverse relation? And should it be manytoone? When does a testcasegroup have many problems?

And for me calling it "parent testcase group" sounds weird, maybe "root testcase group"?

I did push the change for the reverse relation.

nickygerritsen · 2025-08-17T08:20:56Z

webapp/src/Service/ExternalContestSourceService.php

@@ -1855,6 +1859,10 @@ protected function importRun(Event $event, EventData $data): void
            ->setRuntime($runTime)
            ->setResult($judgementTypeId === null ? null : $verdictsFlipped[$judgementTypeId]);

+        if (isset($data->score)) {
+            $run->setScore($data->score);


This comment from PHPStan is not lying 😛

nickygerritsen · 2025-08-17T08:24:54Z

webapp/src/Service/ScoreboardService.php

@@ -259,6 +260,8 @@ public function calculateScoreRow(

        $contestStartTime = $contest->getStarttime();

+        $pointsJury      = "0";


Why not $scoreJury? We have points already for multipoint problems, so this might be confusing?

nickygerritsen · 2025-08-17T08:26:15Z

webapp/src/Service/SubmissionService.php

+    /**
+     * Returns a two-element array:
+     *   - The score for the testcase group, or null if not all results are ready.
+     *   - The result for the testcase group, or null if not all results are ready.
+     * @return array{string|null, string|null}
+     */


I would return a string indexed array to make this more clear:

Suggested change

/**

* Returns a two-element array:

* - The score for the testcase group, or null if not all results are ready.

* - The result for the testcase group, or null if not all results are ready.

* @return array{string|null, string|null}

*/

/**

* Returns a two-element array:

* - The score for the testcase group, or null if not all results are ready.

* - The result for the testcase group, or null if not all results are ready.

* @return array{score: string|null, result: string|null}

*/

…the scoreboard cache

nickygerritsen · 2025-08-17T13:42:28Z

Some notes:

the number of tries on a scoring scoreboard should really be the number of total tries (not until the first correct attempt like for pass-fail)

I changed this to show the total number of submissions. I wonder if this is what we want? Say you have 5 submissions:

AC, score = 50
AC, score = 75
AC, score = 100
WA
WA

Do we want 3 or 5 tries? And what if the fourth one is AC, score = 100?

Also please check my logic. Basically I do not stop processing submissions after the first correct one for scoring problems (and have changed the FTS logic since this was dependent on this assumption).

the layout of the shadow differences table needs to be fixed

Fixed

I also fixed some other things:

I added the reverse relation for problem->testcaseGroup.
We now correctly import (and export) the scoreboard type from contest.yaml/json (but for this I have renamed the SCORING enum case to SCORE as that is what the spec says).
I fixed the single team scoreboard row, it didn't show the scores.
I added the score to the team submission list and detail modal.
I fixed an issue where if we calculated the scorecache for a submission without judgement, it would crash.
I now check the scoreboard type when starting up the shadow script.
I fixed some logic for importing an event with no items, as that is what we had in the replay for the NAC challenge.
I also fixed importing the ZIPs from that challenge that contained folders.

meisterT force-pushed the partial_scoring branch from 3821281 to 5c9d385 Compare August 3, 2025 17:34

meisterT added 3 commits August 4, 2025 21:51

Basic import of score while shadowing. No display or comparison yet.

bebe929

Base scoring scoreboard in shadow mode on external scores.

414b06e

Display more external results.

4ba5683

nickygerritsen reviewed Aug 17, 2025

View reviewed changes

nickygerritsen added 10 commits August 17, 2025 14:08

Add reverse relation for problem parent testcase group

b7c1310

Import/export contest scoreboard type from/to YAML

be6b593

Fix score on single team scoreboard

899583b

Show all tries for scoring scoreboards, not only the first correct one

dcf9cb9

Add score to team submission list and detail page

7ac3835

It can be the case that we don't have a judging yet when calculating …

6c794ee

…the scoreboard cache

Check scoreboard type when shadowing

35c7e53

Do not assume we have one event if we have exactly 0

62f28bd

Do not import directories when importing shadow submissions

8f1b0e9

Fix shadow differences submissions table for scoring contests

97ae23a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Basic implementation of partial scoring. #3047

Basic implementation of partial scoring. #3047

Uh oh!

meisterT commented Aug 3, 2025

Uh oh!

meisterT commented Aug 5, 2025

Uh oh!

nickygerritsen left a comment

Uh oh!

nickygerritsen Aug 17, 2025

Uh oh!

nickygerritsen Aug 17, 2025

Uh oh!

nickygerritsen Aug 17, 2025

Uh oh!

nickygerritsen Aug 17, 2025

Uh oh!

nickygerritsen Aug 17, 2025

Uh oh!

nickygerritsen Aug 17, 2025

Uh oh!

nickygerritsen commented Aug 17, 2025

Uh oh!

Uh oh!

		@@ -259,6 +260,8 @@ public function calculateScoreRow(

		$contestStartTime = $contest->getStarttime();

		$pointsJury = "0";

Basic implementation of partial scoring. #3047

Are you sure you want to change the base?

Basic implementation of partial scoring. #3047

Uh oh!

Conversation

meisterT commented Aug 3, 2025

Uh oh!

meisterT commented Aug 5, 2025

Uh oh!

nickygerritsen left a comment

Choose a reason for hiding this comment

Uh oh!

nickygerritsen Aug 17, 2025

Choose a reason for hiding this comment

Uh oh!

nickygerritsen Aug 17, 2025

Choose a reason for hiding this comment

Uh oh!

nickygerritsen Aug 17, 2025

Choose a reason for hiding this comment

Uh oh!

nickygerritsen Aug 17, 2025

Choose a reason for hiding this comment

Uh oh!

nickygerritsen Aug 17, 2025

Choose a reason for hiding this comment

Uh oh!

nickygerritsen Aug 17, 2025

Choose a reason for hiding this comment

Uh oh!

nickygerritsen commented Aug 17, 2025

Uh oh!

Uh oh!