Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Feat/expectations #307

Draft
wants to merge 27 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
3e5b21d
Partial implementation of expectations proposal 0.4.0
thorehusfeldt Sep 8, 2023
d7165c6
Minimum Viable Implementation
thorehusfeldt Sep 10, 2023
5f7a2bd
Correct semantics of required (some, not all, must exist)
thorehusfeldt Sep 10, 2023
a357011
Change semantics of "accepted" (only "AC" allowed, but nothing required)
thorehusfeldt Sep 10, 2023
ea889a8
is_allowed_verdict: change argument order, give default path
thorehusfeldt Sep 10, 2023
b8bc3dd
bt run: check for required testcase results
thorehusfeldt Sep 10, 2023
c10108e
Add problem/spanishinquisition for testing expectations
thorehusfeldt Sep 10, 2023
c7bbef2
introduce is_allowed_verdict_for_testcase
thorehusfeldt Sep 11, 2023
0d84659
Use bar for logging errors
thorehusfeldt Sep 11, 2023
509461d
spanishinquisition: more example submissions
thorehusfeldt Sep 11, 2023
d4eb2de
Allow regex matching of patterns
thorehusfeldt Sep 11, 2023
9989ebc
Regex match examples in spanishinquisition
thorehusfeldt Sep 11, 2023
2e5861c
Notation change: allowed -> permitted
thorehusfeldt Sep 11, 2023
1dc0e49
Draft documentation
thorehusfeldt Sep 12, 2023
86ebef2
fix markdown syntax for list
thorehusfeldt Sep 12, 2023
1cccbd8
Rename some files in spanishinquisition
thorehusfeldt Sep 12, 2023
3b85390
Verbose verdicts; also hack the progressbar
thorehusfeldt Sep 12, 2023
372e4c1
bt run: more informative logging of violated expectations
thorehusfeldt Sep 12, 2023
4edfd33
refactor, rename some methods in expectations
thorehusfeldt Sep 12, 2023
6c3d945
mixed submission to spanishinquisition
thorehusfeldt Sep 12, 2023
8b3ddae
Refactor Registry again (again)
thorehusfeldt Sep 14, 2023
b68afec
Refactor into BaseExpecations
thorehusfeldt Sep 15, 2023
c783540
short_verdicts
thorehusfeldt Sep 15, 2023
aa6ddd7
bt run uses refactored expectations to detect missing requirements
thorehusfeldt Sep 15, 2023
f6f2bcc
Smoother methods for Registry
thorehusfeldt Sep 16, 2023
db01d9d
Fix doctest
thorehusfeldt Sep 16, 2023
b4d4df9
Document Registry.__setitem__(); better __repr__
thorehusfeldt Sep 17, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
introduce is_allowed_verdict_for_testcase
  • Loading branch information
thorehusfeldt committed Sep 11, 2023
commit c7bbef255c108a2ef4e2ed47223bc1c2f3692e20
14 changes: 11 additions & 3 deletions bin/expectations.py
Original file line number Diff line number Diff line change
Expand Up @@ -176,11 +176,19 @@ def parse_expectations(pattern, expectations):

parse_expectations("", expectations)

def is_allowed_verdict(self, verdict: str, path=""):
def allowed_verdicts_for_testcase(self, path) -> dict[str, str]:
"""Returns a dictionary over the patterns that apply for the given test case path"""
return {
pattern: verdicts
for pattern, verdicts in self._allowed_verdicts.items()
if path.startswith(pattern)
}

def is_allowed_verdict(self, verdict: str, path):
"""Is the result allowed for the testcase at the given path?"""
verdict = shortform(verdict)
for pattern, allowed in self._allowed_verdicts.items():
if path.startswith(pattern) and verdict not in allowed:
for _, verdicts in self.allowed_verdicts_for_testcase(path).items():
if verdict not in verdicts:
return False
return True

Expand Down