-
Notifications
You must be signed in to change notification settings - Fork 101
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add reporting of unit tests results (#5426)
This change adds reporting of unit tests to our PRs using an external github action. The action will create a test run summary as well as annoations (errors and warnings) to failing unit tests. See an example of this in action here: #5422 There are three parts to this PR: - Producing the jUnit XML file format (gotestsum) - Working around bug in file/line number reporting - Processing and uploading results The main that needs commentary is the integration with gotestsum. This is a tool I've been using for a while now because it provides a better local experience for understanding test failures. This PR integrates it as an OPTIONAL dependency. Our various `make test` commands know how to use gotestsum if its present and fall back to `go test` otherwise. gotestsum is compatible (mostly) with `go test` so the only change in behavior is the reporting. However, there's a bug where the test report is missing file/line number. The GH action I'm using doesn't understand how to parse the output of `go test` and retrieve this info, but the file format we're using (jUnit XML) supports it. I wrote a basic script to address this. Potentially this could be fixed inside gotestsum. I looked at a few different github actions and this one seemed like it has the best features and is actively maintained. In general they all work with the jUnit format so we're not getting locked in.
- Loading branch information
Showing
6 changed files
with
177 additions
and
14 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| name: "Process Test Results" | ||
| description: | | ||
| Processes a batch of test results including creating a github artifact and annotating the Pull Request (if applicable). | ||
| This requires using 'gotestum' to run tests, which our makefile can do automatically. Install 'gotestsum' using: | ||
| go install gotest.tools/[email protected] | ||
| You will also need to set the path for the results file using an environment variable. This would output the jUnit | ||
| test results format which is what we require. | ||
| GOTESTSUM_OPTS: '--junitfile ./dist/unit_test_results_raw.xml' | ||
| Then running 'make <test target>' will do the right thing :) | ||
| inputs: | ||
| test_group_name: | ||
| description: 'Name to use for reporting (eg: Unit Tests)' | ||
| required: true | ||
| artifact_name: | ||
| description: 'Name to use for uploading artifacts (eg: unit_test_results)' | ||
| required: true | ||
| result_directory: | ||
| description: 'Directory containing result XML files. These should be in jUnit format. See the description of the action.' | ||
| required: true | ||
| runs: | ||
| using: "composite" | ||
| steps: | ||
| # The test results file output by gotestsum is missing file and line number on the XML elements | ||
| # which is needed for the annotations to work. This script adds the missing information. | ||
| - name: 'Transform ${{ inputs.test_group_name }} Results' | ||
| # Always is REQUIRED here. Otherwise, the action will be skipped when the unit tests fail, which | ||
| # defeats the purpose. YES it is counterintuitive. This applies to all of the actions in this file. | ||
| if: always() | ||
| id: 'process_files' | ||
| shell: 'bash' | ||
| working-directory: ${{ github.workspace }} | ||
| env: | ||
| INPUT_DIRECTORY: ${{ inputs.result_directory }} | ||
| run: | | ||
| echo "repository root is $GITHUB_WORKSPACE" | ||
| INPUT_FILES="$INPUT_DIRECTORY*.xml" | ||
| mkdir -p "$INPUT_DIRECTORY/processed" | ||
| for INPUT_FILE in $INPUT_FILES | ||
| do | ||
| DIRECTORY=$(dirname -- "$INPUT_FILE") | ||
| FILENAME=$(basename -- "$INPUT_FILE") | ||
| FILENAME="${FILENAME%.*}" | ||
| OUTPUT_FILE="${DIRECTORY}/processed/${FILENAME}.xml" | ||
| echo "processing test results in $INPUT_FILE to add line and file info..." | ||
| python3 ./.github/scripts/transform_test_results.py $GITHUB_WORKSPACE "$INPUT_FILE" "$OUTPUT_FILE" | ||
| echo "wrote ${OUTPUT_FILE}" | ||
| done | ||
| - name: 'Create ${{ inputs.test_group_name }} Result Report' | ||
| uses: EnricoMi/publish-unit-test-result-action@v2 | ||
| if: always() | ||
| with: | ||
| files: | | ||
| ${{ inputs.result_directory }}/processed/*.xml | ||
| - name: 'Upload ${{ inputs.test_group_name }} Results' | ||
| uses: actions/upload-artifact@v3 | ||
| if: always() | ||
| with: | ||
| name: ${{ inputs.artifact_name }} | ||
| path: | | ||
| ${{ inputs.result_directory }}/*.xml | ||
| ${{ inputs.result_directory }}processed/*.xml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| # parse an xml file and transform it into a junit xml file | ||
| # that can be used by the github actions junit reporter | ||
| # Path: .github/scripts/transform-test-results.py | ||
|
|
||
| import re | ||
| import sys | ||
| import xml.etree.ElementTree | ||
|
|
||
| def main(): | ||
| if len(sys.argv) != 4: | ||
| print("Usage: transform-test-results.py <repository root> <input file> <output file>") | ||
| sys.exit(1) | ||
|
|
||
| repository_root = sys.argv[1] | ||
| input_file = sys.argv[2] | ||
| output_file = sys.argv[3] | ||
|
|
||
| print(f"Processing {input_file}") | ||
| pattern = re.compile(r"\tError Trace:\t(.*):(\d+)") | ||
| et = xml.etree.ElementTree.parse(input_file) | ||
| for testcase in et.findall('./testsuite/testcase'): | ||
| failure = testcase.find('./failure') | ||
| if failure is None: | ||
| continue | ||
|
|
||
| # Extract file name by matching regex pattern in the text | ||
| # it will look like \tError Trace:\tfilename:line | ||
| match = pattern.search(failure.text) | ||
| if match is None: | ||
| continue | ||
|
|
||
| file = match.group(1) | ||
| line = match.group(2) | ||
|
|
||
| # The filename will contain the fully-qualified path, and we need to turn that into | ||
| # a relative path from the repository root | ||
| if not file.startswith(repository_root): | ||
| print(f"Could not find repository name in file path: {file}") | ||
| continue | ||
|
|
||
| file = file[len(repository_root) + 1:] | ||
|
|
||
| testcase.attrib["file"] = file | ||
| testcase.attrib["line"] = line | ||
| failure.attrib["file"] = file | ||
| failure.attrib["line"] = line | ||
|
|
||
|
|
||
| # Write back to file | ||
| print(f"Writing {output_file}") | ||
| et.write(output_file) | ||
|
|
||
| if __name__ == "__main__": | ||
| main() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| # ------------------------------------------------------------ | ||
| # Copyright (c) Microsoft Corporation. | ||
| # Licensed under the MIT License. | ||
| # ------------------------------------------------------------ | ||
|
|
||
| ##@ Debugging | ||
|
|
||
| .PHONY: dump | ||
| dump: ## Outputs the values of all variables in the makefile. | ||
| $(foreach v, \ | ||
| $(shell echo "$(filter-out .VARIABLES,$(.VARIABLES))" | tr ' ' '\n' | sort), \ | ||
| $(info $(shell printf "%-20s" "$(v)")= $(value $(v))) \ | ||
| ) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters