Actions

This repo is forked from pyTooling/Actions: Reusable steps and workflows for GitHub Actions and modified for use at Zeit Medical.

Reusable steps and workflows for GitHub Actions, focused on Python packages.

Development

This project uses commitizen for conventional commits and follows Semantic Versioning.

The only dependencies are commitizen and pre-commit for development. Run make setup to get started.

Repos that use Actions defined here must reference a release tag. To release a new version, create a tag and bump the version with commitizen. The recommended workflow is:

# Make changes and commit them:
git cz commit # follow prompt and answer "Y" to "Breaking change?"

# Alternatively, create a commit message and add "BREAKING CHANGE" as the
# commit message footer.

cz bump --check-consistency --changelog

There is a Github Action that runs this automatically, so all you really need to do is create new releases (with gh release create or through the UI).

Actions are in github-actions/.github/workflows/.

Use these Actions

These actions are meant to be used for CI in other Zeit repos. The ExamplePipeline.yaml has a sample pipeline implementing all Actions. See docs below for implementation details:

bumpversion

Context

GitHub Actions supports five procedures to reuse code:

• JavaScript Action:
  • docs.github.com: actions/creating-actions/creating-a-javascript-action
• Container Action:
  • docs.github.com: actions/creating-actions/creating-a-docker-container-action
• Container Step:
  • docs.github.com: actions/learn-github-actions/workflow-syntax-for-github-actions#example-using-a-docker-public-registry-action
  • docs.github.com: actions/learn-github-actions/workflow-syntax-for-github-actions#jobsjob_idstepswithargs
• Composite Action:
  • docs.github.com: actions/creating-actions/creating-a-composite-action
  • github.blog/changelog: 2020-08-07-github-actions-composite-run-steps
  • github.blog/changelog: 2021-08-25-github-actions-reduce-duplication-with-action-compositio
• Reusable Workflow:
  • docs.github.com: actions/learn-github-actions/reusing-workflows
  • github.blog/changelog: 2021-10-05-github-actions-dry-your-github-actions-configuration-by-reusing-workflows

Container Actions and Container Steps are almost equivalent: Actions use a configuration file (action.yml), while Steps do not. Leaving JavaScript and Container Actions and Steps aside, the main differences between Composite Actions and Reusable Workflows are the following:

• Composite Actions can be executed from a remote/external path or from the checked out branch, and from any location. However, Reusable Workflows can only be used through a remote/external path ({owner}/{repo}/{path}/{filename}@{ref}), where {path} must be .github/workflows, and @{ref} is required. See actions/runner#1493. As a result:
  • Local Composite Actions cannot be used without a prior repo checkout, but Reusable Workflows can be used without checkout.
  • Testing development versions of local Reusable Workflows is cumbersome, because PRs do not pick the modifications by default.
• Composite Actions can include multiple steps, but not multiple jobs. Conversely, Reusable Workflows can include multiple jobs, and multiple steps in each job.
• Composite Actions can include multiple files, so it's possible to use files from the Action or from the user's repository. Conversely, Reusable Workflows are a single YAML file, with no additional files retrieved by default.

Callable vs dispatchable workflows

Reusable Workflows are defined through the workflow_call event kind. Similarly, any "regular" Workflow can be triggered through a workflow_dispatch event. Both event kinds support input options, which are usable within the Workflow. Therefore, one might intuitively try to write a workflow which is both callable and dispatchable. In other words, which can be either reused from another workflow, or triggered through the API. Unfortunately, that is not the case. Although input options can be duplicated for both events, GitHub's backend exposes them through different objects. In dispatchable Workflows, the object is ${{ github.event.inputs }}, while callable workflows receive ${{ inputs }}.

As a result, in order to make a reusable workflow dispatchable, a wrapper workflow is required. See, for instance, hdl/containers: .github/workflows/common.yml and hdl/containers: .github/workflows/dispatch.yml. Alternatively, a normalisation job might be used, similar to the Parameters in this repo.

Call hierarchy

Reusable Workflows cannot call other Reusable Workflows, however, they can use Composite Actions and Composite Actions can call other Actions. Therefore, in some use cases it is sensible to combine one layer of reusable workflows for orchestrating the jobs, along with multiple layers of composite actions.

Script with post step

JavaScript Actions support defining pre, pre-if, post and post-if steps, which allow executing steps at the beginning or the end of a job, regardless of intermediate steps failing. Unfortunately, those are not available for any other Action type.

Action with-post-step is a generic JS Action to execute a main command and to set a command as a post step. It allows using the post feature with scripts written in bash, python or any other interpreted language available on the environment. See: actions/runner#1478.

Reusable workflows

This repository provides 10+ Reusable Workflows based on the CI pipelines of the repos in this organisation, EDA², VHDL, and others. By combining them, Python packages can be continuously tested and released along with Sphinx documentation sites, to GitHub Releases, GitHub Pages and PyPI. Optionally, coverage and static type check reports can be gathered.

As shown in the screenshots above, the expected order is:

• Global:
  • Parameters: a workaround for the limitations to handle global variables in GitHub Actions workflows (see actions/runner#480). It generates outputs with artifact names and job matrices to be used in later running jobs.
• Code testing/analysis:

  • UnitTesting: run unit test with pytest using multiple versions of Python, and optionally upload results as XML reports. Configuration options to pytest should be given via section [tool.pytest.ini_options] in a pyproject.toml file.

  • CoverageCollection: collect code coverage data (incl. branch coverage) with pytest/pytest-cov/coverage.py using a single version of Python (latest). It generates HTML and Cobertura (XML)reports, upload the HTML report as an artifact, and upload the test results to Codecov and Codacy. Configuration options to pytest and coverage.py should be given via section [tool.pytest.ini_options] and [tool.coverage.*] in a pyproject.toml file.

  • StaticTypeCheck: collect static type check result with mypy, and optionally upload results as an HTML report. Example commands:

    1. Regular package

    commands: mypy --html-report htmlmypy -p ToolName

    2. Parent namespace package

    commands: |
      touch Parent/__init__.py
      mypy --html-report htmlmypy -p ToolName

    3. Child namespace package

    commands: |
      cd Parent
      mypy --html-report ../htmlmypy -p ToolName

  • VerifyDocs: extract code examples from the README and test these code snippets.

• Packaging and releasing:
  • Release: publish GitHub Release.
  • Package: generate source and wheel packages, and upload them as an artifact.
  • PublishOnPyPI: publish source and wheel packages to PyPI.
  • PublishTestResults: publish unit test results through GH action dorny/test-reporter.
• Documentation:
  • BuildTheDocs: build Sphinx documentation with BuildTheDocs, and upload HTML as an artifact.
  • PublishToGitHubPages: publish HTML documentation to GitHub Pages.
• Cleanup:
  • ArtifactCleanUp: delete artifacts.

Example pipeline

ExamplePipeline.yml is an example Workflow which uses all of the Reusable Workflows. Python package/tool developers can copy it into their repos, in order to use al the reusable workflows straightaway. Minimal required modifications are the following:

• Set the name input of job Parameters.
• Specify the commands input of job StaticTypeCheck.

Find further usage cases in the following list of projects:

• edaa-org/pyEDAA.ProjectModel
• edaa-org/pySVModel
• VHDL/pyVHDLModel

References

• hdl/containers#48

Contributors

• Gerrit Egnew

License

This Python package (source code) licensed under Apache License 2.0. The accompanying documentation is licensed under Creative Commons - Attribution 4.0 (CC-BY 4.0).

---

SPDX-License-Identifier: Apache-2.0