🐛 Source Github: add incremental for WorkflowRuns (airbytehq#13115)

* add incremental for `workflowruns`

Signed-off-by: Sergey Chvalyuk <[email protected]>

airbyte-config/init/src/main/resources/seed/source_definitions.yaml

@@ -295,7 +295,7 @@
 - name: GitHub
   sourceDefinitionId: ef69ef6e-aa7f-4af1-a01d-ef775033524e
   dockerRepository: airbyte/source-github
-  dockerImageTag: 0.2.30
+  dockerImageTag: 0.2.31
   documentationUrl: https://docs.airbyte.io/integrations/sources/github
   icon: github.svg
   sourceType: api

airbyte-config/init/src/main/resources/seed/source_specs.yaml

@@ -2493,7 +2493,7 @@
     supportsNormalization: false
     supportsDBT: false
     supported_destination_sync_modes: []
-- dockerImage: "airbyte/source-github:0.2.30"
+- dockerImage: "airbyte/source-github:0.2.31"
   spec:
     documentationUrl: "https://docs.airbyte.com/integrations/sources/github"
     connectionSpecification:

airbyte-integrations/connectors/source-github/Dockerfile

@@ -12,5 +12,5 @@ RUN pip install .
 ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
 
-LABEL io.airbyte.version=0.2.30
+LABEL io.airbyte.version=0.2.31
 LABEL io.airbyte.name=airbyte/source-github

airbyte-integrations/connectors/source-github/acceptance-test-config.yml

@@ -44,6 +44,7 @@ tests:
         review_comments: ["airbytehq/integration-test", "updated_at"]
         reviews: ["airbytehq/integration-test", "pull_request_updated_at"]
         stargazers: ["airbytehq/integration-test", "starred_at"]
+        workflow_runs: ["airbytehq/integration-test", "updated_at"]
         workflows: ["airbytehq/integration-test", "updated_at"]
   full_refresh:
     - config_path: "secrets/config.json"

airbyte-integrations/connectors/source-github/integration_tests/abnormal_state.json

@@ -133,5 +133,10 @@
     "airbytehq/integration-test": {
       "updated_at": "2121-12-31T23:59:59Z"
     }
+  },
+  "workflow_runs": {
+    "airbytehq/integration-test": {
+      "updated_at": "2121-12-31T23:59:59Z"
+    }
   }
 }

airbyte-integrations/connectors/source-github/integration_tests/configured_catalog.json

@@ -375,10 +375,10 @@
       "stream": {
         "name": "workflow_runs",
         "json_schema": {},
-        "supported_sync_modes": ["full_refresh"],
+        "supported_sync_modes": ["full_refresh", "incremental"],
         "source_defined_primary_key": [["id"]]
       },
-      "sync_mode": "full_refresh",
+      "sync_mode": "incremental",
       "destination_sync_mode": "overwrite"
     },
     {

airbyte-integrations/connectors/source-github/source_github/streams.py

@@ -14,6 +14,8 @@
 from airbyte_cdk.sources.streams.http import HttpStream, HttpSubStream
 from requests.exceptions import HTTPError
 
+from .utils import getter
+
 DEFAULT_PAGE_SIZE = 100
 
 
@@ -202,6 +204,8 @@ def __slice_key(self):
             return "repository"
         return "organization"
 
+    record_slice_key = __slice_key
+
     def convert_cursor_value(self, value):
         return value
 
@@ -216,7 +220,7 @@ def get_updated_state(self, current_stream_state: MutableMapping[str, Any], late
         Return the latest state by comparing the cursor value in the latest record with the stream's most recent state
         object and returning an updated state object.
         """
-        slice_value = latest_record[self.__slice_key]
+        slice_value = getter(latest_record, self.record_slice_key)
         updated_state = self.convert_cursor_value(latest_record[self.cursor_field])
         stream_state_value = current_stream_state.get(slice_value, {}).get(self.cursor_field)
         if stream_state_value:
@@ -1051,12 +1055,15 @@ def convert_cursor_value(self, value):
         return pendulum.parse(value).in_tz(tz="UTC").format("YYYY-MM-DDTHH:mm:ss[Z]")
 
 
-class WorkflowRuns(GithubStream):
+class WorkflowRuns(SemiIncrementalMixin, GithubStream):
     """
     Get all workflows of a GitHub repository
     API documentation: https://docs.github.com/en/rest/reference/actions#list-workflow-runs-for-a-repository
     """
 
+    # key for accessing slice value from record
+    record_slice_key = ["repository", "full_name"]
+
     def path(self, stream_slice: Mapping[str, Any] = None, **kwargs) -> str:
         return f"repos/{stream_slice['repository']}/actions/runs"
 

airbyte-integrations/connectors/source-github/source_github/utils.py

@@ -0,0 +1,11 @@
+#
+# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
+#
+
+
+def getter(D: dict, key_or_keys):
+    if not isinstance(key_or_keys, list):
+        key_or_keys = [key_or_keys]
+    for k in key_or_keys:
+        D = D[k]
+    return D

docs/integrations/sources/github.md

@@ -11,52 +11,52 @@ This connector outputs the following full refresh streams:
 * [Assignees](https://docs.github.com/en/rest/reference/issues#list-assignees)
 * [Branches](https://docs.github.com/en/rest/reference/repos#list-branches)
 * [Collaborators](https://docs.github.com/en/rest/reference/repos#list-repository-collaborators)
-* [Commit comment reactions](https://docs.github.com/en/rest/reference/reactions#list-reactions-for-a-commit-comment)
-* [Issue comment reactions](https://docs.github.com/en/rest/reference/reactions#list-reactions-for-an-issue-comment)
 * [Issue labels](https://docs.github.com/en/free-pro-team@latest/rest/reference/issues#list-labels-for-a-repository)
-* [Issue reactions](https://docs.github.com/en/rest/reference/reactions#list-reactions-for-an-issue)
 * [Organizations](https://docs.github.com/en/rest/reference/orgs#get-an-organization)
-* [Pull request comment reactions](https://docs.github.com/en/rest/reference/reactions#list-reactions-for-a-pull-request-review-comment)
 * [Pull request commits](https://docs.github.com/en/rest/reference/pulls#list-commits-on-a-pull-request)
-* [Repositories](https://docs.github.com/en/rest/reference/repos#list-organization-repositories)
 * [Tags](https://docs.github.com/en/rest/reference/repos#list-repository-tags)
 * [TeamMembers](https://docs.github.com/en/rest/teams/members#list-team-members)
 * [TeamMemberships](https://docs.github.com/en/rest/reference/teams#get-team-membership-for-a-user)
 * [Teams](https://docs.github.com/en/rest/reference/teams#list-teams)
 * [Users](https://docs.github.com/en/rest/reference/orgs#list-organization-members)
-* [Workflows](https://docs.github.com/en/rest/reference/actions#workflows)
-* [WorkflowRuns](https://docs.github.com/en/rest/reference/actions#list-workflow-runs-for-a-repository)
 
 This connector outputs the following incremental streams:
 
 * [Comments](https://docs.github.com/en/rest/reference/issues#list-issue-comments-for-a-repository)
+* [Commit comment reactions](https://docs.github.com/en/rest/reference/reactions#list-reactions-for-a-commit-comment)
 * [Commit comments](https://docs.github.com/en/rest/reference/repos#list-commit-comments-for-a-repository)
 * [Commits](https://docs.github.com/en/rest/reference/repos#list-commits)
 * [Deployments](https://docs.github.com/en/rest/reference/deployments#list-deployments)
 * [Events](https://docs.github.com/en/rest/reference/activity#list-repository-events)
+* [Issue comment reactions](https://docs.github.com/en/rest/reference/reactions#list-reactions-for-an-issue-comment)
 * [Issue events](https://docs.github.com/en/rest/reference/issues#list-issue-events-for-a-repository)
 * [Issue milestones](https://docs.github.com/en/rest/reference/issues#list-milestones)
+* [Issue reactions](https://docs.github.com/en/rest/reference/reactions#list-reactions-for-an-issue)
 * [Issues](https://docs.github.com/en/rest/reference/issues#list-repository-issues)
 * [Project cards](https://docs.github.com/en/rest/reference/projects#list-project-cards)
 * [Project columns](https://docs.github.com/en/rest/reference/projects#list-project-columns)
 * [Projects](https://docs.github.com/en/rest/reference/projects#list-repository-projects)
+* [Pull request comment reactions](https://docs.github.com/en/rest/reference/reactions#list-reactions-for-a-pull-request-review-comment)
 * [Pull request stats](https://docs.github.com/en/rest/reference/pulls#get-a-pull-request)
 * [Pull requests](https://docs.github.com/en/rest/reference/pulls#list-pull-requests)
 * [Releases](https://docs.github.com/en/rest/reference/repos#list-releases)
+* [Repositories](https://docs.github.com/en/rest/reference/repos#list-organization-repositories)
 * [Review comments](https://docs.github.com/en/rest/reference/pulls#list-review-comments-in-a-repository)
 * [Reviews](https://docs.github.com/en/rest/reference/pulls#list-reviews-for-a-pull-request)
 * [Stargazers](https://docs.github.com/en/rest/reference/activity#list-stargazers)
+* [WorkflowRuns](https://docs.github.com/en/rest/reference/actions#list-workflow-runs-for-a-repository)
+* [Workflows](https://docs.github.com/en/rest/reference/actions#workflows)
 
 ### Notes
 
-1. Only 4 streams \(`comments`, `commits`, `issues` and `review comments`\) from the above 17 incremental streams are pure incremental meaning that they:
+1. Only 4 streams \(`comments`, `commits`, `issues` and `review comments`\) from the above 24 incremental streams are pure incremental meaning that they:
    * read only new records;
    * output only new records.
 
-2. Other 13 incremental streams are also incremental but with one difference, they:
+2. Other 20 incremental streams are also incremental but with one difference, they:
    * read all records;
    * output only new records.
-   Please, consider this behaviour when using those 13 incremental streams because it may affect you API call limits.
+   Please, consider this behaviour when using those 20 incremental streams because it may affect you API call limits.
 
 3. We are passing few parameters \(`since`, `sort` and `direction`\) to GitHub in order to filter records and sometimes for large streams specifying very distant `start_date` in the past may result in keep on getting error from GitHub instead of records \(respective `WARN` log message will be outputted\). In this case Specifying more recent `start_date` may help.
 **The "Start Date" configuration option does not apply to the streams below, because the Github API does not include dates which can be used for filtering:**
@@ -113,6 +113,7 @@ Your token should have at least the `repo` scope. Depending on which streams you
 
 | Version | Date       | Pull Request | Subject                                                                                                      |
 |:--------|:-----------| :--- |:-------------------------------------------------------------------------------------------------------------|
+| 0.2.31  | 2022-05-24 | [13115](https://github.com/airbytehq/airbyte/pull/13115) | Add incremental support for streams `WorkflowRuns`                                                         |
 | 0.2.30  | 2022-05-09 | [12294](https://github.com/airbytehq/airbyte/pull/12294) | Add incremental support for streams `CommitCommentReactions`, `IssueCommentReactions`, `IssueReactions`, `PullRequestCommentReactions`, `Repositories`, `Workflows` |
 | 0.2.29  | 2022-05-04 | [\#12482](https://github.com/airbytehq/airbyte/pull/12482) | Update input configuration copy |
 | 0.2.28  | 2022-04-21 | [11893](https://github.com/airbytehq/airbyte/pull/11893) | Add new streams `TeamMembers`, `TeamMemberships`                                                           |