🎉 Source HubSpot: support list memberships in Contacts stream (airbyt…

…ehq#9129) * Source Hubspot airbytehq#8477 - separate stream for contacts list memberships * Source Hubspot airbytehq#8477 - reformat * Source Hubspot airbytehq#8477 - fix basic read tests: add contacts_list_memberships to empty streams * Source Hubspot airbytehq#8477 - bump version and update documentation
soufianeodf · Jan 10, 2022 · 7c4ae13 · 7c4ae13
1 parent 44cb30a
commit 7c4ae13
Show file tree

Hide file tree

Showing 11 changed files with 111 additions and 30 deletions.
diff --git a/...ain/resources/config/STANDARD_SOURCE_DEFINITION/36c891d9-4bd9-43ac-bad2-10e12756272c.json b/...ain/resources/config/STANDARD_SOURCE_DEFINITION/36c891d9-4bd9-43ac-bad2-10e12756272c.json
@@ -2,7 +2,7 @@
   "sourceDefinitionId": "36c891d9-4bd9-43ac-bad2-10e12756272c",
   "name": "HubSpot",
   "dockerRepository": "airbyte/source-hubspot",
-  "dockerImageTag": "0.1.29",
+  "dockerImageTag": "0.1.30",
   "documentationUrl": "https://docs.airbyte.io/integrations/sources/hubspot",
   "icon": "hubspot.svg"
 }
diff --git a/airbyte-config/init/src/main/resources/seed/source_definitions.yaml b/airbyte-config/init/src/main/resources/seed/source_definitions.yaml
@@ -287,7 +287,7 @@
 - name: HubSpot
   sourceDefinitionId: 36c891d9-4bd9-43ac-bad2-10e12756272c
   dockerRepository: airbyte/source-hubspot
-  dockerImageTag: 0.1.29
+  dockerImageTag: 0.1.30
   documentationUrl: https://docs.airbyte.io/integrations/sources/hubspot
   icon: hubspot.svg
   sourceType: api

diff --git a/airbyte-config/init/src/main/resources/seed/source_specs.yaml b/airbyte-config/init/src/main/resources/seed/source_specs.yaml
@@ -2846,7 +2846,7 @@
               path_in_connector_config:
               - "credentials"
               - "client_secret"
-- dockerImage: "airbyte/source-hubspot:0.1.29"
+- dockerImage: "airbyte/source-hubspot:0.1.30"
   spec:
     documentationUrl: "https://docs.airbyte.io/integrations/sources/hubspot"
     connectionSpecification:

diff --git a/airbyte-integrations/connectors/source-hubspot/Dockerfile b/airbyte-integrations/connectors/source-hubspot/Dockerfile
@@ -34,5 +34,5 @@ COPY source_hubspot ./source_hubspot
 ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
 
-LABEL io.airbyte.version=0.1.29
+LABEL io.airbyte.version=0.1.30
 LABEL io.airbyte.name=airbyte/source-hubspot
diff --git a/airbyte-integrations/connectors/source-hubspot/acceptance-test-config.yml b/airbyte-integrations/connectors/source-hubspot/acceptance-test-config.yml
@@ -25,7 +25,7 @@ tests:
       # incremental streams: subscription_changes and email_events (it takes a long time to read)
       # and therefore the start date is set at 2021-10-10 for `config_oauth.json`,
       # but the campaign was created on 2021-01-11
-      empty_streams: ["campaigns", "workflows"]
+      empty_streams: ["campaigns", "workflows", "contacts_list_memberships"]
   incremental:
     - config_path: "secrets/config.json"
       configured_catalog_path: "sample_files/configured_catalog.json"

diff --git a/...egrations/connectors/source-hubspot/sample_files/configured_catalog_for_oauth_config.json b/...egrations/connectors/source-hubspot/sample_files/configured_catalog_for_oauth_config.json
@@ -36,6 +36,15 @@
       "sync_mode": "full_refresh",
       "destination_sync_mode": "overwrite"
     },
+    {
+      "stream": {
+        "name": "contacts_list_memberships",
+        "json_schema": {},
+        "supported_sync_modes": ["full_refresh"]
+      },
+      "sync_mode": "full_refresh",
+      "destination_sync_mode": "overwrite"
+    },
     {
       "stream": {
         "name": "deal_pipelines",

diff --git a/airbyte-integrations/connectors/source-hubspot/sample_files/full_refresh_catalog.json b/airbyte-integrations/connectors/source-hubspot/sample_files/full_refresh_catalog.json
@@ -36,6 +36,15 @@
       "sync_mode": "full_refresh",
       "destination_sync_mode": "overwrite"
     },
+    {
+      "stream": {
+        "name": "contacts_list_memberships",
+        "json_schema": {},
+        "supported_sync_modes": ["full_refresh"]
+      },
+      "sync_mode": "full_refresh",
+      "destination_sync_mode": "overwrite"
+    },
     {
       "stream": {
         "name": "deal_pipelines",

diff --git a/airbyte-integrations/connectors/source-hubspot/source_hubspot/api.py b/airbyte-integrations/connectors/source-hubspot/source_hubspot/api.py
@@ -598,6 +598,40 @@ class ContactListStream(IncrementalStream):
     need_chunk = False
 
 
+class ContactsListMembershipsStream(Stream):
+    """Contacts list Memberships, API v1
+    The Stream was created due to issue #8477, where supporting List Memberships in Contacts stream was requested.
+    According to the issue this feature is supported in API v1 by setting parameter showListMemberships=true
+    in get all contacts endpoint. API will return list memberships for each contact record.
+    But for syncing Contacts API v3 is used, where list memberships for contacts isn't supported.
+    Therefore, new stream was created based on get all contacts endpoint of API V1.
+    Docs: https://legacydocs.hubspot.com/docs/methods/contacts/get_contacts
+    """
+
+    url = "/contacts/v1/lists/all/contacts/all"
+    updated_at_field = "timestamp"
+    more_key = "has-more"
+    data_field = "contacts"
+    page_filter = "vidOffset"
+    page_field = "vid-offset"
+
+    def _transform(self, records: Iterable) -> Iterable:
+        """Extracting list membership records from contacts
+        According to documentation Contacts may have multiple vids,
+        but the canonical-vid will be the primary ID for a record.
+        Docs: https://legacydocs.hubspot.com/docs/methods/contacts/contacts-overview
+        """
+        for record in super()._transform(records):
+            canonical_vid = record.get("canonical-vid")
+            for item in record.get("list-memberships", []):
+                yield {"canonical-vid": canonical_vid, **item}
+
+    def list(self, fields) -> Iterable:
+        """Receiving all contacts with list memberships"""
+        params = {"showListMemberships": True}
+        yield from self.read(partial(self._api.get, url=self.url), params)
+
+
 class DealStageHistoryStream(Stream):
     """Deal stage history, API v1
     Deal stage history is exposed by the v1 API, but not the v3 API.

diff --git a/airbyte-integrations/connectors/source-hubspot/source_hubspot/client.py b/airbyte-integrations/connectors/source-hubspot/source_hubspot/client.py
@@ -12,6 +12,7 @@
     API,
     CampaignStream,
     ContactListStream,
+    ContactsListMembershipsStream,
     CRMObjectIncrementalStream,
     DealPipelineStream,
     DealStream,
@@ -38,6 +39,7 @@ def __init__(self, start_date, credentials, **kwargs):
             "companies": CRMObjectIncrementalStream(entity="company", associations=["contacts"], **common_params),
             "contact_lists": ContactListStream(**common_params),
             "contacts": CRMObjectIncrementalStream(entity="contact", **common_params),
+            "contacts_list_memberships": ContactsListMembershipsStream(**common_params),
             "deal_pipelines": DealPipelineStream(**common_params),
             "deals": DealStream(associations=["contacts"], **common_params),
             "email_events": EmailEventStream(**common_params),

diff --git a/...egrations/connectors/source-hubspot/source_hubspot/schemas/contacts_list_memberships.json b/...egrations/connectors/source-hubspot/source_hubspot/schemas/contacts_list_memberships.json
@@ -0,0 +1,24 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "type": "object",
+  "properties": {
+    "canonical-vid": {
+      "type": ["null", "integer"]
+    },
+    "static-list-id": {
+      "type": ["null", "integer"]
+    },
+    "internal-list-id": {
+      "type": ["null", "integer"]
+    },
+    "timestamp": {
+      "type": ["null", "integer"]
+    },
+    "vid": {
+      "type": ["null", "integer"]
+    },
+    "is-member": {
+      "type": ["null", "boolean"]
+    }
+  }
+}
diff --git a/docs/integrations/sources/hubspot.md b/docs/integrations/sources/hubspot.md
@@ -21,6 +21,7 @@ This source is capable of syncing the following tables and their data:
 * [Companies](https://developers.hubspot.com/docs/api/crm/companies) \(Incremental\)
 * [Contact Lists](http://developers.hubspot.com/docs/methods/lists/get_lists) \(Incremental\)
 * [Contacts](https://developers.hubspot.com/docs/methods/contacts/get_contacts) \(Incremental\)
+* [Contacts list memberships](https://legacydocs.hubspot.com/docs/methods/contacts/get_contacts)
 * [Deal Pipelines](https://developers.hubspot.com/docs/methods/pipelines/get_pipelines_for_object_type)
 * [Deals](https://developers.hubspot.com/docs/api/crm/deals) \(including Contact associations\) \(Incremental\)
 * [Email Events](https://developers.hubspot.com/docs/methods/email/get_events) \(Incremental\)
@@ -91,6 +92,7 @@ If you are using Oauth, most of the streams require the appropriate [scopes](htt
 | `companies` | `contacts` |
 | `contact_lists` | `contacts` |
 | `contacts` | `contacts` |
+| `contacts_list_memberships` | `contacts` |
 | `deal_pipelines` | either the `contacts` scope \(to fetch deals pipelines\) or the `tickets` scope. |
 | `deals` | `contacts` |
 | `email_events` | `content` |
@@ -106,29 +108,30 @@ If you are using Oauth, most of the streams require the appropriate [scopes](htt
 
 ## Changelog
 
-| Version | Date | Pull Request | Subject |
-| :--- | :--- | :--- | :--- |
-| 0.1.29 | 2021-12-17 | [8699](https://github.com/airbytehq/airbyte/pull/8699) | Add incremental sync support for `companies`, `contact_lists`, `contacts`, `deals`, `line_items`, `products`, `quotes`, `tickets` streams |
-| 0.1.28 | 2021-12-15 | [8429](https://github.com/airbytehq/airbyte/pull/8429) | Update fields and descriptions |
-| 0.1.27 | 2021-12-09 | [8658](https://github.com/airbytehq/airbyte/pull/8658) | Fixed config backward compatibility issue by allowing additional properties in the spec |
-| 0.1.26 | 2021-11-30 | [8329](https://github.com/airbytehq/airbyte/pull/8329) | Removed 'skip_dynamic_fields' config param |
-| 0.1.25 | 2021-11-23 | [8216](https://github.com/airbytehq/airbyte/pull/8216) | Add skip dynamic fields for testing only |
-| 0.1.24 | 2021-11-09 | [7683](https://github.com/airbytehq/airbyte/pull/7683) | Fix name issue 'Hubspot' -> 'HubSpot' |
-| 0.1.23 | 2021-11-08 | [7730](https://github.com/airbytehq/airbyte/pull/7730) | Fix oAuth flow schema|
-| 0.1.22 | 2021-11-03 | [7562](https://github.com/airbytehq/airbyte/pull/7562) | Migrate Hubspot source to CDK structure |
-| 0.1.21 | 2021-10-27 | [7405](https://github.com/airbytehq/airbyte/pull/7405) | Change of package `import` from `urllib` to `urllib.parse` |
-| 0.1.20 | 2021-10-26 | [7393](https://github.com/airbytehq/airbyte/pull/7393) | Hotfix for `split_properties` function, add the length of separator symbol `,`(`%2C` in HTTP format) to the checking of the summary URL length |
-| 0.1.19 | 2021-10-26 | [6954](https://github.com/airbytehq/airbyte/pull/6954) | Fix issue with getting `414` HTTP error for streams |
-| 0.1.18 | 2021-10-18 | [5840](https://github.com/airbytehq/airbyte/pull/5840) | Add new marketing emails (with statistics) stream |
-| 0.1.17 | 2021-10-14 | [6995](https://github.com/airbytehq/airbyte/pull/6995) | Update `discover` method: disable `quotes` stream when using OAuth config  |
-| 0.1.16 | 2021-09-27 | [6465](https://github.com/airbytehq/airbyte/pull/6465) | Implement OAuth support. Use CDK authenticator instead of connector specific authenticator |
-| 0.1.15 | 2021-09-23 | [6374](https://github.com/airbytehq/airbyte/pull/6374) | Use correct schema for `owners` stream |
-| 0.1.14 | 2021-09-08 | [5693](https://github.com/airbytehq/airbyte/pull/5693) | Include deal\_to\_contact association when pulling deal stream and include contact ID in contact stream |
-| 0.1.13 | 2021-09-08 | [5834](https://github.com/airbytehq/airbyte/pull/5834) | Fixed array fields without items property in schema |
-| 0.1.12 | 2021-09-02 | [5798](https://github.com/airbytehq/airbyte/pull/5798) | Treat empty string values as None for field with format to fix normalization errors |
-| 0.1.11 | 2021-08-26 | [5685](https://github.com/airbytehq/airbyte/pull/5685) | Remove all date-time format from schemas |
-| 0.1.10 | 2021-08-17 | [5463](https://github.com/airbytehq/airbyte/pull/5463) | Fix fail on reading stream using `API Key` without required permissions |
-| 0.1.9 | 2021-08-11 | [5334](https://github.com/airbytehq/airbyte/pull/5334) | Fix empty strings inside float datatype |
-| 0.1.8 | 2021-08-06 | [5250](https://github.com/airbytehq/airbyte/pull/5250) | Fix issue with printing exceptions |
-| 0.1.7 | 2021-07-27 | [4913](https://github.com/airbytehq/airbyte/pull/4913) | Update fields schema |
+| Version | Date       | Pull Request | Subject                                                                                                                                        |
+|:--------|:-----------| :--- |:-----------------------------------------------------------------------------------------------------------------------------------------------|
+| 0.1.30  | 2021-01-10 | [9129](https://github.com/airbytehq/airbyte/pull/9129) | Created Contacts list memberships streams                                                                                                      |
+| 0.1.29  | 2021-12-17 | [8699](https://github.com/airbytehq/airbyte/pull/8699) | Add incremental sync support for `companies`, `contact_lists`, `contacts`, `deals`, `line_items`, `products`, `quotes`, `tickets` streams      |
+| 0.1.28  | 2021-12-15 | [8429](https://github.com/airbytehq/airbyte/pull/8429) | Update fields and descriptions                                                                                                                 |
+| 0.1.27  | 2021-12-09 | [8658](https://github.com/airbytehq/airbyte/pull/8658) | Fixed config backward compatibility issue by allowing additional properties in the spec                                                        |
+| 0.1.26  | 2021-11-30 | [8329](https://github.com/airbytehq/airbyte/pull/8329) | Removed 'skip_dynamic_fields' config param                                                                                                     |
+| 0.1.25  | 2021-11-23 | [8216](https://github.com/airbytehq/airbyte/pull/8216) | Add skip dynamic fields for testing only                                                                                                       |
+| 0.1.24  | 2021-11-09 | [7683](https://github.com/airbytehq/airbyte/pull/7683) | Fix name issue 'Hubspot' -> 'HubSpot'                                                                                                          |
+| 0.1.23  | 2021-11-08 | [7730](https://github.com/airbytehq/airbyte/pull/7730) | Fix oAuth flow schema                                                                                                                          |
+| 0.1.22  | 2021-11-03 | [7562](https://github.com/airbytehq/airbyte/pull/7562) | Migrate Hubspot source to CDK structure                                                                                                        |
+| 0.1.21  | 2021-10-27 | [7405](https://github.com/airbytehq/airbyte/pull/7405) | Change of package `import` from `urllib` to `urllib.parse`                                                                                     |
+| 0.1.20  | 2021-10-26 | [7393](https://github.com/airbytehq/airbyte/pull/7393) | Hotfix for `split_properties` function, add the length of separator symbol `,`(`%2C` in HTTP format) to the checking of the summary URL length |
+| 0.1.19  | 2021-10-26 | [6954](https://github.com/airbytehq/airbyte/pull/6954) | Fix issue with getting `414` HTTP error for streams                                                                                            |
+| 0.1.18  | 2021-10-18 | [5840](https://github.com/airbytehq/airbyte/pull/5840) | Add new marketing emails (with statistics) stream                                                                                              |
+| 0.1.17  | 2021-10-14 | [6995](https://github.com/airbytehq/airbyte/pull/6995) | Update `discover` method: disable `quotes` stream when using OAuth config                                                                      |
+| 0.1.16  | 2021-09-27 | [6465](https://github.com/airbytehq/airbyte/pull/6465) | Implement OAuth support. Use CDK authenticator instead of connector specific authenticator                                                     |
+| 0.1.15  | 2021-09-23 | [6374](https://github.com/airbytehq/airbyte/pull/6374) | Use correct schema for `owners` stream                                                                                                         |
+| 0.1.14  | 2021-09-08 | [5693](https://github.com/airbytehq/airbyte/pull/5693) | Include deal\_to\_contact association when pulling deal stream and include contact ID in contact stream                                        |
+| 0.1.13  | 2021-09-08 | [5834](https://github.com/airbytehq/airbyte/pull/5834) | Fixed array fields without items property in schema                                                                                            |
+| 0.1.12  | 2021-09-02 | [5798](https://github.com/airbytehq/airbyte/pull/5798) | Treat empty string values as None for field with format to fix normalization errors                                                            |
+| 0.1.11  | 2021-08-26 | [5685](https://github.com/airbytehq/airbyte/pull/5685) | Remove all date-time format from schemas                                                                                                       |
+| 0.1.10  | 2021-08-17 | [5463](https://github.com/airbytehq/airbyte/pull/5463) | Fix fail on reading stream using `API Key` without required permissions                                                                        |
+| 0.1.9   | 2021-08-11 | [5334](https://github.com/airbytehq/airbyte/pull/5334) | Fix empty strings inside float datatype                                                                                                        |
+| 0.1.8   | 2021-08-06 | [5250](https://github.com/airbytehq/airbyte/pull/5250) | Fix issue with printing exceptions                                                                                                             |
+| 0.1.7   | 2021-07-27 | [4913](https://github.com/airbytehq/airbyte/pull/4913) | Update fields schema                                                                                                                           |