Add connector builder UI docs (airbytehq#19588)

docs/connector-development/README.md

@@ -8,7 +8,7 @@ To build a new connector in Java or Python, we provide templates so you don't ne
 
 ## Low-code Connector-Development Framework
 
-You can use the [low-code framework](config-based/low-code-cdk-overview.md) to build source connectors for REST APIs by modifying boilerplate YAML files.
+You can use the [low-code framework](config-based/low-code-cdk-overview.md) to build source connectors for REST APIs via a [connector builder UI](config-based/connector-builder-ui.md) or by modifying boilerplate YAML files.
 
 ## Python Connector-Development Kit \(CDK\)
 

docs/connector-development/config-based/connector-builder-ui.md

@@ -0,0 +1,75 @@
+# Connector Builder UI
+
+The connector builder UI provides an ergonomic iteration interface on top of the [low-code YAML format](../understanding-the-yaml-file/yaml-overview). We recommend using it to iterate on your YAML connectors. 
+
+## Getting started
+The high level flow for using the connector builder is as follows: 
+
+1. Run the Airbyte webapp to access the connector builder
+2. Use the connector builder to iterate on your YAML (either starting with a new YAML, or pasting an existing one into the editor)
+3. Export the YAML into a low-code connector module on your local machine
+4. Build the connector's Docker image
+5. Use the built connector image in Airbyte
+
+### Run an Airbyte instance
+:::info
+You need at least Airbyte v0.40.19 to run the connector builder. 
+:::
+
+:::tip
+We recommend using a dedicated Airbyte instance for the connector builder UI. This will allow you to pick up connector builder UI upgrades without impacting your production data pipelines. 
+:::
+
+The connector builder UI is bundled as part of the Airbyte webapp. To run it, follow [the instructions for deploying Airbyte locally](deploying-airbyte/local-deployment) to run Airbyte on your machine. Once you do, Airbyte should be reachable at `http://localhost:8000`.
+
+
+### Visit the connector builder
+Once your Airbyte instance has started and you've moved past the initial setup screen, visit `http://localhost:8000/connector-builder`. You will be redirected to a URL of the form `localhost:8000/workspaces/<UUID>/connector-builder` where `<UUID>` is the ID automatically generated for your workspace by Airbyte. On this page you will find the connector builder UI. It should look like this: 
+
+![The Connector Builder home page](./assets/lowcode_landing_screen.png)
+
+You can now iterate on your connector's YAML manifest. See the [components](#components) section for more information on how to use the UI to iterate on the YAML manifest. 
+
+### Exporting the YAML
+Once you're done iterating on your YAML in the UI, you'll need to export it to your local filesystem into a connector module. 
+
+If you haven't already, create a low-code connector module using the connector generator (see [this YAML tutorial for an example](./tutorial/1-create-source.)) using the name you'd like to use for your connector. For this section, let's assume our connector is called `exchange-rates`. After creating the connector, overwrite the contents of `airbyte-integrations/connectors/source-exchange-rates/source_exchange_rates/exchange_rates.yaml` with the YAML you created in the UI. 
+
+
+<!-- TODO remove this when we bundle schema specification into the YAML -->
+:::info
+Remember that you need to declare the schemas of your streams separately from the YAML manifest. 
+
+You can do this by placing one `.json` file per stream in the `schemas/` directory of your connector module. Each `.json` should be a JSONSchema describing the schema of your stream. More information about this can be found in the [static schemas section of the Python CDK](../cdk-python/schemas#static-schemas). 
+:::
+
+### Building the connector image
+Follow the instructions in the connector README to build the Docker image. Typically this will be something like `docker build . -t airbyte/source-<name>:<version>`. 
+
+Once you've built the connector image, [follow these instructions](../../integrations/custom-connectors#adding-your-connectors-in-the-ui) to add your connector to your Airbyte instance. 
+
+## Connector Builder Components
+
+The UI contains two main components: the text editor where you can edit your YAML (left) according to the [YAML lowcode specification](../understanding-the-yaml-file/yaml-overview), and the testing panel (right) where you can get feedback on how your connector works as a result of your YAML manifest.
+
+![Annotated Test Pane](./assets/lowcode_annotated_test_pane.png)
+
+1. **Input parameters panel**: Configure the input parameters to be used in testing. For example, if the connector requires an API key, use this panel to specify a JSON object containing your API key e.g: `{"api_key": "xyz"}`
+2. **Stream Picker dropdown**: Use this dropdown to choose which stream you want to test
+3. **Endpoint URL**: Displays the URL queried by the CDK to retrieve data for the current stream
+4. **Test button**: When clicked, retrieves the data for the selected stream using the stream configuration setup in the YAML file. This is the equivalent of running the `read` command on the terminal for a single stream (the selected stream). 
+5. **Records tab**: Displays the final output returned by the connector for the selected page of data in this stream
+6. **Request tab**: Displays the outgoing HTTP request made by the connector to retrieve the selected page of data. Useful for debugging. 
+7. **Response tab**: Displays the full HTTP response received by the connector for the selected page of data. Useful for debugging. 
+8. **Results view**: Displays information based on the selected tab
+9. **Page selector** Displays the selected page
+10. **Logs view**: Displays the logs emitted by the connector while running
+
+The following demo video demonstrates these components on a very simple API: 
+
+<div style={{position: "relative", "padding-bottom": "64.90384615384616%", height: 0}}>
+<iframe src="https://www.loom.com/embed/acf899938ef74dec8dd61ba012bc872f" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen style={{position: "absolute", top: 0, left: 0, width: "100%", height: "100%"}}></iframe>
+</div>
+
+## Upgrading
+The connector builder ships as part of the Airbyte platform. So if you want to get the latest and greatest updates to the builder UI, follow the process for [upgrading Airbyte](../../operator-guides/upgrading-airbyte) and you should pick up the upgrades.

docs/connector-development/config-based/low-code-cdk-overview.md

@@ -1,11 +1,29 @@
 # Low-code connector development
 
-Airbyte’s low-code framework enables you to build source connectors for REST APIs by modifying boilerplate YAML files.
+Airbyte’s low-code framework enables you to build source connectors for REST APIs via a [connector builder UI](connector-builder-ui.md) or by modifying boilerplate YAML files via terminal or text editor.
 
 :::caution
 The low-code framework is in [alpha](https://docs.airbyte.com/project-overview/product-release-stages/#alpha), which means it’s still in active development and may include backward-incompatible changes. Share feedback and requests with us on our [Slack channel](https://slack.airbyte.com/) or email us at [[email protected]](mailto:[email protected])
 :::
 
+## Why low-code? 
+
+### API Connectors are common and formulaic
+In building and maintaining hundreds of connectors at Airbyte, we've observed that whereas API source connectors constitute the overwhelming majority of connectors, they are also the most formulaic. API connector code almost always solves small variations of these problems: 
+
+1. Making requests to various endpoints under the same API URL e.g: `https://api.stripe.com/customers`, `https://api.stripe.com/transactions`, etc.. 
+2. Authenticating using a common auth strategy such as Oauth or API keys
+3. Pagination using one of the 4 ubiquitous pagination strategies: limit-offset, page-number, cursor pagination, and header link pagination
+4. Gracefully handling rate limiting by implementing exponential backoff, fixed-time backoff, or variable-time backoff
+5. Describing the schema of the data returned by the API, so that downstream warehouses can create normalized tables
+6. Decoding the format of the data returned by the API (e.g JSON, XML, CSV, etc..) and handling compression (GZIP, BZIP, etc..) 
+7. Supporting incremental data exports by remembering what data was already synced, usually using date-based cursors
+
+and so on. 
+
+### A declarative, low-code paradigm commoditizes solving formulaic problems
+Given that these problems each have a very finite number of solutions, we can remove the need for writing the code to build these API connectors by providing configurable off-the-shelf components to solve them. In doing so, we significantly decrease development effort and bugs while improving maintainability and accessibility. In this paradigm, instead of having to write the exact lines of code to solve this problem over and over, a developer can pick the solution to each problem from an available component, and rely on the framework to run the logic for them. 
+
 ## What connectors can I build using the low-code framework?
 
 Refer to the REST API documentation for the source you want to build the connector for and answer the following questions:
@@ -48,15 +66,19 @@ If the answer to all questions is yes, you can use the low-code framework to bui
 
 To use the low-code framework to build an REST API Source connector:
 
-1. Generate the API key for the source you want to build a connector for
+1. Generate the API key or credentials for the source you want to build a connector for
 2. Set up the project on your local machine
 3. Set up your local development environment
-4. Update the connector spec and config​uration
-5. Update the connector definition
-6. Test the connector
-7. Add the connector to the Airbyte platform
+4. Use the connector builder UI to define the connector YAML manifest and test the connector
+5. Specify stream schemas
+6. Add the connector to the Airbyte platform
 
 For a step-by-step tutorial, refer to the [Getting Started tutorial](./tutorial/0-getting-started.md) or the [video tutorial](https://youtu.be/i7VSL2bDvmw)
+
+## Connector Builder UI
+The main concept powering the lowcode connector framework is the Connector Manifest, a YAML file which describes the features and functionality of the connector. The structure of this YAML file is described in more detail [here](./understanding-the-yaml-file/yaml-overview). 
+
+We recommend iterating on this YAML file is via the [connector builder UI](./connector-builder-ui) as it makes it easy to inspect and debug your connector in greater detail than you would be able to through the commandline. While you can still iterate via the commandline (and the docs contain instructions for how to do it), we're investing heavily in making the UI give you iteration superpowers, so we recommend you check it out!
 
 ## Configuring the YAML file
 
@@ -65,11 +87,13 @@ The low-code framework involves editing a boilerplate YAML file. The general str
 ```
 version: "0.1.0"
 definitions:
- <key-value pairs defining objects which will be reused in the YAML connector>
+  <key-value pairs defining objects which will be reused in the YAML connector>
 streams:
- <list stream definitions>
+  <list stream definitions>
 check:
- <definition of connection checker>
+  <definition of connection checker>
+spec: 
+  <connector spec>
 ```
 
 The following table describes the components of the YAML file:
@@ -80,6 +104,7 @@ The following table describes the components of the YAML file:
 | `definitions` | Describes the objects to be reused in the YAML connector                                                                                               |
 | `streams`     | Lists the streams of the source                                                                                                                        |
 | `check`       | Describes how to test the connection to the source by trying to read a record from a specified list of streams and failing if no records could be read |
+| `spec`       | A [connector specification](../../understanding-airbyte/airbyte-protocol#actor-specification) which describes the required and optional parameters which can be input by the end user to configure this connector |
 
 :::tip
 Streams define the schema of the data to sync, as well as how to read it from the underlying API source. A stream generally corresponds to a resource within the API. They are analogous to tables for a relational database source.

docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md

@@ -4,8 +4,7 @@ We're now ready to start implementing the connector.
 
 Over the course of this tutorial, we'll be editing a few files that were generated by the code generator:
 
-- `source-exchange-rates-tutorial/source_exchange_rates_tutorial/spec.yaml`: This is the [spec file](../../connector-specification-reference.md). It describes the inputs used to configure the connector.
-- `source-exchange-rates-tutorial/source_exchange_rates_tutorial/exchange_rates_tutorial.yaml`: This is the connector definition. It describes how the data should be read from the API source.
+- `source-exchange-rates-tutorial/source_exchange_rates_tutorial/exchange_rates_tutorial.yaml`: This is the connector manifest. It describes how the data should be read from the API source, as well as what inputs can be used to configure the connector.
 - `source-exchange_rates-tutorial/integration_tests/configured_catalog.json`: This is the connector's [catalog](../../../understanding-airbyte/beginners-guide-to-catalog.md). It describes what data is available in a source
 - `source-exchange-rates-tutorial/integration_tests/sample_state.json`: This is a sample state object to be used to test [incremental syncs](../../cdk-python/incremental-stream.md).
 
@@ -17,36 +16,37 @@ We'll also be creating the following files:
 
 ## Updating the connector spec and config
 
-Let's populate the specification (`spec.yaml`) and the configuration (`secrets/config.json`) so the connector can access the access key and base currency.
+Let's populate the specification (`spec`) and the configuration (`secrets/config.json`) so the connector can access the access key and base currency.
 
-1. We'll add these properties to the connector spec in `source-exchange-rates-tutorial/source_exchange_rates_tutorial/spec.yaml`
+1. We'll add these properties to the `spec` block in the `source-exchange-rates-tutorial/source_exchange_rates_tutorial/exchange_rates_tutorial.yaml`
 
 ```yaml
-documentationUrl: https://docs.airbyte.io/integrations/sources/exchangeratesapi
-connectionSpecification:
-  $schema: http://json-schema.org/draft-07/schema#
-  title: exchangeratesapi.io Source Spec
-  type: object
-  required:
-    - access_key
-    - base
-  additionalProperties: true
-  properties:
-    access_key:
-      type: string
-      description: >-
-        Your API Access Key. See <a
-        href="https://exchangeratesapi.io/documentation/">here</a>. The key is
-        case sensitive.
-      airbyte_secret: true
-    base:
-      type: string
-      description: >-
-        ISO reference currency. See <a
-        href="https://www.ecb.europa.eu/stats/policy_and_exchange_rates/euro_reference_exchange_rates/html/index.en.html">here</a>.
-      examples:
-        - EUR
-        - USD
+spec: 
+  documentationUrl: https://docs.airbyte.io/integrations/sources/exchangeratesapi
+  connectionSpecification:
+    $schema: http://json-schema.org/draft-07/schema#
+    title: exchangeratesapi.io Source Spec
+    type: object
+    required:
+      - access_key
+      - base
+    additionalProperties: true
+    properties:
+      access_key:
+        type: string
+        description: >-
+          Your API Access Key. See <a
+          href="https://exchangeratesapi.io/documentation/">here</a>. The key is
+          case sensitive.
+        airbyte_secret: true
+      base:
+        type: string
+        description: >-
+          ISO reference currency. See <a
+          href="https://www.ecb.europa.eu/stats/policy_and_exchange_rates/euro_reference_exchange_rates/html/index.en.html">here</a>.
+        examples:
+          - EUR
+          - USD
 ```
 
 2. We also need to fill in the connection config in the `secrets/config.json`
@@ -169,6 +169,32 @@ streams:
 check:
   stream_names:
     - "rates"
+spec: 
+  documentationUrl: https://docs.airbyte.io/integrations/sources/exchangeratesapi
+  connectionSpecification:
+    $schema: http://json-schema.org/draft-07/schema#
+    title: exchangeratesapi.io Source Spec
+    type: object
+    required:
+      - access_key
+      - base
+    additionalProperties: true
+    properties:
+      access_key:
+        type: string
+        description: >-
+          Your API Access Key. See <a
+          href="https://exchangeratesapi.io/documentation/">here</a>. The key is
+          case sensitive.
+        airbyte_secret: true
+      base:
+        type: string
+        description: >-
+          ISO reference currency. See <a
+          href="https://www.ecb.europa.eu/stats/policy_and_exchange_rates/euro_reference_exchange_rates/html/index.en.html">here</a>.
+        examples:
+          - EUR
+          - USD
 ```
 
 We can now run the `check` operation, which verifies the connector can connect to the API source.