Dagster Helm

TL;DR

Installing from remote

helm repo add dagster https://dagster-io.github.io/helm

helm install my-release dagster/dagster \
    --namespace dagster \
    --create-namespace

Installing from source

helm install my-release . \
    --namespace dagster \
    --create-namespace

Installing a specific Chart version

The Dagster Python version and Helm Chart version have a 1:1 correspondence, and ideally should only be used together when the version numbers match.

On installation, we can pass the --version flag if a specific Chart version is required.

export VERSION=<DAGSTER_VERSION>
helm install my-release . \
    --namespace dagster \
    --create-namespace \
    --version $VERSION

Introduction

Dagster is a Python library for building data applications. This chart will bootstrap a Dagit web server deployment on a Kubernetes cluster using the Helm package manager.

In addition, our helm chart allows for Dagster configuration such as:

• Deploying user code containers separately from Dagster system components
• Specifying the Dagster run launcher
• Specifying the Dagster scheduler to handle recurring pipeline runs

Prerequisites

Setup a Kubernetes Cluster

Please refer to the Kubernetes getting started guide to set up and run a Kubernetes cluster.

Installing Helm

We use Helm to manage our Kubernetes application in a configurable, replicable, and sharable way. Helm can be installed by following the Helm installation guide.

Adding the Dagster Repo

To download and install the Dagster charts, use the following command:

helm repo add dagster https://dagster-io.github.io/helm

We can check the most recent version of the Dagster chart in our chart repository by searching through it:

helm search repo dagster

To use new releases of the Dagster Helm chart, we'll need to update our chart repository cache.

helm repo update

Using Helm

We can now use the Helm client to install the Dagster chart! Refer to the guide on Using Helm for an explanation of useful Helm concepts.

Here, we install a release of the Dagster chart named my-release, under the Kubernetes namespace dagster:

helm install my-release dagster/dagster \
    --namespace dagster \
    --create-namespace

Customizing the Release

Settings can configured to customize the Dagster release.

To see the full list of configurable settings, check out the values.yaml file. Documentation can be found in the comments.

Tutorial

For a introductory guide on using Dagster on Helm, check out our documentation.

Contributing

We cover instructions to get started with developing on our chart.

JSON Schema

JSON Schema can impose a structure on our Dagster chart's values to ensure requirement checks, type validation, range validation, and constraint validation. The Dagster chart's JSON Schema is generated through a Pydantic model of our values.

# Install the cli to generate the JSON Schema
pip install -e ./schema

# Display the resulting schema from the Dagster chart values Pydantic model
dagster-helm schema show

# Update the existing schema
dagster-helm schema apply

Template Testing

We use pytest to assert behaviors about our Helm chart.

Helm values are modelled using Pydantic, and then piped through to helm template to generate the chart's Kubernetes manifests. Kubernetes manifests are then transformed into their pythonic object representations, and assertions are made about these objects to ensure that our templating logic is correct.