Skip to content

Commit

Permalink
Document agent queues (dagster-io#19114)
Browse files Browse the repository at this point in the history
  • Loading branch information
jmsanders authored Jan 10, 2024
1 parent 2ff7aaa commit 4c9450a
Showing 1 changed file with 55 additions and 5 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ platform_type: "cloud"

Each Dagster Cloud full deployment (e.g., `prod`) needs to have at least one agent running. A single agent is adequate for many use cases, but you may want to run multiple agents to provide redundancy if a single agent goes down.

It's recommended to only use multiple agents of the same type (e.g. multiple Kubernetes agents). Submitting runs to a specific agent isn't currently supported.
It's recommended to only use multiple agents of the same type (e.g. multiple Kubernetes agents).

---

Expand Down Expand Up @@ -68,10 +68,6 @@ If using the CloudFormation template provided by Dagster, the number of replicas

If you want to run multiple agents in an environment where each agent can not access the others' resources (for example, multiple kubernetes namespaces or different clusters), you need to enable the `isolated_agents` option. This is supported for all agent types.

<Note>
It is not currently possible to control which agent a request goes to.
</Note>

### In Docker

Add the following to the `dagster.yaml` file:
Expand Down Expand Up @@ -109,3 +105,57 @@ dagsterCloud:
### In Amazon ECS

The `isolated_agents` option can be set as per-deployment configuration on the `dagster.yaml` file used by your agent. See the [ECS configuration reference](/dagster-cloud/deployment/agents/amazon-ecs/configuration-reference#per-deployment-configuration) guide for more information.

## Routing requests to specific agents

Every Dagster Cloud agent serves requests from one or more queues. By default, requests for each code location are placed on a default queue and your agent reads requests only from that default queue. This default configuration is sufficient for most use cases.

In some cases, you might want to route requests for certain code locations to specific agents. For example, you might want to route requests for one code location to an agent running in an on-premise data center but requests for all other code locations to an agent running in AWS. To route requests for a code locations to a specific agent, you can annotate the code locations with the name of a custom queue and configure an agent to serve only requests for that queue.

When configuring your code location, set an agent queue:

```yaml
# dagster_cloud.yaml
locations:
- location_name: data-eng-pipeline
code_source:
package_name: quickstart_etl
executable_path: venvs/path/to/dataengineering_spark_team/bin/python
agent_queue: special-queue
```

Next, configure an agent to handle your agent queue.

### In Docker

Add the following to the `dagster.yaml` file:

```yaml
agent_queues:
include_default_queue: True # Continue to handle requests for code locations that aren't annotated with a specific queue
additional_queues:
- special-queue
```

### In Kubernetes

Add the following options to your Helm command:

```shell
helm upgrade \
...
--set dagsterCloud.agentQueues.additionalQueues={"special-queue"}
```

Or if you're using a `values.yaml` file:

```yaml
dagsterCloued
agentQueues:
# Continue to handle requests for code locations that aren't
# assigned to a specific agent queue
includeDefaultQueue: true
additionalQueues:
- special-queue
```

0 comments on commit 4c9450a

Please sign in to comment.