diff --git a/docs/content/dagster-cloud/deployment/agents/running-multiple-agents.mdx b/docs/content/dagster-cloud/deployment/agents/running-multiple-agents.mdx index dc5c47a43d107..70afd41727489 100644 --- a/docs/content/dagster-cloud/deployment/agents/running-multiple-agents.mdx +++ b/docs/content/dagster-cloud/deployment/agents/running-multiple-agents.mdx @@ -9,7 +9,7 @@ platform_type: "cloud" Each Dagster Cloud full deployment (e.g., `prod`) needs to have at least one agent running. A single agent is adequate for many use cases, but you may want to run multiple agents to provide redundancy if a single agent goes down. -It's recommended to only use multiple agents of the same type (e.g. multiple Kubernetes agents). Submitting runs to a specific agent isn't currently supported. +It's recommended to only use multiple agents of the same type (e.g. multiple Kubernetes agents). --- @@ -68,10 +68,6 @@ If using the CloudFormation template provided by Dagster, the number of replicas If you want to run multiple agents in an environment where each agent can not access the others' resources (for example, multiple kubernetes namespaces or different clusters), you need to enable the `isolated_agents` option. This is supported for all agent types. - - It is not currently possible to control which agent a request goes to. - - ### In Docker Add the following to the `dagster.yaml` file: @@ -109,3 +105,57 @@ dagsterCloud: ### In Amazon ECS The `isolated_agents` option can be set as per-deployment configuration on the `dagster.yaml` file used by your agent. See the [ECS configuration reference](/dagster-cloud/deployment/agents/amazon-ecs/configuration-reference#per-deployment-configuration) guide for more information. + +## Routing requests to specific agents + +Every Dagster Cloud agent serves requests from one or more queues. By default, requests for each code location are placed on a default queue and your agent reads requests only from that default queue. This default configuration is sufficient for most use cases. + +In some cases, you might want to route requests for certain code locations to specific agents. For example, you might want to route requests for one code location to an agent running in an on-premise data center but requests for all other code locations to an agent running in AWS. To route requests for a code locations to a specific agent, you can annotate the code locations with the name of a custom queue and configure an agent to serve only requests for that queue. + +When configuring your code location, set an agent queue: + +```yaml +# dagster_cloud.yaml + +locations: + - location_name: data-eng-pipeline + code_source: + package_name: quickstart_etl + executable_path: venvs/path/to/dataengineering_spark_team/bin/python + agent_queue: special-queue +``` + +Next, configure an agent to handle your agent queue. + +### In Docker + +Add the following to the `dagster.yaml` file: + +```yaml +agent_queues: + include_default_queue: True # Continue to handle requests for code locations that aren't annotated with a specific queue + additional_queues: + - special-queue +``` + +### In Kubernetes + +Add the following options to your Helm command: + +```shell +helm upgrade \ + ... + --set dagsterCloud.agentQueues.additionalQueues={"special-queue"} +``` + +Or if you're using a `values.yaml` file: + +```yaml +dagsterCloued + agentQueues: + # Continue to handle requests for code locations that aren't + # assigned to a specific agent queue + includeDefaultQueue: true + additionalQueues: + - special-queue +```