Skip to content

Commit

Permalink
[docs] - Add new Schedules concept page (DOC-193) (dagster-io#22148)
Browse files Browse the repository at this point in the history
## Summary & Motivation

This PR adds a new concept page for Schedules. The goal is for this page
to explain what a schedule is and why it's beneficial, not to walk users
through how to create/use it.

## How I Tested These Changes

eyes, local
  • Loading branch information
erinkcochran87 authored May 31, 2024
1 parent dd863fd commit 8be1dc9
Show file tree
Hide file tree
Showing 3 changed files with 110 additions and 2 deletions.
108 changes: 108 additions & 0 deletions docs/content/concepts/automation/schedules.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
---
title: "Schedules | Dagster Docs"
description: "Use schedules to run your pipelines at fixed time intervals."
---

# Schedules

Schedules are Dagster's way of supporting traditional methods of [automation](/concepts/automation), which allow you to specify when a [job](/concepts/ops-jobs-graphs/jobs) should run. Using schedules, you can define a fixed time interval to run a pipeline, such as daily, hourly, or Monday at 9:00 AM.

Each interval of a schedule is called a **tick**, which is an indication that a job should execute. Ticks kick off **runs**, which is a single instance of a job being executed.

When viewing a schedule in [Dagster's UI](/concepts/webserver/ui), you can see the schedule's definition, executing timezone, targeted jobs and partitions, as well as its tick and run history.

---

## Benefits

Using schedules helps you:

- Predictably process and deliver data to stakeholders and business-critical applications
- Consistently run data pipelines without the need for manual intervention
- Optimize resource usage by scheduling pipelines to run during off-peak hours

---

## Prerequisites

Before continuing, you should be familiar with:

- [Cron syntax](https://en.wikipedia.org/wiki/Cron)
- [Assets](/concepts/assets/software-defined-assets) and [asset jobs](/concepts/assets/asset-jobs)
- [Ops](/concepts/ops-jobs-graphs/ops) and [op jobs](/concepts/ops-jobs-graphs/op-jobs) _(optional)_

---

## How it works

Schedules run jobs at fixed time intervals and have two main components:

- **A job**, which targets a selection of assets or ops
- [**A cron expression**](https://en.wikipedia.org/wiki/Cron), which defines when the schedule runs. Simple and complex schedules are supported, allowing you to have fine-grained control over when runs are executed. With cron syntax, you can:

- **Create custom schedules** like `Every hour from 9:00AM - 5:00PM` with cron expressions (`0 9-17 * * *`)
- **Quickly create basic schedules** like `Every day at midnight` with predefined cron definitions (`@daily`, `@midnight`)

To make creating cron expressions easier, you can use an online tool like [Crontab Guru](https://crontab.guru/). This tool allows you to create and describe cron expressions in a human-readable format and test the execution dates produced by the expression. **Note**: While this tool is useful for general cron expression testing, always remember to [test your schedules](/concepts/automation/schedules/testing) in Dagster to ensure the results are as expected.

For a schedule to run, it must be turned on and an active [`dagster-daemon` process](/deployment/dagster-daemon) must be running. If you used [`dagster dev` to start the Dagster UI/webserver](/guides/running-dagster-locally), the daemon process will be automatically launched alongside the webserver.

After these criteria are met, the schedule will run at the interval specified in the cron expression. **Schedules will execute in UTC by default**, but [you can specify a custom timezone](/concepts/automation/schedules/customizing-executing-timezones).

---

## Getting started

Check out these guides to get started with schedules:

<div className="w-full inline-flex flex-col space-y-2 md:space-y-0 md:flex-row md:space-x-4">
<Button link="/concepts/automation/schedules/automating-assets-schedules-jobs">
Assets & schedules
</Button>
<Button link="/concepts/automation/schedules/automating-ops-schedules-jobs">
Ops & schedules
</Button>
<Button link="/concepts/automation/schedules/examples" style="secondary">
Examples
</Button>
</div>

From here, you can:

- Construct schedules to run [partitioned jobs](/concepts/automation/schedules/partitioned-schedules)
- Execute jobs in [specific timezones](/concepts/automation/schedules/customizing-executing-timezones)
- Learn to [test your schedules](/concepts/automation/schedules/testing)
- Identify and resolve common issues with our [troubleshooting guide](/concepts/automation/schedules/troubleshooting)

### Limitations and notes

- Dagster supports all [predefined cron definitions](https://en.wikipedia.org/wiki/Cron#Nonstandard_predefined_scheduling_definitions) with the exeception of `@reboot`
- Schedules will execute in UTC [unless a timezone is specified](/concepts/automation/schedules/customizing-executing-timezones)
- When defining a schedule's execution time, keep [Daylight Savings Time (DST) in mind](/concepts/automation/schedules/customizing-executing-timezones#execution-times-and-daylight-savings-time)

---

## Related

<ArticleList>
<ArticleListItem
title="Automation"
href="/concepts/automation"
></ArticleListItem>
<ArticleListItem
title="Running Dagster locally"
href="/guides/running-dagster-locally"
></ArticleListItem>
<ArticleListItem
title="Wikipedia - Cron"
href="https://en.wikipedia.org/wiki/Cron"
></ArticleListItem>
<ArticleListItem
title="dagster-daemon"
href="/deployment/dagster-daemon"
></ArticleListItem>
<ArticleListItem
title="Crontab Guru"
href="https://crontab.guru/"
></ArticleListItem>
</ArticleList>
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ The `cron_schedule` argument accepts standard [cron expressions](https://en.wiki
To run schedules for assets, you can [build a job that materializes assets](/concepts/assets/software-defined-assets#building-jobs-that-materialize-assets) and construct a <PyObject object="ScheduleDefinition" />:

```python file=concepts/partitions_schedules_sensors/schedules/schedules.py startafter=start_basic_asset_schedule endbefore=end_basic_asset_schedule
from dagster import AssetSelection, define_asset_job
from dagster import AssetSelection, ScheduleDefinition, define_asset_job

asset_job = define_asset_job("asset_job", AssetSelection.groups("some_asset_group"))

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ def my_asset():


# start_basic_asset_schedule
from dagster import AssetSelection, define_asset_job
from dagster import AssetSelection, ScheduleDefinition, define_asset_job

asset_job = define_asset_job("asset_job", AssetSelection.groups("some_asset_group"))

Expand Down

0 comments on commit 8be1dc9

Please sign in to comment.