Backfilling ignores task-level start_date and end_date

[SDubrulle]

Apache Airflow version:

2.1.0

OS:

Debian GNU/Linux 10

Apache Airflow Provider versions:

Irrelevant for this issue.

Deployment:

K8S using a custom helm chart

What happened:

When backfilling a DAG with a --mark-success flag, all tasks (within specified timerange) are set to success.

What you expected to happen:

I expected the task-level start_date and end_date to be taken into account. As such, a task-instance would only be created if the dag execution date is within the task date range.

How to reproduce it:

DAG:

from datetime import datetime
from airflow.decorators import dag, task

@dag(
    default_args={},
    description="Start date test DAG",
    schedule_interval="0 0 * * *",
    start_date=datetime(2021, 8, 1),
    catchup=True,
)
def start_date_test_dag():

    @task()
    def since_dag_start_date():
        print("Done")

    @task(start_date=datetime(2021, 8, 10))
    def task_defined_start_date():
        print("Done")

    [since_dag_start_date(), task_defined_start_date()]

dag = start_date_test_dag()

Backfill command:

airflow dags backfill -s 2020-08-01 -e 2021-8-18 --mark-success start_date_test_dag

---

Are you willing to submit a PR?

Sure, if this is unwanted behaviour and there is no workaround (other than backfilling each task individually).

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[eladkal]

It make sense that backfill should respect the task definition for tasks start_date and end_date.

Giving another example if we have a dag as:

with DAG(
    'my_dag',
    default_args=default_args,
    schedule_interval='@daily',
    start_date=datetime(2019, 1, 1),
) as dag:
    t1 = BashOperator(
        task_id='print_date',
        bash_command='date',
    )
      t2 = BashOperator(
        task_id='print_date2',
        bash_command='date',
        start_date=datetime(2021, 1, 1)
    )

and you are running airflow dags backfill my_dag --start-date 2019-01-01 --end-date 2019-06-01
I wouldn't expect print_date2 to be a part of this backfill at all. To my perspective the backfill start-date & stop-date specify the range of runs that needs to be created - in these runs it doesn't make sense to consider tasks that are not part of this window.

If someone has a different take on that I would love to hear the reasons

[uranusjr]

I believe this difference originates in the dependency definitions for (manually-triggered) backfills and scheduler-triggered runs. In dependencies_deps.py, there's a ExecDateAfterStartDateDep in SCHEDULER_QUEUED_DEPS that would skips a task if its start_date is later than the run's logical date (execution_date), but that dependency is not declared in BACKFILL_QUEUED_DEPS.

This difference dates way back to #5079 when BACKFILL_QUEUED_DEPS was introduced, and unfortunately no rationales were provided, from what I can tell. Maybe some of the people involved at the time could remember something?

/cc @kaxil @ashb

[uranusjr]

@dstandish one thing to note when reimplementing backfilling with the scheduler.

[dstandish]

@dstandish one thing to note when reimplementing backfilling with the scheduler.

thanks @uranusjr .

This seems to be desirable behavior, no?

[uranusjr]

Do you mean ignoring is desirable, or respecting? Airflow currently has both, depending on whether the run is triggered by the airflow dags backfill CLI or the scheduler (via catchup=True).