Manage pools in Helm Chart

[mik-laj]

Hello,

Many users need to deployment Airlfow and initialize instances with basic resources. It would be helpful if Helm Chart would facilitate this task.

For connections and variables, all we need to do is add documentation that describes the use of environment variables.

For pools, we will have to add a new script that initializes the database using the CLI.
https://airflow.readthedocs.io/en/latest/cli-and-env-variables-ref.html#pools

Similar tasks can be done with Terraform provider as well, but it's still WIP and not everyone uses Terraform. We still need better support for this use case in the Helm Chart.

Best regards,
Kamil Breguła

[FloChehab]

Hello, giving this issue a big 👍

This feature (creating pools) is already available in another chart, so if someone needs inspirations for this: https://github.com/helm/charts/blob/master/stable/airflow/templates/scheduler/scheduler-deployment.yaml#L250

[OmairK]

Hi @mik-laj, is this issue up for taking?

[mik-laj]

@OmairK Assigned.

[FloChehab]

Hello @OmairK did you have time to start looking at this issue or would you need an extra hand ?

[OmairK]

Hello @FloChehab, I wont be able to work on this issue right now due to some academic commitments. So please go ahead and unassign me from this issue.

[mik-laj]

@FloChehab Assigned. ;-)

[FloChehab]

Hello @FloChehab, I wont be able to work on this issue right now due to some academic commitments. So please go ahead and unassign me from this issue.

No worries @OmairK , good luck with your commitments !

For connections and variables, all we need to do is add documentation that describes the use of environment variables.

@mik-laj Just to make sure, for now we stick with the Readme in the chart folder? Or is something else expected ?

[mik-laj]

in my opinion it is enough to describe the creation of environment variables. Do you have a different idea for a solution?

[FloChehab]

in my opinion it is enough to describe the creation of environment variables. Do you have a different idea for a solution?

No, that what's I was thinking. Just wanted to double check.

[ashb]

Connections and Variables can now come form secrets manager, which is better placed to handle this than baking them in to the chart.

Pools make sense.

[FloChehab]

Hello, just wanted to let you now that I should have time to work on this issue in couple of weeks.

[thesuperzapper]

I want to highlight that the "user-community" chart (this is separate from the "official") supports this.

See "How to create airflow pools?" for documentation.

[csp33]

I'll try to do this one