Inject config objects in elasticsearch/kibana at startup

[nouknouk]

Beforeall, many thanks for this repository ; it helped me save A LOT of time to have a basis for a proper - reproducible - initialization of the elastic stack, as well a good starting point for evolutions. My turn to give back results of my ideas & explorations.

Problem description

In my elastic journey, my goal is to be able to (re)create on demand and from scratch 'sandbox' elastic stacks, configured as close as possible as my 'production' one. It means not only having the docker images running, but also config objects auto-recreated at startup. This involves:

• injection of objects' definitions via elasticsearch API (ilm policies, component templates, data streams & index templates)
• injection of objects' definitions via kibana API (spaces, index patterns, dashboards, ...)

Ideas & suggestions

To my mind, this involves three improvements:

1. inject elasticsearch objects via its API

For each object to inject in elasticsearch I write a .request file containing one elastic API call to make, the same way it is presented in dev tools (first line for HTTP method + path ; lines after is request's body), below my-ilm-policy.request

PUT _ilm/policy/my-ilm-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "30d",
            "max_primary_shard_size": "50gb"
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    },
    "_meta": {
      "managed": true,
      "description": "my super ILM policy (rollover 30d/50GB ; delete after 90d)" 
    }
  }
}

I have several files like that (ilm policies, component templates, data streams & index templates).

Then I wrote a YAML file setup-elasticsearch.yml describing the (ordered) list of request files to process during setup (for easy opt-in/opt-out)

elasticsearch:
  - setup/elastic/ilm-policies/my-datastream-logs-api-ilm-policy.request
  - setup/elastic/component-templates/settings/my-datastream-logs-api-settings.request
  - setup/elastic/component-templates/mappings/my-base-dynamic-mapping.request
  - setup/elastic/component-templates/mappings/my-date-dynamic-mapping.request
  - setup/elastic/component-templates/mappings/my-ecs-base-mapping.request
  - setup/elastic/component-templates/mappings/my-custom-ecs-base-mapping.request
  - setup/elastic/index-templates/my-datastream-logs-api.request

in setup/lib.sh I added a basic function to parse YAML files and transform it in bunch of (prefixed) env vars (cf. https://stackoverflow.com/a/51789677 )

And at the end of entrpoint.sh I added some code to load the yaml file, iterate over the list and make appropriate curl requests.

2. inject kibana objects via its API

Same behavior as for §1 ; work in progress.

It will probably need more logic as not all APIs present an 'upsert-like' behavior (ex: spaces), and thus will need a additional 'first call to check if object already exist => then POST or PUT accordingly)

(foreseen) setup-kibana.yml:

# files to define 'spaces' in kibana instance
# note, there is a specific processing for spaces, because no upsert-like API exists  (creation=POST, update=PUT)

spaces:
    - setup/kibana/spaces/nouknouk.request

# files to define 'index-patterns' in kibana instance
index_patterns:
    - setup/kibana/index-patterns/logs-api.request
    - setup/kibana/index-patterns/my-other-index-pattern.request

3. wait whole setup succeeded before ingesting events

I don't want any log to be ingested by my stack until all has been well setup: index templates, data streams config, etc...

As my setup makes logstash being the sole shipper of data in elasticsearch, therefore I added a dependency for the start of logstash, with the complete & successfull execution of the whole 'setup' container.

in docker-compose.yml

  logstash:
    # ... / ...
    depends_on:
      elasticsearch:
        condition: service_started
      setup:
        condition: service_completed_successfully

Hope it helps.

[antoineco]

Thanks for the feedback, I'm glad this template is being useful to you! 🙌

I'm 100% onboard with the idea of having a better, more flexible setup mechanism in docker-elk. The current one, based on shell scripts, is very primitive, because the initial goal was only to be able to create the users required by the various stack components ("built-in users").

Since I didn't need any specific tool installed, I reused the elasticsearch container image, but we don't necessarily need to. If a Bash-based solution turns complex and hard to maintain, we could also consider introducing any other tool capable of interacting with the Elasticsearch API and applying configurations declaratively (Ansible, etc.). Just a suggestion of course.

[nouknouk]

Hi,

Thanks for feedback.

Your suggestion makes lot of sense to me. Indeed the choice of using yaml to describe things to inject is not a random choice for me:

Long term, using Ansible, the deployment tool choosen by my company WILL be the target.

My long term goal is to have a unique Ansible playbook, to describe a full (indempotent) installation:

• deployable on my company's PRE-PROD/PROD stacks (on-premise, self hosted on virtual machines, air gapped environments)

• as well as on docker-compose sandboxes, to let developpers experiment on their desktops, with a setup up-to-date with the one running in PROD.

This will both let developpers have a 'packaged solution' to start quickly their experiments, and reduce complexity of OPS responsible of the run of ELK in PROD (git repo + pull-requests = review proposed changes, version changes, re-build a stack from scratch, ...)

Having not found so far a 'serious & ready to use' Ansible role already available for that purpose, it will involve a bit of work to write it. Therefore, for the moment, my intermediate goal is to reduce the gap between sandboxes and prod environment, with material (yaml description files) reusable for long term goal.

[antoineco]

YAML should be fine. Probably even dropping files into certain directories should cover most cases (e.g. setup/roles for user roles.