[streams] fork classic stream template

[klacabane]

When applying mappings or lifecycle changes to an unwired stream we don't want to update the current unmanaged index template for several reasons.
Our solution is to fork the effective template and tweak it slightly to make the streams' layer applicable in a controlled way

This change does:

• duplicates the effective template with a higher priority and update the index_patterns to only look for the specific unwired stream
• creates a @stream.layer component template similar to the wired streams. this layer will hold the streams' specific mappings and lifecycle
• extracts the existing template's lifecycle/settings/mappings to a @base component template so stream's layer can override them
• enables ilm lifecycle on unwired streams. the related UI changes will be implemented separately

This change does not:

• implement a way to revert/eject the streams layer
• add a way to specify mappings for unwired streams

these will be implemented separately

Testing

• Updating a classic stream lifecycle will trigger a template fork

PUT kbn:/api/streams/my-unwired-stream
{
  "dashboards": [],
  "stream": {
    "ingest": {
      "lifecycle": {
        "dsl": { "data_retention": "1d" } // "ilm": { "policy": "my-policy" }
      },
      "processing": [],
      "routing": [],
      "unwired": {}
    }
  }
}

TODO

• api tests

[klacabane]

I replaced the executionPlan logic with elasticsearch calls since it does not seem to do much for now. Happy to revert if there are plans to add more to that construct

[elasticmachine]

⏳ Build in-progress, with failures

• Buildkite Build
• Commit: 322f828

Failed CI Steps

• FTR Configs #64
• FTR Configs #64

Test Failures

• [job] [logs] FTR Configs #64 / Stateful Observability - Deployment-agnostic API integration tests Streams Endpoints Lifecycle Unwired streams allows ilm lifecycle on unwired streams
• [job] [logs] FTR Configs #64 / Stateful Observability - Deployment-agnostic API integration tests Streams Endpoints Lifecycle Unwired streams allows ilm lifecycle on unwired streams

History

• 💔 Build #278043 failed c3c699b
• 💔 Build #277559 failed 3c64361