JSON References

This directory showcases some of the capabilities of using JSON References in order to build modular Benthos configurations that can be run and tested in isolation.

Modules

The first config files to check out are within ./modules. These config files each perform a single action:

./modules/group.yaml

Groups JSON documents by a field at the path document.type.

./modules/filter.yaml

Removes JSON documents where the content does not contain the string 2.

./modules/mutate.yaml

Modifies JSON documents by converting the entire contents of the field document.content to upper case.

Run

Each of these modules are functioning Benthos configuration files that you can run:

$ echo '{"document":{"type":"foo","content":"foo type 1"}}
{"document":{"type":"bar","content":"bar type 1"}}
{"document":{"type":"foo","content":"foo type 2"}}
{"document":{"type":"baz","content":"baz type 1"}}
{"document":{"type":"baz","content":"baz type 2"}}
{"document":{"type":"bar","content":"bar type 2"}}' | benthos -c ./modules/group.yaml

Using References

Finally, let's take a look at our main configuration, which uses JSON references in order to extract and combine sections of our modules. The config itself is tiny:

pipeline:
  processors:
  - $ref: ./modules/group.yaml#/pipeline/processors/0
  - $ref: ./modules/filter.yaml#/pipeline/processors/0
  - $ref: ./modules/mutate.yaml#/pipeline/processors/0

And in this case we have simply chained the processor from each module. We might instead wish to select the module that is referenced at the moment when we execute Benthos, in which case we can set the reference using environment variables:

pipeline:
  processors:
  # Always group
  - $ref: ./modules/group.yaml#/pipeline/processors/0

  # Second processor set via environment variable (filter by default)
  - $ref: ./modules/${MODULE:filter}.yaml#/pipeline/processors/0

And then we can execute this config with something like:

$ MODULE=mutate benthos -c ./pipeline.yaml