EnvVar Configuration Provider for Apache Kafka®

Apache Kafka® supports pluggable configuration providers which can load configuration data from external sources. The configuration provider in this repo can be used to load data from environment variables. It can be used in all Kafka components and does not depend on the other Strimzi components. So you could, for example, use it with your producer or consumer applications even if you don't use the Strimzi operators to provide your Kafka cluster. One of the example use-cases is to load certificates or JAAS configuration from environment variables mapped from Kubernetes Secrets.

Using it with Strimzi

From Strimzi Kafka Operators release 0.25.0, the EnvVar Configuration Provider is included in all the Kafka deployments. You can use it for example with Kafka Connect and Kafka Connect connectors. The following example shows how to use it to get a database password from an environment variable in the connector configuration:

1. Deploy Kafka Connect, enable the EnvVar Configuration Provider and map database password to the DB_PASSWORD environment variable:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaConnect
   metadata:
     name: my-connect
     annotations:
       strimzi.io/use-connector-resources: "true"
   spec:
     # ...
     config:
       # ...
       config.providers: env
       config.providers.env.class: io.strimzi.kafka.EnvVarConfigProvider
     # ...
     externalConfiguration:
       env:
         - name: DB_PASSWORD
           valueFrom:
             secretKeyRef:
               name: db-creds
               key: dbPassword
     # ...

2. Create the connector:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaConnector
   metadata:
     name: my-connector
     labels:
       strimzi.io/cluster: my-connect
   spec:
     # ...
     config:
       option: ${env:DB_PASSWORD}
       # ...

Adding the EnvVar Configuration Provider to Apache Kafka clients

You can add EnvVar Configuration Provider as any other Java dependency using Maven or any other build tool. For example:

<dependency>
    <groupId>io.strimzi</groupId>
    <artifactId>kafka-env-var-config-provider</artifactId>
    <version>0.1.0</version>
</dependency>

Adding the EnvVar Configuration Provider to Apache Kafka server components

You can also use the EnvVar Configuration Provider with your own Apache Kafka deployments not managed by Strimzi. To add EnvVar Configuration Provider to your own Apache Kafka server distribution, you can download the ZIP or TAR.GZ files frm the GitHub release page and unpack it into Kafka's libs directory.

Using the configuration provider

First, you need to initialize the configuration provider.

config.providers=env
config.providers.env.class=io.strimzi.kafka.EnvVarConfigProvider

Once you initialize it, you can use it to load data from environment variables. For example:

option=${env:MY_ENV_VAR}

Other Strimzi Configuration Providers

If you run Apache Kafka on Kubernetes, you might also be interested in our Kubernetes Configuration Provider.