Camel-Kafka-connector Google Mail Stream Source Example

This is an example for Camel-Kafka-connector Google Mail Stream Source Example

Standalone

What is needed

• A google account

• A google application authorized to deal with gmail API

Setting up Google Application

The step-by-step guide is a bit long, so follow it with attention:

1. Create a project

2. Note down clientSecret and clientId

3. Create an OAuth consent screen

4. Create OAuth Client credentials (OAuth for WebApp)

5. Add https://developers.google.com/oauthplayground to Authorized redirect URIs

6. Setup https://developers.google.com/oauthplayground using credentials from above

7. Authorize APIs for Gmail API v1 scope https://mail.google.com

8. Exchange authorization code for tokens

9. Note down Access token & Refresh token

10. Mark some email as unread in your account

We auto-sent an email with the following body "Camel Kafka Connector is great!" and marked it as unread.

Running Kafka

$KAFKA_HOME/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties
$KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties
$KAFKA_HOME/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic mytopic

Download the connector package

Download the connector package tar.gz and extract the content to a directory. In this example we’ll use /home/oscerd/connectors/

> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-google-mail-stream-kafka-connector/0.11.5/camel-google-mail-stream-kafka-connector-0.11.5-package.tar.gz
> untar.gz camel-google-mail-stream-kafka-connector-0.11.5-package.tar.gz

Configuring Kafka Connect

You’ll need to set up the plugin.path property in your kafka

Open the $KAFKA_HOME/config/connect-standalone.properties and set the plugin.path property to your choosen location:

...
plugin.path=/home/oscerd/connectors
...

Setup the connectors

Open the Git configuration file at $EXAMPLES/google-mail-stream/google-mail-stream-source/config/CamelGoogleMailStreamSourceConnector.properties

name=CamelGoogle-mail-streamSourceConnector
connector.class=org.apache.camel.kafkaconnector.googlemailstream.CamelGooglemailstreamSourceConnector
tasks.max=1

key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter

topics=mytopic

camel.source.path.index=ckc
camel.source.endpoint.clientId=<client_id>
camel.source.endpoint.accessToken=<access_token>
camel.source.endpoint.refreshToken=<refresh_token>
camel.source.endpoint.clientSecret=<client_secret>
camel.source.endpoint.applicationName=ckc
camel.source.endpoint.delay=60000
camel.source.endpoint.markAsRead=true
camel.source.endpoint.labels=inbox
camel.source.endpoint.query=is:unread -category:(promotions OR social)
camel.source.endpoint.maxResults=20

Running the example

Run the kafka connect with the Google Mail Stream Source connector:

$KAFKA_HOME/bin/connect-standalone.sh $KAFKA_HOME/config/connect-standalone.properties $EXAMPLES/google-mail-stream/google-mail-stream-source/config/CamelGoogleMailStreamSourceConnector.properties

On a different terminal run the kafkacat consumer

> ./kafkacat -b localhost:9092 -t mytopic
Camel Kafka Connector is great!
% Reached end of topic mytopic [0] at offset 1

In the google mail account you should now see the email marked as read.