Contains simple example using small Spark Jobs and Airflow pipeline on Google Cloud
The pipeline consists of ETL 2 json files, matching them in 2 different ways, merging them and storing the result in Elastic Search
Try to change & run the scripts/xx-xxx.sh files in order to create yourown setup
Your experience may differ from mine. :-)
Created for Data Driven Rijnmond meetup https://www.meetup.com/nl-NL/Data-Driven-Rijnmond/events/246008610/
Please join us for other cool talks & demo's
