Data Engineering

This is a summary of my efforts in finding solutions for the tasks of the IBM Data Engineering Professional Specialization.

In this specialization I worked with:

• Create, design and manage relational databases and apply database administration (DBA) concepts to RDBMS such as MySQL, PostgreSQL and IBM Db2.
• Develop and execute SQL queries using SELECT, INSERT, UPDATE, DELETE statements, database functions, stored procedures, nested queries and JOIN.
• Demonstrate working knowledge of NoSQL and Big Data using MongoDB, Cassandra, Cloudant, Hadoop, Apache Spark, Spark SQL, Spark ML, Spark Streaming.
• Implement ETL and Data Pipelines with Bash, Airflow and Kafka; design, populate and implement data warehouses; create BI reports and interactive dashboards.

There are 13 courses throughout the specialization and a capstone project at the end:

1. Introduction to Data Engineer
2. Python for Data Science, AI & Development
3. Python Project for Data Engineer
4. Introduction to Relational Databases (RDBMS)
5. Databases and SQL for Data Science with Python
6. Hands-on Introduction to Linux Commands and Shell Scripting
7. Relational Database Administration (DBA)
8. ETL and Data Pipelines with Shell, Airflow and Kafka
9. Getting Started with Data Warehousing and BI Analytics
10. Introduction to NoSQL Databases
11. Introduction to Big Data with Spark and Hadoop
12. Data Engineering and Machine Learning using Spark
13. Data Engineering Capstone Project

Tools and Technologies

• OLTP database - MySQL
• NoSql database - MongoDB
• Production Data warehouse – DB2 on Cloud
• Staging - Data warehouse – PostgreSQL
• Big data platform - Hadoop
• Big data analytics platform – Spark
• Business Intelligence Dashboard - IBM Cognos Analytics
• Data Pipelines - Apache Airflow

Acknowledgements

© Bladimir Garcia, Republica Dominicana