Getting Started

This is a demo project for education/training purposes of DevOps. All the services used below are in the Cloud to facilitate the understanding. The architecture uses microservices and containerization.

The pipeline uses GitHub Actions that contains a pipeline with 7 phases described below:

1. Compile, Build and Test

All commands of this phase are defined in build.sh file. It checks if there are no compile/build errors. The tools used are:

• Gradle - Tool to automate the build of the code.

2. Code Analysis - SAST (White-box testing)

All commands of this phase are defined in codeAnalysis.sh file. It checks Bugs, Vulnerabilities, Hotspots, Code Smells, Duplications and Coverage of the code. If these metrics don't comply with the defined Quality Gate, the pipeline won't continue. The tools used are:

• Gradle - Tool to automate the SAST analysis of the code.
• Sonar - Service that provides SAST analysis of the code.

Environments variables needed in this phase:

• GITHUB_TOKEN: API Key used by Sonar client to communicate with GitHub.
• SONAR_TOKEN: API Key used by Sonar client to store the generated analysis.

3. Libraries Analysis - SAST (White-box testing)

All commands of this phase are defined in librariesAnalysis.sh file. It checks for vulnerabilities in internal and external libraries used in the code. The tools used are:

• Gradle - Tool to automate the SAST analysis of the libraries.
• Snyk - Service that provides SAST analysis of the libraries.

Environments variables needed in this phase:

• SNYK_TOKEN: API Key used by Snyk to store the generated analysis.

4. Packaging

All commands of this phase are defined in package.sh file. It encapsulates all binaries in a Docker image. Once the code and libraries were checked, it's time build the package to be used in the next phases. The tools/services used are:

• Docker Compose - Tool to build the images.

5. Package Analysis - SAST (White-box testing)

All commands of this phase are defined in packageAnalysis.sh file. It checks for vulnerabilities in the generated package. The tools/services used are:

• Gradle - Tool to automate the SAST analysis of the package.
• Snyk - Service that provides SAST analysis of the package.

Environments variables needed in this phase:

• SNYK_TOKEN: API Key used by Snyk to store the generated analysis.

6. Publishing

All commands of this phase are defined in publish.sh file. It publishes the package in the Docker registry (GitHub Packages). The tools/services used are:

• Docker Compose - Tool to push the images into the Docker registry.
• GitHub Packages - Docker registry where the images are stored.

Environments variables needed in this phase:

• DOCKER_REGISTRY_USER: Username of the Docker registry.
• DOCKER_REGISTRY_PASSWORD: Password of the Docker registry.

7. Deploy

All commands of this phase are defined in deploy.sh file. It deploys the package in a K3S (Kubernetes) multi-cloud cluster. The tools/services used are:

• kubectl - Kubernetes Orchestration tool.
• Portainer - Kubernetes Orchestration Portal.
• Linode - Cloud (Newark/USA) where the cluster manager is installed.
• DigitalOcean - Cloud (Frankfurt/Germany) where the cluster worker is installed.

8. DAST (Black-box testing) and RASP (Runtime Application Self-Protection)

We are doing this phase outside the pipeline but it can be incorporated in the future. The tools/services used are:

• Probely - Services that executes vulnerabilities checks.
• Contrast Security - Services that protects and checks vulnerabilities.

Comments

If any phase got errors or violations, the pipeline will stop.

All environments variables must also have a secret with the same name.

You can define the secret in the repository settings.

DON'T EXPOSE OR COMMIT ANY SECRET IN THE PROJECT.

Architecture

The application uses:

• Java 11 - Programming Language.
• Spring Boot 2.5.4 - Development Framework.
• Gradle 6.8.3 - Automation build tool.
• Mockito 3 - Test framework.
• JUnit 5 - Test framework.
• MariaDB - Database server.
• NGINX 1.18 - Web server.
• Docker 20.10.8 - Containerization tool.
• K3S 1.21.4 - Containerization tool.

For further documentation please check the documentation of each tool/service.

How to install

1. Linux operating system.
2. You need an IDE such as IntelliJ.
3. You need an account in the following services: GitHub, Sonarcloud, Snyk, Contrast Security and Probely.
4. You need to set the environment variables described above in you system.
5. The API Keys for each service must be defined in the UI of each service. Please refer the service documentation.
6. Fork this project from GitHub.
7. Import the project in IDE.
8. Commit some changes in the code and follow the execution of the pipeline in GitHub.

How to run locally

1. In the project directory, execute the scripts below: ./build.sh; ./package.sh; docker-compose up
2. Remember to rename the packages to use your repository id in all YAML and SH files.

How to run in the cloud

1. First, you need to create to find a cloud provider with VPS service (Virtual Private Server).
2. After you provision the VPS and log into, you need to create a Kubernetes cluster using k3s. Follow the instructions of the website.
3. Then, install the Portainer to facilitate the deployment. Follow the instructions of the website.
4. Once Portainer is running, just create the namespace and the applications on the cluster.

Other Resources

• Official Gradle documentation
• Spring Boot Gradle Plugin Reference Guide
• Spring Web
• Spring Data JPA
• Serving Web Content with Spring MVC
• Accessing Data with JPA

All opinions and standard described here are my own.

That's it! Now enjoy and have fun!

Contact

• LinkedIn: https://www.linkedin.com/in/fvilarinho
• e-Mail: [email protected]