Pia's Poetry Cookiecutter for MLOps

Todos

• incorporate modified MLOps File structure
• add documentation generation
• add GCP support
• add DVC support
• add Terraform support
• add updated file tree ascii illustration

Resources and Inspiration

** ML and Data Engineering **

• Secure Production ML Engineering Project Base
• [Cookiecutter MLOps] - https://github.com/Chim-SO/cookiecutter-mlops - Thoughtful ML file structure
• [Cookiecutter template] for MLOps Project based on the MLOps Guide: https://github.com/mlops-guide/mlops-template - includes dvc integration + infrastructure-as-code terraform
• [Minimal ML Cookiecutter Project Structure] https://github.com/jeannefukumaru/cookiecutter-ml
• [AI Singapore's Cookiecutter Template for End-to-end ML Projects] https://github.com/aisingapore/ml-project-cookiecutter-gcp-runai
• [A cookiecutter template to create Docker environment for ML/DL tasks] https://github.com/kazuhirokomoda/cookiecutter-docker-jupyterlab
• [This project is an opinionated template for ML projects which creates package skeletons] https://github.com/project-delphi/cookiecutter-ml-template
• https://github.com/Yann21/cookiecutter-exp-launcher

** Python Best Practices **

• https://github.com/cjolowicz/cookiecutter-hypermodern-python
• https://github.com/sourcery-ai/python-best-practices-cookiecutter

Notes

• use cruft to install packages and update them

🍿 Demo

See My Package for an example of a Python package and app that is scaffolded with this template.

Starting development in My Package can be done with a single click by opening My Package in GitHub Codespaces, or opening My Package in a Dev Container.

🎁 Features

• 🧑‍💻 Quick and reproducible development environments with VS Code's Dev Containers, PyCharm's Docker Compose interpreter, and GitHub Codespaces
• 🌈 Cross-platform support for Linux, macOS (Apple silicon and Intel), and Windows
• 🐚 Modern shell prompt with Starship
• 📦 Packaging and dependency management with Poetry
• 🚚 Installing from and publishing to private package repositories and PyPI
• ⚡️ Task running with Poe the Poet
• ✍️ Code formatting with Ruff
• ✅ Code linting with Pre-commit, Mypy, and Ruff
• 🏷 Optionally follows the Conventional Commits standard to automate Semantic Versioning and Keep A Changelog with Commitizen
• 💌 Verified commits with GPG
• ♻️ Continuous integration with GitHub Actions or GitLab CI/CD
• 🧪 Test coverage with Coverage.py
• 🏗 Scaffolding updates with Cookiecutter and Cruft
• 🧰 Dependency updates with Dependabot

✨ Using

Creating a new Python project

To create a new Python project with this template:

1. Install the latest Cruft and Cookiecutter in your Python environment with:

   pip install --upgrade "cruft>=2.12.0" "cookiecutter>=2.1.1"

2. Create a new repository for your Python project, then clone it locally.
3. Run the following command in the parent directory of the cloned repository to apply the Poetry Cookiecutter template:

   cruft create -f https://github.com/radix-ai/poetry-cookiecutter

4. Optional: if your repository name differs from your project's slugified package name (see package_name in the Template parameters below), you will need to copy the scaffolded project into the repository with:

   cp -r {package-name}/ {repository-name}/

Updating your Python project

To update your Python project with the latest template:

1. Run cruft update to update your project with the latest template.
2. If any of the updates failed, resolve them by inspecting the generated .rej files.

🤓 Template parameters

Parameter	Description
package_name "Spline Reticulator"	The name of the package. Will be slugified to snake_case for importing and kebab-case for installing. For example, My Package will be my_package for importing and my-package for installing.
package_description "A Python package that reticulates splines."	A single-line description of the package.
package_url "https://github.com/user/spline-reticulator"	The URL to the package's repository.
author_name "John Smith"	The full name of the primary author of the package.
author_email "[email protected]"	The email address of the primary author of the package.
python_version "3.8"	The minimum Python version that the package requires.
docker_image "python:$PYTHON_VERSION-slim"	The base Docker image to use for the Dev Container and application. The $PYTHON_VERSION build argument is equal to the python_version value by default, but may be overridden when building the image to test different Python versions. If CUDA support is required, you may use radixai/python-gpu:$PYTHON_VERSION-cuda11.8.
development_environment ["simple", "strict"]	Whether to configure the development environment with a focus on simplicity or with a focus on strictness. In strict mode, additional Ruff rules are added, and tools such as Mypy and Pytest are set to strict mode.
with_conventional_commits ["0", "1"]	If "1", Commitizen will verify that your commits follow the Conventional Commits standard. In return, cz bump may be used to automate Semantic Versioning and Keep A Changelog.
with_fastapi_api ["0", "1"]	If "1", FastAPI is added as a run time dependency, FastAPI API stubs and tests are added, a poe api command for serving the API is added, and an app stage that packages the API is added to the Dockerfile. Additionally, the CI workflow will push the application as a Docker image instead of publishing the Python package.
with_jupyter_lab ["0", "1"]	If "1", JupyterLab is added to Poetry's dev dependencies, and a poe lab command is added to start Jupyter Lab in the notebooks/ directory.
with_pydantic_typing ["0", "1"]	If "1", Pydantic is added as a run time dependency, and the Pydantic mypy plugin is enabled and configured.
with_sentry_logging ["0", "1"]	If "1", Sentry is added as a run time dependency, and a Sentry configuration stub and tests are added.
with_streamlit_app ["0", "1"]	If "1", Streamlit is added as a run time dependency, a Streamlit application stub is added, a poe app command to serve the Streamlit app is added, and an app stage that packages the Streamlit app is added to the Dockerfile. Additionally, the CI workflow will push the application as a Docker image instead of publishing the Python package.
with_typer_cli ["0", "1"]	If "1", Typer is added as a run time dependency, Typer CLI stubs and tests are added, the package itself is registered as a CLI, and an app stage is added to the Dockerfile that packages the CLI.
continuous_integration ["GitHub", "GitLab"]	Whether to include a GitHub Actions or a GitLab CI/CD continuous integration workflow for testing and publishing the package or app.
docstring_style ["NumPy", "Google"]	Whether to use and validate NumPy-style or Google-style docstrings.
private_package_repository_name "Private Package Repository"	Optional name of a private package repository to install packages from and publish this package to.
private_package_repository_url "https://pypi.example.com/simple"	Optional URL of a private package repository to install packages from and publish this package to. Make sure to include the /simple suffix. For instance, when using a GitLab Package Registry this value should be of the form https://gitlab.com/api/v4/projects/ {project_id} /packages/pypi/simple.