Remove poetry from CI (zenml-io#1346)

* Try removing poetry from CI

* Set venv variable

* Add missing shell

* Try fixing windows installation

* Remove debug logs

* Some cleanup

* Fix test

* Investigate linting error

* Use editable install

* Invalidate GH action cache

* Really fix test this time

* Maybe fix HF datasets linting

* Probably fix HF datasets linting

* Remove most mentions of poetry

* Remove more poetry mentions

* Install server dependencies in readmes

* Rename install workflow

.github/actions/setup_environment/action.yml

@@ -1,5 +1,5 @@
-name: "Setup Matrix and Poetry"
-description: "Install Poetry, loads cached venv if applicable, and performs miscellaneous other operations"
+name: "Install ZenML"
+description: "Install ZenML, loads cached venv if applicable, and performs miscellaneous other operations"
 inputs:
   cache_version:
     description: 'Value gets appended to the cache key and will therefore invalidate the cache if it changes'
@@ -32,12 +32,6 @@ runs:
       run: |
         echo "SHELL_EXECUTABLE=C:\Program Files\Git\bin\bash.exe" >> $GITHUB_ENV
 
-    - name: Install Poetry
-      uses: snok/[email protected]
-      with:
-        virtualenvs-create: true
-        virtualenvs-in-project: true
-
     - name: Get current date
       shell: bash
       id: date
@@ -50,19 +44,31 @@ runs:
         CACHE_VERSION : ${{inputs.cache_version}}
 
     - name: Load cached venv
-      id: cached-poetry-dependencies
+      id: cached-virtualenv
       uses: actions/[email protected]
       with:
-        path: |
-          .venv
-          poetry.lock
+        path: .venv
         # Cache the complete venv dir for a given os, python version,
         # pyproject.toml, the current calendar week and the cache version.
         # Providing a different value for the cache version enables manual
         # invalidation of the cache in case it is in some erroneous state.
         key: venv-${{ runner.os }}-python-${{ inputs.python-version }}-${{ hashFiles('pyproject.toml') }}-${{ steps.date.outputs.date }}-${{ hashFiles('cache_version') }}
 
+    - name: Set virtualenv activation script env variable (windows)
+      shell: bash
+      run: echo "VENV=.venv/scripts/activate" >> "$GITHUB_ENV"
+      if: runner.os == 'Windows'
+
+    - name: Set virtualenv activation script env variable (unix)
+      shell: bash
+      run: echo "VENV=.venv/bin/activate" >> "$GITHUB_ENV"
+      if: runner.os != 'Windows'
+
     - name: Install Project
       shell: bash
-      if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
-      run: poetry install --extras server
+      if: steps.cached-virtualenv.outputs.cache-hit != 'true'
+      run: |
+        python -m venv .venv
+        source $VENV
+        python -m pip install --upgrade pip
+        pip install -e ".[server,dev]"

.github/workflows/ci.yml

@@ -16,23 +16,23 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
-  poetry-install:
+  setup-python-environment:
     if: github.event.pull_request.draft == false
-    uses: ./.github/workflows/poetry-install.yml
+    uses: ./.github/workflows/setup-python-environment.yml
     secrets: inherit
 
   lint-code:
-    needs: poetry-install
+    needs: setup-python-environment
     uses: ./.github/workflows/lint.yml
     secrets: inherit
 
   unit-test:
-    needs: poetry-install
+    needs: setup-python-environment
     uses: ./.github/workflows/unit-test.yml
     secrets: inherit
 
   integration-test:
-    needs: poetry-install
+    needs: setup-python-environment
     # Exit if its a commit from Gitbook
     if: ${{ ! startsWith(github.event.head_commit.message, 'GitBook:') }}
     strategy:

.github/workflows/integration-test.yml

@@ -169,7 +169,7 @@ jobs:
           install_components: 'gke-gcloud-auth-plugin'
         if: contains(inputs.test_environment, 'gcp')
 
-      - name: Setup environment with Poetry
+      - name: Setup environment
         uses: ./.github/actions/setup_environment
         with:
           cache_version: ${{ secrets.GH_ACTIONS_CACHE_KEY }}
@@ -242,7 +242,7 @@ jobs:
         run: |
           source $VENV
           # re-install zenml to be sure
-          poetry install --extras server
+          pip install -e ".[server,dev]"
           # install airflow requirements
           pip install "numpy>=1.21.0,<1.24.0"
           pip install apache-airflow-providers-docker

.github/workflows/lint.yml

@@ -27,15 +27,15 @@ jobs:
     steps:
       - uses: actions/checkout@v2
 
-      - name: Setup environment with Poetry
+      - name: Setup environment
         uses: ./.github/actions/setup_environment
         with:
           cache_version: ${{ secrets.GH_ACTIONS_CACHE_KEY }}
           python-version: ${{ matrix.python-version }}
           os: ${{ matrix.os }}
 
       - name: Install Dependencies
-        if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
+        if: steps.cached-virtualenv.outputs.cache-hit != 'true'
         run: |
           source $VENV
           zenml integration export-requirements --output-file requirements.txt --ignore-integration feast --ignore-integration label_studio --ignore-integration kserve --ignore-integration bentoml --ignore-integration tekton --ignore-integration airflow
@@ -45,5 +45,5 @@ jobs:
         
       - name: Lint
         run: |
-          source $VENV 
+          source $VENV
           bash scripts/lint.sh

.github/workflows/publish_api_docs.yml

@@ -40,7 +40,7 @@ jobs:
       - run: npm install
       - run: npm install html-minifier -g
 
-      - name: Setup environment with Poetry
+      - name: Setup environment
         uses: ./.github/actions/setup_environment
         with:
           cache_version: ${{ secrets.GH_ACTIONS_CACHE_KEY }}
@@ -50,7 +50,7 @@ jobs:
       # IMPORTANT: there's a strategy to installing integrations here in a way
       # that avoids conflicts while at the same time making it possible for all
       # ZenML Python modules to be imported, especially the integration modules:
-      # 1. install zenml with all extras with poetry
+      # 1. install zenml with all extras
       # 2. install more restrictive integrations first: feast, seldon and
       # label_studio are currently the ones known to be very restrictive in
       # terms of what versions of dependencies they require
@@ -69,7 +69,7 @@ jobs:
           zenml integration install -y airflow
           zenml integration install -y kserve
           zenml integration install -y --ignore-integration feast --ignore-integration label_studio --ignore-integration seldon --ignore-integration kserve --ignore-integration airflow --ignore-integration bentoml
-          poetry install --extras server
+          pip install -e ".[server,dev]"
           pip install jinja2==3.0.3 protobuf==3.20.0 numpy~=1.21.5
 
       - name: Setup git user
@@ -88,4 +88,6 @@ jobs:
           rm -rf src/zenml/zen_stores/migrations/script.py.mako
 
       - name: Runs docs generation
-        run: poetry run bash scripts/generate-docs.sh -s src/zenml/ --push --latest -v ${{ steps.get_version.outputs.VERSION }}
+        run: |
+          source $VENV
+          bash scripts/generate-docs.sh -s src/zenml/ --push --latest -v ${{ steps.get_version.outputs.VERSION }}

.github/workflows/poetry-install.yml → .github/workflows/setup-python-environment.yml

@@ -26,7 +26,7 @@ jobs:
     steps:
       - uses: actions/checkout@v2
 
-      - name: Setup environment with Poetry
+      - name: Setup environment
         uses: ./.github/actions/setup_environment
         with:
           cache_version: ${{ secrets.GH_ACTIONS_CACHE_KEY }}

.github/workflows/unit-test.yml

@@ -37,7 +37,7 @@ jobs:
     steps:
       - uses: actions/checkout@v2
 
-      - name: Setup environment with Poetry
+      - name: Setup environment
         uses: ./.github/actions/setup_environment
         with:
           cache_version: ${{ secrets.GH_ACTIONS_CACHE_KEY }}

.pre-commit-config.yaml

@@ -5,7 +5,7 @@ repos:
     hooks:
       - id: lint
         name: lint
-        entry: poetry run bash scripts/lint.sh
+        entry: bash scripts/lint.sh
         language: system
         files: ^src/zenml/|^tests/|^examples/
         types: [file, python]
@@ -15,7 +15,7 @@ repos:
     hooks:
       - id: docstringcheck
         name: docstringcheck
-        entry: poetry run bash scripts/docstring.sh
+        entry: bash scripts/docstring.sh
         language: system
         files: ^src/zenml/
         types: [file, python]
@@ -25,7 +25,7 @@ repos:
     hooks:
       - id: pytest
         name: pytest
-        entry: poetry run bash scripts/test-coverage-xml.sh unit
+        entry: bash scripts/test-coverage-xml.sh unit
         language: system
         files: ^src/zenml/|^tests/
         types: [file, python]

CONTRIBUTING.md

@@ -117,20 +117,18 @@ the ["fork-and-pull" Git workflow](https://github.com/susam/gitpr)
 
 ### 🧐 Linting, formatting, and tests
 
-ZenML is mainly developed using [Poetry](https://python-poetry.org/) as the
-dependency management system. In order to
-install all core dev-dependencies, do:
+To install ZenML from your local checked out files includin all core dev-dependencies, run:
 
 ```
-poetry install
+pip install -e ".[server,dev]"
 ```
 
 Optionally, you might want to run the following commands to ensure you have all
 integrations for `mypy` checks:
 
 ```
-poetry run zenml integration install -f
-poetry run zenml integration install -y -i feast
+zenml integration install -f
+zenml integration install -y -i feast
 pip install click~=8.0.3
 mypy --install-types
 ```
@@ -139,13 +137,13 @@ Warning: This might take a while for both (~ 15 mins each, depending on your mac
 time, please run it as it will make the
 next commands error-free.
 
-You can now run the following scripts via Poetry to automatically format your
+You can now run the following scripts to automatically format your
 code and to check whether the code formatting, linting, docstrings, and
 spelling is in order:
 
 ```
-poetry run bash scripts/format.sh
-poetry run bash scripts/run-ci-checks.sh
+bash scripts/format.sh
+bash scripts/run-ci-checks.sh
 ```
 
 If the spell checker catches errors, you will have to add the words to
@@ -156,7 +154,7 @@ manually on your local machine, first install `aspell` (using `brew` or
 Tests can be run as follows:
 
 ```
-poetry run bash scripts/test-coverage-xml.sh
+bash scripts/test-coverage-xml.sh
 ```
 
 Please note that it is good practice to run the above commands before submitting

docs/README.md

@@ -24,11 +24,11 @@ To build them locally follow these steps:
 * Clone the repository
 * Install ZenML and all dependencies
 ```python
-poetry install
-poetry run zenml integration install -y
-poetry run pip install click~=8.0.3 typing-extensions~=3.10.0.2
+pip install ".[dev,server]"
+zenml integration install -y
+pip install click~=8.0.3 typing-extensions~=3.10.0.2
 ```
-* Run `poetry run bash scripts/serve_api_docs.sh` from the repository root - 
+* Run `bash scripts/serve_api_docs.sh` from the repository root - 
 running it from elsewhere can lead to unexpected errors. This script will compose the docs hierarchy
 and serve it (default location is http://127.0.0.1:8000/).
 * In case port 8000 is taken you can also manually go into the docs folder within your terminal and