diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 04c75606b..63bbfee3a 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -11,24 +11,33 @@ jobs: PYTHONPATH: ${{ github.workspace }} steps: - - uses: actions/checkout@v1 - - uses: actions/setup-python@v2 + - name: Checkout code + uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 with: - python-version: '3.9' + python-version: '3.11' architecture: 'x64' - name: Start MySQL service run: | sudo /etc/init.d/mysql start - - name: Install packaged deps + + - name: Install Poetry run: | - sudo apt-get install --yes libdb5.3-dev + curl -sSL https://install.python-poetry.org | python3 - --version 1.8.5 + echo "$HOME/.local/bin" >> $GITHUB_PATH - - name: Prepare Python env - run: | - pip install --upgrade pip setuptools wheel - pip install -r ./requirements-dev.txt + - name: Install dependencies + run: pip install . + + - name: Install dev dependencies + run: pip install -r requirements-dev.txt + + - name: Validate installed packages against poetry.lock file + run: poetry check --lock - name: Dump environment (for troubleshooting) run: | diff --git a/.gitignore b/.gitignore index d2cd9a660..9cbe3ee8d 100644 --- a/.gitignore +++ b/.gitignore @@ -12,6 +12,8 @@ Z* \.old[0-9]*$ \.project$ \.pydevproject$ +\.python-version +.test-venv db dev/css/* dev/legacy-scss/css/* @@ -23,6 +25,7 @@ settings.py static/javascripts/metadata_tooltips.js ui_library ~$ +templates/info # Dirs @@ -33,3 +36,5 @@ templates/info/ *\.egg-info/ \.pytest_cache/ .eggs/ +pytest.ini +.test-venv diff --git a/README.2.7.md b/README.2.7.md new file mode 100644 index 000000000..12d5e6456 --- /dev/null +++ b/README.2.7.md @@ -0,0 +1,141 @@ +# EZID dev setup + +Keeping setup instructions separate from the [original README](README.orig.md) for now. + +## Packaged dependencies + +```shell script + +sudo bash -c ' + apt update + apt dist-upgrade -fy + apt install -y \ + \ + build-essential \ + llvm \ + make \ + \ + python2-dev \ + python-openssl \ + python-setuptools \ + \ + bash \ + curl \ + wget \ + git \ + xz-utils \ + \ + libbz2-dev \ + libc6-dev \ + libdb-dev \ + libedit-dev \ + libffi-dev \ + libgdbm-dev \ + liblzma-dev \ + libmariadbclient-dev + libncurses5-dev \ + libncursesw5-dev \ + libreadline-dev \ + libsqlite3-dev \ + libssl-dev \ + libz-dev \ + tk-dev \ + zlib1g \ + zlib1g-dev \ + \ + sqlite3 \ +' +``` + +## Python 2.7 + +Prepare pyenv + +```shell script +curl https://pyenv.run | bash +``` + +- Follow the instructions printed at the end of the pyenv setup. + +Install Python 2.7 + +```shell script +bash -c ' + pyver=2.7.18 + venv=ezid_${pyver} + export CONFIGURE_OPTS=--enable-shared + export CFLAGS=-O2 + pyenv install ${pyver} + pyenv virtualenv ${pyver} ${venv} + pyenv global ${venv} + pip install --upgrade pip +' +``` + +## EZID + +```shell script +bash -c ' + export ez=$HOME/dev + mkdir -p $ez + cd $ez + + git clone https://github.com/CDLUC3/ezid.git + git clone https://github.com/CDLUC3/ezid-info-pages.git ezid/templates/info + + cd $ez/ezid && git checkout initial-setup + cd $ez/ezid/templates/info && git checkout initial-setup + + cd $ez/ezid + pip install -r requirements.txt + + ./dev_prep_env.sh +' +``` + +DB Init or reinit + +```shell script +bash -c ' + rm -f ./db/*.sqlite3 + ./manage.py migrate + ./manage.py migrate --database=search + ./manage.py loaddata store-init + ./manage.py loaddata search-init --database=search +' +``` + +Optionally, load more complete test database + +```shell script +bash -c ' + yes 'yes' | ./manage.py flush + ./manage.py loaddata store-test -e auth -e contenttypes -e sessions +' +``` + +```shell +# TODO: Outline setup via setup.py +# Dev env with: ./setup.py develop +# pyenv rehash +# Tools now available as ez-* (use tab to get a list) +``` + +Start + +```shell script +$ export ez=$HOME/dev +$ cd $ez/ezid +$ ./manage.py runserver +``` + +## Git + +Set Git to ignore bulk edits in `git blame`: + +```shell script +$ cd +$ git config blame.ignoreRevsFile .git-blame-ignore-revs +``` + + diff --git a/README.md b/README.md index d571c3564..4eaf02367 100644 --- a/README.md +++ b/README.md @@ -1,139 +1,221 @@ -# EZID dev setup - -Keeping setup instructions separate from the [original README](README.orig.md) for now. - -## Packaged dependencies - -```shell script - -sudo bash -c ' - apt update - apt dist-upgrade -fy - apt install -y \ - \ - build-essential \ - llvm \ - make \ - \ - python2-dev \ - python-openssl \ - python-setuptools \ - \ - bash \ - curl \ - wget \ - git \ - xz-utils \ - \ - libbz2-dev \ - libc6-dev \ - libdb-dev \ - libedit-dev \ - libffi-dev \ - libgdbm-dev \ - liblzma-dev \ - libmariadbclient-dev - libncurses5-dev \ - libncursesw5-dev \ - libreadline-dev \ - libsqlite3-dev \ - libssl-dev \ - libz-dev \ - tk-dev \ - zlib1g \ - zlib1g-dev \ - \ - sqlite3 \ -' -``` +Python Dependency Management for EZID +===================================== -## Python 2.7 +Thu May 30 05:18:33 PM PDT 2024 -Prepare pyenv +Our previous installation method using `setup.py install` is no longer +supported for deployments on python `>3.11.x`. We now use `pip install` to +deploy EZID and [poetry](https://python-poetry.org/docs/) for python package +dependency resolution. Both tools rely on file `pyproject.toml` for dependency +management. -```shell script -curl https://pyenv.run | bash -``` +At this point in time we do not use `poetry install` to deploy EZID, because +this forces us to run the application and all django management commands within +a poetry generated virtual environment (e.g. `poetry run command`). The +trade-off is we do not take advantage of poetry's excellent dependency version +locking mechanism. -- Follow the instructions printed at the end of the pyenv setup. - -Install Python 2.7 - -```shell script -bash -c ' - pyver=2.7.18 - venv=ezid_${pyver} - export CONFIGURE_OPTS=--enable-shared - export CFLAGS=-O2 - pyenv install ${pyver} - pyenv virtualenv ${pyver} ${venv} - pyenv global ${venv} - pip install --upgrade pip -' -``` -## EZID - -```shell script -bash -c ' - export ez=$HOME/dev - mkdir -p $ez - cd $ez - - git clone https://github.com/CDLUC3/ezid.git - git clone https://github.com/CDLUC3/ezid-info-pages.git ezid/templates/info - - cd $ez/ezid && git checkout initial-setup - cd $ez/ezid/templates/info && git checkout initial-setup - - cd $ez/ezid - pip install -r requirements.txt - - ./dev_prep_env.sh -' -``` +The `pyproject.toml` file +------------------------- -DB Init or reinit +The +[`pyproject.toml`](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#writing-pyproject-toml) +configuration file is used by a veriety of python packaging and installation +tools. Here is a quick tour of the our EZID `pyproject.toml`: -```shell script -bash -c ' - rm -f ./db/*.sqlite3 - ./manage.py migrate - ./manage.py migrate --database=search - ./manage.py loaddata store-init - ./manage.py loaddata search-init --database=search -' +We declare Poetry as our build tool +``` +[build-system] +requires = ["poetry-core>=1.0.0,<2.0.0"] +build-backend = "poetry.core.masonry.api" ``` -Optionally, load more complete test database +We use Poetry only for dependency management but not for packaging: +``` +[tool.poetry] +# use Poetry only for dependency management but not for packaging +package-mode = false +``` -```shell script -bash -c ' - yes 'yes' | ./manage.py flush - ./manage.py loaddata store-test -e auth -e contenttypes -e sessions -' +We enumerate python dependencies along with minumum semantic version: +``` +[tool.poetry.dependencies] +python = "^3.11" +django = "^4.2.11" +aiomysql = "^0.2.0" +mysqlclient = "^2.2.4" +boto3 = "^1.34.116" +[cut] ``` -```shell -# TODO: Outline setup via setup.py -# Dev env with: ./setup.py develop -# pyenv rehash -# Tools now available as ez-* (use tab to get a list) +We enumerate development tools in a `dev` group section: +``` +[tool.poetry.group.dev.dependencies] +pytest-django = "^4.8.0" +pytest-mock = "^3.14.0" ``` -Start -```shell script -$ export ez=$HOME/dev -$ cd $ez/ezid -$ ./manage.py runserver + +Installing EZID +--------------- + +The target host must support python 3.11 or higher. This can be a python virtual environment. +After install we can use `poetry check` to validate the installed dependencies against the +`poetry.lock` file. See [Installing Poetry](#installing-poetry) below. + +1. Clone the project repostory + ``` + agould@uc3-opspuppet-dev01:~/projects> git clone git@github.com:CDLUC3/ezid.git + Cloning into 'ezid'... + Receiving objects: 100% (32942/32942), 73.36 MiB | 16.44 MiB/s, done. + Resolving deltas: 100% (21345/21345), done. + Updating files: 100% (986/986), done. + ``` + +1. Activate your virtual environment if using such (example uses [pyenv](https://github.com/pyenv/pyenv?tab=readme-ov-file#automatic-installer)) + ``` + agould@uc3-opspuppet-dev01:~/projects> pyenv global + 3.11.9 + agould@uc3-opspuppet-dev01:~/projects> pyenv virtualenv ezid-3.11.9 + agould@uc3-opspuppet-dev01:~/projects> pyenv global ezid-3.11.9 + (ezid-3.11.9) agould@uc3-opspuppet-dev01:~/projects> + ``` + +1. Change directory into the root of the working tree of the cloned project repository + ``` + (ezid-3.11.9) agould@uc3-opspuppet-dev01:~/projects> cd ezid + (ezid-3.11.9) agould@uc3-opspuppet-dev01:~/projects/ezid> ls -l pyproject.toml + -rw-r--r--. 1 agould users 2726 May 31 13:26 pyproject.toml + ``` + +1. Run `pip install .` This tells `pip` to source file `pyproject.toml` and + install all dependencies listed in table `[tool.poetry.dependencies]` + ``` + (ezid-3.11.9) agould@uc3-opspuppet-dev01:~/projects/ezid> pip install . + Processing /home/agould/projects/ezid + Installing build dependencies ... done + Getting requirements to build wheel ... done + Preparing metadata (pyproject.toml) ... done + [cut] + Successfully installed MarkupSafe-2.1.5 aiomysql-0.2.0 amqp-5.2.0 asgiref-3.8.1 beautifulsoup4-4.12.3 billiard-4.2.0 boto3-1.34.117 botocore-1.34.117 ... + ``` + +1. Validate installed packages against `poetry.lock` file + ``` + (ezid-3.11.9) agould@uc3-opspuppet-dev01:~/projects/ezid> poetry check --lock + All set! + ``` + + +Keeping dependencies up-to-date with Poetry +------------------------------------------- + +Occassionally we must ensure the minimum semantic versions of python +dependencies are current. We use a combination of +[Poetry commands](https://python-poetry.org/docs/cli/) to currate the `pyproject.toml` file. + + +#### Installing Poetry + +We recommend to install Poetry globally on your development host, so it may be +used for a veriety of projects regardless of virtual environment. Do this with +[`pipx`](https://python.land/virtual-environments/pipx): +``` +agould@uc3-opspuppet-dev01:~> pipx install poetry +agould@uc3-opspuppet-dev01:~> which poetry +~/.local/bin/poetry ``` -## Git -Set Git to ignore bulk edits in `git blame`: +The `poetry update` command retrieves latest versions of dependencies listed in +`pyproject.toml` and updates the `poetry.lock` file. However, it does not +automatically update minimum semantic versions in `pyproject.toml`. To keep +`pyproject.toml` truly up-to-date, do the following: + +1. Update `pyproject.toml` with command `poetry add ${package}@latest`. This has + the added benefit of updating `poetry.lock` as well. + + Make a list of all dependency package names and run this command in a for loop: + + ``` + # for now we omit `django-matomo-api-tracking` because of how it is sourced. + DEPS=" + django + aiomysql + mysqlclient + boto3 + mysql + myloginpath + jinja2 + pygments + fasteners + feedparser + filelock + freezegun + hjson + lxml + python-dateutil + requests + toml + simplegist + xmltodict + pymysql + configparser + ecs-logging + celery + sqlalchemy + " + for package in $DEPS; do + poetry add ${package}@latest + done + ``` + Note: + * A script `update_pyproject.sh` was created to automate the above process. + * The `django` entry is excluded from the list. Run `poetry add django@version` command to manually upgrade Django to a desired version. + * The `update_pyproject.sh` script includes a final step `poetry update` which is used to update package dependencies in `poetry.lock`. + +1. Review changes with `git diff pyproject.toml`. Make alterations as needed: + ``` + (ezid-3.11.9) agould@uc3-opspuppet-dev01:~/projects/ezid> git diff pyproject.toml + diff --git a/pyproject.toml b/pyproject.toml + index 83bff875..fafbf330 100644 + --- a/pyproject.toml + +++ b/pyproject.toml + @@ -42,7 +42,7 @@ python = "^3.11" + django = "^4.2.11" + aiomysql = "^0.2.0" + mysqlclient = "^2.2.4" + -boto3 = "^1.34.107" + +boto3 = "^1.34.117" + mysql = "^0.0.3" + myloginpath = "^0.0.4" + jinja2 = "^3.1.4" + @@ -54,16 +54,16 @@ freezegun = "^1.5.1" + hjson = "^3.1.0" + lxml = "^5.2.2" + python-dateutil = "^2.9.0.post0" + -requests = "^2.31.0" + +requests = "^2.32.3" + toml = "^0.10.2" + simplegist = "^1.0.1" + xmltodict = "^0.13.0" + -pymysql = "^1.1.0" + +pymysql = "^1.1.1" + configparser = "^7.0.0" + ecs-logging = "^2.1.0" + celery = "^5.4.0" + sqlalchemy = "^2.0.30" + ``` +Note: +* Run the `poetry update` command if you modified the `pyproject.toml` file manually. + +3. Review changes with `git diff poetry.lock` + +4. Commit your updates and cut a new release candidate tag. Open a pull request so + proper integration testing can be scheduled. + + In most cases you will see changes in the `pyproject.toml` and `poetry.lock` files. + -```shell script -$ cd -$ git config blame.ignoreRevsFile .git-blame-ignore-revs -``` diff --git a/README.orig.md b/README.orig.md index b9889a777..40c7d32b7 100644 --- a/README.orig.md +++ b/README.orig.md @@ -256,27 +256,21 @@ lxml is incompatible with mod_wsgi's use of Python sub-interpreters UI development setup ==================== -For UI development purposes only, install: +Reqires: -- NodeJS (recommended to install from NodeSource repository) -- npm (installed automatically with NodeJS) -- Ruby (for the scss-lint plugin) +- Node +- npm -To install all gulp plugins (this relies on files gulpfile.js and -packages.json in .../SITE_ROOT/PROJECT_ROOT): +CD to ezid repository root. - cd .../SITE_ROOT/PROJECT_ROOT - npm update --save-dev +Install UI toolkit: -Then: + npm install - gem install scss_lint - npm install gulp - -For development, to make changes to static elements, components, and +To make changes to static elements, components, and layouts: - gulp + npm run ui:watch This starts a local browser with the index page at http://localhost:3000/, and will reload HTML, CSS, and JavaScript in @@ -284,10 +278,8 @@ the browser immediately upon any changes to these files. To create a build: - gulp build - -This writes static webpages to .../SITE_ROOT/PROJECT_ROOT/ui_library. + npm run ui:build -To validate the HTML of a build using the W3C validation service: +This compiles CSS, JS and static webpages to [ezid repository root]/ui_library. - gulp validateHTML +Copy the compiled HTML, CSS and JS from ui_library into their locations within the ezid app. diff --git a/ansible/files/etc/httpd/conf.modules.d/00-ssl.conf b/ansible/files/etc/httpd/conf.modules.d/00-ssl.conf deleted file mode 100644 index 53235cd76..000000000 --- a/ansible/files/etc/httpd/conf.modules.d/00-ssl.conf +++ /dev/null @@ -1 +0,0 @@ -LoadModule ssl_module modules/mod_ssl.so diff --git a/ansible/group_vars/all b/ansible/group_vars/all index 5d8895b21..749265f34 100644 --- a/ansible/group_vars/all +++ b/ansible/group_vars/all @@ -10,6 +10,7 @@ virtual_environments: ensure: present pyenv_root: "{{ app_dir }}/.pyenv" pyenv_global: ezid-py38 +ansible_python_interpreter: "{{ pyenv_root }}/shims/python" ezid_repo: https://github.com/CDLUC3/ezid.git ezid_version: master @@ -26,7 +27,7 @@ http_protocol: http # AWS SSM Params # Gather any environment specific application settings from AWS SSM ParameterStore -# using aws_ssm pluggin +# using aws_ssm plugin # aws_region: "{{ lookup('env', 'AWS_REGION') | default('us-west-2') }}" ssm_default_path: "{{ lookup('env', 'SSM_DEFAULT_PATH') | default('/no/path/') }}" @@ -82,9 +83,6 @@ admin_username: "{{ ssm_params['admin_username'] }}" allocator_cdl_password: "{{ ssm_params['allocator_cdl_password'] }}" allocator_purdue_password: "{{ ssm_params['allocator_purdue_password'] }}" -binder_url: "{{ ssm_params['binder_url'] }}" -binder_username: "{{ ssm_params['binder_username'] }}" -binder_password: "{{ ssm_params['binder_password'] }}" cloudwatch_instance_name: "{{ ansible_facts.hostname }}" crossref_username: "{{ ssm_params['crossref_username'] }}" crossref_password: "{{ ssm_params['crossref_password'] }}" @@ -102,7 +100,14 @@ django_debug: "{{ ssm_params['django_debug'] }}" ezid_base_url: "{{ ssm_params['ezid_base_url']}}" email_new_account: "{{ ssm_params['email_new_account'] }}" link_checker_admin: "{{ ssm_params['link_checker_admin'] }}" +log_level: "{{ ssm_params['log_level'] }}" +matomo_site_id: "{{ ssm_params['matomo_site_id'] }}" +opensearch_base: "{{ ssm_params['opensearch_base'] }}" +opensearch_index: "{{ ssm_params['opensearch_index'] }}" +opensearch_user: "{{ ssm_params['opensearch_user'] }}" +opensearch_password: "{{ ssm_params['opensearch_password'] }}" resolver_doi: "{{ ssm_params['resolver_doi'] }}" resolver_ark: "{{ ssm_params['resolver_ark'] }}" test_shoulder_dict: "{{ ssm_params['test_shoulder_dict'] }}" secret_key: "{{ ssm_params['secret_key'] }}" +s3_bucket: "{{ ssm_params['s3_bucket'] }}" diff --git a/ansible/notes/notes.migrating_to_poetry b/ansible/notes/notes.migrating_to_poetry new file mode 100644 index 000000000..97f48327b --- /dev/null +++ b/ansible/notes/notes.migrating_to_poetry @@ -0,0 +1,237 @@ +________________________________________________________________________________ +Thu May 23 12:37:59 PM PDT 2024 + +from: https://nanthony007.medium.com/stop-using-pip-use-poetry-instead-db7164f4fc72 + +When using poetry inside containers I recommend installing from pip (I know, I +said not to, but I prefer this over curl-ing and piping something into my +production container). I do recommend locking the poetry version: pip install +poetry=1.1.4 that way you have stability there. I then disable the virtualenv +creation (since we don’t need it inside containers) by running poetry config +virtualenvs.create false after the poetry install. Then the container can copy +our two poetry files (poetry.lock and pyproject.toml) and run poetry install to +install our locked dependencies just as we had them! Furthermore, we now no +longer need to preface our run commands with poetry run since we disabled the +virtual environment! + + + + + + +________________________________________________________________________________ +5/17/2024 + +useful pages from https://packaging.python.org + +https://packaging.python.org/en/latest/overview/ +https://packaging.python.org/en/latest/flow/ +https://packaging.python.org/en/latest/tutorials/packaging-projects/ +https://packaging.python.org/en/latest/guides/installing-stand-alone-command-line-tools/ +https://packaging.python.org/en/latest/guides/writing-pyproject-toml/ +https://packaging.python.org/en/latest/guides/modernize-setup-py-project/ +https://packaging.python.org/en/latest/discussions/install-requires-vs-requirements/ +https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/ +https://packaging.python.org/en/latest/discussions/package-formats/ +https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/ + + + + +This is how I got scotts forked package to resolve: +poetry add git+https://github.com/CDLUC3/django-matomo-api-tracking@0.1.4 + + +________________________________________________________________________________ +Looking at console scripts now. This is how setup.py generates console scripts: + +agould@localhost:~/git/github/cdluc3/ezid> python +Python 3.11.9 (main, May 16 2024, 13:08:52) [GCC 7.5.0] on linux +Type "help", "copyright", "credits" or "license" for more information. +>>> import pathlib +>>> HERE_PATH = pathlib.Path(__file__).parent +Traceback (most recent call last): + File "", line 1, in +NameError: name '__file__' is not defined. Did you mean: '__name__'? +>>> HERE_PATH = pathlib.Path(__name__).parent +>>> def gen_console_scripts(): +... """Generate command line stubs for the modules in the tools folders.""" +... tool_path = HERE_PATH / 'tools' +... stub_list = [] +... for p in tool_path.glob('*.py'): +... if p.name.startswith('_'): +... continue +... module_name = p.with_suffix('').name +... tool_name = 'ez-{}'.format(module_name.replace('_', '-')) +... stub_name = "{}=tools.{}:main".format( tool_name, module_name) +... stub_list.append(stub_name) +... return stub_list +... +>>> gen_console_scripts + +>>> gen_console_scripts() +['ez-batch-register=tools.batch_register:main', 'ez-client=tools.client:main', 'ez-dashboard=tools.dashboard:main', 'ez-decode-file=tools.decode_file:main', 'ez-dump-convert=tools.dump_convert:main', 'ez-dump-crossref=tools.dump_crossref:main', 'ez-dump-datacite=tools.dump_datacite:main', 'ez-dump-diff=tools.dump_diff:main', 'ez-dump-diff-crossref=tools.dump_diff_crossref:main', 'ez-dump-filter=tools.dump_filter:main', 'ez-dump-project=tools.dump_project:main', 'ez-dump-store=tools.dump_store:main', 'ez-filterlog=tools.filterlog:main', 'ez-queue-admin=tools.queue_admin:main', 'ez-shoulderless-identifiers=tools.shoulderless_identifiers:main', 'ez-link-check-emailer=tools.link_check_emailer:main', 'ez-link-check-summary-report=tools.link_check_summary_report:main', 'ez-downserver=tools.downserver:main'] +>>> + +>>> pprint.pp(scripts) +['ez-batch-register=tools.batch_register:main', + 'ez-client=tools.client:main', + 'ez-dashboard=tools.dashboard:main', + 'ez-decode-file=tools.decode_file:main', + 'ez-dump-convert=tools.dump_convert:main', + 'ez-dump-crossref=tools.dump_crossref:main', + 'ez-dump-datacite=tools.dump_datacite:main', + 'ez-dump-diff=tools.dump_diff:main', + 'ez-dump-diff-crossref=tools.dump_diff_crossref:main', + 'ez-dump-filter=tools.dump_filter:main', + 'ez-dump-project=tools.dump_project:main', + 'ez-dump-store=tools.dump_store:main', + 'ez-filterlog=tools.filterlog:main', + 'ez-queue-admin=tools.queue_admin:main', + 'ez-shoulderless-identifiers=tools.shoulderless_identifiers:main', + 'ez-link-check-emailer=tools.link_check_emailer:main', + 'ez-link-check-summary-report=tools.link_check_summary_report:main', + 'ez-downserver=tools.downserver:main'] + + +I don't see how we can generate this list for consumption by poetry. Do we even use these? + +So I just hacked them all in: + +[tool.poetry.scripts] +ez-batch-register = 'tools.batch_register:main' +ez-client = 'tools.client:main' +ez-dashboard = 'tools.dashboard:main' +ez-decode-file = 'tools.decode_file:main' +ez-dump-convert = 'tools.dump_convert:main' +ez-dump-crossref = 'tools.dump_crossref:main' +ez-dump-datacite = 'tools.dump_datacite:main' +ez-dump-diff = 'tools.dump_diff:main' +ez-dump-diff-crossref = 'tools.dump_diff_crossref:main' +ez-dump-filter = 'tools.dump_filter:main' +ez-dump-project = 'tools.dump_project:main' +ez-dump-store = 'tools.dump_store:main' +ez-filterlog = 'tools.filterlog:main' +ez-queue-admin = 'tools.queue_admin:main' +ez-shoulderless-identifiers = 'tools.shoulderless_identifiers:main' +ez-link-check-emailer = 'tools.link_check_emailer:main' +ez-link-check-summary-report = 'tools.link_check_summary_report:main' +ez-downserver = 'tools.downserver:main' + + + + + + + + + + +________________________________________________________________________________ +5/16/2024 + +Name: poetry +Version: 1.8.3 +Summary: Python dependency management and packaging made easy. +Home-page: https://python-poetry.org/ +Author: Sébastien Eustace +Author-email: sebastien@eustace.io +License: MIT +Location: /ezid/.pyenv/versions/3.11.9/lib/python3.11/site-packages +Requires: build, cachecontrol, cleo, crashtest, dulwich, fastjsonschema, installer, keyring, packaging, pexpect, pkginfo, platformdirs, poetry-core, poetry-plugin-export, pyproject-hooks, requests, requests-toolbelt, shellingham, tomlkit, trove-classifiers, virtualenv +Required-by: poetry-plugin-export + + +ezid@uc3-ezidui-dev01:09:20:41:~/install/ezid/ansible$ which poetry +/usr/bin/which: no poetry in (/ezid/.pyenv/shims:/ezid/.pyenv/bin:/ezid/.local/bin:/ezid/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin) + +ezid@uc3-ezidui-dev01:09:22:46:~/install/ezid/ansible$ python -m poetry -h + +Description: + Lists commands. + +Usage: + list [options] [--] [] + +Arguments: + namespace The namespace name + +Options: + -h, --help Display help for the given command. When no command is given display help for the list command. + -q, --quiet Do not output any message. + -V, --version Display this application version. + --ansi Force ANSI output. + --no-ansi Disable ANSI output. + -n, --no-interaction Do not ask any interactive question. + --no-plugins Disables plugins. + --no-cache Disables Poetry source caches. + -C, --directory=DIRECTORY The working directory for the Poetry command (defaults to the current working directory). + -v|vv|vvv, --verbose Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug. + +Help: + The list command lists all commands: + + poetry list + + You can also display the commands for a specific namespace: + + poetry list test + + + + +________________________________________________________________________________ + + + +Starting with a clean pyenv +--------------------------- + +# update pyenv +cd /home/agould/.pyenv/plugins/python-build/../.. && git pull && cd - + +# install python 3.11.9 + 1012 pyenv install 3.11.9 + 1013 pyenv versions + 1017 pyenv global 3.11.9 + 1032 pip list + 1033 pip install -U setuptools + 1034 pip install -U pip + +# install pipx + 1035 pip install pipx + 1036 pip list + 1037 pipx list + +# install poetry + 1072 pipx install poetry + 1073 pipx list + 1075 which poetry + 1076 cat $(which poetry) + 1077 poetry -h + +cat /home/agould/.local/share/pipx/venvs/poetry/pipx_metadat.json + +cat /home/agould/.local/share/pipx/venvs/poetry/pyvenv.cfg +home = /home/agould/.pyenv/versions/3.11.9/bin +include-system-site-packages = false +version = 3.11.9 +executable = /home/agould/.pyenv/versions/3.11.9/bin/python3.11 +command = /home/agould/.pyenv/versions/3.11.9/bin/python3.11 -m venv --without-pip /home/agould/.local/share/pipx/venvs/poetry + + +agould@localhost:~/git/github/cdluc3/ezid> cat pyproject.toml +[tool.black] +skip_string_normalization = true +line_length = 100 + + +agould@localhost:~/git/github/cdluc3/ezid> poetry self show plugins + + - poetry-plugin-export (1.8.0) Poetry plugin to export the dependencies to various formats + 1 application plugin + + Dependencies + - poetry (>=1.8.0,<3.0.0) + - poetry-core (>=1.7.0,<3.0.0) + diff --git a/ansible/roles/ezid/tasks/configure_ezid.yaml b/ansible/roles/ezid/tasks/configure_ezid.yaml index 757fbd633..7f9074808 100644 --- a/ansible/roles/ezid/tasks/configure_ezid.yaml +++ b/ansible/roles/ezid/tasks/configure_ezid.yaml @@ -22,17 +22,30 @@ group: "{{ group }}" mode: "0600" -- name: run setup.py +- name: install poetry + ansible.builtin.pip: + name: poetry==1.8.5 + +- name: run pip install ansible.builtin.command: chdir: "{{ app_dir }}/ezid" argv: - - "{{ pyenv_root }}/versions/{{ pyenv_global }}/bin/python" - - "{{ app_dir }}/ezid/setup.py" - - develop + - "{{ pyenv_root }}/versions/{{ pyenv_global }}/bin/pip" + - install + - . + register: _response1 + +- name: validate poetry.lock + ansible.builtin.command: + chdir: "{{ app_dir }}/ezid" + argv: + - "{{ pyenv_root }}/versions/{{ pyenv_global }}/bin/poetry" + - check + - --lock register: _response1 - name: run pyenv rehash - ansible.builtin.command: pyenv rehash + ansible.builtin.command: "{{ pyenv_root }}/bin/pyenv rehash" - name: run manage.py collectstatic ansible.builtin.command: @@ -43,26 +56,17 @@ - collectstatic - --clear - --no-input + - --settings=settings.settings register: _response2 -#- when: _response1.rc is defined -# block: -# -# - name: report setup.py response std_out -# debug: -# msg="{{ _response1.stdout_lines }}" -# -# - name: report setup.py response std_err -# debug: -# msg="{{ _response1.stderr_lines }}" +- name: report poetry.lock validation + debug: + msg: "{{ _response1.stdout_lines }}" + when: _response1.rc is defined -#- when: _response2.rc is defined -# block: +## too noisy # -# - name: report manage.py response std_out -# debug: -# msg="{{ _response2.stdout_lines }}" -# -# - name: report manage.py response std_err -# debug: -# msg="{{ _response2.stderr_lines }}" +#- name: report `manage.py collectstatic` std_out +# debug: +# msg: "{{ _response2.stdout_lines }}" +# when: _response2.rc is defined diff --git a/ansible/templates/etc/httpd/conf.d/03-ezid-nossl.conf.j2 b/ansible/templates/etc/httpd/conf.d/03-ezid-nossl.conf.j2 index 5ac90c8b4..464a32b54 100644 --- a/ansible/templates/etc/httpd/conf.d/03-ezid-nossl.conf.j2 +++ b/ansible/templates/etc/httpd/conf.d/03-ezid-nossl.conf.j2 @@ -38,13 +38,6 @@ Listen {{ http_port }} Require all granted - - Alias /download {{ app_dir }}/var/www/download/public - - Require all granted - Options -Indexes - ErrorDocument 404 /download_error - Alias /robots.txt {{ app_dir }}/var/www/html/robots.txt Alias /sitemap.xml {{ app_dir }}/ezid/static/sitemap.xml diff --git a/ansible/test_vars.yaml b/ansible/test_vars.yaml index 88414f113..5ec6ef080 100644 --- a/ansible/test_vars.yaml +++ b/ansible/test_vars.yaml @@ -37,8 +37,6 @@ - name: return a single param based on ssm_root_path as variable ##debug: msg="{{ ssm_params['database_host'] }}" debug: msg="database_host {{ database_host }}" - - name: return binder_url - debug: msg="{{ binder_url }}" - name: return resolver_ark debug: msg="{{ resolver_ark }}" diff --git a/dev/README b/dev/README.md similarity index 100% rename from dev/README rename to dev/README.md diff --git a/dev/component_accordion.html b/dev/component_accordion.html index eefe7cffc..19b866536 100755 --- a/dev/component_accordion.html +++ b/dev/component_accordion.html @@ -23,8 +23,8 @@

Accordion Component

- - + + diff --git a/dev/component_citation.html b/dev/component_citation.html index 91ac519fe..e9125ca3a 100755 --- a/dev/component_citation.html +++ b/dev/component_citation.html @@ -23,8 +23,8 @@

Citation

- - + + diff --git a/dev/component_cite.html b/dev/component_cite.html index 650bf5759..3f180df72 100755 --- a/dev/component_cite.html +++ b/dev/component_cite.html @@ -23,8 +23,8 @@

Cite

- - + + diff --git a/dev/component_customize-table.html b/dev/component_customize-table.html index 5ae7b1ac3..d7ddbd6b5 100755 --- a/dev/component_customize-table.html +++ b/dev/component_customize-table.html @@ -23,8 +23,8 @@

Customize Table Component

- - + + diff --git a/dev/component_download.html b/dev/component_download.html index f834955b2..4e512bc2c 100755 --- a/dev/component_download.html +++ b/dev/component_download.html @@ -23,8 +23,8 @@

Download Component

- - + + diff --git a/dev/component_footer.html b/dev/component_footer.html index 4ae98628e..3c05d905b 100755 --- a/dev/component_footer.html +++ b/dev/component_footer.html @@ -29,8 +29,8 @@

Geolocation Off

- - + + diff --git a/dev/component_header.html b/dev/component_header.html index b26b5f060..b092a6eaa 100755 --- a/dev/component_header.html +++ b/dev/component_header.html @@ -23,8 +23,8 @@

Header Component with Login Modal

- - + + diff --git a/dev/component_home-banner.html b/dev/component_home-banner.html index 6633557e6..83b104a61 100755 --- a/dev/component_home-banner.html +++ b/dev/component_home-banner.html @@ -23,8 +23,8 @@

Home Banner Component

- - + + diff --git a/dev/component_identifier-boxes.html b/dev/component_identifier-boxes.html index 2dfe1acba..73de4c27b 100755 --- a/dev/component_identifier-boxes.html +++ b/dev/component_identifier-boxes.html @@ -23,8 +23,8 @@

Identifier Boxes Component

- - + + diff --git a/dev/component_login-menu.html b/dev/component_login-menu.html index 4c4f5ebf5..a7f4ee125 100755 --- a/dev/component_login-menu.html +++ b/dev/component_login-menu.html @@ -23,8 +23,8 @@

Login Menu Component

- - + + diff --git a/dev/component_login-modal.html b/dev/component_login-modal.html index dc68123f7..59725f87d 100755 --- a/dev/component_login-modal.html +++ b/dev/component_login-modal.html @@ -33,8 +33,8 @@

Login Modal Component

- - + + diff --git a/dev/component_pagination.html b/dev/component_pagination.html index fc185aa27..b6d4cd1cb 100755 --- a/dev/component_pagination.html +++ b/dev/component_pagination.html @@ -23,8 +23,8 @@

Pagination Component

- - + + diff --git a/dev/component_share.html b/dev/component_share.html index 17fe86e69..4a70137a3 100755 --- a/dev/component_share.html +++ b/dev/component_share.html @@ -23,8 +23,8 @@

Share Component

- - + + diff --git a/dev/component_showhide.html b/dev/component_showhide.html index b000192da..e3ddf9c3c 100755 --- a/dev/component_showhide.html +++ b/dev/component_showhide.html @@ -23,8 +23,8 @@

Show / Hide Component

- - + + diff --git a/dev/component_sidebox.html b/dev/component_sidebox.html index 11353d526..765ecb829 100755 --- a/dev/component_sidebox.html +++ b/dev/component_sidebox.html @@ -23,8 +23,8 @@

Side Box Component

- - + + diff --git a/dev/component_tab.html b/dev/component_tab.html index 496b17005..c4aece4c2 100755 --- a/dev/component_tab.html +++ b/dev/component_tab.html @@ -23,8 +23,8 @@

Tab Component

- - + + diff --git a/dev/component_table.html b/dev/component_table.html index 94139ac6d..98a83eb50 100755 --- a/dev/component_table.html +++ b/dev/component_table.html @@ -23,8 +23,8 @@

Table Component

- - + + diff --git a/dev/component_table2.html b/dev/component_table2.html index e74edf3bc..7c927483a 100755 --- a/dev/component_table2.html +++ b/dev/component_table2.html @@ -25,8 +25,8 @@

Scrollable Table Component

- - + + diff --git a/dev/component_table3.html b/dev/component_table3.html index ccf307606..eeac26c88 100755 --- a/dev/component_table3.html +++ b/dev/component_table3.html @@ -23,8 +23,8 @@

Stackable Table v.2

- - + + diff --git a/dev/component_testing.html b/dev/component_testing.html index aaee346fe..c04a21a87 100755 --- a/dev/component_testing.html +++ b/dev/component_testing.html @@ -23,8 +23,8 @@

Testing

- - + + diff --git a/dev/element_buttons.html b/dev/element_buttons.html index 70dbe38df..c04ecd61b 100755 --- a/dev/element_buttons.html +++ b/dev/element_buttons.html @@ -23,8 +23,8 @@

Button Elements

- - + + diff --git a/dev/element_fieldsets.html b/dev/element_fieldsets.html index 4c64fe612..ae26b9b88 100755 --- a/dev/element_fieldsets.html +++ b/dev/element_fieldsets.html @@ -23,8 +23,8 @@

Fieldsets

- - + + diff --git a/dev/element_form-controls.html b/dev/element_form-controls.html index bda3c0dca..3f7e87801 100755 --- a/dev/element_form-controls.html +++ b/dev/element_form-controls.html @@ -23,8 +23,8 @@

Form Controls

- - + + diff --git a/dev/element_headings.html b/dev/element_headings.html index 7201320fa..999460080 100755 --- a/dev/element_headings.html +++ b/dev/element_headings.html @@ -23,8 +23,8 @@

Heading Elements

- - + + diff --git a/dev/element_links.html b/dev/element_links.html index 7772e31a5..98a8cfa69 100755 --- a/dev/element_links.html +++ b/dev/element_links.html @@ -23,8 +23,8 @@

Link Elements

- - + + diff --git a/dev/images/icon_cross.svg b/dev/images/icon_cross.svg old mode 100755 new mode 100644 index ba2b3cc1f..a037959bd --- a/dev/images/icon_cross.svg +++ b/dev/images/icon_cross.svg @@ -1,14 +1,3 @@ - - - - - - - - + + + diff --git a/dev/includes/accordion.html b/dev/includes/accordion.html index a361b5518..d8412a3b3 100755 --- a/dev/includes/accordion.html +++ b/dev/includes/accordion.html @@ -15,13 +15,6 @@ -
- Pricing - -

Lorem ipsum dolor sit amet, consectetur adipisicing elit. Laudantium cumque ex, pariatur autem, numquam veniam, ipsam accusantium hic officia qui doloremque soluta, minus cupiditate obcaecati est temporibus expedita at eligendi.

- -
-
Documentation @@ -29,18 +22,4 @@
-
- FAQ - -

Lorem ipsum dolor sit amet, consectetur adipisicing elit. Voluptas molestias quidem quae reprehenderit eveniet fuga sapiente, temporibus aspernatur obcaecati quia illum similique, impedit soluta reiciendis non architecto expedita nesciunt optio!

- -
- -
- Media - -

Lorem ipsum dolor sit amet, consectetur adipisicing elit. Asperiores quam deleniti obcaecati perferendis earum itaque necessitatibus accusantium consequuntur labore in blanditiis illo, corrupti amet maiores! Consequuntur et maxime quia ratione.

- -
- diff --git a/dev/includes/footer_geo-off.html b/dev/includes/footer_geo-off.html index ae9e1fb23..0ac8f1dfd 100755 --- a/dev/includes/footer_geo-off.html +++ b/dev/includes/footer_geo-off.html @@ -19,7 +19,7 @@
- California Digital Library + California Digital Library
diff --git a/dev/includes/footer_geo-on.html b/dev/includes/footer_geo-on.html index 8822084cf..18b2fad81 100755 --- a/dev/includes/footer_geo-on.html +++ b/dev/includes/footer_geo-on.html @@ -25,7 +25,7 @@
- California Digital Library + California Digital Library
diff --git a/dev/includes/header.html b/dev/includes/header.html index a71cbc18c..677694107 100755 --- a/dev/includes/header.html +++ b/dev/includes/header.html @@ -18,17 +18,15 @@ Show or hide navigation -