feat: add status for removing / removalfailed

[mjnagel]

Description

This PR utilizes the "new" ability in a Pepr finalizer to not remove the finalizer. This enables us to update the status while finalizing, and catch errors if cleanup does not work as expected. Changes:

• Skip finalizer if it's already running (based on status)
• Skip finalizer if Package isn't ready/failed yet (for Deleting Package before status is Ready does not properly clean up clients #963)
• Patch Removing status on the CR
• Catch errors on finalization and patch RemovalFailed status and create a failure event
• Retry each cleanup/purge function using retryWithDelay

I also updated the diagram to support these changes, as well as adding test cases for the finalizer function. Diagram update can be previewed on the docs by using this link on docs/reference/configuration/UDS operator/package.md, specific changes:

• Moved finalizer section to the right of reconciler
• Simplified flow of validator (to make more space in the diagram)
• Added new pieces of finalizer flow (failure, status patching, etc)

Related Issue

Fixes #963

Fixes #1159

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Other (security config, docs update, etc)

Steps to Validate

Testing Steps

Test setup:

# Install slim-dev (unicorn flavor to avoid pull rate limiting)
uds run slim-dev --set flavor=unicorn
# Create the test packages
zarf p create src/test --skip-sbom
# Deploy the test packages
zarf p deploy build/zarf-package-uds-core-test-apps-*.tar.zst --confirm
# Validate all package CRs go to Ready status
kubectl get pkg -A # should all show ready

Test that normal deletion works and makes events:

# Delete a package CR
kubectl delete pkg -n test-tenant-app test-tenant-app
# Validate success and events
kubectl get pkg -n test-tenant-app # should show no resources
kubectl get events -n test-tenant-app | grep package # should show 3 removal events

Test that finalizer doesn't run until CR is ready:

# This forces a re-reconcile of the package and then deletes immediately
# If you watch while this happens (k9s, etc) you should see it go to Pending before Removing
kubectl patch pkg httpbin-other -n authservice-test-app --subresource=status --type=json  -p='[{"op": "remove", "path": "/status"}]' && kubectl delete pkg httpbin-other -n authservice-test-app
# Validate that the watcher waited to finalize
kubectl logs -n pepr-system -l app=pepr-uds-core-watcher --tail=-1 | grep "Waiting"
kubectl get events -n authservice-test-app | grep package # should show 3 removal events

Test that finalizer places CR in RemovalFailed state on failed cleanup:

# Deploy the test apps again (we need the sso client)
zarf p deploy build/zarf-package-uds-core-test-apps-*.tar.zst --confirm
# Edit the peprstore
kubectl edit peprstore -n pepr-system pepr-uds-core-store
# Delete the line with `uds-core-operator-v2-sso-client-uds-core-httpbin`, this is the client token and will make Pepr unable to cleanup the client
# Save the peprstore
# Delete the package CR
kubectl delete pkg httpbin-other -n authservice-test-app
# Make sure that status is marked as RemovalFailed (after ~15 seconds)
kubectl get pkg httpbin-other -n authservice-test-app
# Make sure events show up that client failed to be removed
kubectl describe pkg httpbin-other -n authservice-test-app
# Make sure that the SSO client removal was retried 4 times before final failure
kubectl logs -n pepr-system -l app=pepr-uds-core-watcher --tail=-1 | grep "cleanupSSOClients"

Also note the automated jest unit tests and validate those.

Checklist before merging

• Test, docs, adr added or updated as needed
• Contributor Guide followed

[UnicornChance]

looks good to me (LGTM), ran through tests, everything is working as expected