EXECUTION_SWEEP never completing

[BojanZelic]

is there a way to debug why EXECUTION_SWEEP never completes?

I've tried the steps here #21283 (comment) to flush the redis db but nothing ever completes. The EXECUTION_SWEEP job never finishes. Are there any steps for debugging why this job never completes?

I don't see any relevant log entries for sweep that indicates any errors;

kubectl logs -n harbor harbor-jobservice-fd7666b57-ctwkl  | grep -i sweep
2025-01-28T20:46:32Z [INFO] [/jobservice/worker/cworker/c_worker.go:445]: Register job *task.SweepJob with name EXECUTION_SWEEP
2025-01-28T20:46:32Z [INFO] [/pkg/jobmonitor/redis.go:165]: unpause job EXECUTION_SWEEP
2025-01-28T20:46:32Z [INFO] [/jobservice/logger/sweeper_controller.go:121]: 1240 outdated log entries are sweepped by sweeper *sweeper.DBSweeper

[Vad1mo]

chnage the state in the DB

[stonezdj]

You should check it with the following steps.

1. Check if any worker is processing any task related to EXECUTION_SWEEP by checking the log button.
2. Select any running execution related to EXECUTION_SWEEP

docker exec -it harbor-db bash
psql -U postgres -d registry
select * from execution where vendor_type = 'EXECUTION_SWEEP' and status = 'RUNNING' order by start_time desc limit 10;
select job_id from task where execution_id = <execution_id>

Check the logs in /data/job_log/<job_id>.log
3. Because the queue latency is 52 hours, maybe the previous execution might be cleaned up and can not be found in the DB, you can just grep EXECUTION_SWEEP under /data/job_log to find the latest execution sweep job log.

[reasonerjt]

Please let us know what version of Harbor you are using.

[BojanZelic]

I was on the latest version v2.12.2... anyways, problem seems to be fixed thanks for the suggestions. I cleared out the existing executions and tasks;

 delete from task where execution_id IN (select id from execution where vendor_type = 'EXECUTION_SWEEP' and status = 'Running');
 delete from execution where vendor_type = 'EXECUTION_SWEEP' and status = 'Running';

and in-combination with clearing redis, seems to have resolved it.