Skip to content

Commit

Permalink
Catch sqlalchemy's DatabaseError in fetch and gather callback
Browse files Browse the repository at this point in the history
I sometimes see "connection timed out" message which are reported as
sqlalchemy.exc.DatabaseError, so by catching the latter exception, it'd
avoid the harvester to be stuck in "limbo" state.

As DatabaseError is a super-class of OperationalError, the latter would
still be catched.
  • Loading branch information
Denis Laxalde committed Nov 2, 2017
1 parent cc44d03 commit 7bb9a2b
Showing 1 changed file with 4 additions and 2 deletions.
6 changes: 4 additions & 2 deletions ckanext/harvest/queue.py
Original file line number Diff line number Diff line change
Expand Up @@ -274,9 +274,10 @@ def gather_callback(channel, method, header, body):

try:
job = HarvestJob.get(id)
except sqlalchemy.exc.OperationalError:
except sqlalchemy.exc.DatabaseError:
# Occasionally we see: sqlalchemy.exc.OperationalError
# "SSL connection has been closed unexpectedly"
# or DatabaseError "connection timed out"
log.exception('Connection Error during gather of job %s', id)
# By not sending the ack, it will be retried later.
# Try to clear the issue with a remove.
Expand Down Expand Up @@ -374,9 +375,10 @@ def fetch_callback(channel, method, header, body):

try:
obj = HarvestObject.get(id)
except sqlalchemy.exc.OperationalError:
except sqlalchemy.exc.DatabaseError:
# Occasionally we see: sqlalchemy.exc.OperationalError
# "SSL connection has been closed unexpectedly"
# or DatabaseError "connection timed out"
log.exception('Connection Error during fetch of job %s', id)
# By not sending the ack, it will be retried later.
# Try to clear the issue with a remove.
Expand Down

0 comments on commit 7bb9a2b

Please sign in to comment.