feat: Add worker job timeout

[eelkevdbos]

Context
A timeout for jobs would provide the guarantee that a worker does not get stuck on a single job. On top of that, long running scheduled jobs can start to overlap if the scheduled frequency is higher that the job assumed duration is.

Current workaround

import signal
from functools import wraps

class TimeoutException(Exception):
    pass

def timeout(seconds: int):
    def decorator(func):
        def _handle_timeout(signum, frame):
            raise TimeoutException(f"Function '{func.__name__}' timed out after {seconds} seconds")

        @wraps(func)
        def wrapper(*args, **kwargs):
            # Set the signal handler and a timer
            signal.signal(signal.SIGALRM, _handle_timeout)
            signal.alarm(seconds)
            try:
                result = func(*args, **kwargs)
            finally:
                signal.alarm(0)  # Cancel the timer
            return result
        return wrapper
    return decorator


# jobs.py
from plain.worker import register_job, Job
class LongRunningJob(Job):
  @timeout(300)
  def run(self): ...

Proposed interface

from plain.worker import register_job, Job

class LongRunningJob(Job):
  # setting timeout via class prop
  timeout :int = 300

  # providing override via getter
  def get_timeout():
    return 300

  def run(self): ...

[davegaeddert]

Thanks! I can see how this could make sense. I will have to give it a little more thought but maybe the WORKER_JOBS_LOST_AFTER should actually be something like WORKER_JOBS_DEFAULT_TIMEOUT, and we add the option to set per-job timeouts like you're suggesting.