How do you deal with service restarts?

[OrangeDog]

When launching jobs from a web service, the jobs are queued and run in the instance that started them.
However, web services can be restarted at any time, which will stop the processing of those jobs.
How do you ensure that running jobs are restarted when the service is restarted?

I tried this, based on this SO question, which does "work":

@Component
@RequiredArgsConstructor
public static class JobStartup {
    private final JobExplorer jobExplorer;
    private final JobOperator jobOperator;
    private final JobRepository jobRepository;

    @EventListener(ApplicationReadyEvent.class)
    public void restartIncompleteJobs() throws JobExecutionException {
        for (String jobName : jobExplorer.getJobNames()) {
            for (JobExecution execution : jobExplorer.findRunningJobExecutions(jobName)) {
                execution.getStepExecutions().stream()
                        .filter(step  -> step.getStatus().isRunning())
                        .forEach(step -> {
                            step.setStatus(BatchStatus.STOPPED);
                            jobRepository.update(step);
                        });
                execution.setStatus(BatchStatus.STOPPED);
                jobRepository.update(execution);
                jobOperator.restart(execution.getId());
            }
        }
    }
}

There are a number of problems with this hack:

1. It doesn't work if there are multiple service instances, as a "running" job may be running in a different instance.
2. It cannot be made atomic because restart runs its own transaction, which will not see the updates made before it.
3. It just feels wrong having to use three separate APIs to achieve "one" thing.

Is there a better way to ensure jobs continue across service restarts, or anything Spring Batch can add to do this properly?

[fmbenhassine]

How do you ensure that running jobs are restarted when the service is restarted?

There is no reliable way to do that. If jobs are terminated abruptly, their status remains STARTED in the job repository and Spring Batch cannot distinguish effectively running jobs from abruptly terminated ones. You have to mark those executions as failed or abandoned and restart them manually (see next point).

So to answer your question: It is not possible to ensure that "running" jobs (or more precisely, abruptly terminated jobs) are restarted on service restart.

There is an interesting thread about this here. And you might also check my experiment with abrupt shutdowns here.

What do you think? Does that help?

It just feels wrong having to use three separate APIs to achieve "one" thing.

We recently added an API for that: #4876

--

That said, there are some improvements in this area that are planned for v6 (#4431, #4023, #1530).