[12.x] Improve Queues and Horizon documentation

horizon.md

@@ -3,9 +3,15 @@
 - [Introduction](#introduction)
 - [Installation](#installation)
     - [Configuration](#configuration)
-    - [Balancing Strategies](#balancing-strategies)
     - [Dashboard Authorization](#dashboard-authorization)
+    - [Max Job Attempts](#max-job-attempts)
+    - [Job Timeout](#job-timeout)
+    - [Job Backoff](#job-backoff)
     - [Silenced Jobs](#silenced-jobs)
+- [Balancing Strategies](#balancing-strategies)
+    - [Auto Balancing](#auto-balancing)
+    - [Simple Balancing](#simple-balancing)
+    - [No Balancing](#no-balancing)
 - [Upgrading Horizon](#upgrading-horizon)
 - [Running Horizon](#running-horizon)
     - [Deploying Horizon](#deploying-horizon)
@@ -123,41 +129,6 @@ While your application is in [maintenance mode](/docs/{{version}}/configuration#
 
 Within Horizon's default configuration file, you will notice a `defaults` configuration option. This configuration option specifies the default values for your application's [supervisors](#supervisors). The supervisor's default configuration values will be merged into the supervisor's configuration for each environment, allowing you to avoid unnecessary repetition when defining your supervisors.
 
-<a name="balancing-strategies"></a>
-### Balancing Strategies
-
-Unlike Laravel's default queue system, Horizon allows you to choose from three worker balancing strategies: `simple`, `auto`, and `false`. The `simple` strategy splits incoming jobs evenly between worker processes:
-
-    'balance' => 'simple',
-
-The `auto` strategy, which is the configuration file's default, adjusts the number of worker processes per queue based on the current workload of the queue. For example, if your `notifications` queue has 1,000 pending jobs while your `render` queue is empty, Horizon will allocate more workers to your `notifications` queue until the queue is empty.
-
-When using the `auto` strategy, you may define the `minProcesses` and `maxProcesses` configuration options to control the minimum number of processes per queue and the maximum number of worker processes in total Horizon should scale up and down to:
-
-```php
-'environments' => [
-    'production' => [
-        'supervisor-1' => [
-            'connection' => 'redis',
-            'queue' => ['default'],
-            'balance' => 'auto',
-            'autoScalingStrategy' => 'time',
-            'minProcesses' => 1,
-            'maxProcesses' => 10,
-            'balanceMaxShift' => 1,
-            'balanceCooldown' => 3,
-            'tries' => 3,
-        ],
-    ],
-],
-```
-
-The `autoScalingStrategy` configuration value determines if Horizon will assign more worker processes to queues based on the total amount of time it will take to clear the queue (`time` strategy) or by the total number of jobs on the queue (`size` strategy).
-
-The `balanceMaxShift` and `balanceCooldown` configuration values determine how quickly Horizon will scale to meet worker demand. In the example above, a maximum of one new process will be created or destroyed every three seconds. You are free to tweak these values as necessary based on your application's needs.
-
-When the `balance` option is set to `false`, the default Laravel behavior will be used, wherein queues are processed in the order they are listed in your configuration.
-
 <a name="dashboard-authorization"></a>
 ### Dashboard Authorization
 
@@ -184,6 +155,82 @@ protected function gate(): void
 
 Remember that Laravel automatically injects the authenticated user into the gate closure. If your application is providing Horizon security via another method, such as IP restrictions, then your Horizon users may not need to "login". Therefore, you will need to change `function (User $user)` closure signature above to `function (User $user = null)` in order to force Laravel to not require authentication.
 
+<a name="max-job-attempts"></a>
+### Max Job Attempts
+
+> [!NOTE]
+> Before refining these options, make sure you are familiar with Laravel's default [queue services](/docs/{{version}}/queues#max-job-attempts-and-timeout) and the concept of 'attempts'.
+
+You can define the maximum number of attempts a job can consume within a supervisor's configuration:
+
+```php
+'environments' => [
+    'production' => [
+        'supervisor-1' => [
+            // ...
+            'tries' => 10,
+        ],
+    ],
+],
+```
+
+> [!NOTE]
+> This option is similar to the `--tries` option when using the Artisan command to process queues.
+
+Adjusting the `tries` option is essential when using middlewares such as `WithoutOverlapping` or `RateLimited` because they consume attempts. To handle this, adjust the `tries` configuration value either at the supervisor level or by defining the `$tries` property on the job class.
+
+If you don’t set the `tries` option, Horizon defaults to a single attempt, unless the job class defines `$tries`, which takes precedence over the Horizon configuration.
+
+Setting `tries` or `$tries` to 0 allows unlimited attempts, which is ideal when the number of attempts is uncertain. To prevent endless failures, you can limit the number of exceptions allowed by setting the `$maxExceptions` property on the job class.
+
+<a name="job-timeout"></a>
+### Job Timeout
+
+Similarly, you can set a `timeout` value at the supervisor level, which specifies how many seconds a worker process can run a job before it's forcefully terminated. Once terminated, the job will either be retried or marked as failed, depending on your queue configuration:
+
+```php
+'environments' => [
+    'production' => [
+        'supervisor-1' => [
+            // ...¨
+            'timeout' => 60,
+        ],
+    ],
+],
+```
+
+> [!WARNING]
+> The `timeout` value should always be at least a few seconds shorter than the `retry_after` value defined in your `config/queue.php` configuration file. Otherwise, your jobs may be processed twice.
+
+<a name="job-backoff"></a>
+### Job Backoff
+
+You can define the `backoff` value at the supervisor level to specify how long Horizon should wait before retrying a job that encounters an unhandled exception:
+
+```php
+'environments' => [
+    'production' => [
+        'supervisor-1' => [
+            // ...
+            'backoff' => 10,
+        ],
+    ],
+],
+```
+
+You may also configure "exponential" backoffs by using an array for the `backoff` value. In this example, the retry delay will be 1 second for the first retry, 5 seconds for the second retry, 10 seconds for the third retry, and 10 seconds for every subsequent retry if there are more attempts remaining:
+
+```php
+'environments' => [
+    'production' => [
+        'supervisor-1' => [
+            // ...
+            'backoff' => [1, 5, 10],
+        ],
+    ],
+],
+```
+
 <a name="silenced-jobs"></a>
 ### Silenced Jobs
 
@@ -208,6 +255,174 @@ class ProcessPodcast implements ShouldQueue, Silenced
 }
 ```
 
+<a name="balancing-strategies"></a>
+## Balancing Strategies
+
+Each supervisor can process one or more queues but unlike Laravel's default queue system, Horizon allows you to choose from three worker balancing strategies: `auto`, `simple`, and `false`.
+
+<a name="auto-balancing"></a>
+### Auto Balancing
+
+The `auto` strategy, which is the default strategy, adjusts the number of worker processes per queue based on the current workload of the queue. For example, if your `notifications` queue has 1,000 pending jobs while your `default` queue is empty, Horizon will allocate more workers to your `notifications` queue until the queue is empty.
+
+When using the `auto` strategy, you may also configure the `minProcesses` and `maxProcesses` configuration options:
+
+<div class="content-list" markdown="1">
+
+- `minProcesses` defines the minimum number of worker processes per queue. This value must be greater than or equal to 1.
+- `maxProcesses` defines the maximum total number of worker processes Horizon may scale up to across all queues. This value should typically be greater than the number of queues multiplied by the `minProcesses` value. To prevent the supervisor from spawning any processes, you may set this value to 0.
+
+</div>
+
+For example, you may configure Horizon to maintain at least one process per queue and scale up to a total of 10 worker processes:
+
+```php
+'environments' => [
+    'production' => [
+        'supervisor-1' => [
+            'connection' => 'redis',
+            'queue' => ['default', 'notifications'],
+            'balance' => 'auto',
+            'autoScalingStrategy' => 'time',
+            'minProcesses' => 1,
+            'maxProcesses' => 10,
+            'balanceMaxShift' => 1,
+            'balanceCooldown' => 3,
+        ],
+    ],
+],
+```
+
+The `autoScalingStrategy` configuration option determines how Horizon will assign more worker processes to queues. You can choose between two strategies:
+
+<div class="content-list" markdown="1">
+
+- The `time` strategy will assign workers based on the total estimated amount of time it will take to clear the queue.
+- The `size` strategy will assign workers based on the total number of jobs on the queue.
+
+</div>
+
+The `balanceMaxShift` and `balanceCooldown` configuration values determine how quickly Horizon will scale to meet worker demand. In the example above, a maximum of one new process will be created or destroyed every three seconds. You are free to tweak these values as necessary based on your application's needs.
+
+<a name="auto-queue-priorities"></a>
+#### Queue Priorities and Auto Balancing
+
+When using the `auto` balancing strategy, Horizon does not enforce strict priority between queues. The order of queues in a supervisor's configuration does not affect how worker processes are assigned. Instead, Horizon relies on the selected `autoScalingStrategy` to dynamically allocate worker processes based on queue load.
+
+For example, in the following configuration, the high queue is not prioritized over the default queue, despite appearing first in the list:
+
+```php
+'environments' => [
+    'production' => [
+        'supervisor-1' => [
+            // ...
+            'queue' => ['high', 'default'],
+            'minProcesses' => 1,
+            'maxProcesses' => 10,
+        ],
+    ],
+],
+```
+
+If you need to enforce a relative priority between queues, you may define multiple supervisors and explicitly allocate processing resources:
+
+```php
+'environments' => [
+    'production' => [
+        'supervisor-1' => [
+            // ...
+            'queue' => ['default'],
+            'minProcesses' => 1,
+            'maxProcesses' => 10,
+        ],
+        'supervisor-2' => [
+            // ...
+            'queue' => ['images'],
+            'minProcesses' => 1,
+            'maxProcesses' => 1,
+        ],
+    ],
+],
+```
+
+In this example, the default `queue` can scale up to 10 processes, while the `images` queue is limited to one process. This configuration ensures that your queues can scale independently.
+
+> [!NOTE]
+> When dispatching resource-intensive jobs, it's sometimes best to assign them to a dedicated queue with a limited `maxProcesses` value. Otherwise, these jobs could consume excessive CPU resources and overload your system.
+
+<a name="simple-balancing"></a>
+### Simple Balancing
+
+The `simple` strategy distributes worker processes evenly across the specified queues. With this strategy, Horizon does not automatically scale the number of worker processes. Rather, it uses a fixed number of processes:
+
+```php
+'environments' => [
+    'production' => [
+        'supervisor-1' => [
+            // ...
+            'queue' => ['default', 'notifications'],
+            'balance' => 'simple',
+            'processes' => 10,
+        ],
+    ],
+],
+```
+
+In the example above, Horizon will assign 5 processes to each queue, splitting the total of 10 evenly.
+
+If you'd like to control the number of worker processes assigned to each queue individually, you can define multiple supervisors:
+
+```php
+'environments' => [
+    'production' => [
+        'supervisor-1' => [
+            // ...
+            'queue' => ['default'],
+            'balance' => 'simple',
+            'processes' => 10,
+        ],
+        'supervisor-notifications' => [
+            // ...
+            'queue' => ['notifications'],
+            'balance' => 'simple',
+            'processes' => 2,
+        ],
+    ],
+],
+```
+
+With this configuration, Horizon will assign 10 processes to the `default` queue and 2 processes to the `notifications` queue.
+
+<a name="no-balancing"></a>
+### No Balancing
+
+When the `balance` option is set to `false`, Horizon processes queues strictly in the order they're listed, similar to Laravel’s default queue system. However, it will still scale the number of worker processes if jobs begin to accumulate:
+
+```php
+'environments' => [
+    'production' => [
+        'supervisor-1' => [
+            // ...
+            'queue' => ['default', 'notifications'],
+            'balance' => false,
+            'minProcesses' => 1,
+            'maxProcesses' => 10,
+        ],
+    ],
+],
+```
+
+In the example above, jobs in the `default` queue are always prioritized over jobs in the `notifications` queue. For instance, if there are 1,000 jobs in `default` and only 10 in `notifications`, Horizon will fully process all `default` jobs before handling any from `notifications`.
+
+You can control Horizon's ability to scale worker processes using the `minProcesses` and `maxProcesses` options:
+
+<div class="content-list" markdown="1">
+
+- `minProcesses` defines the minimum number of worker processes in total. This value must be greater than or equal to 1.
+- `maxProcesses` defines the maximum total number of worker processes Horizon may scale up to.
+
+</div>
+
 <a name="upgrading-horizon"></a>
 ## Upgrading Horizon
 

queues.md

@@ -577,6 +577,8 @@ public function middleware(): array
 }
 ```
 
+Releasing an overlapping job back onto the queue will still increment the job's total number of attempts. You may wish to tune your `tries` and `maxExceptions` properties on your job class accordingly. For example, leaving the `tries` property to 1 as it is by default would prevent any overlapping job from being retried later.
+
 Any overlapping jobs of the same type will be released back to the queue. You may also specify the number of seconds that must elapse before the released job will be attempted again:
 
 ```php
@@ -1213,7 +1215,26 @@ class ProcessPodcast implements ShouldQueue
 <a name="max-attempts"></a>
 #### Max Attempts
 
-If one of your queued jobs is encountering an error, you likely do not want it to keep retrying indefinitely. Therefore, Laravel provides various ways to specify how many times or for how long a job may be attempted.
+Job attempts are a core concept of Laravel's queue system and power many advanced features. While they may seem confusing at first, it's important to understand how they work before modifying the default configuration.
+
+When a job is dispatched, it is pushed onto the queue. A worker then picks it up and attempts to execute it. This is a job attempt.
+
+However, an attempt does not necessarily mean the job's `handle` method was executed. Attempts can also be "consumed" in several ways:
+
+<div class="content-list" markdown="1">
+
+- The job encounters an unhandled exception during execution.
+- The job is manually released back to the queue using `$this->release()`.
+- Middleware such as `WithoutOverlapping` or `RateLimited` fails to acquire a lock and releases the job.
+- The job timed out.
+- The job's `handle` method runs and completes without throwing an exception.
+
+</div>
+
+You likely do not want to keep attempting a job indefinitely. Therefore, Laravel provides various ways to specify how many times or for how long a job may be attempted.
+
+> [!NOTE]
+> By default, Laravel will only attempt a job once. If your job uses middleware like `WithoutOverlapping` or `RateLimited`, or if you're manually releasing jobs, you will likely need to increase the number of allowed attempts via the `tries` option.
 
 One approach to specifying the maximum number of times a job may be attempted is via the `--tries` switch on the Artisan command line. This will apply to all jobs processed by the worker unless the job being processed specifies the number of times it may be attempted:
 
@@ -1370,6 +1391,9 @@ If you would like to indicate that a job should be marked as [failed](#dealing-w
 public $failOnTimeout = true;
 ```
 
+> [!NOTE]
+> By default, when a job times out, it consumes one attempt and is released back to the queue (if retries are allowed). However, if you configure the job to fail on timeout, it will not be retried, regardless of the value set for tries.
+
 <a name="error-handling"></a>
 ### Error Handling
 
@@ -2249,6 +2273,18 @@ class ProcessPodcast implements ShouldQueue
 > [!WARNING]
 > A new instance of the job is instantiated before invoking the `failed` method; therefore, any class property modifications that may have occurred within the `handle` method will be lost.
 
+A failed job is not necessarily one that encountered an unhandled exception. A job may also be considered failed when it has exhausted all of its allowed attempts. These attempts can be consumed in several ways:
+
+<div class="content-list" markdown="1">
+
+- The job timed out.
+- The job encounters an unhandled exception during execution.
+- The job is released back to the queue either manually or by a middleware.
+
+</div>
+
+If the final attempt fails due to an exception thrown during job execution, that exception will be passed to the job's failed method. However, if the job fails because it has reached the maximum number of allowed attempts, the `$exception` will be an instance of `Illuminate\Queue\MaxAttemptsExceededException`. Similarly, if the job fails due to exceeding the configured timeout, the `$exception` will be an instance of `Illuminate\Queue\TimeoutExceededException`.
+
 <a name="retrying-failed-jobs"></a>
 ### Retrying Failed Jobs