Add docs for graceful shutdown (#2931)

docs/internals/graceful-shutdown.md

@@ -0,0 +1,37 @@
+# Shutdown Management
+In any application it is important to shut down gracefully, to avoid dropping already accepted work or 
+creating inconsistent state. Misk handles this through use of a special `ReadyService`
+and its [Service Manager Module](service-management.md)
+
+## Orchestrating Graceful Shutdown
+Misk ensures a graceful shutdown by dividing services into those that ingest or create work (e.g. SQS, Cron, Jetty), 
+and those that are needed to process work (e.g. JDBC, Redis). To ensure the work created by an incoming API request, 
+SQS subscription, cron job, or other work producing service is handled correctly even during shutdown, these services need
+
+1. To stop generating new work
+2. For the services they depend on to process their work to remain running until they have processed all existing work
+
+### Ensuring needed services remain running
+Because Misk cannot know ahead of time which services an application might or might not need, we cannot create 
+hard dependencies from these work producing services to the various services needed for work processing. Instead,
+we configure the work producing services to depend on - and services needed for work processing to be enhanced by -
+the `ReadyService`, a special service that does no work but exists only to orchestrate a graceful shutdown.
+
+By having work producing services depend on the `ReadyService` and work processing services enhanced by it, Misk will 
+guarantee that services startup as follows:
+
+1. Work processing services (e.g. Redis)
+2. The `ReadyService`
+3. Work generating services (e.g. Jetty)
+
+At shutdown time, we walk the dependency graph in reverse, shutting down services as follows:
+
+1. Work generating services (e.g. Jetty)
+2. The `ReadyService`
+3. Work processing services (e.g. Redis)
+
+This ensures services that are needed for work processing remain up until all ingested work has been processed.
+
+## Notes
+* The mechanism for enhancing one service with another 
+is [described in the service management doc](service-management.md#enhancements) 

docs/internals/service-management.md

@@ -0,0 +1,76 @@
+# Misk Services
+
+Services in Misk can depend on other services. We need to reconcile these dependencies to ensure an orderly application
+startup and [shutdown](graceful-shutdown.md)
+
+## Dependencies
+
+Suppose we have a `DatabaseService` and a `MovieService`, with the `MovieService` depending on
+the `DatabaseService`.
+
+ ```
+ DatabaseService
+   depended on by MovieService
+ ```
+
+When you install a service via this module, start-up and shut-down of its dependencies are
+handled automatically, so that a service can only run when the services it depends on are
+running. In the example above, the `MovieService` doesn't enter the `STARTING` state until the
+`DatabaseService` has entered the `RUNNING` state. Conversely, the `MovieService` must enter the
+`TERMINATED` state before the DatabaseService enters the `STOPPING` state.
+
+Dependencies can have their own dependencies, so there's an entire graph to manage of what starts
+and stops when.
+
+## Enhancements
+
+Some services exist to enhance the behavior of another service.
+
+For example, a `DatabaseService` may manage a generic connection to a MySQL database, and the
+`SchemaMigrationService` may create tables specific to the application.
+
+We treat such enhancements as implementation details of the enhanced service: they depend on the
+service, but downstream dependencies like the `MovieService` don't need to know that they exist.
+
+ ```
+ DatabaseService
+   enhanced by SchemaMigrationService
+   depended on by MovieService
+ ```
+
+In the above service graph we start the `DatabaseService` first, the `SchemaMigrationService`
+second, and finally the `MovieService`. The `MovieService` doesn't need to express a dependency
+on the `SchemaMigrationService`, that happens automatically for enhancements.
+
+## What does this look like?
+
+### Configuration
+
+Instead of using the regular service multi-bindings you might be used to, in the `configure`
+block of a Guice [KAbstractModule], you would set up the above relationship as follows:
+
+ ```kotlin
+ override fun configure() {
+   install(ServiceModule<SchemaMigrationService())
+   install(
+     ServiceModule<DatabaseService>()
+       .enhancedBy<SchemaMigrationService>()
+   )
+   install(
+     ServiceModule<MoviesService>()
+       .dependsOn<DatabaseService>()
+   )
+ }
+ ```
+
+### How does this work?
+
+Bindings are hooked up for a `ServiceManager` provider, which decorates the service with its
+dependencies and enhancements to defer its start up and shut down until its dependent services
+are ready.
+
+This service will stall in the `STARTING` state until all upstream services are `RUNNING`.
+Symmetrically it stalls in the `STOPPING` state until all dependent services are `TERMINATED`.
+
+## Notes
+* This doc was lifted from the doc string on the `ServiceModule` class

mkdocs.yml

@@ -67,6 +67,9 @@ nav:
       - 'misk-testing': 0.x/misk-testing/index.md
       - 'misk-zookeeper': 0.x/misk-zookeeper/index.md
       - 'misk-zoopkeeper-testing': 0.x/misk-zoopkeeper-testing/index.md
+  - 'Internals':
+      - 'Graceful Shutdown': internals/graceful-shutdown.md
+      - 'Service Management': internals/service-management.md
   - 'Change Log': changelog.md
   - 'Releasing': releasing.md
   - 'Manual Releasing': releasing-manual.md