diff --git a/.DS_Store b/.DS_Store index 91e14dc..ca5efbb 100644 Binary files a/.DS_Store and b/.DS_Store differ diff --git a/SUMMARY.md b/SUMMARY.md index cc8bfa2..28f7161 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -1,214 +1,178 @@ -# Summary - -## Overview - -* [Show me the code!](/contents/ShowMeTheCode.md) - * [Dispatching A Request](/contents/ShowMeTheCode.md#dispatching-a-request) - * [Using an External Bus](/contents/ShowMeTheCode.md#using-an-external-bus) - * [Returning Query Results](/contents/ShowMeTheCode.md) -* [Basic Concepts](/contents/BasicConcepts.md) - -## Brighter Configuration - -* [Basic Configuration](/contents/BrighterBasicConfiguration.md) - * [Using .NET Core Dependency Injection](/contents/BrighterBasicConfiguration.md#using-net-core-dependency-injection) - * [Configuring the Command Processor](/contents/BrighterBasicConfiguration.md#configuring-the-command-processor) - *[Command Processor Service Collection Extensions](/contents/BrighterBasicConfiguration.md#command-processor-service-collection-extensions) - * [Adding Polly Policies](/contents/BrighterBasicConfiguration.md#adding-polly-policies) - * [Configuring Lifetimes](/contents/BrighterBasicConfiguration.md#configuring-lifetimes) - * [Brighter Builder Fluent Interface](/contents/BrighterBasicConfiguration.md#brighter-builder-fluent-interface) - * [Type Registration](/contents/BrighterBasicConfiguration.md#type-registration) - * [Using an External Bus](/contents/BrighterBasicConfiguration.md#using-an-external-bus) - * [Publications](/contents/BrighterBasicConfiguration.md#publications) - * [Transports & Gateways](/contents/BrighterBasicConfiguration.md#transports-and-gateways) - * [Transport NuGet Packages](/contents/BrighterBasicConfiguration.md#transport-nuget-packages) - * [Bus Example](/contents/BrighterBasicConfiguration.md#bus-example) - * [Retry and Circuit Breaker](/contents/BrighterBasicConfiguration.md#retry-and-circuit-breaker-with-an-external-bus) - * [Outbox Support](/contents/BrighterBasicConfiguration.md#outbox-support) - * [Putting It All Together](/contents/BrighterBasicConfiguration.md#putting-it-all-together) - * [Configuring the Service Activator](/contents/BrighterBasicConfiguration.md#configuring-the-service-activator) - * [ServiceActivator Service Collection Extensions](/contents/BrighterBasicConfiguration.md#serviceactivator-service-collection-extensions) - * [Subscriptions](/contents/BrighterBasicConfiguration.md#subscriptions) - * [Gateway Connections & Channel Factories](/contents/BrighterBasicConfiguration.md#gateway-connections--channel-factories) - * [Configuring Service Activator Lifetimes](/contents/BrighterBasicConfiguration.md#configuring-service-activator-lifetimes) - * [Service Activator Brighter Builder Fluent Interface](/contents/BrighterBasicConfiguration.md#service-activator-brighter-builder-fluent-interface) - *[Inbox](/contents/BrighterBasicConfiguration.md#inbox) - * [Running Service Activator](/contents/BrighterBasicConfiguration.md#running-service-activator) - * [A Complete Service Activator Example](/contents/BrighterBasicConfiguration.md#a-complete-service-activator-example) - * [Samples](/contents/BrighterBasicConfiguration.md#samples) -* [How Configuring the Command Processor Works](/contents/HowConfiguringTheCommandProcessorWorks.md) - * [What you need to provide](/contents/HowConfiguringTheCommandProcessorWorks.md#what-you-need-to-provide) - * [Subscriber Registry](/contents/HowConfiguringTheCommandProcessorWorks.md#subscriber-registry) - * [Handler Factory](/contents/HowConfiguringTheCommandProcessorWorks.md#handler-factory) - * [Policy Registry](/contents/HowConfiguringTheCommandProcessorWorks.md#policy-registry) - * [Request Context Factory](/contents/HowConfiguringTheCommandProcessorWorks.md#request-context-factory) - * [Putting it all together](/contents/HowConfiguringTheCommandProcessorWorksmd#putting-it-all-together) -* [How Configuring a Dispatcher for an External Bus Works](/contents/HowConfiguringTheDispatcherWorks.md) - * [Configuring the Dispatcher](/contents/HowConfiguringTheDispatcherWorks.md#configuring-the-dispatcher) - * [Message Mappers](/contents/HowConfiguringTheDispatcherWorks.md#message-mappers) - * [Channel Factory](/contents/HowConfiguringTheDispatcherWorks.md#channel-factory) - * [Connection List](/contents/HowConfiguringTheDispatcherWorks.md#connection-list) - * [Creating a Builder](/contents/HowConfiguringTheDispatcherWorks.md#creating-a-builder) - * [The Dispatcher](/contents/HowConfiguringTheDispatcherWorks.md#running-the-dispatcher) -* [RabbitMQ Configuration](/contents/RabbitMQConfiguration.md) -* [AWS SNS/SQS Configuration](/contents/AWSSQSConfiguration.md) -* [Kafka Configuration](/contents/KafkaConfiguration.md) -* [Azure Service Bus Configuration](/contents/AzureServiceBusConfiguration.md) -* [Azure Archive Provider Configuration](/contents/AzureBlobConfiguration.md) - -## Darker Configuration - -* [Basic Configuration](/contents/DarkerBasicConfiguration.md) - -## Brighter Request Handlers & Middleware Pipelines - -* [How to Implement a Request Handler](/contents/ImplementingAHandler.md) -* [How to Implement an Async Request Handler](/contents/ImplementingAsyncHandler.md) -* [Requests, Commands and an Events](/contents/Requests%2C%20Commands%20and%20Events.md) -* [Dispatching Requests](/contents/DispatchingARequest.md) - * [Usage](/contents/DispatchingARequest.md#usage) - * [Registering a Handler](/contents/DispatchingARequest.md#registering-a-handler) - * [Dispatching Requests](/contents/DispatchingARequest.md#dispatching-requests) -* [Dispatching An Async Request](/contents/AsyncDispatchARequest.md) - * [Usage](/contents/AsyncDispatchARequest.md#usage) - * [Registering a Handler](/contents/AsyncDispatchARequest.md#registering-a-handler) - * [Dispatching Requests](/contents/AsyncDispatchARequest.md#dispatching-requests) -* [Returning results from a Handler](/contents/ReturningResultsFromAHandler.md) - * [Handling Failure](/contents/ReturningResultsFromAHandler.md#handling-failure) - * [Communicating the Outcome](/contents/ReturningResultsFromAHandler.md#communicating-the-outcome-of-a-command) -* [Using an External Bus ](/contents/ImplementingExternalBus.md) - * [Brighter's External Bus Architecture](/contents/ImplementingExternalBus.md#brighters-external-bus-architecture) - * [Sending via the External Bus](/contents/ImplementingExternalBus.md#sending-via-the-external-bus) - * [Receiving via the External Bus](/contents/ImplementingExternalBus.md#receiving-via-the-external-bus) -* [Message Mappers](/contents/MessageMappers.md) - * [Writing A Message Mapper](/contents/MessageMappers.md#writing-a-message-mapper) - * [Transformers](/contents/MessageMappers.md#transformers) - * [Claim Check](/contents/ClaimCheck.md) - * [Claim Check & Retrieve Claim](/contents/ClaimCheck.md#claim-check-and-retrieve-claim) - * [The Luggage Store](/contents/ClaimCheck.md#the-luggage-store) - * [S3LuggageStore](/contents/S3LuggageStore.md) - * [Compression](/contents/Compression.md) -* [Routing](/contents/Routing.md) - * [Publish-Subscribe](/contents/Routing.md#publish-subscribe) - * [Direct Messaging](/contents/Routing.md#direct-messaging) - * [Summary](/contents/Routing.md#summary) -* [Building a Pipeline of Request Handlers](/contents/BuildingAPipeline.md) - * [The Pipes and Filters Architectural Style](/contents/BuildingAPipeline.md#the-pipes-and-filters-architectural-style) - * [The Russian Doll Model](/contents/BuildingAPipeline.md#the-russian-doll-model) - * [Implementing a Pipeline](/contents/BuildingAPipeline.md#implementing-a-pipeline) - * [Using a Manual Approach](/contents/BuildingAPipeline.md#using-a-manual-approach) -* [Building an Async Pipeline of Request Handlers](/contents/BuildingAnAsyncPipeline.md) - * [Implementing a Pipeline](/contents/BuildingAnAsyncPipeline.md#implementing-a-pipeline) -* [Passing information between Handlers in the Pipeline](/contents/UsingTheContextBag.md) -* [Failure and Dead Letter Queues](/contents/HandlerFailure.md) - * [Retry (and Circuit Break) the *Request* on the Internal Bus](/contents/HandlerFailure.md#retry-and-circuit-break-the-request-on-the-internal-bus) - * [Retry (with Delay) the *Request* on the External Bus](/contents/HandlerFailure.md#retry-with-delay-the-request-on-the-external-bus) - * [Terminate processing of that *Request*](/contents/HandlerFailure.md#terminate-processing-of-that-request) - * [Run a Fallback](/contents/HandlerFailure.md#run-a-fallback) - * [Use Custom Middleware](/contents/HandlerFailure.md#use-custom-middleware) -* [Supporting Retry and Circuit Breaker](/contents/PolicyRetryAndCircuitBreaker.md) - * [Using Brighter’s UsePolicy Attribute](/contents/PolicyRetryAndCircuitBreaker.md#using-brighters-usepolicy-attribute) - * [Timeout](/contents/PolicyRetryAndCircuitBreaker.md#timeout) -* [Failure and Fallback](/contents/PolicyFallback.md) - * [Calling the Fallback Pipeline](/contents/PolicyFallback.md#calling-the-fallback-pipeline) - * [Using the FallbackPolicy Attribute](/contents/PolicyFallback.md#using-the-fallbackpolicy-attribute) -* [Feature Switches](/contents/FeatureSwitches.md) - * [Using the Feature Switch Attribute](/contents/FeatureSwitches.md#using-the-feature-switch-attribute) - * [Building a config for Feature Switches with FluentConfigRegistryBuilder](/contents/FeatureSwitches.md#building-a-config-for-feature-switches-with-fluentconfigregistrybuilder) - * [Implementing a custom Feature Switch Registry](/contents/FeatureSwitches.md#implementing-a-custom-feature-switch-registry) - * [Setting Feature Switching Registry](/contents/FeatureSwitches.md#setting-feature-switching-registry) - -## Guaranteed, At Least Once - -* [Outbox Support](/contents/BrighterOutboxSupport.md) - * [Post](/contents/BrighterOutboxSupport.md#post) - * [Deposit and Clear](/contents/BrighterOutboxSupport.md#deposit-and-clear) - * [Participating in Transactions](/contents/BrighterOutboxSupport.md#participating-in-transactions) - * [Implicit or Explicit Clearing of Messages from the Outbox](/contents/BrighterOutboxSupport.md#implicit-or-explicit-clearing-of-messages-from-the-outbox) - * [Outbox Sweeper](/contents/BrighterOutboxSupport.md#outbox-sweeper) - * [Outbox Arcguver](/contents/BrighterOutboxSupport.md#outbox-archiver) - * [Outbox Configuration](/contents/BrighterOutboxSupport.md#outbox-configuration) - * [Outbox Builder](/contents/BrighterOutboxSupport.md#outbox-builder) - * [EFCore Outbox](/contents/EFCoreOutbox.md) - * [Dapper Outbox](/contents/DapperOutbox.md) - * [Dynamo Outbox](/contents/DynamoOutbox.md) - * [Azure Blob Archive Provider](/contents/AzureBlobArchiveProvider.md) -* [Inbox Support](/contents/BrighterInboxSupport.md) - * [Guaranteed, At Least Once](/contents/BrighterInboxSupport.md#guaranteed-at-least-once) - * [Guaranteed, Once Only](/contents/BrighterInboxSupport.md#guaranteed-once-only) - * [Inbox](/contents/BrighterInboxSupport.md#inbox) - * [Adding an Inbox to a Handler](/contents/BrighterInboxSupport.md#adding-an-inbox-to-a-handler) - * [Inbox Configuration](/contents/BrighterInboxSupport.md#inbox-configuration) - * [Inbox Builder](/contents/BrighterInboxSupport.md#inbox-builder) - * [MSSQL Inbox](/contents/MSSQLInbox.md) - * [MySQL Inbox](/contents/MySQLInbox.md) - * [Postgres Inbox](/contents/PostgresInbox.md) - * [Sqlite Inbox](/contents/SqliteInbox.md) - * [Dynamo Inbox](/contents/DynamoInbox.md) - -## Darker Query Handlers - -* [How to Implement a Query Handler](/contents/ImplementAQueryHandler.md) - -## Health Checks & Observability - -* [Logging](/contents/Logging.md) -* [Monitoring](/contents/Monitoring.md) - * [Configuring Monitoring](/contents/Monitoring.md#configuring-monitoring) - * [Config file](/contents/Monitoring.md#config-file) - * [Handler configuration](/contents/Monitoring.md#handler-configuration) - * [Monitor message format](/contents/Monitoring.md#monitor-message-format) -* [Health Checks](/contents/HealthChecks.md) -* [Telemetry](/contents/Telemetry.md) - -## FAQ -* [FAQ](/contents/FAQ.md) - * [Dispatching a list of requests](/contents/FAQ.md#iterating-over-a-list-of-requests-to-dispatch-them) - * [Internal Bus & Asynchronous vs. External Bus](/contents/AsyncDispatchARequest.md#internal-bus--asynchronous-vs-external-bus) - -## Under the Hood - -* [How The Command Processor Works](/contents/HowBrighterWorks.md) - * [The Dispatcher](/contents/HowBrighterWorks.md#the-dispatcher) -* [How Service Activator Works](/contents/HowServiceActivatorWorks) - -## Commands, Processors, & Dispatchers - -* [Commands Patterns](/contents/CommandsCommandDispatcherandProcessor.md) - * [Command](/contents/CommandsCommandDispatcherandProcessor.md#command) - * [Command Dispatcher](/contents/CommandsCommandDispatcherandProcessor.md#command-dispatcher) - * [Command Processor](/contents/CommandsCommandDispatcherandProcessor.md#command-processor) - -## Event Driven Architectures - -* [Microservices](/contents/Microservices.md) - * [Boundaries are explicit](/contents/Microservices.md#boundaries-are-explicit) - * [Services are autonomous](/contents/Microservices.md#services-are-autonomous) - * [Share Schema not type](/contents/Microservices.md#share-schema-not-type) - * [Compatibility is based on policy](/contents/Microservices.md#compatibility-is-based-on-policy) -* [Event Driven Collaboration](/contents/EventDrivenCollaboration.md) - * [Messaging](/contents/EventDrivenCollaboration.md#messaging) - * [Temporal Coupling](/contents/EventDrivenCollaboration.md#temporal-coupling) - * [Behavioral Coupling](/contents/EventDrivenCollaboration.md#behavioral-coupling) - * [Event Driven Collaboration](/contents/EventDrivenCollaboration.md#event-driven-collaboration-1) -* [Event Carried State Transfer](/contents/EventCarriedStateTransfer.md) - * [Outside and Inside Data](/contents/EventCarriedStateTransfer.md#outside-and-inside-data) - * [Reference Data](/contents/EventCarriedStateTransfer.md#reference-data) - * [Caching](/contents/EventCarriedStateTransfer.md#caching)) - * [Event Carried State Transfer](/contents/EventCarriedStateTransfer.md#event-carried-state-transfer-1) - * [Alternatives to Event Carried State Transfer](/contents/EventCarriedStateTransfer.md#alternatives-to-event-carried-state-transfer) - * [Worked Scenario](/contents/EventCarriedStateTransfer.md#worked-scenario) - * [A Pipeline](/contents/EventCarriedStateTransfer.md#a-pipeline) - * [ECST](/contents/EventCarriedStateTransfer.md#ecst) -* [Outbox Pattern](/contents/OutboxPattern.md) - * [Producer Correctness](/contents/OutboxPattern.md#producer-correctness) - * [Solutions](/contents/OutboxPattern.md#solutions) - * [Ignore](/contents/OutboxPattern.md#ignore) - * [Compensate](/contents/OutboxPattern.md#compensate) - * [The Outbox Pattern](/contents/OutboxPattern.md#the-outbox-pattern) - • [Log Tailing](/contents/OutboxPattern.md#log-tailing) - -## Task Queues - -* [Using a Task Queue](/contents/TaskQueuePattern.md) - * [Doing Work Asynchronously](/contents/TaskQueuePattern.md#doing-work-asynchronously) +## 9 + +### Overview + + * [Show me the code!](/contents/9/howMeTheCode.md) + * [Basic Concepts](/contents/9/BasicConcepts.md) + +### Brighter Configuration + + * [Basic Configuration](/contents/9/BrighterBasicConfiguration.md) + * [How Configuring the Command Processor Works](/contents/9/HowConfiguringTheCommandProcessorWorks.md) + * [How Configuring a Dispatcher for an External Bus Works](/contents/9/HowConfiguringTheDispatcherWorks.md) + * [RabbitMQ Configuration](/contents/9/RabbitMQConfiguration.md) + * [AWS SNS Configuration](/contents/9/AWSSQSConfiguration.md) + * [Kafka Configuration](/contents/9/KafkaConfiguration.md) + * [Azure Service Bus Configuration](/contents/9/AzureServiceBusConfiguration.md) + * [Azure Archive Provider Configuration](/contents/9/) + +### Darker Configuration + + * [Basic Configuration](/contents/9/DarkerBasicConfiguration.md) + +### Brighter Request Handlers and Middleware Pipelines + + * [Building an Async Pipeline of Request Handlers](/contents/9/BuildingAnAsyncPipeline.md) + * [Basic Configuration](/contents/9/DarkerBasicConfiguration.md) + * [How to Implement an Async Request Handler](/contents/9/ImplementingAsyncHandler.md) + * [Requests, Commands and an Events](/contents/9/Requests%2C%20Commands%20and%20Events.md) + * [Dispatching Requests](/contents/9/DispatchingARequest.md) + * [Dispatching An Async Request](/contents/9/AsyncDispatchARequest.md) + * [Returning results from a Handler](/contents/9/ReturningResultsFromAHandler.md) + * [Using an External Bus](/contents/9/ImplementingExternalBus.md) + * [Message Mappers](/contents/9/MessageMappers.md) + * [Routing](/contents/9/Routing.md) + * [Building a Pipeline of Request Handlers](/contents/9/BuildingAPipeline.md) + * [Passing information between Handlers in the Pipeline](/contents/9/UsingTheContextBag.md) + * [Failure and Dead Letter Queues](/contents/9/HandlerFailure.md) + * [Supporting Retry and Circuit Breaker](/contents/9/PolicyRetryAndCircuitBreaker.md) + * [Failure and Fallback](/contents/9/PolicyFallback.md) + * [Feature Switches](/contents/9/FeatureSwitches.md) + +### Guaranteed At Least Once + + * [Outbox Support](/contents/9/BrighterOutboxSupport.md) + * [Inbox Support](/contents/9/BrighterInboxSupport.md) + * [EFCore Outbox](/contents/9/EFCoreOutbox.md) + * [Dapper Outbox](/contents/9/DapperOutbox.md) + * [Dynamo Outbox](/contents/9/DynamoOutbox.md) + * [MSSQL Inbox](/contents/9/MSSQLInbox.md) + * [MySQL Inbox](/contents/9/MySQLInbox.md) + * [Postgres Inbox](/contents/9/PostgresInbox.md) + * [Sqlite Inbox](/contents/9/SqliteInbox.md) + * [Dynamo Inbox](/contents/9/DynamoInbox.md) + +### Darker Query Handlers and Middleware Pipelines + + * [How to Implement a Query Handler](/contents/9/ImplementAQueryHandler.md) + +### Health Checks and Observability + + * [Logging](/contents/9/Logging.md) + * [Monitoring](/contents/9/Monitoring.md) + * [Health Checks](/contents/9/HealthChecks.md) + * [Telemetry](/contents/9/Telemetry.md) + +### Command, Processors and Dispatchers + + * [Command, Processor and Dispatcher Patterns](/contents/9/CommandsCommandDispatcherandProcessor.md) + +### Under the Hood + + * [How The Command Processor Works](/contents/9/HowBrighterWorks.md) + * [How Service Activator Works](/contents/9/HowServiceActivatorWorks.md) + +### Event Driven Architectures + + * [Microservices](/contents/9/Microservices.md) + * [Event Driven Collaboration](/contents/9/EventDrivenCollaboration.md) + * [Event Carried State Transfer](/contents/9/EventCarriedStateTransfer.md) + * [Outbox Pattern](/contents/9/OutboxPattern.md) + +### Task Queues + + * [Using a Task Queue](/contents/9/TaskQueuePattern.md) + +### FAQ + + * [FAQ](/contents/9/FAQ.md) + +## 10 + +### Overview + + * [Show me the code!](/contents/10/howMeTheCode.md) + * [Basic Concepts](/contents/10/BasicConcepts.md) + +### Brighter Configuration + + * [Basic Configuration](/contents/10/BrighterBasicConfiguration.md) + * [How Configuring the Command Processor Works](/contents/10/HowConfiguringTheCommandProcessorWorks.md) + * [How Configuring a Dispatcher for an External Bus Works](/contents/10/HowConfiguringTheDispatcherWorks.md) + * [RabbitMQ Configuration](/contents/10/RabbitMQConfiguration.md) + * [AWS SNS Configuration](/contents/10/AWSSQSConfiguration.md) + * [Kafka Configuration](/contents/10/KafkaConfiguration.md) + * [Azure Service Bus Configuration](/contents/10/AzureServiceBusConfiguration.md) + * [Azure Archive Provider Configuration](/contents/10/) + +### Darker Configuration + + * [Basic Configuration](/contents/10/DarkerBasicConfiguration.md) + +### Brighter Request Handlers and Middleware Pipelines + + * [Building an Async Pipeline of Request Handlers](/contents/10/BuildingAnAsyncPipeline.md) + * [Basic Configuration](/contents/10/DarkerBasicConfiguration.md) + * [How to Implement an Async Request Handler](/contents/10/ImplementingAsyncHandler.md) + * [Requests, Commands and an Events](/contents/10/Requests%2C%20Commands%20and%20Events.md) + * [Dispatching Requests](/contents/10/DispatchingARequest.md) + * [Dispatching An Async Request](/contents/10/AsyncDispatchARequest.md) + * [Returning results from a Handler](/contents/10/ReturningResultsFromAHandler.md) + * [Using an External Bus](/contents/10/ImplementingExternalBus.md) + * [Message Mappers](/contents/10/MessageMappers.md) + * [Routing](/contents/10/Routing.md) + * [Building a Pipeline of Request Handlers](/contents/10/BuildingAPipeline.md) + * [Passing information between Handlers in the Pipeline](/contents/10/UsingTheContextBag.md) + * [Failure and Dead Letter Queues](/contents/10/HandlerFailure.md) + * [Supporting Retry and Circuit Breaker](/contents/10/PolicyRetryAndCircuitBreaker.md) + * [Failure and Fallback](/contents/10/PolicyFallback.md) + * [Feature Switches](/contents/10/FeatureSwitches.md) + +### Guaranteed At Least Once + + * [Outbox Support](/contents/10/BrighterOutboxSupport.md) + * [Inbox Support](/contents/10/BrighterInboxSupport.md) + * [EFCore Outbox](/contents/10/EFCoreOutbox.md) + * [Dapper Outbox](/contents/10/DapperOutbox.md) + * [Dynamo Outbox](/contents/10/DynamoOutbox.md) + * [MSSQL Inbox](/contents/10/MSSQLInbox.md) + * [MySQL Inbox](/contents/10/MySQLInbox.md) + * [Postgres Inbox](/contents/10/PostgresInbox.md) + * [Sqlite Inbox](/contents/10/SqliteInbox.md) + * [Dynamo Inbox](/contents/10/DynamoInbox.md) + +### Darker Query Handlers and Middleware Pipelines + + * [How to Implement a Query Handler](/contents/10/ImplementAQueryHandler.md) + +### Health Checks and Observability + + * [Logging](/contents/10/Logging.md) + * [Monitoring](/contents/10/Monitoring.md) + * [Health Checks](/contents/10/HealthChecks.md) + * [Telemetry](/contents/10/Telemetry.md) + +### Command, Processors and Dispatchers + + * [Command, Processor and Dispatcher Patterns](/contents/10/CommandsCommandDispatcherandProcessor.md) + +### Under the Hood + + * [How The Command Processor Works](/contents/10/HowBrighterWorks.md) + * [How Service Activator Works](/contents/10/HowServiceActivatorWorks.md) + +### Event Driven Architectures + + * [Microservices](/contents/10/Microservices.md) + * [Event Driven Collaboration](/contents/10/EventDrivenCollaboration.md) + * [Event Carried State Transfer](/contents/10/EventCarriedStateTransfer.md) + * [Outbox Pattern](/contents/10/OutboxPattern.md) + +### Task Queues + + * [Using a Task Queue](/contents/10/TaskQueuePattern.md) + +### FAQ + + * [FAQ](/contents/10/FAQ.md) + diff --git a/bin/Rewind b/bin/Rewind new file mode 100755 index 0000000..23cb6b6 Binary files /dev/null and b/bin/Rewind differ diff --git a/contents/AWSSQSConfiguration.md b/contents/10/AWSSQSConfiguration.md similarity index 100% rename from contents/AWSSQSConfiguration.md rename to contents/10/AWSSQSConfiguration.md diff --git a/contents/AsyncDispatchARequest.md b/contents/10/AsyncDispatchARequest.md similarity index 100% rename from contents/AsyncDispatchARequest.md rename to contents/10/AsyncDispatchARequest.md diff --git a/contents/AzureBlobArchiveProvider.md b/contents/10/AzureBlobArchiveProvider.md similarity index 100% rename from contents/AzureBlobArchiveProvider.md rename to contents/10/AzureBlobArchiveProvider.md diff --git a/contents/AzureBlobConfiguration.md b/contents/10/AzureBlobConfiguration.md similarity index 100% rename from contents/AzureBlobConfiguration.md rename to contents/10/AzureBlobConfiguration.md diff --git a/contents/AzureServiceBusConfiguration.md b/contents/10/AzureServiceBusConfiguration.md similarity index 100% rename from contents/AzureServiceBusConfiguration.md rename to contents/10/AzureServiceBusConfiguration.md diff --git a/contents/BasicConcepts.md b/contents/10/BasicConcepts.md similarity index 100% rename from contents/BasicConcepts.md rename to contents/10/BasicConcepts.md diff --git a/contents/10/BrighterBasicConfiguration.md b/contents/10/BrighterBasicConfiguration.md new file mode 100644 index 0000000..e36537a --- /dev/null +++ b/contents/10/BrighterBasicConfiguration.md @@ -0,0 +1,742 @@ +# **Basic Configuration** + +Configuration is the most labor-intensive part of using Brighter.Once you have configured Brighter, using its model of requests and handlers is straightforward + +## **Using .NET Core Dependency Injection** + +This section covers using .NET Core Dependency Injection to configure Brighter. If you want to use an alternative DI container then see the section [How Configuration Works](/contents/HowConfigurationWorks.md) + +We divide configuration into two sections, depending on your requirements: + +* [**Configuring The Command Processor**](#configuring-the-command-processor): This section covers configuring the **Command Processor**. Use this if you want to dispatch requests to handlers, or publish messages from your application on an external bus +* [**Configuring The Service Activator**](#configuring-the-service-activator): This section covers configuring the **Service Activator**. Use this if you want to read messages from a transport (and then dispatch to handlers). + + +## **Configuring The Command Processor** + +### **Command Processor Service Collection Extensions** + +Brighter's package: + +* **Paramore.Brighter.Extensions.DependencyInjection** + + provides extension methods for **ServiceCollection** that can be used to add Brighter to the .NET Core DI Framework. + +By adding the package you can call the **AddBrighter()** extension method. + +If you are using a **Startup** class's **ConfigureServices** method call the following: + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) +} + +``` + +if you are using .NET 6 you can make the call direction on your **HostBuilder**'s Services property. + +The **AddBrighter()** method takes an **`Action`** delegate. The extension method supplies the delegate with a **BrighterOptions** object that allows you to configure how Brighter runs. + +The **AddBrighter()** method returns an **IBrighterBuilder** interface. **IBrighterBuilder** is a [fluent interface](https://en.wikipedia.org/wiki/Fluent_interface) that you can use to configure additional Brighter properties (see [Brighter Builder Fluent Interface](#brighter-builder-fluent-interface)). + +#### **Adding Polly Policies** + +Brighter uses Polly policies for both internal reliability, and to support adding a custom policy to a handler for reliability. + +To use a Polly policy with Brighter you need to register it first with a Polly **PolicyRegistry**. In this example we register both Synchronous and Asynchronous Polly policies with the registry. + +``` csharp + var retryPolicy = Policy.Handle().WaitAndRetry(new[] + { + TimeSpan.FromMilliseconds(50), + TimeSpan.FromMilliseconds(100), + TimeSpan.FromMilliseconds(150) }); + + var circuitBreakerPolicy = Policy.Handle().CircuitBreaker(1, + TimeSpan.FromMilliseconds(500)); + + var retryPolicyAsync = Policy.Handle() + .WaitAndRetryAsync(new[] { TimeSpan.FromMilliseconds(50), TimeSpan.FromMilliseconds(100), TimeSpan.FromMilliseconds(150) }); + + var circuitBreakerPolicyAsync = Policy.Handle().CircuitBreakerAsync(1, TimeSpan.FromMilliseconds(500)); + + var policyRegistry = new PolicyRegistry() + { + { "SyncRetryPolicy", retryPolicy }, + { "SyncCircuitBreakerPolicy", circuitBreakerPolicy }, + { "AsyncRetryPolicy", retryPolicyAsync }, + { "AsyncCircuitBreakerPolicy", circuitBreakerPolicyAsync } + }; + +``` + +And you can use them in you own handler like this: + +``` csharp +internal class MyQoSProtectedHandler : RequestHandler +{ + static MyQoSProtectedHandler() + { + ReceivedCommand = false; + } + + [UsePolicy(policy: "SyncRetryPolicy", step: 1)] + public override MyCommand Handle(MyCommand command) + { + /*Do work that could throw error because of distributed computing reliability*/ + } +} +``` + +See the section [Policy Retry and Circuit Breaker](/contents/PolicyRetryAndCircuitBreaker.md) for more on using Polly policies with handlers. + +With the Polly Policy Registry filled, you need to tell Brighter where to find the Policy Registry: + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(options => + options.PolicyRegistry = policyRegistry + ) +} + +``` + +#### **Configuring Lifetimes** + +Brighter can register your *Request Handlers* and *Message Mappers* for you (see [IBrighter Builder Fluent Interface](#ibrighterbuilder-fluent-interface)). When we register types for you with ServiceCollection, we need to register them with a given lifetime (see [Dependency Injection Service Lifetimes](https://docs.microsoft.com/en-us/dotnet/core/extensions/dependency-injection#service-lifetimes)). + +We also allow you to set the lifetime for the CommandProcessor. + +We recommend the following lifetimes: + +* If you are using *Scoped* lifetimes, for example with EF Core, make your *Request Handlers* and your *Command Processor* Scoped as well. +* If you are not using *Scoped* lifetimes you can use *Transient* lifetimes for *Request Handlers* and a *Singleton* lifetime for the *Command Processor*. +* Your *Message Mappers* should not have state and can be *Singletons*. + +(Be cautious about using *Singleton* lifetimes for *Request Handlers*. Even if your *Request Handler* is stateless today, and so does not risk carrying state across requests, a common bug is that state is added to an existing *Request Handler* which has previously been registered as a *Singleton*.) + +You configure the lifetimes for the different types that Brighter can create at run-time as follows: + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(options => + options.HandlerLifetime = ServiceLifetime.Scoped; + options.CommandProcessorLifetime = ServiceLifetime.Scoped; + options.MapperLifetime = ServiceLifetime.Singleton; + ); +} + +``` + +### **Brighter Builder Fluent Interface** + +#### **Type Registration** +The **IBrighterBuilder** fluent interface can scan your assemblies for your *Request Handlers* (inherit from **IHandleRequests<>** or **IHandleRequestsAsync<>**) and *Message Mappers* (inherit from **IAmAMessageMapper<>**) and register then with the **ServiceCollection**. This is the most common way to register your code. + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .AutoFromAssemblies(); +} + +``` + +The code scans any loaded assemblies. If you need to register types from assemblies that are not yet loaded, you can provide a list of additional assemblies to scan as an argument to the call to **AutoFromAssemblies()**. + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .AutoFromAssemblies(typeof(MyRequestHandlerAsync).Assembly); +} + +``` + +Instead of using **AutoFromAssemblies** you can exert more fine-grained control over the registration, by explicitly registering your *Request Handlers* and *Message Mappers*. We don't recommend this, but make it available for cases where the automatic registration does not meet your needs. + +* **MapperRegistryFromAssemblies()**, **HandlersFromAssemblies()** and **AsyncHandlersFromAssemblies** are the methods called by **AutoFromAssemblies()** and can be called explicitly. +* **Handlers()**, **AsyncHandlers()** and **MapperRegistry()** accept an **Action<>** delegate that respectively provide you with **IAmASubscriberRegistry** or **IAmAnAsyncSubscriberRegistry** to register your RequestHandlers explicitly or a **ServiceCollectionMapperRegistry** to register your mappers. This gives you explicit control over what you register. + +#### **Using an External Bus** + +Using an *External Bus* allows you to send messages between processes using a message-oriented middleware transport (such as RabbitMQ or Kafka). (For symmetry, we refer to the usage of the *Command Processor* without an external bus as using an *Internal Bus*). + +When raising a message on the *Internal Bus*, you use one of the following methods on the *Command Processor*: + +* **Send()** and **SendAsync()** - Sends a *Command* to one *Request Handler*. +* **Publish()** and **PublishAsync()** - Broadcasts an *Event* to zero or more *Request Handlers*. + +When raising a message on an *External Bus*, you use the following methods on the *CommandProcessor*: + +* **Post()** and **PostAsync()** - Immediately posts a *Command* or *Event* to another process via the external Bus +* **DepositPost()** and **DepositPostAsync()** - Puts on or many *Command*(s) or *Event*(s) in the *Outbox* for later delivery +* **ClearOutbox()** and **ClearOutboxAsync()** - Clears the *Outbox*, posting un-dispatched messages to another process via the *External Bus*. +* **ClearAsyncOutbox()** - Implicitly clears the **Outbox**, similar to above however allows bulk dispatching of messages onto a **Transport**. + +The major difference here is whether or not you wish to use an *Outbox* for Transactional Messaging. (See [Outbox Pattern](/contents/OutboxPattern.md) and [Brighter Outbox Support](/contents/BrighterOutboxSupport.md) for more on Brighter and the Outbox Pattern). + +To use an *External Bus*, you need to supply Brighter with configuration information that tells Brighter what middleware you are using and how to find it. (You don't need to do anything to configure an *Internal Bus*, it is always available.) + +In order to provide Brighter with this information we need to provide it with an implementation of **IAmAProducerRegistry** for the middleware you intend to use for the *External Bus*. + +#### **Transports and Gateways** + +*Transports* are how Brighter supports specific Message-Oriented-Middleware (MoM). *Transports* are provided in separate NuGet packages so that you can take a dependency only on the transport that you need. Brighter supports a number of different *transports*. + +A *Gateway Connection* is how you configure connection to MoM within a *transport*. As an example, the *Gateway Connection* **RMqGatewayConnection** is used to connect to RabbitMQ. Internally the *Gateway Connection* is used to create a *Gateway* object which wraps the client SDK for the MoM. + +We go into more depth on the fields you set here in sections dealing with specific transports. + +#### **Publications** + +A *Publication* configures a transport for sending a message to it's associated MoM. So an **RmqPublication** configures how we publish a message to RabbitMQ. There are a number of common properties to all publications. + +* **MakeChannels**: Do you want Brighter to create the infrastructure? Brighter can create infrastructure that it needs, and is aware of: **OnMissingChannel.Create**. So a publication can create the topic to send messages to. Alternatively if you create the channel by another method, such as IaaC, we can verify the infrastructure on startup: **OnMissingChannel.Validate**. Finally, you can avoid the performance cost of runtime checks by assuming your infrastructure exists: **OnMissingChannel.Assume**. +* **MaxOutstandingMessages**: How large can the number of messages in the Outbox grow before we stop allowing new messages to be published and raise an **OutboxLimitReachedException**. +* **MaxOutStandingCheckIntervalMilliSeconds**: How often do we check to see if the Outbox is full. +* **Topic**: A Topic is the key used within the MoM to route messages. Publishers publish to a topic and subscribers, subscribe to it. We use a class **RoutingKey** to encapsulate the identifier used for a topic. The name the MoM uses for a topic may vary. Kafka & SNS use *topic* whilst RMQ uses *routingkey* + +#### **Transport NuGet Packages** + +We use the naming convention **Paramore.Brighter.MessagingGateway.{TRANSPORT}** for *transports* where {TRANSPORT} is the name of the middleware. + +In this example we will show using an implementation of **IAmAProducerRegistry** for RabbitMQ, provided by the NuGet package: + +* **Paramore.Brighter.MessagingGateway.RMQ** + +See the documentation for detail on specific *transports* on how to configure them for use with Brighter, for now it is enough to know that you need to provide a *Messaging Gateway* which tells us how to reach the middleware and a *Publication* which tells us how to configure the middleware. + +*Transports* provide an **IAmAProducerRegistryFactory()** to allow you to create multiple *Publications* connected to the same middleware. + +#### Retry and Circuit Breaker with an External Bus + +When posting a request to the External Bus we use a Polly policy internally to control Retry and Circuit Breaker in case the External Bus is not available. These policies have defaults but you can configure the behavior using the policy keys: + +* **Paramore.RETRYPOLICY** +* **Paramore.CIRCUITBREAKER** + +#### **Bus Example** + +Putting this together, an example configuration for an External Bus for a local RabbitMQ instance could look like this: + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .UseExternalBus(new RmqProducerRegistryFactory( + new RmqMessagingGatewayConnection + { + AmpqUri = new AmqpUriSpecification(new Uri("amqp://guest:guest@localhost:5672")), + Exchange = new Exchange("paramore.brighter.exchange"), + }, + new RmqPublication[]{ + new RmqPublication + { + Topic = new RoutingKey("GreetingMade"), + MaxOutStandingMessages = 5, + MaxOutStandingCheckIntervalMilliSeconds = 500, + WaitForConfirmsTimeOutInMilliseconds = 1000, + MakeChannels = OnMissingChannel.Create + }} + ).Create() + ) + + ... +} +``` + +#### **Outbox Support** + +TODO: V10 Changes + +If you intend to use Brighter's *Outbox* support for Transactional Messaging then you need to provide us with details of your *Outbox*. + +Brighter provides a number of *Outbox* implementations for common Dbs (and you can write your own for a Db that we do not support). For this discussion we will look at Brighter's support for working with EF Core. See the documentation for working with specific *Outbox* implementations. + +EF Core supports a number of databases and you should pick the packages that match the Dy you want to use with EF Core. In this case we will choose MySQL. + +For this we will need the *Outbox* packages for the MySQL *Outbox*. + +* **Paramore.Brighter.MySql** +* **Paramore.Brighter.Outbox.MySql** + +For a given backing store the pattern should be Paramore.Brighter.{DATABASE} and Paramore.Brighter.Outbox.{DATABASE} where {DATABASE} is the name of the Db that you are using. + +In addition for an ORM you will need to add the package that supports the ORM, in this case EF Core: + +* **Paramore.Brighter.MySql.EntityFrameworkCore** + +For a given ORM the pattern should be Paramore.Brighter.{ORM}.{DATABASE} where {ORM} is the ORM you are choosing and {DATABASE} is the Db you are using with the ORM. + +To configure our *Outbox* we then need to use the Use{DATABASE}Outbox method call, where {DATABASE} is the {DATABASE} that we want, passing in the configuration for our Db so that we can access it. In our case this will be **UseMySqlOutbox()**. + +As we want to use an ORM, in our case EF Core, we have to tell the Outbox how to access EF Core transactions - as we need to participate in a transaction with the ORM. We call a method for the Db, Use{DATABASE}TransactionConnectionProvider, where {DATABASE} is our Db, so in our case **UseMySqlTransactionConnectionProvider()**. + +As a parameter to Use{DATABASE}TransactionConnectionProvider we need to provide a *Transaction Provider* for the ORM we are using, in our case this is *MySqlEntityFrameworkConnectionProvider<>). + +Finally, if we want the *Outbox* to use a background thread to clear un-dispatched items from the *Outbox*, and we do in most circumstances, otherwise they will not be dispatched, we need to run an *Outbox Sweeper* to do this work. + +To add the *Outbox Sweeper* you will need to take a dependency on another NuGet package: + +* **Paramore.Brighter.Extensions.Hosting** + +This results in: + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .UseExternalBus(...) + .UseMySqlOutbox(new MySqlConfiguration(DbConnectionString(), _outBoxTableName), typeof(MySqlConnectionProvider), ServiceLifetime.Singleton) + .UseMySqTransactionConnectionProvider(typeof(MySqlEntityFrameworkConnectionProvider), ServiceLifetime.Scoped) + .UseOutboxSweeper() + + ... +} + +``` + +(**UseExternalBus()** has optional parameters for use with Request-Reply support for some transports. We don't cover that here, instead see [Direct Messaging](/contents/Routing.md#direct-messaging) for more). + +#### **Configuring JSON Serialization** + +Brighter defines a set of serialization options for use when it needs to serialize messages to JSON. Internally we use these options in our transports, when serializing messages to an external bus and deserializing from an external bus. You may wish to use these options in your own [*Message Mapper*](/contents/MessageMappers.md) implementation. + +By default our JSONSerialization Options are configured as follows: + +``` csharp +static JsonSerialisationOptions() +{ + var opts = new JsonSerializerOptions + { + PropertyNameCaseInsensitive = true, + PropertyNamingPolicy = JsonNamingPolicy.CamelCase, + NumberHandling = JsonNumberHandling.AllowReadingFromString, + AllowTrailingCommas = true + }; + + opts.Converters.Add(new JsonStringConverter()); + opts.Converters.Add(new DictionaryStringObjectJsonConverter()); + opts.Converters.Add(new ObjectToInferredTypesConverter()); + opts.Converters.Add(new JsonStringEnumConverter()); + + Options = opts; +} +``` + +You can use the **IBrighterBuilder** extension **ConfigureJsonSerialisation** to override these values. The method takes an **Action\** lambda expression that allows you to override these defaults. For example: + +```csharp + +.ConfigureJsonSerialisation((options) => +{ + options.PropertyNameCaseInsensitive = true; +}) + +``` + +If you want to use this configured set of JSON Serialization options in your own code, you can, by using the static property JsonSerialisationOptions.Options. For example: + +```csharp +public GreetingMade MapToRequest(Message message) +{ + return JsonSerializer.Deserialize(message.Body.Value, JsonSerialisationOptions.Options); +} +``` + +### **Putting It All Together** + +Putting all this together, a typical configuration might looks as follows: + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(options => + { + options.HandlerLifetime = ServiceLifetime.Scoped; + options.MapperLifetime = ServiceLifetime.Singleton; + options.PolicyRegistry = policyRegistry; + }) + .ConfigureJsonSerialisation((options) => + { + options.PropertyNameCaseInsensitive = true; + }) + .UseExternalBus(new RmqProducerRegistryFactory( + new RmqMessagingGatewayConnection + { + AmpqUri = new AmqpUriSpecification(new Uri("amqp://guest:guest@rabbitmq:5672")), + Exchange = new Exchange("paramore.brighter.exchange"), + }, + new RmqPublication[] { + new RmqPublication + { + Topic = new RoutingKey("GreetingMade"), + MaxOutStandingMessages = 5, + MaxOutStandingCheckIntervalMilliSeconds = 500, + WaitForConfirmsTimeOutInMilliseconds = 1000, + MakeChannels = OnMissingChannel.Create + }} + ).Create() + ) + .UseMySqlOutbox(new MySqlConfiguration(DbConnectionString(), _outBoxTableName), typeof(MySqlConnectionProvider), ServiceLifetime.Singleton) + .UseMySqTransactionConnectionProvider(typeof(MySqlEntityFrameworkConnectionProvider), ServiceLifetime.Scoped) + .UseOutboxSweeper() + .AutoFromAssemblies(); +} + +``` + +## **Configuring The Service Activator** + +A *consumer* reads messages from Message-Oriented Middleware (MoM), and a *producer* puts messages onto the MoM for the *consumer* to read. + +A *consumer* waits for messages to appear on the queue, reads them, and then calls your *Request Handler* code to react. Because the •consumer* runs your code in response to an external request, a message being placed on the External Bus, we call the component that listens for messages and dispatches them a [*Service Activator*](https://www.enterpriseintegrationpatterns.com/patterns/messaging/MessagingAdapter.html) + +To use Brighter's Service Activator you will need to take a dependency on the NuGet package: + +* **Paramore.Brighter.ServiceActivator** + +### **ServiceActivator Service Collection Extensions** + +We provide support for configuring .NET Core's **HostBuilder** as a *ServiceActivator* for use with MoM. We use Brighter's Command Processor to dispatch the messages read by a *Dipatcher*. If you are not using **HostBuilder** then you will need to configure the Dispatcher yourself. See [How Configuring the Dispatcher Works](/contents/HowConfiguringTheDispatcherWorks.md) for more. + +To use Brighter's *Service Activator* with **HostBuilder** you will need to take a dependency on the following NuGet packages: + +* **Paramore.Brighter.ServiceActivator.Extensions.Hosting** +* **Paramore.Brighter.ServiceActivator.Extensions.DependencyInjection** + +These provide an extension method **AddServiceActivator()** that can be used to add Brighter to the .NET Core DI Framework. + +By adding the package you can call the **AddServiceActivator()** extension method. + +If you are using a **HostBuilder** class's **ConfigureServices** method call the following: + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + services.AddServiceActivator(...) + } + +``` + +if you are using .NET 6 you can make the call direction on your **HostBuilder**'s Services property. + +The **AddServiceActivator()** method takes an **`Action`** delegate. The extension method supplies the delegate with a **ServiceActivatorOptions** object that allows you to configure how Brighter runs. + +The **AddServiceActivator()** method returns an **IBrighterBuilder** interface. **IBrighterBuilder** is a [fluent interface](https://en.wikipedia.org/wiki/Fluent_interface) that you can use to configure Brighter *Command Processor* properties. It is discussed above at [Brighter Builder Fluent Interface](#brighter-builder-fluent-interface)) and the same options apply. We discuss one additional option that becomes important when receiving requests the *Inbox* in [Additional Brighter Builder Options](/contents/BasicConfiguration.md#the-inbox). + +#### **Subscriptions** + +When configuring your application's *Service Activator*, your *Subscriptions* indicate configure how your application will receive messages from the associated MoM queues or streams. + +All *Subscriptions* lets you configure the following common properties. + +* **Buffer Size**: The number of messages to hold in memory. Where the buffer is not shared, a single thread or Performer can access these; where the buffer is shared, multiple threads can access the same buffer of work. Work in a buffer is locked on queue based middleware, and thus not available to other consumers (threads or process depending if the buffer is shared or not) until *Acknowledged* or *Rejected*. +* **Channel Factory**: Creates or finds the necessary infrastructure for messaging on the MoM and wraps it in an object. +* **Channel *Name**: If queues are primitives in the MoM this names the queue, otherwise just used for diagnostics. +* **Channel Failure Delay**: How long should we delay if a channel fails before trying again, to give problems time to clear. +* **Data Type**: We use a [Datatype Channel](https://www.enterpriseintegrationpatterns.com/DatatypeChannel.html). What is the type of this channel? +* **Empty Channel Delay**: If there are no messages in the queue or stream when we read, how long should we pause before reading again? +* **MakeChannels**: Do you want Brighter to create the infrastructure? Brighter can create infrastructure that it needs, and is aware of: **OnMissingChannel.Create**. So a subscription can create the topic to send messages to, and any subscription to that topic required by the MoM, including a queue (which uses the *Channel Name*). Alternatively if you create the channel by another method, such as IaaC, we can verify the infrastructure on startup: **OnMissingChannel.Validate**. Finally, you can avoid the performance cost of runtime checks by assuming your infrastructure exists: **OnMissingChannel.Assume**. +* **Name**: What do we call this subscription for diagnostic purposes. +* **NoOfPerformers**: Effectively, how many threads do we use to read messages from the queue. As Brighter uses a Single-Threaded Apartment model, each thread has it's own message pump and is thus an in-process implementation of the [Competing Consumers](https://www.enterpriseintegrationpatterns.com/CompetingConsumers.html) pattern. +* **RequeueCount**: How many times can you retry a message before we declare it a poison pill message? +* **RequeueDelayInMilliseconds**: When we requeue a message how long should we delay it by? +* **RoutingKey**: The identifier used to routed messages to subscribers on MoM. You publish to this, and subscriber from this. This has different names; in Kafka or SNS this is a Topic, in RMQ this is the routing key. +* **RunAsync**: Is this an async pipeline? Your pipeline must be sync or async. An async pipeline can increase throughput where a handler is I/O bound by allowing the message pump to read another message whilst we await I/O completion. The cost of this is that strict ordering of messages will now be lost as processing of I/O bound requests may complete out-of-sequence. Brighter provides its own synchronization context for async operations. We recommend scaling via increasing the number of performers, unless you know that I/O is your bottleneck. +* **TimeoutInMilliseconds**: How long does a read 'wait' before assuming there are no pending messages. +* **UnaceptableMessageLimit**: Brighter will ack a message that throws an unhandled exception, thus removing it from a queue. + +For a more detailed discussion of using Requeue (with Delay) for Handler failure, (**RequeueCount** and **RequeueDelayInMilliseconds**) along with termination of a consumer due to message failure (**UnacceptableMessageLimit**) see [Handler Failure](/contents/HandlerFailure.md) + +In addition, individual transports that provide access to specific MoM sub-class *Subscription* to provide properties unique to the chosen middleware. We discuss those under a section for that transport. + +For RabbitMQ for example, this would look like this: + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + var subscriptions = new Subscription[] + { + new RmqSubscription( + new SubscriptionName("paramore.sample.salutationanalytics"), + new ChannelName("SalutationAnalytics"), + new RoutingKey("GreetingMade"), + runAsync: true, + timeoutInMilliseconds: 200, + isDurable: true, + makeChannels: OnMissingChannel.Create), //change to OnMissingChannel.Validate if you have infrastructure declared elsewhere + }; + + services.AddServiceActivator(options => + { + options.Subscriptions = subscriptions; + }) +} + +... + +``` + +#### **Gateway Connections & Channel Factories** + +A *Gateway Connection* tells Brighter how to connect to MoM for a particular transport. The transport package will contain a *Gateway Connection*, you need to provide the information to connect to your middleware (URIs, ports, credentials etc.) Your transport package provides a *Gateway Connection* + +A *Channel Factory* connects Brighter to MoM. Depending on the configuration settings for your *Subscription* it may create the required primitives (topics/routing keys, queues, streams) on MoM or simply attach to ones that you have created via Infrastructure as Code (IaC). Your transport provides a *Channel Factory* and you need to pass it a *Gateway Connection*. + +For RabbitMQ, this would look like: + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + + var rmqConnection = new RmqMessagingGatewayConnection + { + AmpqUri = new AmqpUriSpecification(new Uri($"amqp://guest:guest@local:5672")), + Exchange = new Exchange("paramore.brighter.exchange") + }; + + var rmqMessageConsumerFactory = new RmqMessageConsumerFactory(rmqConnection); + + services.AddServiceActivator(options => + { + options.ChannelFactory = new ChannelFactory(rmqMessageConsumerFactory); + }) +} + +... + +``` + +#### **Configuring Service Activator Lifetimes** + +Under the hood your *Service Activator* uses a *Command Processor* and you will need to configure lifetimes [as discussed above](#configuring-lifetimes). + +An additional requirement is configuring the lifetime of the *Command Processor* itself. Within the context of an ASP.NET application, configuring the lifetime of the **Command Processor** relies on ASP.NET creating an instance of the *Command Processor* in a request pipeline. When you are using *Service Activator* there is no ASP.NET pipeline, instead Brighter's *Dispatcher* manages the lifetime of the *Command Processor* that we pass a request to. By setting the **ServiceActivatorOptions.UseScoped** field to true, you instruct *Brighter* to use a new *Command Processor* instance for each request. This is important if you take the *Command Processor* as a dependency in any of your *Request Handlers* with a **Scoped** lifetime. If in doubt, just set **ServiceActivatorOptions.UseScoped** field to true. + + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + services.AddServiceActivator(options => + { + options.UseScoped = true; + options.HandlerLifetime = ServiceLifetime.Scoped; + options.MapperLifetime = ServiceLifetime.Singleton; + options.CommandProcessorLifetime = ServiceLifetime.Scoped; + }) +} + +... + +``` + + +### **Service Activator Brighter Builder Fluent Interface** + +The call to **AddServiceActivator()** returns an **IBrighterBuilder** fluent interface. This means that you can use any of the options described in [Brighter Build Fluent Interfaces](#brighter-builder-fluent-interface) to configure the associated *Command Processor* such as scanning assemblies for *Request Handlers* and adding an *External Bus* and *Outbox*. + +An option is intended for the context of a Service Activator is described below. + +#### **Inbox** + +As described in the [Outbox Pattern](/contents/OutboxPattern.md) an *Outbox* offers **Guaranteed, At Least Once** delivery. It explicitly may result in you sending duplicate messages. In addition, MoM tends to offer "At Least Once" guarantees only, further creating the risk that you will receive a duplicate message. + +If the request is not idempotent, you can use an Inbox to de-duplicate it. See [Inbox Support](/contents/BrighterInboxSupport.md) for more. + +Configuring an *Inbox* has two elements. The first is the type of *Inbox*, the second configuration for the *Inbox* behavior. + +Brighter provides a number of *Inbox* implementations for common Dbs (and you can write your own for a Db that we do not support). For this discussion we will look at Brighter's support for working with MySQL. See the documentation for working with specific *Inbox* implementations. + +For this we will need the *Inbox* packages for the MySQL *Inbox*. + +* **Paramore.Brighter.Inbox.MySql** + +For a given backing store the pattern should be Paramore.Brighter.Inbox.{DATABASE} where {DATABASE} is the name of the Db that you are using. + +To configure our *Inbox* we then need to use the UseExternalInbox method call and pass in an instance of a class that implements **IAmAnInbox**, taken from our package, and an instance of **InboxConfiguration** that tells Brighter how we want to use the Inbox. + +For *Inbox Configuration* you set the following properties: + +* **ActionOnExists**: What do we do if the request has been handled? The default,**OnceOnlyAction.Throw** is to throw a **OnceOnlyException**. If you take no other action this will cause the message to be rejected and sent to a DLQ if one is configured (See [Handler Failure](/contents/HandlerFailure.md)). The alternative is **OnceOnlyAction.Warn** simply logs that the request is a duplicate, but takes no other action. +* **OnceOnly**: This defaults to *true* and will check for a duplicate and take the action indicated by **ActionOnExists**. If *false* the *Inbox* will record the request, but will take no further action. (This tends to be set to *false* if you are using the *Inbox* to record what requests caused current state only and not de-duplicate). +* **Scope**: This indicates the type of request (*Command* or *Event*) to store in the *Inbox*. By default this is set to **InboxScope.All** and captures everything but you can be explicit and just capture **InboxScope.Commands** or **InboxScope.Events**. (This tends to be set to **InboxScope.Commands** when only commands cause changes to state that are not idempotent). +* **Context**: Used to uniquely identify receipt of this request via this handler. If you are recording *Events* and have multiple handlers, then the first event handler to receive the message will block the others from doing so, unless you disambiguate the handler identity by supplying a context method. + +A typical *Inbox* configuration for MySQL would be: + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + services.AddServiceActivator(options => + { ... }) + .UseExternalInbox( + new MySqlInbox(new MySqlInboxConfiguration("server=localhost; port=3306; uid=root; pwd=root; database=Salutations", "Inbox"); + new InboxConfiguration( + scope: InboxScope.Commands, + onceOnly: true, + actionOnExists: OnceOnlyAction.Throw + ) + ); +} + +... + +``` +Typically you would obtain the connection string for the Db from configuration (as opposed to hard coding the string), likewise for the table name for your *Inbox*. + + +### Running Service Activator + +To run *Service Activator* we add it as a [Hosted Service](https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-6.0&tabs=visual-studio). + +We provide the class **ServiceActivatorHostedService** for this in the NuGet package: + +* **Paramore.Brighter.ServiceActivator.Extensions.Hosting** + +The **ServiceActivatorHostedService** calls the **Dispatcher.Receive** method which starts message pumps for the configured *Subscriptions*. + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + ... + + services.AddHostedService(); +} + +... + +``` + +On shutdown Brighter will allow the current *Request Handler* to complete, then end the message pump loop and exit. If you have long-running handlers it is possible that they will not complete in the default 5s for graceful shutdown of the MS Generic Host. In this case, you need to [increase the timeout](https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/generic-host?view=aspnetcore-6.0#shutdowntimeout) of the host shutdown. + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + services.Configure(options => + { + options.ShutdownTimeout = TimeSpan.FromSeconds(20); + }); + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + ... + + services.AddHostedService(); +} + +``` + +### A Complete Service Activator Example + +When all of the relevant configuration sections are added together, your code will look something like this, with variations for your transport and stores. + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + services.Configure(options => + { + options.ShutdownTimeout = TimeSpan.FromSeconds(20); + }); + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + var subscriptions = new Subscription[] + { + new RmqSubscription( + new SubscriptionName("paramore.sample.salutationanalytics"), + new ChannelName("SalutationAnalytics"), + new RoutingKey("GreetingMade"), + runAsync: true, + timeoutInMilliseconds: 200, + isDurable: true, + makeChannels: OnMissingChannel.Create), //change to OnMissingChannel.Validate if you have infrastructure declared elsewhere + }; + + var rmqConnection = new RmqMessagingGatewayConnection + { + AmpqUri = new AmqpUriSpecification(new Uri($"amqp://guest:guest@localhost:5672")), + Exchange = new Exchange("paramore.brighter.exchange") + }; + + var rmqMessageConsumerFactory = new RmqMessageConsumerFactory(rmqConnection); + + services.AddServiceActivator(options => + { + options.Subscriptions = subscriptions; + options.ChannelFactory = new ChannelFactory(rmqMessageConsumerFactory); + options.UseScoped = true; + options.HandlerLifetime = ServiceLifetime.Scoped; + options.MapperLifetime = ServiceLifetime.Singleton; + options.CommandProcessorLifetime = ServiceLifetime.Scoped; + }) + .AutoFromAssemblies() + .UseExternalInbox( + new MySqlInbox(new MySqlInboxConfiguration("server=localhost; port=3306; uid=root; pwd=root; database=Salutations", "Inbox"); + new InboxConfiguration( + scope: InboxScope.Commands, + onceOnly: true, + actionOnExists: OnceOnlyAction.Throw + ) + ) + + services.AddHostedService(); + +} + +``` + +## Samples + +Brighter includes a comprehensive set of [Samples](https://github.com/BrighterCommand/Brighter/tree/master/samples) in its main repo that you can review for clarity on how Brighter works and should be configured. + + diff --git a/contents/BrighterInboxSupport.md b/contents/10/BrighterInboxSupport.md similarity index 100% rename from contents/BrighterInboxSupport.md rename to contents/10/BrighterInboxSupport.md diff --git a/contents/BrighterOutboxSupport.md b/contents/10/BrighterOutboxSupport.md similarity index 100% rename from contents/BrighterOutboxSupport.md rename to contents/10/BrighterOutboxSupport.md diff --git a/contents/BuildingAPipeline.md b/contents/10/BuildingAPipeline.md similarity index 100% rename from contents/BuildingAPipeline.md rename to contents/10/BuildingAPipeline.md diff --git a/contents/BuildingAnAsyncPipeline.md b/contents/10/BuildingAnAsyncPipeline.md similarity index 100% rename from contents/BuildingAnAsyncPipeline.md rename to contents/10/BuildingAnAsyncPipeline.md diff --git a/contents/ClaimCheck.md b/contents/10/ClaimCheck.md similarity index 100% rename from contents/ClaimCheck.md rename to contents/10/ClaimCheck.md diff --git a/contents/CommandsCommandDispatcherandProcessor.md b/contents/10/CommandsCommandDispatcherandProcessor.md similarity index 100% rename from contents/CommandsCommandDispatcherandProcessor.md rename to contents/10/CommandsCommandDispatcherandProcessor.md diff --git a/contents/Compression.md b/contents/10/Compression.md similarity index 100% rename from contents/Compression.md rename to contents/10/Compression.md diff --git a/contents/DapperOutbox.md b/contents/10/DapperOutbox.md similarity index 100% rename from contents/DapperOutbox.md rename to contents/10/DapperOutbox.md diff --git a/contents/DarkerBasicConfiguration.md b/contents/10/DarkerBasicConfiguration.md similarity index 100% rename from contents/DarkerBasicConfiguration.md rename to contents/10/DarkerBasicConfiguration.md diff --git a/contents/DispatchingARequest.md b/contents/10/DispatchingARequest.md similarity index 100% rename from contents/DispatchingARequest.md rename to contents/10/DispatchingARequest.md diff --git a/contents/DynamoInbox.md b/contents/10/DynamoInbox.md similarity index 100% rename from contents/DynamoInbox.md rename to contents/10/DynamoInbox.md diff --git a/contents/DynamoOutbox.md b/contents/10/DynamoOutbox.md similarity index 100% rename from contents/DynamoOutbox.md rename to contents/10/DynamoOutbox.md diff --git a/contents/EFCoreOutbox.md b/contents/10/EFCoreOutbox.md similarity index 100% rename from contents/EFCoreOutbox.md rename to contents/10/EFCoreOutbox.md diff --git a/contents/EventCarriedStateTransfer.md b/contents/10/EventCarriedStateTransfer.md similarity index 100% rename from contents/EventCarriedStateTransfer.md rename to contents/10/EventCarriedStateTransfer.md diff --git a/contents/EventDrivenCollaboration.md b/contents/10/EventDrivenCollaboration.md similarity index 100% rename from contents/EventDrivenCollaboration.md rename to contents/10/EventDrivenCollaboration.md diff --git a/contents/FAQ.md b/contents/10/FAQ.md similarity index 100% rename from contents/FAQ.md rename to contents/10/FAQ.md diff --git a/contents/FeatureSwitches.md b/contents/10/FeatureSwitches.md similarity index 100% rename from contents/FeatureSwitches.md rename to contents/10/FeatureSwitches.md diff --git a/contents/HandlerFailure.md b/contents/10/HandlerFailure.md similarity index 100% rename from contents/HandlerFailure.md rename to contents/10/HandlerFailure.md diff --git a/contents/HealthChecks.md b/contents/10/HealthChecks.md similarity index 100% rename from contents/HealthChecks.md rename to contents/10/HealthChecks.md diff --git a/contents/HowBrighterWorks.md b/contents/10/HowBrighterWorks.md similarity index 100% rename from contents/HowBrighterWorks.md rename to contents/10/HowBrighterWorks.md diff --git a/contents/HowConfiguringTheCommandProcessorWorks.md b/contents/10/HowConfiguringTheCommandProcessorWorks.md similarity index 100% rename from contents/HowConfiguringTheCommandProcessorWorks.md rename to contents/10/HowConfiguringTheCommandProcessorWorks.md diff --git a/contents/HowConfiguringTheDispatcherWorks.md b/contents/10/HowConfiguringTheDispatcherWorks.md similarity index 100% rename from contents/HowConfiguringTheDispatcherWorks.md rename to contents/10/HowConfiguringTheDispatcherWorks.md diff --git a/contents/HowServiceActivatorWorks.md b/contents/10/HowServiceActivatorWorks.md similarity index 100% rename from contents/HowServiceActivatorWorks.md rename to contents/10/HowServiceActivatorWorks.md diff --git a/contents/ImplementAQueryHandler.md b/contents/10/ImplementAQueryHandler.md similarity index 100% rename from contents/ImplementAQueryHandler.md rename to contents/10/ImplementAQueryHandler.md diff --git a/contents/ImplementingAHandler.md b/contents/10/ImplementingAHandler.md similarity index 100% rename from contents/ImplementingAHandler.md rename to contents/10/ImplementingAHandler.md diff --git a/contents/ImplementingAsyncHandler.md b/contents/10/ImplementingAsyncHandler.md similarity index 100% rename from contents/ImplementingAsyncHandler.md rename to contents/10/ImplementingAsyncHandler.md diff --git a/contents/ImplementingExternalBus.md b/contents/10/ImplementingExternalBus.md similarity index 100% rename from contents/ImplementingExternalBus.md rename to contents/10/ImplementingExternalBus.md diff --git a/contents/KafkaConfiguration.md b/contents/10/KafkaConfiguration.md similarity index 100% rename from contents/KafkaConfiguration.md rename to contents/10/KafkaConfiguration.md diff --git a/contents/Logging.md b/contents/10/Logging.md similarity index 100% rename from contents/Logging.md rename to contents/10/Logging.md diff --git a/contents/MSSQLInbox.md b/contents/10/MSSQLInbox.md similarity index 100% rename from contents/MSSQLInbox.md rename to contents/10/MSSQLInbox.md diff --git a/contents/MessageMappers.md b/contents/10/MessageMappers.md similarity index 100% rename from contents/MessageMappers.md rename to contents/10/MessageMappers.md diff --git a/contents/Microservices.md b/contents/10/Microservices.md similarity index 100% rename from contents/Microservices.md rename to contents/10/Microservices.md diff --git a/contents/Monitoring.md b/contents/10/Monitoring.md similarity index 100% rename from contents/Monitoring.md rename to contents/10/Monitoring.md diff --git a/contents/MySQLInbox.md b/contents/10/MySQLInbox.md similarity index 100% rename from contents/MySQLInbox.md rename to contents/10/MySQLInbox.md diff --git a/contents/OutboxPattern.md b/contents/10/OutboxPattern.md similarity index 100% rename from contents/OutboxPattern.md rename to contents/10/OutboxPattern.md diff --git a/contents/PolicyFallback.md b/contents/10/PolicyFallback.md similarity index 100% rename from contents/PolicyFallback.md rename to contents/10/PolicyFallback.md diff --git a/contents/PolicyRetryAndCircuitBreaker.md b/contents/10/PolicyRetryAndCircuitBreaker.md similarity index 100% rename from contents/PolicyRetryAndCircuitBreaker.md rename to contents/10/PolicyRetryAndCircuitBreaker.md diff --git a/contents/PostgresInbox.md b/contents/10/PostgresInbox.md similarity index 100% rename from contents/PostgresInbox.md rename to contents/10/PostgresInbox.md diff --git a/contents/RabbitMQConfiguration.md b/contents/10/RabbitMQConfiguration.md similarity index 100% rename from contents/RabbitMQConfiguration.md rename to contents/10/RabbitMQConfiguration.md diff --git a/contents/Requests, Commands and Events.md b/contents/10/Requests, Commands and Events.md similarity index 100% rename from contents/Requests, Commands and Events.md rename to contents/10/Requests, Commands and Events.md diff --git a/contents/ReturningResultsFromAHandler.md b/contents/10/ReturningResultsFromAHandler.md similarity index 100% rename from contents/ReturningResultsFromAHandler.md rename to contents/10/ReturningResultsFromAHandler.md diff --git a/contents/Routing.md b/contents/10/Routing.md similarity index 100% rename from contents/Routing.md rename to contents/10/Routing.md diff --git a/contents/S3LuggageStore.md b/contents/10/S3LuggageStore.md similarity index 100% rename from contents/S3LuggageStore.md rename to contents/10/S3LuggageStore.md diff --git a/contents/ShowMeTheCode.md b/contents/10/ShowMeTheCode.md similarity index 100% rename from contents/ShowMeTheCode.md rename to contents/10/ShowMeTheCode.md diff --git a/contents/SqliteInbox.md b/contents/10/SqliteInbox.md similarity index 100% rename from contents/SqliteInbox.md rename to contents/10/SqliteInbox.md diff --git a/contents/TaskQueuePattern.md b/contents/10/TaskQueuePattern.md similarity index 100% rename from contents/TaskQueuePattern.md rename to contents/10/TaskQueuePattern.md diff --git a/contents/Telemetry.md b/contents/10/Telemetry.md similarity index 100% rename from contents/Telemetry.md rename to contents/10/Telemetry.md diff --git a/contents/UsingTheContextBag.md b/contents/10/UsingTheContextBag.md similarity index 100% rename from contents/UsingTheContextBag.md rename to contents/10/UsingTheContextBag.md diff --git a/contents/9/AWSSQSConfiguration.md b/contents/9/AWSSQSConfiguration.md new file mode 100644 index 0000000..05f99bc --- /dev/null +++ b/contents/9/AWSSQSConfiguration.md @@ -0,0 +1,168 @@ +# AWS SQS Configuration + +## General + +SNS and SQS are proprietary message-oriented-middleware available on the AWS platform. Both are well documented: see [SNS](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) and [SQS](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html). Brighter handles the details of sending to SNS using an SQS queue for the consumer. You might find the [documentation for the AWS .NET SDK](https://docs.aws.amazon.com/sdk-for-net/) helpful when debugging, but you should not have to interact with it directly to use Brighter. + +It is useful to understand the relationship between these two components: + +- **SNS**: A routing table, [SNS](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) provides routing for messages to subscribers. Subscribers include, but are not limited to, SQS [see SNS Subscribe Protocol](https://docs.aws.amazon.com/sns/latest/api/API_Subscribe.html). An entry in the table is a **Topic**. +- **SQS**: A store-and-forward queue over which a consumer receives messages. A message is locked whilst a consumer has read it, until they ack it, upon which it is deleted from the queue, or nack it, upon which it is unlocked. A policy controls movement of messages that cannot be delivered to a DLQ. [SQS](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html) may be used for point-to-point integration, and does not require SNS. + +Brighter only supports the scenario where SNS is used as a routing table, and an SQS subscribes to a **topic**. It does not support stand-alone SQS queues. Point-to-point scenarios can be modelled as an SNS **topic** with one subscribing queue. + +## Connection + +The Connection to AWS is provided by an **AWSMessagingGatewayConnection**. This is a wrapper around AWS credentials and region, that allows us to create the .NET clients that abstract various AWS HTTP APIs. We require the following parameters: + +- **Credentials**: An instance of *AWSCredentials*. Storing and retrieving the credentials is a detail for your application and may vary by environment. There is AWS discussion of credentials resolution [here](https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/creds-assign.html) +- **Region**: The *RegionEndpoint* to use. SNS is a regional service, so we need to know which region to provision infrastructure in, or find it from. + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + if (!new CredentialProfileStoreChain().TryGetAWSCredentials("default", out var credentials) + throw InvalidOperationException("Missing AWS Credentials); + + services.AddBrighter(...) + .UseExternalBus(new SnsProducerRegistryFactory + new AwsMessagingGatewayConnection(credentials, Environment.GetEnvironmentVariable("AWS_REGION")) + , + ... //publication, see below + ).Create() +} +``` +## Publication + +For more on a *Publication* see the material on an *External Bus* in [Basic Configuration](/contents/BrighterBasicConfiguration.md#using-an-external-bus). + +Brighter's **Routing Key** represents the [SNS Topic Name](https://docs.aws.amazon.com/sns/latest/api/API_CreateTopic.html). + +### Finding and Creating Topics +Depending on the option you choose for how we handle required messaging infrastructure (Create, Validate, Assume), we will need to determine if a **Topic** already exists, when we want to create it if missing, or validate it. + +Naively using the AWS SDK's **FindTopic** method is an expensive operation. This enumerates all the **Topics** in that region, looking for those that have a matching name. Under-the-hood the client SDK pages through your topics. If you have a significant number of topics, this is expensive and subject to rate limiting. + +As creating a **Topic** is an *idempotent* operation in SNS, if asked to Create we do so without first searching to see if it already exists because of the cost of validation. + +If you create your infrastructure out-of-band, and ask us validate it exists, to mitigate the cost of searching for topics, we provide several options under **FindTopicBy**. + +- **FindTopicBy**: How do we find the topic: + - **TopicFindBy.Arn** -> On a *Publication*, the routing key is the **Topic** name, but you explicitly supply the ARN in another field: **TopicArn**. On a *Subscription* the routing key is the **Topic** ARN. + - **TopicFindBy.Convention** -> The routing key is the **Topic** name, and we use convention to construct the ARN from it + - **TopicFindBy.Name** -> The routing key is the **Topic** name & we use ListTopics to find it (rate limited 30/s) + +#### TopicFindBy.Arn +We use **GetTopicAttributesAsync** SDK method to request attributes of a Topic with the ARN supplied in **TopicArn**. If this call fails with a NotFoundException, we know that the Topic does not exist. This is a *hack*, but is much more efficient than enumeration as a way of determining if the ARN exists. + +#### TopicFindBy.Convention +If you supply only the **Topic** name via the routing key, we construct the ARN by convention as follows: + +``` csharp +var arn = new Arn + { + Partition = //derived from the partition of the region you supplied to us, + Service = "sns", + Region = //derived from the system name of the region you supplied to us, + AccountId = //your account id - derived from the credentials you supplied, + Resource = topicName + } +``` + +These assumptions work, if the topic is created by the account your credentials belong to. If not, you can't use by convention. + +Once we obtain an ARN by convention, we can then use the optimized approach described under [TopicFindBy.Arn](#topicfindbyarn) to confirm that your topic exists. + +#### TopicFindBy.Name +If you supply a name, but we can't construct the ARN via the above conventions, we have to fall back to the **SDKs** **FindTopic** approach. + +Because creation is idempotent, and **FindTopic** is expensive, you are almost always better off choosing to create over validating a topic by name. + +If you are creating the topics out-of-band, by CloudFormation for example, and so do not want Brighter the risk that Brighter will create them, then you will have an ARN. In that case you should use [TopicFindBy.Arn](#topicfindbyarn) or assume that any required infrastructure exists. + +### Other Attributes +- **SNSAttributes**: This property lets you pass through an instance of **SNSAttributes** which has properties representing the attributes used when creating a **Topic**. These are only used if you are creating a **Topic**. + - **DeliveryPolicy**: The policy that defines how Amazon SNS retries failed deliveries to HTTP/S endpoints. + - **Policy**: The policy that defines who can access your topic. By default, only the topic owner can publish or subscribe to the topic. + - **Tags**: A list of resource tags to use. + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + if (!new CredentialProfileStoreChain().TryGetAWSCredentials("default", out var credentials) + throw InvalidOperationException("Missing AWS Credentials); + + services.AddBrighter(...) + .UseExternalBus( + ...,//connection, see above + new SnsPublication[] + { + new SnsPublication() + { + Topic = new RoutingKey("GreetingEvent"), + FindTopicBy = TopicFindBy.Convention + } + } + ).Create() +} +``` + +## Subscription + +As normal with Brighter, we allow **Topic** creation from the *Subscription*. Because this works in the same way as the *Publication* see the notes under [Publication](#publication) for further detail on the options that you can configure around creation or validation. + +In SNS you need to [subscribe](https://docs.aws.amazon.com/sns/latest/api/API_Subscribe.html) to a **Topic** to receive messages from that **Topic**. Brighter subscribes using an SQS queue (there are other options for SNS, but Brighter does not use those). Much of the *Subscription* configuration allows you to control the parameters of that *Subscription*. + +We support the following properties on an *SQS Subscription* most of which relate to the creation of the [SQS Queue](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_CreateQueue.html) with which we subscribe: + +- **LockTimeout**: How long, in seconds, a 'lock' is held on a message for one consumer before it times out(*VisibilityTimeout*). Default is 10s. +- **DelaySeconds**: The length of time, in seconds, for which the delivery of all messages in the queue is delayed. Default is 0. +- **MessageRetentionPeriod**: The length of time, in seconds, for which Amazon SQS retains a message on a queue before deleting it. Default is 4 days. +- **IAMPolicy**: The queue's policy. A valid [AWS policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html). +- **RawMessageDelivery**: Indicate that the Raw Message Delivery setting is enabled or disabled. Defaults to true. +- **RedrivePolicy**: The parameters for the dead-letter queue functionality of the source queue. An instance of the **RedrivePolicy** class, which has the following parameters: + - **MaxReceiveCount**: The maximum number of requeues for a message before we push it to the DLQ instead + - **DeadlLetterQueueName**: The name of the dead letter queue we want to associate with any redrive policy. + + +``` csharp +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + (AWSCredentials credentials, RegionEndpoint region) = CredentialsChain.GetAwsCredentials(); + var awsConnection = new AWSMessagingGatewayConnection(credentials, region); + + var subscriptions = new Subscription[] + { + new SqsSubscription( + name: new SubscriptionName("Subscription-Name), + channelName: new ChannelName("Channel-Name"), + routingKey: new RoutingKey("arn:aws:sns:us-east-2:444455556666:MyTopic"), + findTopicBy: TopicFindBy.Arn, + makeChannels: OnMissingChannel.Validate + ); + } + + var sqsMessageConsumerFactory = new SqsMessageConsumerFactory(awsConnection); + + services.AddServiceActivator(options => + { + options.Subscriptions = subscriptions; + options.ChannelFactory = new ChannelFactory(sqsMessageConsumerFactory); + ... //see Basic Configuration + }) +``` + +### Ack and Nack + +As elsewhere, Brighter only Acks after your handler has run to process the message. We will Ack unless you throw a **DeferMessageAction**. See [Handler Failure](/contents/HandlerFailure.md) for more. + +An Ack will delete the message from the SQS queue using the SDK's **DeleteMessageAsync**. + +In response to a DeferMessageAction we will requeue, using the SDK's **ChangeMessageVisibilityAsync** to make the message available again to other consumers. + +On a Nack, we will move the message to a DLQ, if there is one. We Nack when we exceed the requeue count for a message, or we raise a ConfigurationException. + + + + + diff --git a/contents/9/AsyncDispatchARequest.md b/contents/9/AsyncDispatchARequest.md new file mode 100644 index 0000000..6673712 --- /dev/null +++ b/contents/9/AsyncDispatchARequest.md @@ -0,0 +1,98 @@ +# Dispatching Requests Asynchronously + +Once you have [implemented your Request Handler](ImplementingAHandler.html), you will want to dispatch **Commands** or **Events** to that Handler. + +## Usage + +In the following example code we register a handler, create a command processor, and then use that command processor to send a command to the handler asynchronously. + + +``` csharp + public class Program + { + private static async Task Main(string[] args) + { + var host = Host.CreateDefaultBuilder() + .ConfigureServices((hostContext, services) => + + { + services.AddBrighter() + .AutoFromAssemblies(); + } + ) + .UseConsoleLifetime() + .Build(); + + var commandProcessor = host.Services.GetService(); + + await commandProcessor.SendAsync(new GreetingCommand("Ian")); + + await host.RunAsync(); + } +``` + +## Registering a Handler + +In order for a **Command Dispatcher** to find a Handler for your **Command** or **Event** you need to register the association between that **Command** or **Event** and your Handler. + +Brighter's **HostBuilder** support provides **AutoFromAssemblies** to register any *Request Handlers* in the project. See [Basic Configuration](/contents/BrighterBasicConfiguration.md) for more. + +### Pipelines Must be Homogeneous + +Brighter only supports pipelines that are solely **IHandleRequestsAsync** or **IHandleRequests**. In particular, note that middleware (attributes on your handler) must be of the same type as the rest of your pipeline. A common mistake is to **UsePolicy** when you mean **UsePolicyAsync**. + +## Dispatching Requests + +Once you have registered your Handlers, you can dispatch requests to them. To do that you simply use the **commandProcessor.SendAsync()** (or **commandProcessor.PublishAsync()**) method passing in an instance of your command. *Send* expects one handler, *Publish* expects zero or more. (You can use **commandProcessor.DepositPostAsync** and **commandProcessor.ClearOutboxAsync** with an External Bus). + +``` csharp +await commandProcessor.SendAsync(new GreetingCommand("Ian")); +``` + +### Returning results to the caller. + +A Command does not have return value and **CommandDispatcher.Send()** does not return anything. Please see a discussion on how to handle this in [Returning Results from a Handler](/contents/ReturningResultsFromAHandler.md). + +### Cancellation + +Brighter supports the cancellation of asynchronous operations. + +The asynchronous methods: **SendAsync** and **PublishAsync** accept a **CancellationToken** and pass this token down the pipeline. The parameter defaults to default(CancellationToken) where the call does not intend to cancel. + +The responsibility for checking for a cancellation request lies with the individual handlers, which must determine what action to take if cancellation had been signalled. + +### Async Callback Context + +When an awaited method completes, what thread runs any completion code? There are two options: + +- The original thread that was running when the await began +- A new thread allocated from the thread pool + +Why does this matter? Because if you needed to access anything that is thread local, being called back on a new thread means you will not have access to those variables. + +As a result, when awaiting it is possible to configure how the continuation runs. + +- To run on the original thread, requires the CLR to capture information on the thread you were using. This is the SynchronizationContext; because the CLR must record this information, we refer to it as a captured context. Your execution will be queued back on to the original context, which has a performance cost. +- To run on a new thread, using the Task Scheduler to allocate from the thread pool. + +You can use ConfigureAwait to control this. This article explains why you might wish to use [ConfigureAwait](https://devblogs.microsoft.com/dotnet/configureawait-faq/), in more depth. + +As a library, we need to allow you to make this choice for your handler chain. For this reason, our *Async methods support the parameter **continueOnCapturedContext**. + +Library writers are encouraged to default to false i.e. use the Task Scheduler instead of the SychronizationContext. Brighter adopts this default, but it might not be what you want if your handler needs to run in the context of the original thread. As a result we let you use this parameter on the **\*Async** calls to change the behaviour throughout your pipeline. + +``` csharp +await commandProcessor.SendAsync(new GreetingCommand("Ian"), continueOnCapturedContext: true); +``` + +A handler exposes the parameter you supply via the property **ContinueOnCapturedContext**. + +You should pass this value via **ConfigureAwait** if you need to be able to support making this choice at the call site. For example, when you call the base handler in your return statement, to ensure that the decision as to whether to use the scheduler or the context flows down the pipeline. + +``` csharp +return await base.HandleAsync(command, ct).ConfigureAwait(ContinueOnCapturedContext); +``` + +You can ignore this, if you want to default to using the Task Scheduler. + + diff --git a/contents/9/AzureBlobArchiveProvider.md b/contents/9/AzureBlobArchiveProvider.md new file mode 100644 index 0000000..1fb9810 --- /dev/null +++ b/contents/9/AzureBlobArchiveProvider.md @@ -0,0 +1,40 @@ +# Azure Blob Archive Provider + +## Usage +The Azure Blob Archive Provider is a provider for [Outbox Archiver](/contents/BrighterOutboxSupport.md#outbox-archiver). + +For this we will need the *Archive* packages for the Azure *Archive Provider*. + +* **Paramore.Brighter.Archive.Azure** + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + services.AddBrighter(options => + { ... }) + .UseOutboxArchiver( + new AzureBlobArchiveProvider(new AzureBlobArchiveProviderOptions() + { + BlobContainerUri = "https://brighterarchivertest.blob.core.windows.net/messagearchive", + TokenCredential = New AzCliCredential(); + } + ), + options => { + TimerInterval = 5; // Every 5 seconds + BatchSize = 500; // 500 messages at a time + MinimumAge = 744; // 1 month + } + ); +} + +... + +``` + diff --git a/contents/9/AzureBlobConfiguration.md b/contents/9/AzureBlobConfiguration.md new file mode 100644 index 0000000..3af6ed9 --- /dev/null +++ b/contents/9/AzureBlobConfiguration.md @@ -0,0 +1,24 @@ +# Azure Archive Provider Configuration + +## General +Azure Service Bus (ASB) is a fully managed enterprise message broker and is [well documented](https://docs.microsoft.com/en-us/azure/service-bus-messaging/) Brighter handles the details of sending to or receiving from ASB. You may find it useful to understand the [concepts](https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-queues-topics-subscriptions) of the ASB. + +## Connection +At this time Azure Blob Archive Provider only supports Token Credential for authentication + +## Permissions +For the archiver to work the calling credential will require the role **Storage Blob Data Owner** however if **TagBlobs** is set to False then **Storage Blob Data Contributor** will be adequate. If you feel that Data Owner is too high you can create a custom role encompasing Contributor and 'Microsoft.Storage/storageAccounts/blobServices/containers/blobs/tags/write' + +## Options + +* **BlobContainerUri** : The URI of the Blob container to store messages in (i.e. "https://BlobTest.blob.core.windows.net/messagearchive) +* **TokenCredential** : The Credential to use when writing the Blob +* **AccessTier** : The Access Tier to write to the blob +* **TagBlobs** : if this is set to True the defined in **TagsFunc** will be written to the blobs +* **TagsFunc** : The function to arrange the tags to add when storing, please note that **TagBlobs** must be True for these to be used, default Tags : + - topic + - correlationId + - message_type + - timestamp + - content_type +* **StorageLocationFunc** : The function to provide the location to store the message inside of the Blob container, default location : The Id of the message at the root of the **BlobContainerUri** \ No newline at end of file diff --git a/contents/9/AzureServiceBusConfiguration.md b/contents/9/AzureServiceBusConfiguration.md new file mode 100644 index 0000000..77fadd5 --- /dev/null +++ b/contents/9/AzureServiceBusConfiguration.md @@ -0,0 +1,121 @@ +# Azure Service Bus Configuration + +## General +Azure Service Bus (ASB) is a fully managed enterprise message broker and is [well documented](https://docs.microsoft.com/en-us/azure/service-bus-messaging/) Brighter handles the details of sending to or receiving from ASB. You may find it useful to understand the [concepts](https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-queues-topics-subscriptions) of the ASB. + +## Connection +The connection to ASB id defined by an **IServiceBusClientProvider**, Brighter proviedes the following Implimentations + +* **ServiceBusChainedClientProvider**: A client provider that allows you to specific a chain of **TokenCredentials** to authenticate with. + +* **ServiceBusConnectionStringClientProvider**: A client provider that accepts a connection string (containg Authentication information) + +* **ServiceBusDefaultAzureClientProvider**: A client provider that uses the Default Azure Credential to authenticate. + +* **ServiceBusManagedIdentityClientProvider**: A client provider that uses Azure Managed Identity to authenticate. + +* **ServiceBusVisualStudioCredentialClientProvider**: A client provider that uses Visual Studio Credential to authenticate. + +In Brighter's implementation of the Messaging Gateway *Publications* and *Subscriptions* have their own Individual configuration. + +## Publication + +No custom properties are supported for ASB + +Basic Brighter configutarion publications is as follows + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .UseExternalBus( + new AzureServiceBusProducerRegistryFactory( + asbConnection, + new AzureServiceBusPublication[] + { + new() { Topic = new RoutingKey("greeting.event") }, + new() { Topic = new RoutingKey("greeting.addGreetingCommand") }, + new() { Topic = new RoutingKey("greeting.Asyncevent") } + } + ) + .Create() + ) +} +``` + +For more on a *Publication* see the material on an *External Bus* in [Basic Configuration](/contents/BrighterBasicConfiguration.md#using-an-external-bus). + +## Subscription + +For more on a *Subscription* see the material on configuring *Service Activator* in [Basic Configuration](/contents/BrighterBasicConfiguration.md#configuring-the-service-activator). + +When + +We support a number of ASB specific *Subscription* options: + +* **MaxDeliveryCount**: The Maximum amount of times that a Message can be delivered before it is dead Lettered. This differs from **requeue count** as this is used by the transport in the event of lock expiry (in the event of process failure or processing taking too long) **default:** 5 + +* **DeadLetteringOnMessageExpiration**: Dead letter a message when it expires **default:** true + +* **LockDuration**: How long message locks are held for **default:** true + +* **DefaultMessageTimeToLive**: How long messages sit in the queue before they expire **default:** 1 minute + +* **SqlFilter**: A Sql Filter to apply to the *subscription* see [Topic Filters](https://docs.microsoft.com/en-us/azure/service-bus-messaging/topic-filters) **default:** none + + +This is a typical *Subscription* configuration in a Consumer application: + +``` csharp +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + var subscriptions = new Subscription[] + { + new AzureServiceBusSubscription( + new SubscriptionName(GreetingEventAsyncMessageMapper.Topic), + new ChannelName(subscriptionName), + new RoutingKey(GreetingEventAsyncMessageMapper.Topic), + timeoutInMilliseconds: 400, + makeChannels: OnMissingChannel.Create, + requeueCount: 3, + isAsync: true, + noOfPerformers: 2, unacceptableMessageLimit: 1), + new AzureServiceBusSubscription( + new SubscriptionName(GreetingEventMessageMapper.Topic), + new ChannelName(subscriptionName), + new RoutingKey(GreetingEventMessageMapper.Topic), + timeoutInMilliseconds: 400, + makeChannels: OnMissingChannel.Create, + requeueCount: 3, + isAsync: false, + noOfPerformers: 2), + new AzureServiceBusSubscription( + new SubscriptionName(AddGreetingMessageMapper.Topic), + new ChannelName(subscriptionName), + new RoutingKey(AddGreetingMessageMapper.Topic), + timeoutInMilliseconds: 400, + makeChannels: OnMissingChannel.Create, + requeueCount: 3, + isAsync: true, + noOfPerformers: 2) + }; + + var clientProvider = new ServiceBusVisualStudioCredentialClientProvider("my-awesome-asb.servicebus.windows.net"); + + var asbConsumerFactory = new AzureServiceBusConsumerFactory(clientProvider); + + builder.Services.AddServiceActivator(options => + { + options.Subscriptions = subscriptions; + options.ChannelFactory = new AzureServiceBusChannelFactory(asbConsumerFactory); + + } +``` + +## Complete Reject + +We use ASB's *Subscription* to surscribe to a Topic on a namespace. + +When we Complete a message, in response to a handler chain completing, we Complete the message on ASB using **messageReceiver.CompleteMessageAsync**. Note that we only Complete a message once we have completed running the chain and only if AckOnRead is set to false (as the messages is removed from the queue otherwise). + +When we Dead Letter a message (see [Handler Failure](/contents/HandlerFailure.md) for more on failure) then we use **messageReceiver.DeadLetterMessageAsync** to delete the message, and move it to a DLQ. \ No newline at end of file diff --git a/contents/9/BasicConcepts.md b/contents/9/BasicConcepts.md new file mode 100644 index 0000000..88267c0 --- /dev/null +++ b/contents/9/BasicConcepts.md @@ -0,0 +1,121 @@ +# Basic Concepts + +## Command + +A command is an instruction to carry out work. It exercises the domain and results in a change of state. It expects a single handler. + +An [event](#event) may be used to indicate the outcome of a command. + +## Command Processor + +In Brighter, a command processor allows you to use the *Command Pattern* to separate caller from the executor, typically when separating I/O from domain code. It acts both as a *Command Dispatcher* which allows the separation of the parameters of a [request](#request) from the [handler](#request-handler) that executes that request and a *Command Processor* that allows you to use a middleware [pipeline](#pipeline) to provide additional and re-usable behaviors when processing that request. + +The Command Processor may dispatch to an [Internal Bus](#internal-bus) or an [External Bus](#external-bus). + +## Command-Query Separation (CQS) + +Command-Query separation is the principle that because a [query](#query) should never have the unexpected *side-effect* of updating state, a query should clearly be distinguished from a [command](#request). A query reports on the state of a domain, a command changes it. + +## Event + +An event is a fact. The domain may be updated to reflect the fact represented by the event. There may be no subscribers to an event. It may be skinny, a notification, where the fact is the event itself, or fat, a document, where the event provides facts describing a change. + +An event may be used to indicate the outcome of a [command](#command). + +## Event Stream + +In [message oriented middleware](#message-oriented-middleware-mom), an event stream delivers [messages](#message) (or records) via a steam. A consumer reads the stream at a specific offset from the start. Consumers can store their offsets to resume reading the stream for where they left off, or reset their offset to re-read a stream. Consumers neither lock, nor delete messages from the stream. For consuming apps to scale, the stream can be partitioned, allowing offsets to be maintained of a partition of the stream. By using separate consumer threads or processes to read a partition, an application can ensure that it is able to reduce the latency of reading the stream. + +Examples: Kafka, Kinesis, Redis Streams + +## External Bus + +An external bus allows a [command](#command) or [event](#event) to be turned into a [message](#message) and sent over message-oriented-middleware via broker to a [message queue](#message-queue) or [event stream](#event-stream). + +Brighter also offers a [service activator](#service-activator) to listen for messages published to a queue or stream and forward them to an [internal bus](#internal-bus) within another process. + +## Internal Bus + +A [command](#command), [event](#event) or [query](#query) is executed in-process, passed from the [command processor](#command-processor) or [query processor](#) to a [handler](#request-handler) [pipeline](#pipeline). + +## Message + +A message is a packet of data sent over message-oriented-middleware. It's on-the-wire representation is defined by the protocol used by [message-oriented-middleware](#message-oriented-middleware-mom). + +## Message Oriented Middleware (MoM) + +The class of applications that deliver a [message](#message) from one process to another. MoM may send messages either point-to-point (with just a [message queue](#message-queue)) between sender and receiver, or via a broker, which acts as a dynamic router for messages between sender and receiver. With a broker, the receiver often establishes a subscription to a routing table entry (a *routing key* or *topic*) via a [message queue](#message-queue) or an [event stream](#event-stream). + +Brighter abstracts a specific type of message-oriented middleware by a *Transport*. + +For simplicity, Brighter only supports transports that have a broker configuration, not point-to-point. If you need point-to-point semantics, configure your routing table entry so that it only delivers to one consuming queue or stream. + +## Message Mappers + +A message mapper turns domain code into a message: a header and a body, or turns a message into domain code. Because [message oriented middleware](#message-oriented-middleware-mom) typically looks in a header for routing information, it is also where you add routing information via the header. + +Each individual transport has code to turn a Brighter format message into a message oriented middleware compatible message, and vice versa, so your code only needs to translate to and from the Brighter format. + +## Message Queue + +In [message oriented middleware](#message-oriented-middleware-mom), a message queue delivers [messages](#message) via a queue. A consumer locks a message, processes it, and when it acknowledges it, it is deleted from the queue. Other consumers can process the same queue, and read past any locked messages. This allows scaling via the competing consumers pattern. A nack will release the lock and make a message visible in the queue again, sometimes with a delay. A dead-letter-queue (DLQ) can be used with a nack, to limit the number of retries before a message is considered to be "posion pill" and moved to another queue for undeliverable messages. + +Examples: SQS, AMQP 0-9-1 (Rabbit MQ), AMQP 1-0 (Azure Service Bus). + +## Pipeline + +A pipeline is a sequence of [handlers](#request-handler) that respond to a [request](#request) or [query](#query). The last handler in the sequence is the "target" handler, which forms the pipeline sink. Handlers prior to that form "middleware" that can transform or respond to the request before it reaches the target handler. + +Brighter and Darker's pipelines use a "Russian Doll Model" that is, each handler in the pipeline encompasses the call to the next handler, allowing the handler chain to behave like a call stack. + +## Query + +A query asks the domain for facts. The [result](#result) of the query reports these facts - the state of the domain. A query does not change the state of the domain, for that use a [request](#request). + +## Query Handler + +A handler is the entry point to domain code. It receives a query and returns a [result](#result) to the caller. A handler is always part of an [internal bus](#internal-bus). As such a handler forms part of a [pipeline](#pipeline). + +It is analogous to a method on an ASP.NET Controller. + +## Query Processor + +In Darker, a query processor allows you to use the *Query Object Pattern* to separate caller from the executor, typically when separating the code required to execute a query on a specific database/backing store from the parameters of that query. It acts both as a *Query Dispatcher* which allows the separation of the parameters of a [query](#query) from the [handler](#query-handler) that executes that query and a *Query Processor* that allows you to use a middleware [pipeline](#pipeline) to provide additional and re-usable behaviors when processing that query. + +The Query Processor dispatches to an [Internal Bus](#internal-bus). + +The Query Processor returns a [result](#result). + +## Result + +The return value from a [query](#query). The result is returned from a [query handler](#query-handler) and exposed to the caller via the [QueryProcessor](#query-processor). + +## Request + +In Brighter, either a [command](#command) or an [event](#event), a request for the domain to (potentially) change state in response to an instruction or new facts. + +## Request Handler + +A handler is the entry point to domain code. It receives a request, which may be a [command](#command) or an [event](#event). A handler is always part of an [internal bus](#internal-bus) even when the call to the handler was triggered by a [service activator](#service-activator) receiving a [message](#message) sent by another process to an [external bus](#external-bus). As such a handler forms part of a [pipeline](#pipeline). + +It is analogous to a method on an ASP.NET Controller. + +## Request-Reply + +Request-Reply is a pattern in which there is a request for work and a response. + +To enforce [Command-Query Separation](#command-query-separation-cqs) Brighter handles commands/events and Darker handles queries. + +Where the request changes state, Brighter models this as a [command](#command) and a matching [event](#event) which describes the change. (See [Returning Results from a Handler](/contents/ReturningResultsFromAHandler.md) for a discussion of returning a response directly to the sender of a Command). + +Where the request queries for state, Darker models this as a [query](#query), which returns a [result](#result) directly to the caller. + +A common approach is to change state via Brighter and query for the results of that state change via Darker (and return those results to the caller). + +If the call to Brighter results in a new entity, and the id for the new entity was not given to the command (for example it relies on the Database generating the id), a common problem is how to then request the details of that newly created entity via Darker. A simple solution is to update the command with the id (as a conceptual *out* parameter), and then retrieve it from there to use in the Darker query. See [update a field on a command](/contents/ReturningResultsFromAHandler.md#update-a-field-on-the-command) for more. + +## Service Activator + +A Service Activator triggers execution of your code due to an external input, such as an HTTP call, or a [message](#message) sent over middleware. + +In Brighter, the *Dispatcher* acts as a Service Activator, listening for a message from middleware, which it delivers via the [command processor](#command-processor) to a [handler](#request-handler). As such, it turns messages sent over middleware to a call on your [internal bus](#internal-bus). diff --git a/contents/BrighterBasicConfiguration.md b/contents/9/BrighterBasicConfiguration.md similarity index 100% rename from contents/BrighterBasicConfiguration.md rename to contents/9/BrighterBasicConfiguration.md diff --git a/contents/9/BrighterInboxSupport.md b/contents/9/BrighterInboxSupport.md new file mode 100644 index 0000000..0b04294 --- /dev/null +++ b/contents/9/BrighterInboxSupport.md @@ -0,0 +1,116 @@ +# Brighter Inbox Support + +## Guaranteed, At Least Once + +Messaging makes the *guaranteed, at least once* promise. + +- Guaranteed: A broker writes a copy of your message to disk, so that it is not lost. An [Outbox](/contents/OutboxPattern.md) writes the message to the application's database to ensure it is not lost. +- At Least Once: In a distributed system you cannot guarantee that a writer will receive a response that it has successfully persisted a message, it must choose to retry the persistence if it does not receive an acknowledgement. This means that duplicates will occur, hence *at least once*. + +## Guaranteed, Once Only + +There are two possible reactions to the *at least once* problem. + +- Idempotency: Ensure that when receiving a message, handling it multiple times will not have side-effects +- De-duplication: Ensure that when receiving a message, you check whether you have handled it before, and discard if you have already processed it. + +Events are often idempotent, whilst commands often require de-duplication. + +## Inbox + +An inbox records messages that you have received and processed. Brighter provides inbox implementations built over a range of databases. If your preferred database is not included, see [implementing an inbox](#implementing-an-inbox). Brighter also makes available an in-memory inbox which is intended for development only as it will not work across multiple consumers, or survive process restarts. + +You can configure the usage of an inbox for de-duplication on a per-handler, or per command processor basis. + +### Adding an Inbox to a Handler + +The inbox is middleware and forms a part of the internal bus pipeline. When added to the pipeline, it will run before your handler, and after subsequent handlers (see the [Russian Doll Model](/contents/BuildingAnAsyncPipeline.md)). + +- Before: Check to see if we have already seen this request +- After: Add this request to those that we have already seen. + +You also need to configure what action the inbox takes when it has seen a request: + +- OnceOnly: Does the inbox reject duplicates, or does it simply record requests. WARNING: Defaults to false - so it won't reject duplicates unless you set this parameter to true, just log requests that pass through this handler. +- OnceOnlyAction: What does the inbox do, when a duplicate is encountered. Defaults to Throw. + - Warn: Just log that a duplicate was received + - Throw: Throws a OnceOnlyException + +**If you wish to terminate processing on a duplicate, you should set OnceOnly to true, which is what you need to terminate processing; the OnceOnlyAction will default to Throw** + +In the context of the Service Activator (listening to messages over middleware) throwing a OnceOnlyException will result in the message being acked (because it has already been processed). + +The inbox is global to your application and uses the request id; you will want to distinguish requests in the inbox if you need to store the same request id for different pipelines. For example, if you deliver an event to multiple handlers, each handler has a request with the same request id. + +Use the **contextKey** parameter to the attribute to disambiguate the request id. We recommend using the type of the handler, as the id usually needs to be unique via pipeline. + +There are two versions of the attribute: sync and async. Ensure that you choose the correct version, which should [match your handler](/contents/DispatchingARequest.md#pipelines-must-be-homogeneous). + +``` csharp + [UseInboxAsync(step:0, contextKey: typeof(GreetingMadeHandlerAsync), onceOnly: true )] + public override async Task HandleAsync(GreetingMade @event, CancellationToken cancellationToken = default(CancellationToken)) + { + Console.WriteLine($"Greeting Received: {@event.Greeting}"); + + return await base.HandleAsync(@event, cancellationToken); + } +``` + +### Inbox Configuration + +Your inbox is configured as part of the Brighter extensions to ServiceCollection. See [Inbox Configuration](/contents/BrighterBasicConfiguration.md#inbox) for more. + +### Inbox Builder + +Brighter contains DDL to configure your Inbox. For each supported database we include an **InboxBuilder**. The Inbox Builder **GetDDL** which allows you to obtain the DDL statements required to create an Inbox. You can use this as part of your application start up to configure the Inbox if it does not already exist. + +The following example shows creation of a MySql inbox. + +We assume that INBOX_TABLE_NAME is a constant, shared with the code that configures your inbox. + +``` csharp + +private static void CreateInbox(IConfiguration config, IHostEnvironment env) +{ + try + { + var connectionString = config.GetConnectionString("Salutations") + + using var sqlConnection = new MySqlConnection(connectionString); + sqlConnection.Open(); + + using var existsQuery = sqlConnection.CreateCommand(); + existsQuery.CommandText = MySqlInboxBuilder.GetExistsQuery(INBOX_TABLE_NAME); + bool exists = existsQuery.ExecuteScalar() != null; + + if (exists) return; + + using var command = sqlConnection.CreateCommand(); + command.CommandText = MySqlInboxBuilder.GetDDL(INBOX_TABLE_NAME); + command.ExecuteScalar(); + } + catch (System.Exception e) + { + Console.WriteLine($"Issue with creating Inbox table, {e.Message}"); + throw; + } +} + +``` + +## Clearing the Inbox + +As of V9, clearing the inbox is deferred to the implementer i.e. Brighter will not do this for you. Typically this involves creating a cron job, or agent, that clears inbox entries that are outside of the window during which they may be resent. + +Later versions of Brighter may include data retention policy options that let you configure clearing an inbox. + +## Non-Transactional Inbox + +As of V9 Brighter's inbox is not transactional, that is it does not participate in the transaction that may write to disk as a result of processing a message. This means that the inbox could fail if your changes to state as a result of processing a request are made, but the inbox is not updated. + +Later versions of Brighter may address including the inbox within a transaction, as outbox does today. + +## Implementing an Inbox + +You can refer to existing inbox implementations if you need to implement an inbox that Brighter does not support. + diff --git a/contents/9/BrighterOutboxSupport.md b/contents/9/BrighterOutboxSupport.md new file mode 100644 index 0000000..febc3d4 --- /dev/null +++ b/contents/9/BrighterOutboxSupport.md @@ -0,0 +1,181 @@ +# Outbox Support + +Brighter supports storing messages that are sent via an External Bus in an Outbox, as per the [Outbox Pattern](/contents/OutboxPattern.md) + +This allows you to determine that a change to an entity owned by your application should always result in a message being sent i.e. you have Transactional Messaging. + +There are two approaches to using Brighter's Outbox: + +* Post: This does not offer Transactional Messaging, but does offer replay +* Deposit and Clear: This approach offers Transactional Messaging. + +The **Post** method on the CommandProcessor in Brighter writes first to the **Outbox** and if that succeeds to the Message-Oriented Middleware. If you use Post, then your correctness options are **Ignore/Retry** or **Compensation**. You can use **Post** with **Log Tailing** or **Event Change Capture** but you have to implement those yourself. + +The **DepositPost** and **ClearOutbox** methods allow you to use the **Outbox** pattern instead. + +## Post + +In this approach you choose to **CommandProcessor.Post** a message after your Db transaction writes entity state to the Db. You intend to rely on the *retrying* the call to the broker if it fails. You should make sure that you have setup your **CommandProcessor.RETRYPOLICY** policy with this in mind. + +One caveat here is to look at the interaction of the retry on Post and any **UsePolicy** attribute for the handler. If your **CommandProcessor.RETRYPOLICY** policy bubbles up an exception following the last Retry attempt, and your **UsePolicy** attribute for the handler then catches that exception for your handler and forces a Retry, you will end up re-running the database transaction, which may result in duplicate entries. Your **UsePolicy** attribute for the handler needs to explicitly catch the Db errors you wish to retry, and not errors Posting to the message queue in this case. + +(As an aside, you should generally write Retry policies to catch specific errors that you know you can retry, not all errors anyway). + +In this case, you might also need to consider using a **Fallback** method via the FallbackPolicy attribute to catch **CommandProcessor.Post** exceptions that bubble out and issue a reversing transaction to kill any Db entries made +in error, or raise a log to ensure that there will be manual compensation. + +**CommandProcessor.Post** still uses the **Outbox** to store messages you send, but you are not including them in the Db transaction scope, so you have no **guarantees**. + +If the failure was on the call to the transport, and not the write to the **Outbox**, you will still have a **Outbox** entry that you can resend via manual compensation later. If the message is posted to the +broker, it **must** have already been written to the **Outbox**. + +In you fail to write to the **Outbox**, but have successfully written the entity to the Db, you would need to compensate by reversing the write to the Db in a **Fallback** handler. + +## Deposit and Clear + +Brighter allows the write to the **Outbox** and the write to the Broker to be separated. This form or Brighter allows you to support Producer-Consumer correctness via the **Outbox Pattern**. + +Metaphorically, you can think of this as a post box. You deposit a letter in a post box. Later the postal service clears the post box of letters and delivers them to their recipients. + +Within your database transaction you write the message to the Outbox with **CommandProcessor.DepositPost**. This means that if the entity write succeeds, the corresponding write to the **Outbox** will have +taken place. This method returns the Id for that message. + +(Note that we use **CommandProcessor.RETRYPOLICY** on the write, but this will only impact the attempt to write within the transaction, not the success or failure of the overall Db transaction, which is under +your control. You can safely ignore Db errors on this policy within this approach for this reason.) + +You can then call **CommandProcessor.ClearPostBox** to flush one or more messages from the **Outbox** to the broker. We support multiple messages as your entity write might possibly involve sending multiple downstream messages, which you want to include in the transaction. + +It provides a stronger guarantee than the **CommandProcessor.Post** outside Db transaction with Retry approach as the write to the **Outbox** shares a transaction with the persistence of entity state. + + +## Bulk Deposit + +Starting in v9.2.1 Brighter allows a batch of Messages to be written to the **Outbox**. If your outbox suoports Bulk (This will become a requirement in v10) **CommandProcessor.DepositPost** can be used to deposit a large number of messages in much quicker than individually. + +When creating your **CommandProcessor** you can set an outbox bulk chunk size, if the amount of mesages to be deposited into the **Outbox** is greater than this number it will be broken up into chunks of no more than this size. + +## Participating in Transactions + +Brighter has the functionality to allow the **Outbox** to participate in the database transactions of your application so that you can ensure that distributed requests will be persisted (or fail to persist) inline with application changes. + +To have the Brighter **Outbox** participate in Database transactions the command process must be built specifying a **IAmABoxTransactionConnectionProvider**, this connection provider will be used when **CommandProcessor.DepositPost** is called and if there is an active transactions the **Outbox** will participate in the active transaction provider by the specified **IAmABoxTransactionConnectionProvider**. + +It is important to note that **CommandProcessor.Clear** and **CommandPorcessor.Post** will never participate in transactions as the purpose transaction participation is to ensure that **Outbox** messages are committed (or fail to commit) in the same transaction as application entity changes. + +Below is an example using a UnitOfWork that wraps the database connection for your application +``` csharp +//Begin Database transaction +unitOfWork.BeginTransaction(); + +try +{ + //Update applicationEntities + var updatedContact = contactsService.UpdateContact(contact); + + //Deposit the message in the outbox + commandProcess.DepositPost(updatedContact.ToBrighterMessage()); + + //Commit Transaction + unitOfWork.CommitTransaction(); +} +catch(Exception e) +{ + // If there was an error during processing, rollback all changes + unitOfWork.RollbackTransaction(); +} +``` + +## Implicit or Explicit Clearing of Messages from the Outbox + +There are two approaches to dispatching messages from Brighter's **Outbox** + * Implicitly: This relies on a **Sweeper** to dispatch messages out of process + * Explicitly: This ensures that your message is sent sooner but will processing time to your application code. + +To explicitly clear a message you can call **CommandProcessor.ClearOutbox** directly in your handler, after the Db transaction completes. This has the lowest latency. You are responsible for tracking the ids of messages that you wish to send in **CommandProcessor.ClearOutbox**, we do not maintain this state for you. Note that you cannnot guarantee that this will succeed, although you can Retry. We use **CommandProcessor.RETRYPOLICY** on the write to the Broker, and you should retry errors writing to the Broker in that policy. However, as the message is now in the **Outbox** you can compensate for eventual failure to write to the Broker by replaying the message from the **MessageStore** at a later time. + +To implicitly clear messages from your outbox, configure a **Outbox Sweeper** to listen to your **Outbox** and dispatch messages for you. Once an **Outbox Sweeper** is running you no longer need to call **CommandProcessor.ClearOutbox** however you still have the choice to if you feel a specific message is time sensitive. + +## Outbox Sweeper + +The **Outbox Sweeper** is an out of process service that monitors an **Outbox** and dispatches messages that have yet to be dispatches. Using **Outbox Sweeper** has a lower latency impact for your application, but because it keeps trying to send the messages until it succeeds is the recommended approach to *Guranteed, At Least Once, Delivery*. + +The benefits of using an **Outbox Sweeper** are: + * If there is a failure dispatch a message after it is committed to the **Outbox** it will be retried until it is dispatches + * The ability to choose between the implicit and explicit clearing of messages + +The **Timed Outbox Sweeper** has the following configurables + * TimerInterval: The amount of seconds to wait between checks for undispatches messages (default: 5) + * MinimumMessageAge: The age a message (in miliseconds) that a messages should be before the **OutboxSweeper** should attempt to dispatch it. (default: 5000) + * BatchSize: The number of messages to attempt to dispatch in each check (default: 100) + * UseBulk: Use Bulk dispatching of messages on your **Messaging Gateway** (default: false), note: not all **messaging Gateway**s support Bulk dispatching. + +It is important to note that the lower the Minimum Message age is the more likely it is that your message will be dispatches more than once (as if you are explicitly clearing messages your application may have instructed the clearing of a message at the same time as the **Outbox Sweeper**) + +## Outbox Archiver + +The **Outbox Archiver** is an out of process services that monitors an **Outbox** and will archive messages of older than a certain age. + +The **Timed Outbox Archiver** has the following configurables + * TimerInterval: The number of seconds to wait between checked for messages eligable for archival (default: 15) + * BatchSize: The maximum number of messages to archive for each check (default: 100) + * MinimunAge: The time ellapsed since a message was dispated in hours before it is eligable for archival (default: 24) + +### Outbox Configuration + +Your outbox is configured as part of the Brighter extensions to ServiceCollection. See [Outbox Configuration](/contents/BrighterBasicConfiguration.md#outbox-support) for more. + +### Outbox Builder + +Brighter contains DDL to configure your Outbox. For each supported database we include an **OutboxBuilder**. The Inbox Builder **GetDDL** which allows you to obtain the DDL statements required to create an Outbox. You can use this as part of your application start up to configure the Outbox if it does not already exist. + +The following example shows creation of a MySql outbox. + +We assume that OUTBOX_TABLE_NAME is a constant, shared with the code that configures your inbox. + +``` csharp + +public static IHost CreateOutbox(this IHost webHost) +{ + using (var scope = webHost.Services.CreateScope()) + { + var services = scope.ServiceProvider; + var env = services.GetService(); + var config = services.GetService(); + + CreateOutbox(config, env); + } + + return webHost; +} + +private static void CreateOutbox(IConfiguration config, IWebHostEnvironment env) +{ + try + { + var connectionString = config.GetConnectionString("Greetings"); + + using var sqlConnection = new MySqlConnection(connectionString); + sqlConnection.Open(); + + using var existsQuery = sqlConnection.CreateCommand(); + existsQuery.CommandText = MySqlOutboxBuilder.GetExistsQuery(OUTBOX_TABLE_NAME); + bool exists = existsQuery.ExecuteScalar() != null; + + if (exists) return; + + using var command = sqlConnection.CreateCommand(); + command.CommandText = MySqlOutboxBuilder.GetDDL(OUTBOX_TABLE_NAME); + command.ExecuteScalar(); + + } + catch (System.Exception e) + { + Console.WriteLine($"Issue with creating Outbox table, {e.Message}"); + //Rethrow, if we can't create the Outbox, shut down + throw; + } +} + +``` + + diff --git a/contents/9/BuildingAPipeline.md b/contents/9/BuildingAPipeline.md new file mode 100644 index 0000000..31ee49b --- /dev/null +++ b/contents/9/BuildingAPipeline.md @@ -0,0 +1,175 @@ +# Building a Pipeline of Request Handlers + +Once you are using the features of Brighter to act as a [command dispatcher](CommandsCommandDispatcherAndProcessor.html#command-dispatcher) and send or publish messages to a target handler, you may want to use +its [command processor](CommandsCommandDispatcherAndProcessor.html#command-processor) features to handle orthogonal operations. + +Common examples of orthogonal operations include: + +- Logging the Command +- Providing integration with tools for monitoring performance and + availability +- Validating the Command +- Supporting idempotency of messages +- Supporting re-sequencing of messages +- Handling exceptions +- [Providing Timeout, Retry, and Circuit Breaker + support](QualityOfServicePatterns.html) +- Providing undo support, or rollback + +## The Pipes and Filters Architectural Style + +To handle these orthogonal concerns our [command processor](CommandsCommandDispatcherAndProcessor.html#command-processor) uses a pipes and filters architectural style: the filters are where +processing occurs, they do not share state with other filters, nor do they know about adjacent filters. The pipe is the connector between the filters in our case this is provided by the +**IHandleRequests\** interface which has a method **IHandleRequests\ Successor** that allows us to chain filters together. + +![PipesAndFilters](_static/images/PipesAndFilters.png) + +The sink handler is handler that is the receiver you wish to invoke the action on. The pump is the **Command Dispatcher**. We occasionally use *target handler* as a synonym for *sink handler* + +## The Russian Doll Model + +Our pipes and filters approach supports the *Russian Doll Model* of calling the handler pipeline, a context bag for the pipeline, and support for generating a request path description out-of-the-box. + +The *Russian Doll Model* is names for the [Matryoshka](https://en.wikipedia.org/wiki/Matryoshka_doll) wooden dolls, in which dolls of decreasing sizes are nested one inside another. The importance of this for a [pipes and filters pattern](https://msdn.microsoft.com/en-us/library/dn589788.aspx) style is that each filter in the pipeline is called within the scope of a previous filter in the pipeline. + +![RussianDoll](_static/images/RussianDoll.png) + +This is significant because you may desire to act before and after a subsequent filter step. One particular use case is exception handling: a try-catch block that wraps the call to a subsequent step can react to +exceptions raised by subsequent steps. This allows us to create policy decisions around exceptions using a library such as [Polly](https://github.com/App-vNext/Polly) and thus support [Retry](https://msdn.microsoft.com/en-us/library/dn589788.aspx) and [Circuit Breaker](https://msdn.microsoft.com/en-gb/library/dn589784.aspx?f=255&MSPPError=-2147217396) + +Our usage of the Russian Doll Model was inspired by [FubuMVC](http://codebetter.com/jeremymiller/2011/01/09/fubumvcs-internal-runtime-the-russian-doll-model-and-how-it-compares-to-asp-net-mvc-and-openrasta/) + +## Implementing a Pipeline + +The first step in building a pipeline is to decide that we want an orthogonal operation in our pipeline. Let us assume that we want to do basic request logging. + +Because you do not want to write an orthogonal handler for every Command or Event type, these handlers should remain generic types. At runtime the HandlerFactory creates an instance of the generic type specialized for the type parameter of the Command or Event being passed along the pipeline. + +The limitation here is that you can only make assumptions about the type you receive into the pipeline from the constraints on the generic type. + +Although it is possible to implement the [IHandleRequests](https://github.com/BrighterCommand/Brighter/blob/master/src/Paramore.Brighter/IHandleRequests.cs) interface directly, we recommend deriving your handler from [RequestHandler](https://github.com/BrighterCommand/Brighter/blob/master/src/Paramore.Brighter/RequestHandler.cs\). + +Let us assume that we want to log all requests travelling through the pipeline. (We provide this for you in the Brighter.CommandProcessor packages so this for illustration only). We could implement a generic +handler as follows: + +``` csharp +using System; +using Newtonsoft.Json; +using Brighter.commandprocessor.Logging; + +namespace Brighter.commandprocessor +{ + public class RequestLoggingHandler + : RequestHandler where TRequest : class, IRequest + { + private HandlerTiming _timing; + + public override void InitializeFromAttributeParams( + params object[] initializerList + ) + { + _timing = (HandlerTiming)initializerList[0]; + } + + public override TRequest Handle(TRequest command) + { + LogCommand(command); + return base.Handle(command); + } + + private void LogCommand(TRequest request) + { + logger.InfoFormat("Logging handler pipeline call. Pipeline timing {0} target, for {1} with values of {2} at: {3}", + _timing.ToString(), + typeof(TRequest), + JsonConvert.SerializeObject(request), + DateTime.UtcNow); + } + } +} +``` + +Our Handle method is the method which will be called by the pipeline to service the request. After we log we call **return base.Handle(command)** to ensure that the next handler in the chain is +called. If we failed to do this, the *target handler* would not be called nor any subsequent handlers in the chain. This call to the next item in the chain is how we support the \'Russian Doll\' model - because +the next handler is called within the scope of this handler, we can manage when it is called handle exceptions, units of work, etc. + +It is worth remembering that handlers may be called after the target handler (in essence you can designate an orthogonal handler as the sink handler when configuring your pipeline). For this reason **all** handlers should remember to call their successor, **even your target handler**. + +We now need to tell our pipeline to call this orthogonal handler before our target handler. To do this we use attributes. The code we want to write looks like this: + +``` csharp +class GreetingCommandHandler : RequestHandler +{ + [RequestLogging(step: 1, timing: HandlerTiming.Before)] + public override GreetingCommand Handle(GreetingCommand command) + { + Console.WriteLine("Hello {0}", command.Name); + return base.Handle(command); + } +} +``` + +The **RequestLogging** Attribute tells the Command Processor to insert a Logging handler into the request handling pipeline before (**HandlerTiming.Before**) we run the target handler. It tells the Command Processor that we want it to be the first handler to run if we have multiple orthogonal handlers i.e. attributes (**step: 1**). + +We implement the **RequestLoggingAttribute** by creating our own Attribute class, derived from **RequestHandlerAttribute**. + +``` csharp +public class RequestLoggingAttribute : RequestHandlerAttribute +{ + public RequestLoggingAttribute(int step, HandlerTiming timing) + : base(step, timing) + { } + + public override object[] InitializerParams() + { + return new object[] { Timing }; + } + + public override Type GetHandlerType() + { + return typeof(RequestLoggingHandler<>); + } +} +``` + +The most important part of this implementation is the GetHandlerType() method, where we return the type of our handler. At runtime the Command Processor uses reflection to determine what attributes are on the target handler and requests an instance of that type from the user-supplied **Handler Factory**. + +Your Handler Factory needs to respond to requests for instances of a **RequestHandler\** specialized for a concrete type. For example, if you create a **RequestLoggingHandler\** we will ask you for a **RequestLoggingHandler\** etc. Depending on your implementation of HandlerFactory, you may need to register an implementation for every concrete instance of your handler with your +underlying IoC container etc. + +Note that as we rely on an user supplied implementation of **IAmAHandlerFactory** to instantiate Handlers, you can have any dependencies in the constructor of your handler that you can resolve at +runtime. In this case we pass in an ILog reference to actually log to. + +You may wish to pass parameter from your Attribute to the handler. Attributes can have constructor parameters or public members that you can set when adding the Attribute to a target method. These can only be +compile time constants, see the documentation [here](https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/attributes/). After the Command Processor calls your Handler Factory to create an +instance of your type it calls the **RequestHandler.InitializeFromAttributeParams** method on that created type and passes it the object array defined in the **RequestHandlerAttribute.InitializerParams**. By this approach, you can pass parameters to the handler, for example the Timing parameter is passed to the handler above. + +It is worth noting that you are limited when using Attributes to provide constructor values that are compile time constants, you cannot pass dynamic information. To put it another way you are limited to value set +at design time not at run time. + +In fact, you can use this approach to pass any data to the handler on initialization, not just attribute constructor or property values, but you are constrained to what you can access from the context of the +Attribute at run time. it can be tempting to set retrieve global state via the [Service Locator](https://en.wikipedia.org/wiki/Service_locator_pattern) pattern at this point. Avoid that temptation as it creates coupling between your Attribute and global state reducing modifiability. + +## Using a Manual Approach + +Using an attribute based approach is not an approach favoured by everyone. Some people prefer a more explicit approach to configuring the pipeline. + +The trick is to remember that any handler that derives from **IHandleRequests\** has a **Successor** and you can build a chain by having the first handler call the second handler\'s +**Handle()** method i.e. **Successor.Handle()**. You can derive from **RequestHandler\** and call **base.Handle()** for this, even if you don\'t want to use the Attribute based pipelines. + +In the SubscriberRegistry you just register the first Handler in your pipeline. When we lookup the Handler for the Command in the SubscriberRegistry we will call it\'s Handle method. It can execute your +code, and then call it\'s Successor (using the Russian Doll approach). + +``` csharp +var myCommandHandler = new MyCommandHandler(); +var myLoggingHandler = new MyLoggingHandler(log); + +myLoggingHandler.Successor = myCommandHandler; + +var subscriberRegistry = new SubscriberRegistry(); +subscriberRegistry.Register(); +``` + +It is worth noting that as you control the HandlerFactory, you could also register the sink handler, but when instantiating an instance of it on request, build the pipeline of handlers yourself. + +We think it is easier to use attributes, but there may be circumstances where that approach does not work, and so this option is supported as well. diff --git a/contents/9/BuildingAnAsyncPipeline.md b/contents/9/BuildingAnAsyncPipeline.md new file mode 100644 index 0000000..0a5fedd --- /dev/null +++ b/contents/9/BuildingAnAsyncPipeline.md @@ -0,0 +1,107 @@ +# Building a Pipeline of Async Request Handlers + +Once you are using the features of Brighter to act as a [command dispatcher](CommandsCommandDispatcherAndProcessor.html#command-dispatcher) and send or publish messages to a target handler, you may want to use +its [command processor](CommandsCommandDispatcherAndProcessor.html#command-processor) features to handle orthogonal operations. + +# Implementing a Pipeline + +The first step in building a pipeline is to decide that we want an orthogonal operation in our pipeline. Let us assume that we want to do command sourcing. + +Because you do not want to write an orthogonal handler for every Command or Event type, these handlers should remain generic types. At runtime the framework will request HandlerFactory creates an instance of the +generic type specialized for the type parameter of the Command or Event being passed along the pipeline. + +The limitation here is that you can only make assumptions about the type you receive into the pipeline from the constraints on the generic type. + +Although it is possible to implement the +[IHandleRequestsAsync](https://github.com/BrighterCommand/Brighter/blob/master/src/Paramore.Brighter/IHandleRequestsAsync.cs) +interface directly, we recommend deriving your handler from +[RequestHandlerAsync\ +\]{.title-ref}\_\_. + +Let us assume that we want to log all requests travelling through the pipeline. (We provide this for you in the Brighter.CommandProcessor packages so this for illustration only). We could implement a generic +handler as follows: + +``` csharp +public class CommandSourcingHandlerAsync : RequestHandlerAsync where T : class, IRequest +{ + private readonly IAmACommandStoreAsync _commandStore; + + public CommandSourcingHandlerAsync(IAmACommandStoreAsync commandStore) + { + _commandStore = commandStore; + } + + public override async Task HandleAsync(T command, CancellationToken? ct = null) + { + await _commandStore.AddAsync(command, -1, ct).ConfigureAwait(ContinueOnCapturedContext); + } +} +``` + +Our HandleAsync method is the method which will be called by the pipeline to service the request. After we log we call **return await base.HandleAsync(command, ct)** to ensure that the next handler in the +chain is called. + +If we failed to do this, the *target handler* would not be called nor any subsequent handlers in the chain. This call to the next item in the chain is how we support the \'Russian Doll\' model - because the next +handler is called within the scope of this handler, we can manage when it is called handle exceptions, units of work, etc. + +It is worth remembering that handlers may be called after the target handler (in essence you can designate an orthogonal handler as the sink handler when configuring your pipeline). For this reason **all** handlers should remember to call their successor, **even your target handler**. + +We now need to tell our pipeline to call this orthogonal handler before our target handler. To do this we use attributes. The code we want to write looks like this: + +``` csharp +internal class GreetingCommandRequestHandlerAsync : RequestHandlerAsync +{ + [UseCommandSourcingAsync(step: 1, timing: HandlerTiming.Before)] + public override async Task HandleAsync(GreetingCommand command, CancellationToken? ct = null) + { + var api = new IpFyApi(new Uri("https://api.ipify.org")); + + var result = await api.GetAsync(ct); + + Console.WriteLine("Hello {0}", command.Name); + Console.WriteLine(result.Success ? "Your public IP addres is {0}" : "Call to IpFy API failed : {0}", result.Message); + return await base.HandleAsync(command, ct).ConfigureAwait(base.ContinueOnCapturedContext); + } +} +``` + +The **UseCommandSourcingAsync** Attribute tells the Command Processor to insert a Logging handler into the request handling pipeline before (**HandlerTiming.Before**) we run the target handler. It tells the +Command Processor that we want it to be the first handler to run if we have multiple orthogonal handlers i.e. attributes (**step: 1**). + +We implement the **UseCommandSourcingAsyncAttribute** by creating our own Attribute class, derived from **RequestHandlerAttribute**. + +``` csharp +public class UseCommandSourcingAsyncAttribute : RequestHandlerAttribute +{ + + public UseCommandSourcingAsyncAttribute(int step, HandlerTiming timing = HandlerTiming.Before) + : base(step, timing) + { } + + + public override Type GetHandlerType() + { + return typeof (CommandSourcingHandlerAsync<>); + } +} +``` + +The most important part of this implementation is the GetHandlerType() method, where we return the type of our handler. At runtime the Command Processor uses reflection to determine what attributes are on the target handler and requests an instance of that type from the user-supplied **Handler Factory**. + +Your Handler Factory needs to respond to requests for instances of a **RequestHandlerAsync\** specialized for a concrete type. For example, if you create a **CommandSourcingHandlerAsync\** we +will ask you for a **CommandSourcingHandlerAsync\** etc. Depending on your implementation of HandlerFactory, you may need to register an implementation for every concrete instance of your handler +with your underlying IoC container etc. + +Note that as we rely on an user supplied implementation of **IAmAHandlerFactoryAsync** to instantiate Handlers, you can have any dependencies in the constructor of your handler that you can resolve at +runtime. In this case we pass in an ILog reference to actually log to. + +You may wish to pass parameter from your Attribute to the handler. Attributes can have constructor parameters or public members that you can set when adding the Attribute to a target method. These can only be +compile time constants, see the documentation [here](https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/language-specification/attributes). + +After the Command Processor calls your Handler Factory to create an instance of your type it calls the **RequestHandler.InitializeFromAttributeParams** method on that created type and passes it the object array defined in the **RequestHandlerAttribute.InitializerParams**. By this approach, you can pass parameters to the handler, for example the Timing parameter is passed to the handler above. + +It is worth noting that you are limited when using Attributes to provide constructor values that are compile time constants, you cannot pass dynamic information. To put it another way you are limited to value set +at design time not at run time. + +In fact, you can use this approach to pass any data to the handler on initialization, not just attribute constructor or property values, but you are constrained to what you can access from the context of the +Attribute at run time. It can be tempting to set retrieve global state via the [Service Locator](https://en.wikipedia.org/wiki/Service_locator_pattern) pattern at this point. Avoid that temptation as it creates coupling between your Attribute and global state reducing modifiability. diff --git a/contents/9/ClaimCheck.md b/contents/9/ClaimCheck.md new file mode 100644 index 0000000..912dbe2 --- /dev/null +++ b/contents/9/ClaimCheck.md @@ -0,0 +1,66 @@ +# Claim Check + +The [Claim Check](https://www.enterpriseintegrationpatterns.com/patterns/messaging/StoreInLibrary.html) pattern helps us reduce the size of our messages, without losing information that we need to exchange. + +Instead of being transmitted in the body of the message, the payload is written to a distributed file storage and a token to retrieve the payload is sent instead. The receiver can read the payload by taking the reference and requesting it from the distributed file storage. The metaphor here is a luggage check. Instead of carrying large items of luggage aboard an aircraft we check them into the hold of the aircraft. The airline gives us a claim check for our luggage, that matches a tag on the bag. This pattern is sometimes called Reference Based Messaging. + +## Claim Check and Retrieve Claim + +We treat the Claim Check pattern as [Transformer](/contents/MessageMappers.md#message-transformer-factory) middleware. + +We provide a **WrapWithAttribute** of **ClaimCheck** that will use the **ClaimCheckTransformer** to upload the body of your **Message** to a *luggage store* replacing it with a body that contains an claim check for the body, as well as setting a message header of "claim_check_header" with the claim. The trigger for this behavior can be controlled by a threshold parameter that sets the size above which the message body should be moved to the *luggage store*. + +In the following example we add the **ClaimCheck** attribute to the *Message Mapper* with a trigger at 256Kb + +``` csharp +[ClaimCheck(step:0, thresholdInKb: 256)] +public Message MapToMessage(GreetingEvent request) +{ + var header = new MessageHeader(messageId: request.Id, topic: typeof(GreetingEvent).FullName.ToValidSNSTopicName(), messageType: MessageType.MT_EVENT); + var body = new MessageBody(JsonSerializer.Serialize(request, JsonSerialisationOptions.Options)); + var message = new Message(header, body); + return message; +} +``` + +We provide a matching **UnwrapWithAttribute** of **RetrieveClaim** that will use the **ClaimCheckTransformer** to download the body of your **Message** from a luggage store and replace the existing body (likely a claim check reference) with the downloaded content. + +``` csharp +[RetrieveClaim(0, retain:false)] +public GreetingEvent MapToRequest(Message message) +{ + var greetingCommand = JsonSerializer.Deserialize(message.Body.Value, JsonSerialisationOptions.Options); + + return greetingCommand; +} + +``` + +An optional parameter 'retain' determines if we keep the body in storage after it is retrieved or delete it. The default is to delete it. + +The outcome of these attributes is that the uploading of the body to the *luggage store* and downloading from it is transparent to your code. You serialize your **IRequest** to a **Message** as normal, or serialize your **Message** to an **IRequest** as normal - everything happens in the middleware pipeline. + +## The Luggage Store + +The *luggage store* is where we store the body of the message for later retrieval. We provide implementations of the Luggage Store interface for popular distributed stores, but you can implement the interface for any that we do not provide. + +```csharp + + public interface IAmAStorageProviderAsync + { + Task DeleteAsync(string claimCheck, CancellationToken cancellationToken); + Task DownloadAsync(string claimCheck, CancellationToken cancellationToken); + Task HasClaimAsync(string claimCheck, CancellationToken cancellationToken); + Task UploadAsync(Stream stream, CancellationToken cancellationToken); + } + +``` + +* DeleteAsync: Deletes a item from the store +* DownloadAsync: Creates a stream for a download from the store +* HasClaimAsync: Does the claim check exist in the store +* UploadAsync: Uploads a stream to the store and returns a claim, an identifier that can later be used to delete, download or check for the existence of the file uploaded to the store. + +We provide the following implementations of **IAmAStorageProviderAsync: + +* [S3LuggageStore](/contents/S3LuggageStore.md) \ No newline at end of file diff --git a/contents/9/CommandsCommandDispatcherandProcessor.md b/contents/9/CommandsCommandDispatcherandProcessor.md new file mode 100644 index 0000000..950ca4b --- /dev/null +++ b/contents/9/CommandsCommandDispatcherandProcessor.md @@ -0,0 +1,109 @@ +# Command Patterns + +## Command + +The **Command** design pattern encapsulates a request as an object, allowing reuse, queuing or logging of requests, or undoable operations. It also serves to decouple the implementation of the request from the +requestor. The caller of a Command object does not need to understand how the Command is implemented, only that the Command exists. When the caller and the implementer are decoupled it becomes easy to replace or +refactor the implementation of the request, without impacting the caller - our system is more modifiable. Our ability to test the Command in isolation of the caller - allows us to implement the ports and +adapters model easily - we can instantiate the Command, provide \'fake\' parameters to it and confirm the results. We can also use the command from multiple callers, although this is not a differentiator from the +service class approach. + +![Command](_static/images/Command.png) + +**Command** - Declares an interface for executing an operation. + +**ConcreteCommand** --Defines a binding between a Receiver object and an action. Implements Execute by invoking the corresponding operation(s) on the Receiver. + +**Client** -- creates a ConcreteCommand object and sets its receiver. + +**Invoker** - asks the command to carry out the request. + +![CommandWorkflow](_static/images/CommandWorkflow.png) + +An **Invoker** object knows about the **Concrete Command** object. The Invoker issues a request by calling Execute on the **Command**. When commands are un-doable, the Command stores state for undoing the command +prior to invoking Execute.The Command object invokes operations on its **Receiver** to carry out the request + +In addition we can structure a system transactionally using Commands. A Command is a transactional boundary. Because a Command is a transactional boundary, when using the [Domain Driven Design](https://en.wikipedia.org/wiki/Domain-driven_design) technique of an aggregate there is a natural affinity between the Command, which operates on a transactional boundary and the Aggregate which is a transactional boundary within the domain model. The Aggregate is the Receiver stereotype within the Command Design pattern. Because we want to separate use of outgoing Adapters via a secondary Port, such as a Repository +[Repository](https://martinfowler.com/eaaCatalog/repository.html) in the DDD case, this can lead to a pattern for implementation of a Command: + +> 1. Begin Transaction +> 2. Load from Repository +> 3. Operate on Aggregate +> 4. Flush to Repository +> 5. Commit Transaction + +In the Repository pattern we may need to notify other Aggregates that can be eventually consistent of the change within the transactionaly consistent boundary. The pattern suggested there is a notification. Because the handling of that notification is in itself likely to be a transactional boundary for a different aggregate we can encapsulate this domain event with the Command design pattern as well, which gives rise to the following additional step to the sequence,outside the original transactional boundary: + +> 6. Invoke Command Encapsulating Notification + +This has obvious similarities to the [actor model](https://en.wikipedia.org/wiki/Actor_model), particularly if you use an External Bus. + +The problems with the Command pattern are that the caller is coupled to a specific Command at the call site - which undermines the promise of being extensible through use of Commands. To change that Command, or +call orthogonal services before calling the command requires us to amend the calling code, wherever the Command is used. To decouple a higher and lower layers we want to be able alter the implementation of the commands that we call on the lower layer without altering the calling layer. + +## Command Dispatcher + +Brighter is a .NET implementation of the **Command Dispatcher** pattern. + +*This pattern increases the flexibility of applications by enabling their services to be changed, by adding, replacing or removing any command handlers at any point in time without having to modify, recompile or statically relink the application. By simulating the command-evaluation feature common in interpreted languages, this pattern supports the need for continual, incremental evolution of applications.* + +A [Command Dispatcher](https://en.wikipedia.org/wiki/Command_pattern) is often used with a hierarchical architecture to avoid the [Fat Controller problem](https://github.com/BrighterCommand/Brighter/wiki/Fat-Controllers) and allow us to [decouple from the caller](https://github.com/BrighterCommand/Brighter/wiki/Why-use-a-Command-Processor). + +An Action-Request object is an object that both encapsulates the identity of the action we want to fire and the parameters for this action, i.e. the extrinsic state of the action to undertake. In other words, an Action-Request object is a representation of the action to undertake, which is identified using a key, possibly a string such as \'set_depth\'. An Action-Handler is the object that knows how to perform a particular action, and is passed the parameters at run-time. It is therefore a shared object that can be used in multiple contexts simultaneously. The Command-Dispatcher is the object that links the Action-Request to the appropriate Action Handler object. It has a dictionary that contains a reference to all the registered Action-Handlers. The Command-Dispatcher uses the Action-Request\'s key to find the right entry and dispatches the appropriate Action-Handler. The Action Handler can then perform the requested action. + +We want to separate an Action-Request object that contains the identity of the action we want to perform, and the parameter for that action from the Action-Handler which knows how to perform that action. + +A Command Dispatcher is an object that links the Action-Request with the appropriate Action-Handler. + +We may distinguish between a Command Action-Request that has one Action Handler and an Event Action-Request that has many + +The Command Dispatcher allows dynamic registration and removal of Command Handlers, it is an administrative entity that manages linking of commands to the appropriate command handlers. + +It relates to the Observer pattern in that hooks together publishers and subscribers. + +Command Dispatcher registration requires a key -- provided by the Command Dispatcher for the Commands it can service, using getKey(). \[In practice we often use RTTI for this\]. + +The Command Handler is fired, when a command with the same name (key) is sent to the Command Dispatcher. + +The Command Dispatcher is a repository of key-value pairs (key., Command Handler) and when the Command Dispatcher is called it looks up the command's key in the repository. If there is a match it calls the +appropriate method(s) on the handler to process the Command. + +![CommandDispatcher](_static/images/CommandDispatcher.png) + +**Invoker** - has a lit of Commands that are to be executed + +**Command** - represents the request to be processed, encapsulating the parameters to be passed to the command-handler to perform the request + +**Command Handler** - specifies the interface that any command handler must implement + +**Concrete Command Handler** -- implements the request + +**Command Dispatcher** -- Allows dynamic registration of Command Handlers and looks up handlers for commands, by matching command and handler key. + +**Client** -- registers Commands with the Command Dispatcher. + +![CommandExtendedWorkflow](_static/images/CommandExtendedWorkflow.png) + +A Command Dispatcher can also act as the port layer in a [Ports & Adapters architecture](http://alistair.cockburn.us/Hexagonal+architecture). + +## Command Processor + +Brighter is a .NET implementation of the [Command Processor pattern](https://wiki.hsr.ch/APF/files/CommandProcessor.pdf). + +The Command Processor pattern separates the request for a service from its execution. A Command Processor component manages requests as separate objects, schedules their execution, and provides additional +services such as the storing of request objects for later undo. + +A Command Dispatcher and a Command Processor are similar in that both divorce the caller of a Command from invoker of that Command. However, the motivation is different. A Dispatcher seeks to decouple the caller from the invoker to allow us to easily extend the system without modification to the caller. Conversely the motivation behind a Command Processor is to allows us to implement orthogonal operations such as logging, or scheduling without forcing the sender or receiver to be aware of them. It does this by giving those responsibilities to the invoker. + +Of course as both patterns separate the invoker from sender and receiver, it is possible for us to combine them by having the Command Dispatcher\'s invoker support executing orthogonal concerns when it invokes the Command. + +![CommandProcessor](_static/images/CommandProcessor.png) + +The central command processor easily allows the addition of services related to command execution. An advanced command processor can log or store commands to a file for later examination or replay. A command +processor can queue commands and schedule them at a later time. This is useful if commands should execute at a specified time, if they are handled according to priority, or if they will execute in a separate +thread of control. An additional example is a single command processor shared by several concurrent applications that provides a transaction control mechanism with logging and rollback of commands. + +A Command Processor enforces quality of service and maximizes throughput. A Command Processor forms a juncture at which concerns like: [retry, timeout and circuit breaker](PolicyRetryAndCircuitBreaker.html) +can be implemented for all commands. + +![CommandProcesorCapitalize](_static/images/CommandProcesorCapitalize.png) diff --git a/contents/9/Compression.md b/contents/9/Compression.md new file mode 100644 index 0000000..650e6cc --- /dev/null +++ b/contents/9/Compression.md @@ -0,0 +1,51 @@ +# Compression + +The Compression transform helps us reduce the size of a message using a compression algorithm. It is an efficient approach to reducing the size of a payload. + +We offer [gzip](https://en.wikipedia.org/wiki/Gzip) on netstandard2.0 and add [deflate](https://en.wikipedia.org/wiki/Deflate) and [brotli](https://en.wikipedia.org/wiki/Brotli) on net6+. + +## Compress and Decompress + +We treat Compress and Decompress as [Transformer](/contents/MessageMappers.md#message-transformer-factory) middleware. + +We provide a **WrapWithAttribute** of **Compress** that will use the **CompressPayloadTransformer** to compress the body of your message using your choice of compression algorithm. The claim check has a threshold, over which messages will be compressed. + +In the following example we compress any string larger than 150K + +``` csharp +[Compress(0, CompressionMethod.GZip, CompressionLevel.Optimal, 150)] +public Message MapToMessage(GreetingEvent request) +{ + var header = new MessageHeader(messageId: request.Id, topic: typeof(GreetingEvent).FullName.ToValidSNSTopicName(), messageType: MessageType.MT_EVENT); + var body = new MessageBody(JsonSerializer.Serialize(request, JsonSerialisationOptions.Options)); + var message = new Message(header, body); + return message; +} +``` + +We provide a matching **UnwrapWithAttribute** of **Decompress** that will use the **CompressPayloadTransformer** to decompress the body of your message using the algorithm the message body was compressed with. If the string is not compressed, we take no action. This supports the scenario where some messages on a channel are small enough not to cross the threshold for compression, but others will be large and require compression. (If you want compress all messages on a channel, regardless of individual size, just set your threshold to zero). + +In this example, we look for a GZip compressed string and if we find it, decompress the body. + +```csharp +[Decompress(0, CompressionMethod.GZip)] +public GreetingEvent MapToRequest(Message message) +{ + var greetingCommand = JsonSerializer.Deserialize(message.Body.Value, JsonSerialisationOptions.Options); + + return greetingCommand; +} +``` + + +### Impact of Compression + +When we compress a message we change the *Content Type* header (content-type) for the message to reflect the compressed type: "application/gzip" for GZip, "application/deflate" for Deflate and "application/br" for Brotli. We store the pre-compression content type, in the *Original Content Type* (originalContentType) header. + +Compression produces binary content. Where middleware requires that we transmit the message as text (for example over HTTPs such as SNS) we use a base64 string to ensure that the translation to and from text does not corrupt the data. Because turning binary data into a base64 string inflates it, you may need to adjust for that. As an example, if the limit of the middleware is 256K, a string that compresses to more than 192K will breach your limit. This is particularly useful to note if your strategy is to compress a string, and then use a [Claim Check](ClaimCheck.md) to offload any payloads that remain too large. In the example case your claim check would need to be at 192K and not 256K. + + + + + + diff --git a/contents/9/DapperOutbox.md b/contents/9/DapperOutbox.md new file mode 100644 index 0000000..a7f573b --- /dev/null +++ b/contents/9/DapperOutbox.md @@ -0,0 +1,93 @@ +# Dapper Outbox + +## Usage +The Dapper Outbox allows integration between Dapper and [Brighter's outbox support](/contents/BrighterOutboxSupport.md). The configuration is described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#outbox-support). + +For this we will need the *Outbox* package for Dapper. Packages for Dapper exist for the following RDBMS: MSSQL, MYSQL, and Sqlite. Packages have the naming convention: + +* **Paramore.Brighter.{DB}.Dapper** + +In addition, you will need the Outbox package for the relevant RDBMS: + +* **Paramore.Brighter.Outbox.{DB}** + +Obviously, {DB} should match. In the example below we use MySql, so we would need the following packages: + +* **Paramore.Brighter.MySql.Dapper** +* **Paramore.Brighter.Outbox.MySql** + +**Paramore.Brighter.MySql.Dapper** will pull in another two packages: + +* **Paramore.Brighter.MySql** +* **Paramore.Brighter.Dapper** + +As described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#outbox-support), we configure Brighter to use an outbox with the Use{DB}Outbox method call. + +As we want to use Dapper, we also call: Use{DB}TransactionConnectionProvider so that we can share your transaction scope when persisting messages to the outbox. + + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .UseExternalBus(...) + .UseMySqlOutbox(new MySqlConfiguration(DbConnectionString(), _outBoxTableName), typeof(MySqlConnectionProvider), ServiceLifetime.Singleton) + .UseMySqTransactionConnectionProvider(typeof(Paramore.Brighter.MySql.Dapper.UnitOfWork), ServiceLifetime.Scoped) + .UseOutboxSweeper() + + ... +} + +``` + +In our handler we take a dependency on Brighter's Dapper Unit of Work. We explicitly start a transaction within the handler on the Database within the Unit of Work. Dapper provides extension methods on a DbConnection for typical CRUD operations. Our Unit of Work wraps that DbConnection, and allows you to create a DB transaction associated with that DbConnection. You must use our method, and not create the transaction directly via the connection, because we cannot obtain that transaction. Sharing that transaction allows us to insert a message into the Outbox within the same transaction. + +We call **DepositPostAsync** within that transaction to write the message to the Outbox. Once the transaction has closed we can call **ClearOutboxAsync** to immediately clear, or we can rely on the Outbox Sweeper, if we have configured one to clear for us. (There are equivalent synchronous versions of these APIs).x + +``` csharp + public override async Task HandleAsync(AddGreeting addGreeting, CancellationToken cancellationToken = default(CancellationToken)) +{ + var posts = new List(); + + //We use the unit of work to grab connection and transaction, because Outbox needs + //to share them 'behind the scenes' + + var tx = await _uow.BeginOrGetTransactionAsync(cancellationToken); + try + { + var searchbyName = Predicates.Field(p => p.Name, Operator.Eq, addGreeting.Name); + var people = await _uow.Database.GetListAsync(searchbyName, transaction: tx); + var person = people.Single(); + + var greeting = new Greeting(addGreeting.Greeting, person); + + //write the added child entity to the Db + await _uow.Database.InsertAsync(greeting, tx); + + //Now write the message we want to send to the Db in the same transaction. + posts.Add(await _postBox.DepositPostAsync(new GreetingMade(greeting.Greet()), cancellationToken: cancellationToken)); + + //commit both new greeting and outgoing message + await tx.CommitAsync(cancellationToken); + } + catch (Exception e) + { + _logger.LogError(e, "Exception thrown handling Add Greeting request"); + //it went wrong, rollback the entity change and the downstream message + await tx.RollbackAsync(cancellationToken); + return await base.HandleAsync(addGreeting, cancellationToken); + } + + //Send this message via a transport. We need the ids to send just the messages here, not all outstanding ones. + //Alternatively, you can let the Sweeper do this, but at the cost of increased latency + await _postBox.ClearOutboxAsync(posts, cancellationToken:cancellationToken); + + return await base.HandleAsync(addGreeting, cancellationToken); +} +``` + +## Brighter Unit of Work without Dapper + +Because the Brighter Unit of Work just wraps a DbConnection and is's associated transaction, it can be used to provide a DbTransaction that works with the outbox whenever you want to use DbConnection to interface with a database. Whilst Dapper adds value on top of DbConnection, it just a set of extension methods, and our unit of work does not depend upon Dapper itself. + + diff --git a/contents/9/DarkerBasicConfiguration.md b/contents/9/DarkerBasicConfiguration.md new file mode 100644 index 0000000..464505b --- /dev/null +++ b/contents/9/DarkerBasicConfiguration.md @@ -0,0 +1 @@ +# Basic Configuration \ No newline at end of file diff --git a/contents/9/DispatchingARequest.md b/contents/9/DispatchingARequest.md new file mode 100644 index 0000000..aa12211 --- /dev/null +++ b/contents/9/DispatchingARequest.md @@ -0,0 +1,168 @@ +# Dispatching Requests + +Once you have [implemented your Request Handler](ImplementingAHandler.html), you will want to dispatch **Commands** or **Events** to that Handler. + +## Usage + +In the following example code we register a handler, create a *Command Processor*, and then use that *Command Processor* to dispatch a request to the handler. + + +``` csharp + public class Program + { + private static void Main() + { + var host = Host.CreateDefaultBuilder() + .ConfigureServices((context, collection) => + { + collection.AddBrighter().AutoFromAssemblies(); + }) + .UseConsoleLifetime() + .Build(); + + var commandProcessor = host.Services.GetService(); + + commandProcessor.Send(new GreetingCommand("Ian")); + + host.WaitForShutdown(); + } + } +``` + +## Registering a Handler + +In order for the *Command Processor* to find a Handler for your **Command** or **Event** you need to register the association between that **Command** or **Event** and your Handler. + +Brighter's **HostBuilder** support provides **AutoFromAssemblies** to register any *Request Handlers* in the project. See [Basic Configuration](/contents/BrighterBasicConfiguration.md) for more. If you are not using **HostBuilder** and or **ServiceCollection** you will need to register your handlers yourself. See [How Configuring the Command Processor Works](/contents/HowConfiguringTheCommandProcessorWorks.md). + +### Taking a Dependency on a Command Processor + +#### Producers + +Typically, a producer is an ASP.NET WebAPI or MVC app. In this case you take a dependency in your *Controller* on the **IAmACommandProcessor** interface, which is satisfied via *ServiceCollection*. + +If you intend to dispatch messages to another app, via message oriented middleware, your Brighter configuration will need a **Publication** which identifies how to to do that. + +#### Consumer + +An *Internal Bus* consumer is just a handler, typically registered through Brighter's ServiceCollection integration via our HostBuilder extension. It can thus take dependencies on other registered services within your app. + +An *External Bus* consumer is just a handler, but typically you host it using Brighter's *Service Activator*. You configure *Service Activator* to listens to messages flowing over message oriented middleware through a **Subscription**. *Service Activator* takes care of listening to messages arriving via the middleware, and delivering them to your handler code. In this way the complexity of using middleware is abstracted away from you, and you can focus on the business logic in your handler that you want to run in response to a message. + +### Pipelines Must be Homogeneous + +Brighter only supports pipelines that are solely **IHandleRequestsAsync** or **IHandleRequests**. In particular, note that middleware (attributes on your handler) must be of the same type as the rest of your pipeline. A common mistake is to **UsePolicy** when you mean **UsePolicyAsync**. + +## Dispatching Requests + +Once you have registered your Handlers, you can dispatch requests to them. + +### Internal Bus: Send & Publish + +When using an *Internal Bus*, the *Command Processor* has two options for dispatching messages: + +* **Send**: Used with a **Command**, send expects one, and only one, receiver. +* **Publish**: used with an **Event**, publish expects zero or more receivers. + +All methods have versions that support async...await. + +#### Internal Bus: Sending a Command + +A **Command** is an instruction to do work. We only expect one recipient to do the work, and side-effects mean that we want to ensure that only one receiver actions it as it typically mutates state. + +To send a **Command** you simply use **CommandProcessor.Send()** + +``` csharp +commandProcessor.Send(new GreetingCommand("Ian")); +``` + +NOTE: On a call to **CommandProcessor.Send()** the execution path flows to the handler. The Internal Bus is not buffered. + +#### Internal Bus: Returning results of a Command to the caller. + +Brighter follows Command-Query separation, and a Command does not have return value. So **CommandDispatcher.Send()** does not return anything. Please see a discussion on how to handle this in [Returning Results from a Handler](/contents/ReturningResultsFromAHandler.md). Also note that **Darker** provides our support for a **Query** over an Internal Bus. + +#### Internal Bus: Publishing an Event + +An **Event** is a fact, often the results of work that has been done. It is not atypical to raise an event to indicate the results of a **Command** having been actioned. + +``` csharp +commandProcessor.Publish(new GreetingEvent("Ian has been greeted")); +``` + +NOTE: On a call to **CommandProcessor.Publish()** the execution path flows to all handlers in a loop. The Internal Bus is not buffered. + +### External Bus: Post, Deposit and Clear + +When using an [External Bus](/contents/ImplementingExternalBus.md) the *Command Processor* has two options for dispatching a message: + +* **DepositPost** and **ClearOutbox**: This is a two-step approach to dispatching a message via middleware. It allows you to include the **DepositPost** call that puts the message in your [Outbox](/contents/BrighterOutboxSupport.md) within a database transation, so that you can achieve transactional messaging (either the message is placed in the Outbox and the change is made to any entities, or nothing is written to either). +* **Post**: This is a one-step approach to dispatching a message via middleware. Use it if you do not need transactional messaging, as described above. + +All methods have versions that support async...await. + +In both cases, if you use an Outbox with external storage, the message will be eventually delivered if it is written to the Outbox, provided that you run an *Outbox Sweeper* to dispatch any messages in the Outbox that have not been marked as dispatched. + +In this example we use **CommandProcessor.Post()** to dispatch a message over middleware. + +``` csharp +commandProcessor.Post(new GreetingCommand("Ian")); +``` + +In this exaple, we use **CommandProcessor.DepositPost()** and **CommandProcessor.ClearOutbox** to raise a transactional message. We then immediately clear it to lower latency. (We could have relied on an Outbox Sweeper and you should have an Outbox Sweeper in case this was to fail). + +In this example we are using Dapper as the library for writing our entities to the Db, and have used Brighter's Unit of Work support for that (passed into the handler constructor). + +```csharp +public override async Task HandleAsync(AddGreeting addGreeting, CancellationToken cancellationToken = default(CancellationToken)) +{ + var posts = new List(); + + //We use the unit of work to grab connection and transaction, because Outbox needs + //to share them 'behind the scenes' + + var conn = await _uow.GetConnectionAsync(cancellationToken); + await conn.OpenAsync(cancellationToken); + var tx = _uow.GetTransaction(); + try + { + var searchbyName = Predicates.Field(p => p.Name, Operator.Eq, addGreeting.Name); + var people = await conn.GetListAsync(searchbyName, transaction: tx); + var person = people.Single(); + + var greeting = new Greeting(addGreeting.Greeting, person); + + //write the added child entity to the Db + await conn.InsertAsync(greeting, tx); + + //Now write the message we want to send to the Db in the same transaction. + posts.Add(await _postBox.DepositPostAsync(new GreetingMade(greeting.Greet()), cancellationToken: cancellationToken)); + + //commit both new greeting and outgoing message + await tx.CommitAsync(cancellationToken); + } + catch (Exception e) + { + _logger.LogError(e, "Exception thrown handling Add Greeting request"); + //it went wrong, rollback the entity change and the downstream message + await tx.RollbackAsync(cancellationToken); + return await base.HandleAsync(addGreeting, cancellationToken); + } + + //Send this message via a transport. We need the ids to send just the messages here, not all outstanding ones. + //Alternatively, you can let the Sweeper do this, but at the cost of increased latency + await _postBox.ClearOutboxAsync(posts, cancellationToken:cancellationToken); + + return await base.HandleAsync(addGreeting, cancellationToken); +} +``` + +#### Message Mapper, MT_COMMAND and MT_EVENT + +When sending a message, the [Message Mapper](/contents/MessageMappers.md) is invoked to map your request to a Brighter **Message** which can be sent over message oriented middleware. + +Given you may have both a **Command** and an **Event** how do we preserve that behavior (a command expects one handler, an event zero or more) in listening applications? + +By setting the **Message.MessageType** to **MT_COMMAND** or **MT_EVENT** you indicate whether you expect this message to be treated as a **Command** or an **Event**. We flow that information in the message headers when sending over middleware. + +When *Service Activator* listens to messages it expects that the **MessageType** matches the type of **IRequest**, either **Command** or **Event** that your message mapper code transforms the message into. It will then use **CommandProcessor.Send()** to dispatch messages to a single handler, or **CommandProcessor.Publish** to dispatch messages to zero or more handlers, as appropriate. diff --git a/contents/9/DynamoInbox.md b/contents/9/DynamoInbox.md new file mode 100644 index 0000000..f2c6c27 --- /dev/null +++ b/contents/9/DynamoInbox.md @@ -0,0 +1,36 @@ +# Dynamo Inbox + +## Usage +The DynamoDb Inbox allows use of DynamoDb for [Brighter's inbox support](/contents/BrighterInboxSupport.md). The configuration is described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#inbox). + +For this we will need the *Inbox* packages for the DynamoDb *Inbox*. + +* **Paramore.Brighter.Inbox.DynamoDb** + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + var dynamoDb = new AmazonDynamoDBClient(credentials, new AmazonDynamoDBConfig { ServiceURL = "http://dynamodb.us-east-1.amazonaws.com"; }); + + services.AddServiceActivator(options => + { ... }) + .UseExternalInbox( + new DynamoDbInbox(dynamoDb); + new InboxConfiguration( + scope: InboxScope.Commands, + onceOnly: true, + actionOnExists: OnceOnlyAction.Throw + ) + ); +} + +... + +``` \ No newline at end of file diff --git a/contents/9/DynamoOutbox.md b/contents/9/DynamoOutbox.md new file mode 100644 index 0000000..d10e30b --- /dev/null +++ b/contents/9/DynamoOutbox.md @@ -0,0 +1,81 @@ +# DynamoDb Outbox + +## Usage +The DynamoDb Outbox allows integration between DynamoDb and [Brighter's outbox support](/contents/BrighterOutboxSupport.md). The configuration is described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#outbox-support). + +To support transactional messaging when using DynamoDb requires us to use DynamoDb's support for ACID transactions. You should understand best practices for using transactions with DynamoDb. + +For this we will need the *Outbox* package for DynamoDb: + +* **Paramore.Brighter.Outbox.DynamoDB** + +**Paramore.Brighter.Outbox.DynamoDb** will pull in another package: + +* **Paramore.Brighter.DynamoDb** + +As described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#outbox-support), we configure Brighter to use an outbox with the Use{DB}Outbox method call. + +As we want to use DynamoDb with the outbox, we also call: Use{DB}TransactionConnectionProvider so that we can share your transaction scope when persisting messages to the outbox. + + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .UseExternalBus(...) + .UseDynamoDbOutbox(ServiceLifetime.Singleton) + .UseDynamoDbTransactionConnectionProvider(typeof(DynamoDbUnitOfWork), ServiceLifetime.Scoped) + .UseOutboxSweeper() + + ... +} + +``` + +In our handler we take a dependency on Brighter's **IAmABoxTransactionConnectionProvider** interface and convert it to a **DynamoDbUnitofWork**. We explicitly start a transaction within the handler on the Database within the Unit of Work. + +We call **DepositPostAsync** within that transaction to write the message to the Outbox. Once the transaction has closed we can call **ClearOutboxAsync** to immediately clear, or we can rely on the Outbox Sweeper, if we have configured one to clear for us. (There are equivalent synchronous versions of these APIs).x + +``` csharp +public override async Task HandleAsync(AddGreeting addGreeting, CancellationToken cancellationToken = default(CancellationToken)) +{ + var posts = new List(); + + //We use the unit of work to grab connection and transaction, because Outbox needs + //to share them 'behind the scenes' + var context = new DynamoDBContext(_unitOfWork.DynamoDb); + var transaction = _unitOfWork.BeginOrGetTransaction(); + try + { + var person = await context.LoadAsync(addGreeting.Name); + + person.Greetings.Add(addGreeting.Greeting); + + var document = context.ToDocument(person); + var attributeValues = document.ToAttributeMap(); + + //write the added child entity to the Db - just replace the whole entity as we grabbed the original + //in production code, an update expression would be faster + transaction.TransactItems.Add(new TransactWriteItem{Put = new Put{TableName = "People", Item = attributeValues}}); + + //Now write the message we want to send to the Db in the same transaction. + posts.Add(await _postBox.DepositPostAsync(new GreetingMade(addGreeting.Greeting), cancellationToken: cancellationToken)); + + //commit both new greeting and outgoing message + await _unitOfWork.CommitAsync(cancellationToken); + } + catch (Exception e) + { + _logger.LogError(e, "Exception thrown handling Add Greeting request"); + //it went wrong, rollback the entity change and the downstream message + _unitOfWork.Rollback(); + return await base.HandleAsync(addGreeting, cancellationToken); + } + + //Send this message via a transport. We need the ids to send just the messages here, not all outstanding ones. + //Alternatively, you can let the Sweeper do this, but at the cost of increased latency + await _postBox.ClearOutboxAsync(posts, cancellationToken:cancellationToken); + + return await base.HandleAsync(addGreeting, cancellationToken); +} +``` diff --git a/contents/9/EFCoreOutbox.md b/contents/9/EFCoreOutbox.md new file mode 100644 index 0000000..97b6c30 --- /dev/null +++ b/contents/9/EFCoreOutbox.md @@ -0,0 +1,87 @@ +# EF Core Outbox + +## Usage +The EFCore Outbox allows integration between EF Core and [Brighter's outbox support](/contents/BrighterOutboxSupport.md). The configuration is described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#outbox-support). + +For this we will need the *Outbox* package for EF Core. Packages for EF Core exist for the following RDBMS: MSSQL, MYSQL, Postgres, and Sqlite. Packages have the naming convention: + +* **Paramore.Brighter.{DB}.EntityFrameworkCore** + +In addition, you will need the Outbox package for the relevant RDBMS: + +* **Paramore.Brighter.Outbox.{DB}** + +Obviously, {DB} should match. In the example below we use MySql, so we would need the following packages: + +* **Paramore.Brighter.MySql.EntityFrameworkCore** +* **Paramore.Brighter.Outbox.MySql** + +**Paramore.Brighter.MySql.EntityFrameworkCore** will pull in another package + +* **Paramore.Brighter.MySql** + +As described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#outbox-support), we configure Brighter to use an outbox with the Use{DB}Outbox method call. + +As we want to use EF Core, we also call: Use{DB}TransactionConnectionProvider so that we can share your transaction scope when persisting messages to the outbox. + + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .UseExternalBus(...) + .UseMySqlOutbox(new MySqlConfiguration(DbConnectionString(), _outBoxTableName), typeof(MySqlConnectionProvider), ServiceLifetime.Singleton) + .UseMySqTransactionConnectionProvider(typeof(MySqlEntityFrameworkConnectionProvider), ServiceLifetime.Scoped) + .UseOutboxSweeper() + + ... +} + +``` + +In our handler we take a dependency on our EF Core Context (derived from Db context). We explicitly start a transaction within the handler, because the Outbox is not within the Db Context we cannot rely on the DBContext's implicit transaction. + +We call **DepositPostAsync** within that transaction to write the message to the Outbox. Once the transaction has closed we can call **ClearOutboxAsync** to immediately clear, or we can rely on the Outbox Sweeper, if we have configured one to clear for us. (There are equivalent synchronous versions of these APIs).x + +``` csharp + public override async Task HandleAsync(AddGreeting addGreeting, CancellationToken cancellationToken = default(CancellationToken)) +{ + var posts = new List(); + + //We span a Db outside of EF's control, so start an explicit transactional scope + var tx = await _uow.Database.BeginTransactionAsync(cancellationToken); + try + { + var person = await _uow.People + .Where(p => p.Name == addGreeting.Name) + .SingleAsync(cancellationToken); + + var greeting = new Greeting(addGreeting.Greeting); + + person.AddGreeting(greeting); + + //Now write the message we want to send to the Db in the same transaction. + posts.Add(await _postBox.DepositPostAsync(new GreetingMade(greeting.Greet()), cancellationToken: cancellationToken)); + + //write the changed entity to the Db + await _uow.SaveChangesAsync(cancellationToken); + + //write new person and the associated message to the Db + await tx.CommitAsync(cancellationToken); + } + catch (Exception) + { + //it went wrong, rollback the entity change and the downstream message + await tx.RollbackAsync(cancellationToken); + return await base.HandleAsync(addGreeting, cancellationToken); + } + + //Send this message via a transport. We need the ids to send just the messages here, not all outstanding ones. + //Alternatively, you can let the Sweeper do this, but at the cost of increased latency + await _postBox.ClearOutboxAsync(posts, cancellationToken:cancellationToken); + + return await base.HandleAsync(addGreeting, cancellationToken); +} + +``` + diff --git a/contents/9/EventCarriedStateTransfer.md b/contents/9/EventCarriedStateTransfer.md new file mode 100644 index 0000000..46a9154 --- /dev/null +++ b/contents/9/EventCarriedStateTransfer.md @@ -0,0 +1,114 @@ +# Event Carried State Transfer (ECST) + +## Outside and Inside Data + +In his white paper \"Data on the Outside vs. Data on the Inside\", Pat Helland classifies data according to whether it exists inside a service boundary or outside that boundary. He calls the former Inside Data and the latter Outside Data. + +![ReferenceData](_static/images/ReferenceData.png) + +Inside Data is the data inside the service boundary. No one outside the service boundary can take a dependency on Inside Data. The service is the single writer of this data, the system of record, and is at liberty to change its schema. The backing store for a service holds Inside Data. + +Outside Data is what the service communicates to consumers at its boundary. It has three interesting properties: + +1: **It is immutable.** Changing this data does not impact the Inside Data of the originating service. That data; can only change from outside by using a provided API, if any, at the boundary. + +2: **It is stale.** As soon as data leaves our service it risks being stale. Any update to the data in the originating service will not be reflected in that data. Consider that if we use an HTTP API and GET the +state of a resource, a PUT that arrives to change that resource immediately after will mean the copy that we have is now stale. + +3: **It should be versioned.** Because the data risks being stale, we need to version it, so that we can compare it against potentially fresher versions. For example in HTTP we can use the If-None-Match +header with an ETag to determine if our data is stale, or if the resource would be the same if we retrieved it again. + +(It\'s worth noting that data supplied by a client as part of a command or request sent to the service is also Outside Data.) + +### Reference Data + +Pat Helland uses the term Reference Data to describe the types of data suitable for sharing as outside data. + +1\. Shared Collections. This is ubiquitous data that everyone needs to use to do work, such as a list of users, products, suppliers, brokers etc. It is so common for code to need to join this data that it makes +sense to copy it to each service that needs it. + +2\. Operand Data. Constructing requests to other microservices may require a service to understand a set of available options such as customer billing plans, or product categories. Operand data is where we +share the range of available options we can use to construct requests. + +3\. Snapshots. Where we want to query across multiple microservices we can end up with chatty solutions making requests to other microservices which we then need to join in the caller. An alternative is to listen to events so as to build an model that we can query. This is the model used by many Big Data pipelines or by Composite View Models. + +### Caching + +Outside Data is amenable to caching. Indeed this is how the web scales, we expose Outside Data from the Origin Server that is immutable (changing the response body has no impact on Inside Data), stale (post our GET a subsequent POST may modify the Inside Data our response was built from, perhaps before we have even parsed the result, and versioned (use of a Last Modified or ETag header allows the Origin Server to version the result that it responds with). + +## Event Carried State Transfer + +It\'s not just HTTP APIs whose results can be cached, we could also cache an event from the origin server, raised via AMQP, Kafka, ATOM or some other protocol. + +If the origin server raises events whenever a change occurs to the resources it manages, then consuming services downstream can cache those results, thus preventing the need to make a request to the origin server +to GET the current state of the entity. We can think of this as a push based cache instead of a pull based one. + +![ECST](_static/images/EventCarriedStateTransfer.png) + +The name given to building a cache upstream of the origin server from events is Event Carried State Transfer or ECST. + +There are some valuable aspects to this kind of cache. + +When we need to combine the state from two or more microservices to answer a query or respond to a command, we want to avoid making requests to that other service. This is because we temporally couple the two +services together - for our service to work, the other services must also be available. + +To solve this problem, we could work with a cache of the data we need to enrich our service built by ECST. By listening to the microservice we need the data from, we are able to join to the cache of data we require without making a request to that service. + +Usually a worker process listens for events from the other service and populates our local cache from the other microservice\'s events. + +Note that what we are putting in the cache is Outside Data, not Inside Data. We do not want to couple our consuming microservice to the the internals of the producing microservices model. We want to store the +equivalent to what we would recieve if we queried for it. This is why we prefer ECST over simple replication of data between services which would couple us to the details. + +### Alternatives to Event Carried State Transfer + +We could also meet the constraint that our service needs data from another to respond by ensuring that the request has all the data we require to process the event. If we think of the sender as the data +source, and our service as the data sink, we can build a pipeline where the original request is enriched with the required data by the microservices that own that data as a filter step. + +Where no central process controls this pipeline we refer to it as choreography. + +![Choreography](_static/images/Choreography.png) + +And we refer to it as orchestration when a process controls the pipeline. Orchestration uses commands, whereas choreography uses events. For this reason we may prefer choreography, as it has lower behavioural +coupling, unless we need confirmed rollback, or use of the reservation pattern which are easier with a Process Manager controlling the workflow. + +![Orchestration](_static/images/Orchestration.png) + +Whilst this could also work for a query, it is less common to take this approach to populating a response due to the likely latency of the response. + + +## Worked Scenario + +Imagine that we are writing software for a hotel. We have identified a number of microservices for our hotel: + +![HotelMicroservices](_static/images/HotelMicroservices.png) + +DirectBooking: Lets a customer reserve a room. May be a customer with an account or a guest. Credit Card Payments: Handles taking payments from a customer. Accounts: Holds information on account holders, including card details Housekeeping: Prepares rooms for a guest\'s stay and provides upkeep of the room during the stay Channel Manager: Markets our hotel rooms via various aggregator sites. + +When an account holder books a room they use the DirectBooking API to POST a booking. DirectBooking validates the booking and then raises an event to indicate that there has been a BookingMadeOnAccount. A number +of services listen for this message: + +Channel Manager: Decrements the rooms available on aggregator sites. Housekeeping: Schedules occupancy, cleaning of the room prior to occupancy, during and after. Credit Card Payments: Takes a payment from the Account holder. + +How does the Credit Card Payments system take the payment, when Accounts holds the account holders credit card details? We don\'t want to call a credit card details HTTP directly as this moves us back to a request driven architecture. + +We have two options. + +### A Pipeline + +Accounts listens for DirectBookingMadeOnAccount. It adds the credit card details to the booking and raises a DirectBookingMadeOnAccountWithCardDetails message. It is this message that Credit Card Payments listens to and then takes the card payment +via. + +![Choreography](_static/images/Choreography.png) + +### ECST + +Accounts publishes an event whenever an account holder changes name, address, or credit card details, called AccountDetailsChanged. Credit Card Payments subscribes to this event and caches the data in its own +backing store. Then when a payment request comes in via BookingMadeOnAccount it is able to look up the credit card details and take the payment. When we cross-check we can see that account details would seem to be a clear case of Shared Collection Reference Data and suitable for use in ECST. + +![ECST](_static/images/EventCarriedStateTransfer.png) + +Our preference for the two may depend on the extent to which we want to allow Credit Card Payments to take a payment even if Accounts is down, as Credit Card Payments is working with a cache. we may decide that a bulkhead is valuable enough to us to use ECST over choreography via a pipeline. + +## Next + +See [Correctness in Brighter](BrighterOutboxSupport.html) for guidance on how to use Brighter's support for the Outbox pattern to ensure producer-consumer correctness. diff --git a/contents/9/EventDrivenCollaboration.md b/contents/9/EventDrivenCollaboration.md new file mode 100644 index 0000000..05a293e --- /dev/null +++ b/contents/9/EventDrivenCollaboration.md @@ -0,0 +1,101 @@ +# Event Driven Collaboration + +Event Driven Architectures (EDA) are a major use case for Brighter's External Bus-you want processes to collaborate via messaging. + +(For another use case, offloading work to be performed asynchronously, see [Task Queues](/contents/TaskQueuePattern.md)) + +## Messaging + +Messages are packets of data, sent asynchronously over middleware + +- Commands, Documents, and Events are types of messages. +- Commands expect another service to handle the request, and possibly to respond. +- Documents represent the transfer of data; a query is a command, with a document response. +- Events are a notification that something happened. + +Commands can use a point-to-point channel i.e. only the sender and receiver are aware of the channel. Events can either use a publish-subscribe channel, or raise the event via a router that manages +dynamic subscriptions. The significant difference is that a sender of a command knows who the receiver is, the raiser of an event does not. + +## Temporal Coupling + +Why is it important that this integration is asynchronous? The answer is that we want to avoid Temporal Coupling. + +When we move from a monolithic to a microservices architecture, functionality and data becomes distributed across multiple microservices. In order to satisfy a given use case of our software, we +may need to multiple microservices to collaborate. + +Temporal Coupling occurs when in order to satisfy this use case, a set of microservices must all be available at the same time. + +Commonly, this occurs in systems that use synchronous communication protocols to integrate, such as HTTP + JSON, or gRPC. We refer to this as a Request Driven Architecture (or perhaps even Request Driven Collaboration). + +Let\'s take an example. In the illustration below we imagine hotel software and the use case of booking a room at the hotel. + +![RequestDrivenArchitecture](_static/images/RequestDrivenArchitecture.png) + +When a booking is made by an actor, an HTTP POST is made to our Direct Booking microservice with the details in the body of the request. The Direct Booking microservice validates the supplied details, and then +validates the request for missing information, room availability and so on. + +Once the booking is made the workflow for the operation suggests that we need to take a payment. In this use case the payment is being made by an existing account holder, and so the POST body does not contain the +payment details, instead, details already held on account are being used. + +In addition to taking a payment, we want to inform housekeeping of the booking, and use our Channel Manager to lower the availability of rooms on aggregator sites. + +How does the Direct Booking microservice communicate these steps to other microservices? + +In a Request Driven Architecture we would use a synchronous protocol such as HTTP+JSON or gRPC to call the API exposed by the other microservices. The issue here is that for our service to work, all these other services must also be available. + +A useful metaphor here is a phone call. If we make a phone call, the other party must be available i.e. present and not busy with another call. + +If the Channel Manager is not available, as in the diagram, does our transaction, the room booking, fail. + +Now, we can try to mitigate the risks of a Request Driven Architecture. We can call through a proxy that load balances across a pool of upstream instances of the service. The proxy can retry failed requests to +alleviate transient availability issues and take bad instances out of the pool. + +This is what a \'service mesh\' does - it improves the availability of a Request Driven Architecture by seeking to lower Temporal Coupling. + +## Behavioral Coupling + +Why does it matter whether we know about the receiver (a command) or are ignorant as to receivers? The answer is behavioral coupling. + +Let\'s look at our hotel example again. + +What happens if we decide that we no longer want a single housekeeping service, but a set of services such a laundry, room cleaning etc.? In a Request Driven Architecture the caller knows about the sequence of steps to complete a booking. This means that any change to the sequence, to call the new Room Cleaning microservice instead of the housekeeping service requires a change to Direct Booking. + +![BehavioralCoupling](_static/images/BehavioralCoupling.png) + +This coupling, through knowledge of other services, if a form of behavioral coupling. It hampers our goal of independent deployability because downstream components are impacted by changes to the partitions of upstream components, or to the shape of their APIs. + +## Event Driven Collaboration + +Event Driven Collaboration helps us solve the problems of temporal and behavioural coupling. + +When we publish an event, a subscriber uses a queue to receive events asynchronously. Because messages are held in a queue, the subscriber does not have to be available when the publisher produces the message, +only the queue does. If the subscriber is not available, the queue holds the message until the subcriber can process it. + +This removes temporal coupling - we do not need both services to be available at the same time. + +Let\'s look at the hotel example again. When the direct booking is made, the Payments microservice the Housekeeping service and the Channel Manager service can be unavailable. Their subscription just queues the +message until the service is available. + +![EventDrivenArchitecture1](_static/images/EventDrivenArchitecture1.png) + +![EventDrivenArchitecture2](_static/images/EventDrivenArchitecture2.png) + +This allows us to take the booking, even if these services are not available - we have given our microservices \'bulkheads\' against failure. Our system can keep offering service, even if parts of it are +not available. + +We do have to design our workflow, such that the customer expects an asynchronous operation i.e. \"We will mail you to confirm your booking\". This may seem like a limitation, but many workflows were +traditionally asynchronous before widespread automation, so processes exist for this approach, and customers expect \'tasks\', such as making a booking may need confirmation. + +If the metaphor for Request Driven Architectures is a phone call, for Event Drive Architectures it is SMS or a messaging app such as WhatsApp or Slack (or even a phone answering service). You don\'t need both +parties to be available, when you use messaging. + +In addition, a publisher does not know who it\'s subscribers are. That is the function of message oriented middleware - a broker. The broker routes messages from a publisher to subscribers. Because of this, +changes to the subscribers don\'t impact the publisher. The publisher remains independently deployable of its subscribers. + +So an Event Driven Architecture benefits from a lack of behavioral coupling too. + +(Note that if we use commands, and not events, between microservices i.e. the sender knows who should receive this instruction, we do not have temporal coupling, but we do have behavioral coupling). + +## Next + +See [Event Carried State Transfer](EventCarriedStateTransfer.html) for guidance on how to \'join\' data between two microservices, when you need data from more than one service to carry out an operation. diff --git a/contents/9/FAQ.md b/contents/9/FAQ.md new file mode 100644 index 0000000..186e900 --- /dev/null +++ b/contents/9/FAQ.md @@ -0,0 +1,51 @@ +# FAQ + +## Asynchronous or External Bus + +When should you use an asynchronous pipeline to handle work and when should you use an External Bus. + +Using an asynchronous handler allows you to avoid blocking I/O. This can increase your throughput by allowing you to re-use threads to service new requests. Using this approach, even a single-threaded application can +achieve high throughput, if it is not CPU-bound. + +Using an External Bus allows you to hand-off work to another process, to be executed at some point in the future. This also allows you to improve throughput by freeing up the thread to service new requests. We assume that we can accept dealing with that work at some point in the future i.e. we can be eventually consistent. + +One disadvantage of an External Bus is that the pattern - ack to callers, and then do the work, can create additional complexity because we must deal with notifying the user of completion, or errors. Because an async operation simply has the caller wait, the programming model is simpler. The trade-off here is that the client of our process is still using resources awaiting for the request with the async operation. If the operation takes time to complete the client may not know if the operation failed and should be timed out, or is still running. + +Where work is long-running there is a risk that the server faults, and we lose the long-running work. An External Bus provides reliability here, through guaranteed delivery. The queue keeps the work until it is +successfully processed and acknowledged. + +Our recommendation is to use the async pattern to improve throughput where the framework supports async, such as ASP.NET WebAPI but to continue to hand-off work that takes a long time to complete to a work +queue. You may choose to define your own thresholds but we recommend that operations that take longer than 200ms to complete be handed-off. We also recommend that operations that are CPU bound be handed-off as +they diminish the throughput of your application. + +## Iterating over a list of requests to dispatch them + +All **Command** or **Event** messages derive from **IRequest** and **ICommand** and **IEvent** respectively. So it may seem natural to create a collection of them, for example **List\**, and then +process a set of messages by enumerating over them. + +When you try this, you will encounter the issue that we dispatch based on the concrete type of the **Command** or **Event**. In other words the type you register via the **SubscriberRegistry.** Because +**CommandProcessor.Send()** is actually **CommandProcessor.Send\()** you need to provide the concrete type in the call for the compiler to determine the type to use with the cool as the concrete type. + +If you try this: + +``` csharp +ICommand command = new GreetingCommand("Ian"); +commandProcessor.Send(command); +``` + +Then you will get this error: *\"ArgumentException \"No command handler was found for the typeof command Brighter.commandprocessor.ICommand - a command should have exactly one handler.\"\"* + +Now, you don\'t see this issue if you pass the concrete type in, so the compiler can correctly resolve the run-time type. + +``` csharp +commandProcessor.Send(new GreetingCommand("Ian")); +``` + +So what can you do if you must pass the base class to the **Command Processor** i.e. because you are using a list. + +The workaround is to use the dynamic keyword. Using the dynamic keyword means that the type will be evaluated using RTTI, which will successfully pick up the type that you need. + +``` csharp +ICommand command = new GreetingCommand("Ian"); +commandProcessor.Send((dynamic)command); +``` diff --git a/contents/9/FeatureSwitches.md b/contents/9/FeatureSwitches.md new file mode 100644 index 0000000..18f6955 --- /dev/null +++ b/contents/9/FeatureSwitches.md @@ -0,0 +1,123 @@ +# Feature Switches + +We provide a **FeatureSwitch** Attribute and **FeatureSwitchAsync** Attribute that you can use on your **IHandleRequests\.Handle()** method, and **IHandleRequests\.HandleAysnc()** method. The **FeatureSwitch** Attribute and **FeatureSwitchAsync** Attribute that you have configured will determine whether or not the +**IHandleRequests\.Handle()** and **IHandleRequests\.HandleAsync()** will be executed. + +## Using the Feature Switch Attribute + +By adding the **FeatureSwitch** Attribute or **FeatureSwitchAsync** Attribute, you instruct the Command Processor to do one of the following: + +- run the handler as normal, this is **FeatureSwitchStatus.On**. +- not execute the handler, this is **FeatureSwitchStatus.Off**. +- detemine whether to run the handler based on a **Feature Switch + Registry**, [creating of which is described + later](FeatureSwitches.html#building-a-config-for-feature-switches-with-fluentconfigregistrybuilder). + +In the following example, **MyFeatureSwitchedHandler** will only be run if it has been configured in the **Feature Switch Registry** and set to **FeatureSwitchStatus.On**. + +``` csharp +class MyFeatureSwitchedHandler : RequestHandler +{ + [FeatureSwitch(typeof(MyFeatureSwitchedHandler), FeatureSwitchStatus.Config, step: 1)] + public override MyCommand Handle (MyCommand command) + { + /* Do work */ + return base.Handle(command); + } +} +``` + +In the second example, **MyIncompleteHandlerAsync** will not be run in the pipeline. + +``` csharp +class MyIncompleteHandlerAsync : RequestHandlerAsync +{ + [FeatureSwitchAsync(typeof(MyIncompleteHandlerAsync), FeatureSwitchStatus.Off, step: 1)] + public override Task HandleAsync(MyCommand command, CancellationToken cancellationToken = default) + { + /* Nothing implmented so we're skipping this handler */ + return await base.HandleAsync(command, cancellationToken); + } +} +``` + +## Building a config for Feature Switches with FluentConfigRegistryBuilder + +We provide a **FluentConfigRegistryBuilder** to build a mapping of request handlers to **FeatureSwitchStatus**. For each Handler that you wish to feature switch you supply a type and a status using a fluent +API. The valid statuses used in the builder are **FeatureSwitchStatus.On** and **FeatureSwitchStatus.Off**. + +``` csharp +var featureSwitchRegistry = FluentConfigRegistryBuilder + .With() + .StatusOf().Is(FeatureSwitchStatus.On) + .StatusOf().Is(FeatureSwitchStatus.Off) + .Build(); +``` + +## Implementing a custom Feature Switch Registry + +The **FluentConfigRegistryBuilder** provides compile time configuration of **FeatureSwitch** Attributes. If this is not suitable to your needs then you can write you own Feature Switch Registry using the [IAmAFeatureSwitchRegistry](https://github.com/BrighterCommand/Brighter/blob/master/src/Paramore.Brighter/FeatureSwitch/IAmAFeatureSwitchRegistry.cs) interface. The two requirements of this interface is a [MissingConfigStrategy](https://github.com/BrighterCommand/Brighter/blob/master/src/Paramore.Brighter/FeatureSwitch/MissingConfigStrategy.cs), and an implementation of **StatusOf(Type type)** which returns a +**FeatureSwitchStatus**. + +The **MissingConfigStrategy** determines how the Command Processor should behave when a Handler is decorated with a **FeatureSwitch** Attribute that is set to **FeatureSwitchStatus.Config** does not exist +in the registry. + +Your implementation of the **StatusOf** method is used to determine the **FeatureSwitchStatus** of the Handler type that is passed in as a parameter. How you store and retrieve these configurations is then up to +you. + +In the following example there are two FeatureSwitches configured in the **AppSettings.config**. We then build an **AppSettingsConfigRegistry**. The **StatusOf** method is implemetned to read the config from the App Settings and return the status for the given type. + +``` xml + + + + +``` + +``` csharp +class AppSettingsConfigRegistry : IAmAFeatureSwitchRegistry +{ + public MissingConfigStrategy MissingConfigStrategy { get; set; } + + public FeatureSwitchStatus StatusOf(Type handler) + { + var configStatus = ConfigurationManager.AppSettings[$"FeatureSwitch::{handler}"].ToLower(); + + switch (configStatus) + { + case "on": + return FeatureSwitchStatus.On; + case "off": + return FeatureSwitchStatus.Off; + default: + return MissingConfigStrategy is MissingConfigStrategy.SilentOn + ? FeatureSwitchStatus.On + : MissingConfigStrategy is MissingConfigStrategy.SilentOff + ? FeatureSwitchStatus.Off + : throw new InvalidOperationException($"No Feature Switch configuration for {handler} specified"); + } + } +} +``` + +## Setting Feature Switching Registry + +We associate a **Feature Switch Registry** with a **Command Processor** by passing it into the constructor of the **Command Processor**. For convenience, we provide a **Commmand Processor Builder** that helps you +configure new instances of **Command Processor**. + +``` csharp +var featureSwitchRegistry = FluentConfigRegistryBuilder + .With() + .StatusOf().Is(FeatureSwitchStatus.Off) + .Build(); + +var builder = CommandProcessorBuilder + .With() + .Handlers(new HandlerConfiguration(_registry, _handlerFactory)) + .DefaultPolicy() + .NoTaskQueues() + .ConfigureFeatureSwitches(fluentConfig) + .RequestContextFactory(new InMemoryRequestContextFactory()); + +var commandProcessor = builder.Build(); +``` diff --git a/contents/9/HandlerFailure.md b/contents/9/HandlerFailure.md new file mode 100644 index 0000000..f36f8db --- /dev/null +++ b/contents/9/HandlerFailure.md @@ -0,0 +1,48 @@ +# Failure and Dead Letter Queues + +When a *Request* is passed to **RequestHandler.Handle()** it runs in your application code. If your application code fails, you have a number of options: + +- [Retry (and Circuit Break) the *Request* on the Internal Bus](#retry-and-circuit-break-the-request-on-the-internal-bus) +- [Retry (with Delay) the *Request* on the External Bus](#retry-with-delay-the-request-on-the-external-bus) +- [Terminate processing of that *Request*](#terminate-processing-of-that-request) +- [Run a Fallback](#run-a-fallback) +- [Use Custom Middleware](#use-custom-middleware) + +Any unhandled exception that leaves the *Request Handling Pipeline* (in other words is not intercepted by middleware) will [Terminate Processing of the Request](#terminate-processing-of-that-request). + +## **Retry (and Circuit Break) the *Request* on the Internal Bus** + +You can use Brighter's support for Polly policies to retry the operation on the Internal Bus. See [Retry and Circuit Breaker](/contents/PolicyRetryAndCircuitBreaker.md) + +A circuit breaker is triggered when all Retry attempts fail, and will prevent further requests from succeeding. + +Both the triggering of the circuit breaker, and requests passed to the *Request Handler Pipeline* while the circuit breaker is open will [Terminate processing of that *Request*](#terminate-processing-of-that-request) + +## Retry (with Delay) the *Request* on the External Bus + +If you the failure is potentially retriable, but you want to retry on the External Bus (by making the message available to be consume from the External Bus again) then you can throw a **DeferMessageAction** exception. Upon receipt of a **DeferMessageAction** the pump will Reject the message and Requeue it. with a delay. The delay is configured by the External Bus **Subscription.RequeueDelayInMilliseconds** property. + +You can configure a limit on the number of requeue attempts by setting the **Subscription.RequeueCount**. A value of -1 will allow infinite retries. + +## Terminate processing of that *Request* + +An unhandled exception leaving the pipeline results in us terminating processing of the *Request*. + +- On an Internal Bus that exception will bubble out to the caller. +- On an External Bus we will nack (or reject) the message. On a queue this will delete the message for all consumers, on a stream we will increment the offset past that message for a consumer group. + +We do this because you have responsibility to handle exceptions thrown in your code, not the framework and we assume that non-recovered errors are not potentially retriable. + +On and Exernal Bus if your middleware supports a Dead Letter Queue (DLQ), and it is configured in your subscription, when we reject a message it will be copied to the DLQ. + +On an External Bus, to prevent discarding too many messages, you can set an **Subscription.UnacceptableMessageLimit**. If the number of messages terminated due to unhandled exceptions equals or exceeds this limit, the message pump processing the External Bus will terminate. + +## Run a Fallback + +If you want to take action before exiting a handler, due to a failure you can use a Fallback policy. See [Fallback Policy](/contents/PolicyFallback.md) for more details + +## Use Custom Middleware + +If none of the above options meet your needs, you can define custom approaches to exception handling by building your own middleware, see [Pipeline](BuildingAPipeline.html). + + diff --git a/contents/9/HealthChecks.md b/contents/9/HealthChecks.md new file mode 100644 index 0000000..98f0860 --- /dev/null +++ b/contents/9/HealthChecks.md @@ -0,0 +1,75 @@ +# Health Checks + +Brighter provides an AspNet Core Health check for **Service Activator** + +## Configure Health Checks + +The below will configure ASP.Net Core Health checks for Brighter's **Service Activator**, for more information on [ASP.NET Core Health Check](https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/health-checks?view=aspnetcore-6.0) + +```csharp +// Web Application Builder code goes here + +builder.Services.AddHealthChecks() + .AddCheck("Brighter", HealthStatus.Unhealthy); + +var app = builder.Build(); + +app.UseEndpoints(endpoints => +{ + endpoints.MapHealthChecks("/health"); + endpoints.MapHealthChecks("/health/detail", new HealthCheckOptions + { + ResponseWriter = async (context, report) => + { + var content = new + { + Status = report.Status.ToString(), + Results = report.Entries.ToDictionary(e => e.Key, + e => new + { + Status = e.Value.Status.ToString(), + Description = e.Value.Description, + Duration = e.Value.Duration + }), + TotalDuration = report.TotalDuration + }; + + context.Response.ContentType = "application/json"; + await context.Response.WriteAsync(JsonSerializer.Serialize(content, JsonSerialisationOptions.Options)); + } + }); +}); + +app.Run(); + +``` + +The /health endpoing will return a Status 200 with the Body of the Health status (i.e. Healthy) + +The /health/detail endpoint will return a detailed response with all of your information for example: + +```json +{ + "status": "Healthy", + "results": { + "Brighter": { + "status": "Healthy", + "description": "21 healthy consumers.", + "duration": "00:00:00.0000132" + } + }, + "totalDuration": "00:00:00.0029747" +} +``` + +## Health Status + +The following will be produced + +| Scenario | Status | +| -------- | ------ | +| All Message Pumps are running | Healthy | +| Some Message Pumps are running | Degraded | +| No Message Pumps are running | Unhealthy | + +In the event on a Degraded status the /health/details page can be used to find out which Dispatchers have failed pumps \ No newline at end of file diff --git a/contents/9/HowBrighterWorks.md b/contents/9/HowBrighterWorks.md new file mode 100644 index 0000000..ecb8e10 --- /dev/null +++ b/contents/9/HowBrighterWorks.md @@ -0,0 +1,87 @@ +# How The Command Processor Works + +You don\'t need to understand how Brighter works under the hood to use it, but if you want to debug, or contribute to the project, it can help to know what is going on. + +## The Dispatcher + +Ignoring attributes, which create a pipeline, for now, all of CommandProcessor\'s dispatch methods: + +- Accept a IAmARequest derived class as an argument +- Find the registered handler(s) for the type of the command +- Ask the factory you provide to create an instance of that handler +- Call Handle() or HandleAsync() as appropriate on that handler, passing in the command + +Let\'s look at that sequence in more detail, for those of you who want to understand the detail, so that you can walk the code if required. We\'ll show the code looking for a SendAsync() but the Send() works the +same way. Publish() and PublishAsync() are a minor variation in that they may dispatch to multiple handler chains, not just the one. + +![SequenceDiagram](_static/images/Brighter_SendAsync_Pipeline.png) + +Let\'s model what happens, if you use Brighter as the Ports layer behind the an ASP.NET Core web controller. + +1: The client makes an HTTP POST request to ASP.NET Core which marshalls the parameters and calls your web controller\'s POST method. + +2: The web controller takes those parameters, along with any other relevant information, such as identity from the bearer token, and creates a Command. Let\'s call it MyCommand + +3: The web controller calls the SendAsync() method on CommandProcessor because it is a Command and we expect one handler. If this was an Event, perhaps raised by the command handler itself, we would call +PublishAsync() instead. + +4: The SendAsync() method creates a RequestContext, using the supplied RequestContextFactory. Here we use the supplied InMemoryRequestContextFactory, which does not try to persist the state of the request, and is adequate for most purposes. + +5: The CommandProcessor then creates a new PipelineBuilder. The PipelineBuilder is a generic type specialized to the type of the Request. (Both Command and Event inherit from Request.) The PipelineBuilder orchestrates building the chain of responsibility that will handle our request. + +6: The PipelineBuilder creates an Interpreter. Again this is a generic type, specialized to the type of the Request. The interpreter is going to find the \'target handlers\'. A \'target handler\' is your code that implements IAmARequestHandlerAsync (usually via RequestHandlerAsync\<\>) that you register via the SubscriberRegistry. it is where the code which exercises the entities of your domain should live. The PipelineBuilder also creates a LifeTimeScope It will track the handlers we have created as part of this request. + +7: The CommandProcessor calls Build() on the PipelineBuilder to create the pipeline. + +8: The PipelineBuilder asks the Interpreter for an instance of the registered RequestHandlerAsync\<\>. + +9: The Interpreter uses the SubscriberRegistry to lookup the RequestHandlerAsync\<\> that we have associated with this Command. We don\'t show it here, but we passed the SubscriberRegistry to the +CommandProcessor when we built it, so it\'s not created here. The SubscriberRegistry returns the type of RequestHandlerAsync\<\>. In our example MyCommandHandlerAsync. + +10: The Interpreter now tries to build an instance of the handler by calling the HandlerFactory. The HandlerFactory is supplied to the CommandProcessor when it is built. It is user implemented because we +don\'t know how to build your handler, which has its dependencies in your code. However, we do integrate with ServiceCollection if desired. In our case we build a MyHandlerAsync. + +11: Having constructed a handler, we now need to build the pipeline based on the attributes that you have decorated the handler with (and any global attributes). We call the BuildPipeline method to create the +pipeline. + +12: We start with the part of the chain called before the registered handler - attributes flagged HandlerTiming.Before. + +13: Because your pipeline\'s configuration can only change at design time (when you write code) and not at runtime (when you execute) we only want to figure out the attributes of the chain once. So we store the +attribute list, once it\'s been determined, in a memento collection. This collection is a static. The first thing we do when building is see if we have already determined the configuration. If we have, we will use that. + +14: If we have not got a pipeline configuration, then we need to build one. Our first step is to call FindHandlerMethod() to get the handler method from our target handler i.e. MyCommandHandler in this case. RequestHandlerAsync does the heavy lifting for you here. + +15: We then use RTTI to find the attributes you have tagged your handler method with. In this case, let\'s pretend we just have RequestLoggingAsync\<\>. We order them via the Step value on the attribute (note that step collision order behavior is undefined). + +16: Once we have the per-handler list of attributes, we check if the CommandProcessor has been configured to use a global inbox. If it already has a UseInboxAsync attribute, or has a NoGlobalInbox attribute in the preAttributes list, we are done. Otherwise, if we have set up a global inbox, we add UseInboxAsync into the list of handlers. + +17: We then add the preAttributes into the collection of preAttributes, so that we will not need to use RTTI to build them again. Let\'s assume that we have configured out CommandProcessor with a global inbox. + +18: Next we have to construct this preAttribute chain. We iterate over all of the attributes and create the handlers by calling the supplied HandlerFactory. Again, this is because only your code knows how to create your handlers. If you use our support for ServiceCollection, then we will search the assemblies in the project for classes that implement IAmARequestHandlerAsync (including those we supply) and register them for you. The hander we create is determined from the type information supplied by the Attribute. In our example we will construct a RequestLoggingHandlerAsync and a UseInboxHandlerAsync. + +19: We construct the pre-Attributes. once we have constructed a Handler, we set it\'s successor property to be the next handler in the chain. + +20: We then repeat this process for any post handler attributes i.e. those tagged as HandlerTiming.After. We don\'t have any of those here, so we don\'t show that again this time. This decision was mainly for +simplicity. + +21: We then add the handlers to the LifeTimeScope of the pipeline. + +22: We then return the handler chain that we just constructed to the CommandProcessor. + +23: The CommandProcessor checks that we have a valid pipeline; for a SendAsync we expect exactly one pipeline will handle the Command. For PublishAsync we allow zero or more. + +24: Now we call the pipeline by passing our MyCommand to the first handler in the chain, in our case the UseInboxHandlerAsync. + +25: The UseInboxHanlderAsync has an inbox as a private member, that was passed in via the constructor via the HandlerFactoryAsync. This is a data store specific implementation of IAmAnInbox. if the OnceOnly parameter is set on the attribute then we call the Inbox\'s ExistsAsync method to determine if we have already processed the command. If the command has not already been processed, we call the base class\'s HandleAsync method. + +26: The base class\'s HandleAsync() method we use the successor field (see 19 above) to determine the next handler in the chain and we call it\'s HandleAsync() method. In this case we call RequestLoggingAsync\<\>\'s HandleAsync method. + +27: The RequestLoggingAsync\<\>\'s HandleAsync method logs the call, and again calls the base class\'s HandleAsync() method to pass the call down the pipeline. + +28: Finally, we call MyCommandHandlerAsync whose HandleAsync() command runs our business logic. Again we call the base class\'s HandleAsync() method, but as there is no successor we return. + +29: We return from RequestLoggingAsync\<\> which has no work left to do. + +30: UseInboxHandlerAsync calls IAmAnIbox\'s AddAsync method to write the command to the Inbox. Then it returns. + +31: SendAsync returns, and we are done. diff --git a/contents/9/HowConfiguringTheCommandProcessorWorks.md b/contents/9/HowConfiguringTheCommandProcessorWorks.md new file mode 100644 index 0000000..2a5f014 --- /dev/null +++ b/contents/9/HowConfiguringTheCommandProcessorWorks.md @@ -0,0 +1,165 @@ +# How Configuring the Command Processor Works + +Brighter does not have a dependency on an Inversion Of Control (IoC) framework. This gives you freedom to choose the DI libraries you want for your project. + +We follow an approach outlined by Mark Seeman in his blog on a [DI Friendly +Framework](http://blog.ploeh.dk/2014/05/19/di-friendly-framework/) and +[Message Dispatching without Service +Location](http://blog.ploeh.dk/2011/09/19/MessageDispatchingwithoutServiceLocation/). + +This means that we can support any approach to DI that you choose, provided you implement a range of interfaces that we require to create instances of your classes at runtime. + +For .NET Core's DI framework we provide the implementation of these interfaces. If you are using that approach, just follow the outline in [Basic Configuration](/contents/BrighterBasicConfiguration.md). This chapter is 'interest only' at that point, and you don't need to read it. It may be helpful for debugging. + +If you choose another DI framework, this document explains what you need to do to support that DI framework. + +## CommandProcessor Configuration Dependencies + +- You need to provide a **Subscriber Registry** with all of the **Command**s or **Event**s you wish to handle, mapped to their **Request Handlers**. +- You need to provide a **Handler Factory** to create your Handlers +- You need to provide a **Policy Registry** if you intend to use [Polly](https://github.com/App-vNext/Polly) to support Retry and Circuit-Breaker. +- You need to provide a **Request Context Factory** + +## Subscriber Registry + +The Command Dispatcher needs to be able to map **Command**s or **Event**s to a **Request Handlers**. + +For a **Command** we expect one and only one **Request Handlers** for an event we expect many. + +YOu can use our **SubcriberRegistry** regardless of your DI framework. + +Register your handlers with your **Subscriber Registry** + +``` csharp +var registry = new SubscriberRegistry(); +registry.Register(); +``` + +We also support an initializer syntax + +``` csharp +var registry = new SubscriberRegistry() +{ + {typeof(GreetingCommand), typeof(GreetingCommandHandler)} +} +``` + +## Handler Factory + +We don't know how to construct your handler so we call a factory, that you provide, to build your handler (and its entire dependency chain). + +Instead, we take a dependency on an interface for a handler factory, and you implement that. Within the handler factory you need to construct instances of your types in response to our request to create one. + +For this you need to implement the interface: **IAmAHandlerFactory**. + +Brighter manages the lifetimes of handlers, as we consider the request pipeline to be a scope, and we will call your factory again informing that we have terminated the pipeline and finished processing the request. You should take any required action to clear up the handler and its dependencies in response to that call. + +You can implement the Handler Factory using an IoC container. This is what Brighter does with .NET Core + +For example using [TinyIoC Container](https://github.com/grumpydev/TinyIoC): + +``` csharp +internal class HandlerFactory : IAmAHandlerFactory +{ + private readonly TinyIoCContainer _container; + + public HandlerFactory(TinyIoCContainer container) + { + _container = container; + } + + public IHandleRequests Create(Type handlerType) + { + return (IHandleRequests)_container.GetInstance(handlerType); + } + + public void Release(IHandleRequests handler) + { + _container.Release(handler); + } +} +``` + +## Policy Registry + +If you intend to use a [Polly](https://github.com/App-vNext/Polly) Policy to support [Retry and Circuit-Breaker](PolicyRetryAndCircuitBreaker.html) then you will need to register the Policies in the **Policy Registry**. + +This is just the Polly **PolicyRegistry**. + +Registration requires a string as a key, that you will use in your [UsePolicy] attribute to choose the policy. + +The two keys: CommandProcessor.RETRYPOLICY and CommandProcessor.CIRCUITBREAKER are used within Brighter to control our response to broker issues. You can override them if you wish to change our behavior from the default. + +You can also use them for a generic retry policy, though we recommend building retry policies that handle the kind of exceptions that will be thrown from your handlers. + +In this example, we set up a policy. To make it easy to reference the string, instead of adding it everywhere, we use a global readonly reference, not shown here. + +``` csharp +var retryPolicy = + Policy.Handle().WaitAndRetry( + new[] { + TimeSpan.FromMilliseconds(50), + TimeSpan.FromMilliseconds(100), + TimeSpan.FromMilliseconds(150) }); + +var circuitBreakerPolicy = Policy.Handle().CircuitBreaker( + 1, TimeSpan.FromMilliseconds(500)); + +var policyRegistry = new PolicyRegistry() { + { Globals.MYRETRYPOLICY, retryPolicy }, + { Globals.MYCIRCUITBREAKER, circuitBreakerPolicy } + }; +``` + +When you attribute your code, you then use the key to attach a specific policy: + +``` csharp +[RequestLogging(step: 1, timing: HandlerTiming.Before)] +[UsePolicy(Globals.MYRETRYPOLICY, step: 2)] +public override TaskReminderCommand Handle(TaskReminderCommand command) +{ + _mailGateway.Send(new TaskReminder( + taskName: new TaskName(command.TaskName), + dueDate: command.DueDate, + reminderTo: new EmailAddress(command.Recipient), + copyReminderTo: new EmailAddress(command.CopyTo) + )); + + return base.Handle(command); +} +``` + +If you need multiple policies then you can pass them as an array. We evaluate them left to right. + +``` csharp +[RequestLogging(step: 1, timing: HandlerTiming.Before)] +[UsePolicy(new [] {Globals.MYRETRYPOLICY, Globals.MYCIRCUITBREAKER}, step: 2)] +public override TaskReminderCommand Handle(TaskReminderCommand command) +{ + _mailGateway.Send(new TaskReminder( + taskName: new TaskName(command.TaskName), + dueDate: command.DueDate, + reminderTo: new EmailAddress(command.Recipient), + copyReminderTo: new EmailAddress(command.CopyTo) + )); + + return base.Handle(command); +} +``` + +## Request Context Factory + +You need to provide a factory to give us instances of a [Context](UsingTheContextBag.html). If you have no implementation to use, just use the default **InMemoryRequestContextFactory**. Typically you would replace ours if you wanted to support initializing the context outside of our pipeline, for tracing for example. + +## Command Processor Builder + +All these individual elements can be passed to a **Command Processor Builder** to help build a **Command Processor**. This has a fluent interface to help guide you when configuring Brighter. The result looks like this: + +``` csharp +var commandProcessor = CommandProcessorBuilder.With() + .Handlers(new HandlerConfiguration(subscriberRegistry, handlerFactory)) + .Policies(policyRegistry) + .NoExternalBus() + .RequestContextFactory(new InMemoryRequestContextFactory()) + .Build(); +``` diff --git a/contents/9/HowConfiguringTheDispatcherWorks.md b/contents/9/HowConfiguringTheDispatcherWorks.md new file mode 100644 index 0000000..ce17b49 --- /dev/null +++ b/contents/9/HowConfiguringTheDispatcherWorks.md @@ -0,0 +1,111 @@ +# How Configuring a Dispatcher for an External Bus Works + +In order to receive messages from Message Oriented Middleware (MoM) such as RabbitMQ or Kafka you have to configure a *Dispatcher*. The *Dispatcher* works with a *Command Processor* to deliver messages read from a queue or stream to your *Request Handler*. You write a Request Handler as you would for a request sent over an Internal Bus, and hook it up to Message Oriented Middleware via a *Dispatcher*. + +For each message source (queue or stream) that you listen to, the Dispatcher lets you run one or more *Performers*. A *Performer* is a single-threaded message pump. As such, ordering is guaranteed on a *Peformer*. You can run multiple *Peformers* to utilize the [Competing Consumers](https://www.enterpriseintegrationpatterns.com/CompetingConsumers.html) pattern, at the cost of ordering. + +If you are using .NET Core Dependency Injection, we provide extension methods to **HostBuilder** to help you configure a Dispatcher. This information is then for background only, but may be useful when debugging. Just follow the steps outlined in [BasicConfiguration](/contents/BrighterBasicConfiguration.md). + +If you are not using **HostBuilder** you will need to perform the following steps explicitly in your code. + +## Configuring the Dispatcher + +We provide a Dispatch Builder that has a progressive interface to assist you in configuring a **Dispatcher** + +You need to consider the following when configuring the Dispatcher + +- Command Processor +- Message Mappers +- Channel Factory +- Connection List + +Configuring the **Command Processor** is covered in [How Configuring the Command Processor Works](/contents/HowConfiguringTheCommandProcessorWorks.md). + +### Message Mappers + +You need to register your [Message Mapper](/contents/MessageMappers.md) so that we can find it. The registry must implement **IAmAMessageMapperRegistry**. We recommend using Brighter's **MessageMapperRegistry** unless you have more specific requirements. + +``` csharp +var messageMapperRegistry = new MessageMapperRegistry(messageMapperFactory) +{ + { typeof(GreetingCommand), typeof(GreetingCommandMessageMapper) } +}; +``` + +### Channel Factory + +The Channel Factory is where we take a dependency on a specific Broker. We pass the **Dispatcher** an instances of **InputChannelFactory** which in turn has a dependency on implementation of **IAmAChannelFactory**. The channel factory is used to create channels that wrap the underlying Message-Oriented Middleware that you are using. + +### Creating a Builder + +This code fragment shows putting the whole thing together + +``` csharp +// create message mappers +var messageMapperRegistry = new MessageMapperRegistry(messageMapperFactory) +{ + { typeof(GreetingCommand), typeof(GreetingCommandMessageMapper) } +}; + +// create the gateway +var rmqMessageConsumerFactory = new RmqMessageConsumerFactory(logger); +_dispatcher = DispatchBuilder.With() + .CommandProcessor(CommandProcessorBuilder.With() + .Handlers(new HandlerConfiguration(subscriberRegistry, handlerFactory)) + .Policies(policyRegistry) + .NoExternalBus() + .RequestContextFactory(new InMemoryRequestContextFactory()) + .Build()) + .MessageMappers(messageMapperRegistry) + .ChannelFactory(new InputChannelFactory(rmqMessageConsumerFactory)) + .Subscribers(subscriptions) + .Build(); +``` + +## Running The Dispatcher + +To ensure that messages reach the handlers from the queue you have to run a **Dispatcher**. + +The Dispatcher reads messages of input channels. Internally it creates a message pump for each channel, and allocates a thread to run that message pump. The pump consumes messages from the channel, using the +**Message Mapper** to translate them into a **Message** and from there a **Command** or **Event**. It then dispatches those to handlers (using the Brighter **Command Processor**). + +To use the Dispatcher you need to host it in a consumer application. Usually a console application or Windows Service is appropriate. + +We recommend using HostBuilder, but if not you will need to use something like [Topshelf](http://topshelf-project.com/) to host your consumers. + +The following code shows an example of using the **Dispatcher** from Topshelf. The key methods are **Dispatcher.Receive()** to start the message pumps and **Dispatcher.End()** to shut them. + +We do allow you to start and stop individual channels, but this is an advanced feature for operating the services. + +``` csharp +internal class GreetingService : ServiceControl +{ + private Dispatcher _dispatcher; + + public GreetingService() + { + /* Configfuration Code Goes here*/ + } + + public bool Start(HostControl hostControl) + { + _dispatcher.Receive(); + return true; + } + + public bool Stop(HostControl hostControl) + { + _dispatcher.End().Wait(); + _dispatcher = null; + return false; + } + + public void Shutdown(HostControl hostcontrol) + { + if (_dispatcher != null) + _dispatcher.End(); + return; + } +} +``` + diff --git a/contents/9/HowServiceActivatorWorks.md b/contents/9/HowServiceActivatorWorks.md new file mode 100644 index 0000000..883d55c --- /dev/null +++ b/contents/9/HowServiceActivatorWorks.md @@ -0,0 +1,3 @@ +# How The Service Activator Works + +Coming Soon... \ No newline at end of file diff --git a/contents/9/ImplementAQueryHandler.md b/contents/9/ImplementAQueryHandler.md new file mode 100644 index 0000000..c858f1e --- /dev/null +++ b/contents/9/ImplementAQueryHandler.md @@ -0,0 +1,3 @@ +# Implementing a Query Handler + +TODO \ No newline at end of file diff --git a/contents/9/ImplementingAHandler.md b/contents/9/ImplementingAHandler.md new file mode 100644 index 0000000..c0cdb16 --- /dev/null +++ b/contents/9/ImplementingAHandler.md @@ -0,0 +1,32 @@ +# How to Implement a Request Handler + +To implement a handler, derive from RequestHandler\ where T should be the **Command** or **Event** derived type that you wish to handle. Then override the base class Handle() method to implement your handling +for the Command or Event. + +For example, assume that you want to handle the **Command** GreetingCommand + +``` csharp +public class GreetingCommand : Command +{ + public GreetingCommand(string name) + : base(Guid.NewGuid()) + { + Name = name; + } + + public string Name { get; private set; } +} +``` + +Then derive your handler from **RequestHandler\** and accept a parameter of that type on the overriden **Handle()** method. + +``` csharp +public class GreetingCommandHandler : RequestHandler +{ + public override GreetingCommand Handle(GreetingCommand command) + { + Console.WriteLine("Hello {0}", command.Name); + return base.Handle(command); + } +} +``` diff --git a/contents/9/ImplementingAsyncHandler.md b/contents/9/ImplementingAsyncHandler.md new file mode 100644 index 0000000..a7378bb --- /dev/null +++ b/contents/9/ImplementingAsyncHandler.md @@ -0,0 +1,52 @@ +# How to Implement an Asynchronous Request Handler + +To implement an asynchronous handler, derive from **RequestHandlerAsync\** where *T* should be the **Command** or **Event** derived type that you wish to handle. Then override the base +class **RequestHandlerAsync\.HandleAsync()** method to implement your handling for the Command or Event. + +For example, assume that you want to handle the **Command** GreetingCommand + +``` csharp +public class GreetingCommand : Command +{ + public GreetingCommand(string name) + : base(Guid.NewGuid()) + { + Name = name; + } + + public Guid Id { get; set; } + public string Name { get; private set; } +} +``` + +Then derive your handler from **RequestHandlerAsync\** and accept a parameter of that type on the overriden **HandleAsync()** method, along with a nullable cancellation token - which you should +default to null. + +To ensure that the pipeline runs, you should return the result of the next handler in the chain, by awaiting the base class **HandleAsync()**. + +(Because the next element in the pipeline should also be async, you should always await the result of this call.) + +``` csharp +public class GreetingCommandRequestHandlerAsync : RequestHandlerAsync +{ + public override async Task HandleAsync(GreetingCommand command, CancellationToken? ct = null) + { + var api = new IpFyApi(new Uri("https://api.ipify.org")); + + var result = await api.GetAsync(ct); + + Console.WriteLine("Hello {0}", command.Name); + Console.WriteLine(result.Success ? "Your public IP addres is {0}" : "Call to IpFy API failed : {0}", + result.Message); + return await base.HandleAsync(command, ct).ConfigureAwait(base.ContinueOnCapturedContext); + } +} +``` + +Note how we use **ConfigureAwait()** when calling the next handler in the chain, and set the value to the **RequestHandlerAsync\.ContinueOnCapturedContext** property. This ensures that we utilize any override of the default (which is to use the Task Scheduler) made when the call to **SendAsync**, **PublishAsync**, or **PostAsync** was made. + +It is worth noting that although the override forces you to return a **Task\** it does not force you to add the **async** keyword to the method to compile. This risks introducing a subtle bug. You can await a +method that returns a **Task\** but creation of the state machine in the caller depends on the presence of the **async** keyword. If your handler does not await anything, you will not be forced to add the +**async** keyword. Your handler will run sychronously in this context, which may not be what you expect. + +Remembering to always await the base class **HandleAsync()** mitigates against this as even if your handler does not do asynchronous work, you will be forced to add **async** to the signature. diff --git a/contents/9/ImplementingExternalBus.md b/contents/9/ImplementingExternalBus.md new file mode 100644 index 0000000..4790968 --- /dev/null +++ b/contents/9/ImplementingExternalBus.md @@ -0,0 +1,106 @@ +# Using an External Bus + +Brighter provides support for an External Bus. Instead of handling a command or event, synchronously and in-process, (an Internal Event Bus) work can be dispatched to a distributed queue to be handled +asynchronously and out-of-process. The trade-off here is between the cost of distribution (see [The Fallacies of Distributed Computing](https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing). + +An External Bus allows you offload work to another process, to be handled asynchronously (once you push the work onto the queue, you don\'t wait) and in parallel (you can use other cores to process the +work). It also allows you to ensure delivery of the message, eventually (the queue will hold the work until a consumer is available to read it). + +As part of the [Microservices](https://martinfowler.com/articles/microservices.html) architectural style an External Bus let's you implement an [Event Driven Architecture](/contents/EventDrivenCollaboration.md). + +In addition use of an External Bus allows you to throttle requests - you can hand work off from the web server to a queue that only needs to consume at the rate you have resources to support. This +allows you to scale to meet unexpected demand, at the price of [eventual consistency.](https://en.wikipedia.org/wiki/Eventual_consistency). See the [Task Queue Pattern](/contents/TaskQueuePattern.md) + +## Brighter\'s External Bus Architecture + +Brighter implements an External Bus using a [Message Broker](http://www.enterpriseintegrationpatterns.com/MessageBroker.html). The software that provides the Message Broker is referred to as Message-Oriented Middleware (MoM). Brighter calls its abstraction over MoM a *Transport*. + +The producer sends a **Command** or **Event** to a [Message Broker](http://www.enterpriseintegrationpatterns.com/MessageBroker.html) using **CommandProcessor.Post()** or **CommandProcessor.DepositPost** and **CommandProcessor.ClearOutBox** (or their \*Async equivalents). + +We use an **IAmAMessageMapper** to map the **Command** or **Event** to a **Message**. (Usually we just serialize the object to JSON and add to the **MessageBody**), but if you want to use higher performance +serialization approaches, such as [protobuf-net](https://github.com/mgravell/protobuf-net), the message mapper is agnostic to the way the body is formatted.) + +When we deserialize we set the **MessageHeader** which includes a topic (often we use a namespaced name for the **Command** or **Event**). + +We store the created **Message** in a [Outbox](https://microservices.io/patterns/data/transactional-outbox.html) for use by **CommandProcessor.ClearOutbox()** if we need to resend a failed message. + +The Message Broker manages a [Recipient List](http://www.enterpriseintegrationpatterns.com/RecipientList.html) of subscribers to a topic. When it receives a **Message** the Broker looks at the topic (in Brighter terms the *Routing Key*) in the **MessageHeader** and dispatches the **Message** to the [Recipient Channels](http://www.enterpriseintegrationpatterns.com/MessageChannel.html) identified by the Recipient List. + +The consumer registers a [Recipient Channel](http://www.enterpriseintegrationpatterns.com/MessageChannel.html) to receive messages on a given topic. In other words when the consumer\'s registered topic matches the producer\'s topic, the broker dispatches the message to the consumer when it receives it from the producer. + +A **Message** may be delivered to multiple Consumers, all of whom get their own copy. + +in addition, we can support a [Competing Consumers](http://www.enterpriseintegrationpatterns.com/CompetingConsumers.html) approach by having multiple consumers read from the same [Channel](http://www.enterpriseintegrationpatterns.com/MessageChannel.html) to allow us to scale out to meet load. + +![TaskQueues](_static/images/TaskQueues.png) + + +## Sending via the External Bus + +Instead of using **CommandProcessor.Send()** you use **CommandProcessor.Post()** or **CommandProcessor.DepositPost** and **CommandProcessor.ClearOutbox** to send the message + +``` csharp +var reminderCommand = new TaskReminderCommand( + taskName: reminder.TaskName, + dueDate: DateTime.Parse(reminder.DueDate), + recipient: reminder.Recipient, + copyTo: reminder.CopyTo); + + _commandProcessor.Post(reminderCommand); +``` + +You add a message mapper to tell Brighter how to serialize the message for sending to your consumers. + +``` csharp +public class TaskReminderCommandMessageMapper : IAmAMessageMapper +{ + public Message MapToMessage(TaskReminderCommand request) + { + var header = new MessageHeader(messageId: request.Id, topic: "Task.Reminder", messageType: MessageType.MT_COMMAND); + var body = new MessageBody(JsonConvert.SerializeObject(request)); + var message = new Message(header, body); + return message; + } + + public TaskReminderCommand MapToRequest(Message message) + { + return JsonConvert.DeserializeObject(message.Body.Value); + } +} +``` +## Receiving via the External Bus + +A consumer reads the **Message** using the [Service Activator](http://www.enterpriseintegrationpatterns.com/MessagingAdapter.html) pattern to map between an [Event Driven Consumer](http://www.enterpriseintegrationpatterns.com/EventDrivenConsumer.html) and a Handler. + +The use of the Service Activator pattern means the complexity of the distributed task queue is hidden from you. You just write a handler as normal, but call it via post and create a message mapper, the result is +that your command is handled reliably, asynchronously, and in parallel with little cognitive overhead. It just works! + +``` csharp +public class MailTaskReminderHandler : RequestHandler +{ + private readonly IAmAMailGateway _mailGateway; + + public MailTaskReminderHandler(IAmAMailGateway mailGateway, IAmACommandProcessor commandProcessor) + : this(mailGateway, commandProcessor, LogProvider.GetCurrentClassLogger()) + {} + + public MailTaskReminderHandler(IAmAMailGateway mailGateway, ILog logger) : base(logger) + { + _mailGateway = mailGateway; + } + + [RequestLogging(step: 1, timing: HandlerTiming.Before)] + [UsePolicy(new [] {CommandProcessor.CIRCUITBREAKER, CommandProcessor.RETRYPOLICY}, step: 2)] + public override TaskReminderCommand Handle(TaskReminderCommand command) + { + _mailGateway.Send(new TaskReminder( + taskName: new TaskName(command.TaskName), + dueDate: command.DueDate, + reminderTo: new EmailAddress(command.Recipient), + copyReminderTo: new EmailAddress(command.CopyTo) + )); + + return base.Handle(command); + } +} +``` \ No newline at end of file diff --git a/contents/9/KafkaConfiguration.md b/contents/9/KafkaConfiguration.md new file mode 100644 index 0000000..3626f78 --- /dev/null +++ b/contents/9/KafkaConfiguration.md @@ -0,0 +1,317 @@ +# Kafka Configuration + +## General + +Kafka is OSS message-oriented-middleware and is [well documented](https://kafka.apache.org/documentation/#gettingStarted). Brighter handles the details of sending to or receiving from Kafka. You may find it useful to understand the [building blocks](https://kafka.apache.org/documentation/#introduction) of the protocol. Brighter's Kafka support is implemented on top of the Confluent .NET client, and you might find the [documentation for the .NET client](https://docs.confluent.io/kafka-clients/dotnet/current/overview.html) helpful when debugging, but you should not have to interact with it directly to use Brighter (although we expose many of its configuration options). + +Kafka has two main roles: + +- **Producer**: A producer sends events to a **Topic** on a Kafka broker. +- **Consumer**: A consumer reads events from a **Topic** on a Kafka broker. + +**Topics** are append-only streams of events. Multiple producers can write to a topic, and multiple consumers can read from one. A **consumer** uses an **offset** into the stream to indicate the event it wants to read. Kafka does not delete an event from the stream when it is ack'd by the consumer; instead a **consumer** increments its **offset** once an item has been read so that it can avoid processing the same event twice. See [Offset Management](#offset-management) for more on how Brighter manages **consumer offsets**. As a result the lifetime of events on a stream is instead a configuration setting for the stream. + +As a **consumer** manages an **offset** to record events that is has read, you cannot scale an application that wishes to consume a **topic** by increasing the number of **consumers**--they don't share an offset--without partitioning the **topic**. If you supply a **partition key**, a **partition** uses consistent hashing to slice a **topic** into a number of streams; otherwise it will use round-robin. See [this documentation](https://jaceklaskowski.gitbooks.io/apache-kafka/content/kafka-producer-internals-DefaultPartitioner.html) for more. Each **partition** is only read by a single **consumer** within the application. All of the consumers for an application should share the same group id, called a **consumer group** in Kafka. As each **consumer** tracks the **offset** for the **partitions** it is reading, it is possible to have multiple **consumers** read and process the same **topic**. + +A **consumer** may read from *multiple* **partitions**, but only one **consumer** may read from a **partition** at one time in a given **consumer group**. Kafka will assign partitions across the pool of consumers for the **consumer group**. When the pool changes, a **rebalance** occurs, which may mean that a consumer changes the **partition** that it is assigned within the **consumer group**. Brighter favors *sticky assignment of partitions* to avoid unnecessary churn of partitions. + +In addition to the Producer API and Consumer API Kafka streams have features such as the Streams API and the Connect API. We do not use either of these from Brighter. + +## Connection + +The Connection to Kafka is provided by an **KafkaMessagingGatewayConnection** which allows you to configure the following: + +- **BootstrapServers**: A **bootstrap** server is a well-known broker through which we discover the servers in the Kafka cluster that we can connect to. You should supply a comma-separated list of host and port pairs. These are the addresses of the Kafka brokers in the "bootstrap" Kafka cluster. +- **Debug**: A comma-separated list of debug contexts to enable. Producer: broker, topic, msg. Consumer: consumer, cgrp, topic, fetch. +- **Name**: An identifier to use for the client. +- **SaslMechanisms**: If any, what is the protocol used for authenticated connection to the Kafka broker: plain, scram-sha-256, scram-sha-256, gssapi (kerberos), oauthbearer +- **SaslKerberosName**: If using kerberos, what is the connection name. +- **SaslUsername**: SASL username for use with PLAIN and SASL-SCRAM +- **SaslPassword**: SASL password for use with PLAIN and SASL-SCRAM +- **SecurityProtocol**: How are messages between client and server encrypted, if at all: plaintext, ssl, saslplaintext, saslssl +- **SslCaLocation**: Where is the CA certificate located (see [here](https://docs.confluent.io/platform/current/tutorials/examples/clients/docs/csharp.html) for guidance). +- **SslKeystoreLocation**: Path to the client's keystore +- **SslKeystorePassword**: Password for the client's keystore + +The following code connects to a local Kafka instance (for development): + +``` csharp + services.AddBrighter(...) + .UseExternalBus( + new KafkaProducerRegistryFactory( + new KafkaMessagingGatewayConfiguration() + { + Name = "paramore.brighter.greetingsender", + BootStrapServers = new[] {"localhost:9092"} + }, + ...//publication, see below + ) + .Create()) + ... + +``` + +The following code connects to a remote Kafka instance. The settings here will depend on how your production broker is configured for access. We show getting secrets from environment variables for simplicity, again you will need to adjust this for your approach to secrets management: + +``` csharp + services.AddBrighter(...) + .UseExternalBus( + new KafkaProducerRegistryFactory( + new KafkaMessagingGatewayConfiguration() + { + Name = "paramore.brighter.greetingsender", + BootStrapServers = new[] { Environment.GetEnvironmentVariable("BOOSTRAP_SERVER")}, + SecurityProtocol = Paramore.Brighter.MessagingGateway.Kafka.SecurityProtocol.SaslSsl, + SaslMechanisms = Paramore.Brighter.MessagingGateway.Kafka.SaslMechanism.Plain, + SaslUsername = Environment.GetEnvironmentVariable("SASL_USERNAME"), + SaslPassword = Environment.GetEnvironmentVariable("SASL_PASSWORD"), + SslCaLocation = RuntimeInformation.IsOSPlatform(OSPlatform.OSX) ? "/usr/local/etc/openssl@1.1/cert.pem" : null; + }, + ...//publication, see below + ) + .Create()) + ... + +``` + +## Publication + +For more on a *Publication* see the material on an *External Bus* in [Basic Configuration](/contents/BrighterBasicConfiguration.md#using-an-external-bus). + +We allow you to configure properties for both Brighter and the Confluent .NET client. Because there are many properties on the Confluent .NET Client we also configure a callback to let you inspect and modify the configuration that we will pass to the client if you so desire. This can be used to add properties we do not support or adjust how we set them. + +- **Replication**: how many ISR nodes must receive the record before the producer can consider the write successful. Default is Acks.All. +- **BatchNumberMessages**: Maximum number of messages batched in one MessageSet. Default is 10. +- **EnableIdempotence**: Messages are produced once only. Will adjust the following if not set: `max.in.flight.requests.per.connection=5` (must be less than or equal to 5), `retries=INT32_MAX` (must be greater than 0), `acks=all`, `queuing.strategy=fifo`. Default is true. +- **LingerMs**: Maximum time, in milliseconds, for buffering data on the producer queue. Default is 5. +- **MessageSendMaxRetries**: How many times to retry sending a failing MessageSet. Note: retrying may cause reordering, set the max in flight to 1 if you need ordering by when sent. Default is 3. +- **MessageTimeoutMs**: Local message timeout. This value is only enforced locally and limits the time a produced message waits for successful delivery. A time of 0 is infinite. Default is 5000. +- **MaxInFlightRequestsPerConnection**: Maximum number of in-flight requests the client will send. We default this to 1, so as to allow retries to not de-order the stream. +- **NumPartitions**: How many partitions for this topic. We default to 1. +- **Partitioner**: How do we partition? Defaults to Partitioner.ConsistentRandom. +- **QueueBufferingMaxMessages**: Maximum number of messages allowed on the producer queue. Defaults to 10. +- **QueueBufferingMaxKbytes**: Maximum total message size sum allowed on the producer queue. Defaults to 1048576 bytes (so for 10 messages about 104Kb per message). +- **ReplicationFactor**: What is the replication factor? How many nodes is the topic copied to on the broker? Defaults to 1. +- **RetryBackoff**: The backoff time before retrying a message send. Defaults to 100. +- **RequestTimeoutMs**: The ack timeout of the producer request. This value is only enforced by the broker and relies on Replication being != AcksEnum.None. Defaults to 500. +- **TopicFindTimeoutMs**: How long to wait when asking for topic metadata. Defaults to 5000. +- **TransactionalId**: The unique identifier for this producer, used with transactions + +The following example shows how a *Publication* might be configured: + +``` csharp + services.AddBrighter(...) + .UseExternalBus( + new KafkaProducerRegistryFactory( + ...,//connection see above + new KafkaPublication[] {new KafkaPublication() + { + Topic = new RoutingKey("MyTopicName"), + NumPartitions = 3, + ReplicationFactor = 3, + MessageTimeoutMs = 1000, + RequestTimeoutMs = 1000, + MakeChannels = OnMissingChannel.Create + } + ) + .Create()) + ... + +``` + +### Configuration Callback + +The Confluent .NET client has a range of configuration options. Some of those can be controlled through the publication. But, to allow you the full range of configuration options for the Confluent client, including new options that may appear, we provide a callback on the **KafkaProducerRegistryFactory**. The registry exposes a method, **SetConfigHook(Action hook)**. The method takes a *delegate* (you can pass a lambda). Your delegate will be called with the *proposed* ProducerConfig (taking into account the *Publication* settings). You can adjust additional parameters at this point. + +You can use it as follows: + +``` csharp + + var publication = new KafkaPublication() + { + Topic = new RoutingKey("MyTopicName"), + NumPartitions = 3, + ReplicationFactor = 3, + MessageTimeoutMs = 1000, + RequestTimeoutMs = 1000, + MakeChannels = OnMissingChannel.Create + }; + publication.SetConfigHook(config => config.EnableGaplessGuarantee = true) + + services.AddBrighter(...) + .UseExternalBus( + new KafkaProducerRegistryFactory( + ...,//connection see above + new KafkaPublication[] {publication}) + .Create()) + ... + +``` + +### Kafka Topic Auto Create + +Brighter uses the Kafka AdminClient for topic creation. For this to work as expected you should set the server property of **auto.create.topics.enable** to **false**; otherwise the topic will be auto-created with the values defined by your server for new topics, such as the number of partitions. This error can be insidious because your code will still work against this topic, but without inspection you will not observe that its properties do not match those requested. + +If you want to specify the topic through Brighter, or through your own IaaS code, we recommend always setting this setting to false; we recommend only setting it to true if you tell Brighter to assume that the infrastructure exists, as it will then be created on the first write. + +## Subscription + +For more on a *Subscription* see the material on configuring *Service Activator* in [Basic Configuration](/contents/BrighterBasicConfiguration.md#configuring-the-service-activator). + +We support a number of Kafka specific *Subscription* options: + +- **CommitBatchSize**: We commit processed work (marked as acked or rejected) when a batch size worth of work has been completed (see [below](#offset-management)). +- **GroupId**: Only one consumer in a group can read from a partition at any one time; this preserves ordering. We do not default this value, and expect you to set it. +- **IsolationLevel**: Default to read only committed messages, change if you want to read uncommitted messages. May cause duplicates. +- **MaxPollIntervalMs**: How often the consumer needs to poll for new messages to be considered alive, polling greater than this interval triggers a re-balance. Kafka default to 300000ms +- **NumPartitions**: How many partitions does the topic have? Used for topic creation, if required. +- **OffsetDefault**: What do we do if there is no offset stored in ZooKeeper for this consumer. Defaults to AutoOffsetReset.Earliest - Begin reading the stream from the start. Options include AutOffsetRest.Latest - Start from now i.e. only consume messages after we start and AutoOffsetReset.Error - which considers it an error if not reset is found +- **ReadCommittedOffsetsTimeOutMs**: How long before attempting to read back committed offsets (mainly used in debugging) is an error. Defaults to 5000. +- **ReplicationFactor**: What is the replication factor? How many nodes is the topic copied to on the broker? Defaults to 1. Used for topic creation if required. +- **SessionTimeoutMs**: If Kafka does not receive a heartbeat from the consumer within this time window, trigger a re-balance. Default is Kafka default of 10s. +- **SweepUncommittedOffsetsIntervalMs**: The interval at which we sweep, looking for offsets that have not been flushed (see [below](#offset-management)). + +The following example shows how a subscription might be configured: + +``` csharp + private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + var subscriptions = new KafkaSubscription[] + { + new KafkaSubscription( + new SubscriptionName("paramore.example.greeting"), + channelName: new ChannelName("greeting.event"), + routingKey: new RoutingKey("greeting.event"), + groupId: Environment.GetEnvironmentVariable("KAFKA_GROUPID"), + timeoutInMilliseconds: 100, + commitBatchSize: 5, + sweepUncommittedOffsetsIntervalMs: 3000 + ) + }; + + //create the gateway + var consumerFactory = new KafkaMessageConsumerFactory( + new KafkaMessagingGatewayConfiguration {...} // see connection information above + ); + + services.AddServiceActivator(options => + { + options.Subscriptions = subscriptions; + options.ChannelFactory = new ChannelFactory(consumerFactory); + }).AutoFromAssemblies(); + + + services.AddHostedService(); +} +``` + +## Offset Management + +It is important to understand how Brighter manages the **offset** of any **partitions** assigned to your **consumer**. + +- Brighter manages committing **offsets** to Kafka. This means we set the Confluent client's *auto store* and *auto commit* properties to *false*. +- The **CommitBatchSize** setting on the *Subscription* determines the size of your buffer. A smaller buffer is less efficient, but if your consumer crashes any **offsets** pending commit in the buffer will be lost, and you will be represented with those records when you next read from the **partition**. We default this value to 10. +- We do not add an offset commit to the buffer until you Ack the request. The message pump will Ack for you once you exit your handler (via return or [throwing an exception](/contents/HandlerFailure.md)). +- Flushing the commit buffer happens on a separate thread. We only run one flush at a time, and we flush a **CommitBatchSize** number of items from the buffer. + - A busy consumer may not flush on every increment of the **CommitBatchSize**, as it may need to wait for the last flush to finish. + - We won't flush again until we cross the next multiple of the **CommitBatchSize**. For example if the **CommitBatchSize** is 10, and the handler is busy so that by the time the buffer flushes there are 13 pending commits in the buffer, the buffer would only flush 10, and 3 would remain in the buffer; we would not flush the next 10 until the buffer hit 20. + - If your **CommitBatchSize** is too low for the throughput, you might find that you miss a flush interval, because you are already flushing. + - If you miss a flush on a busy consumer, your buffer will begin to back up. If this continues, you will not catch up with subsequent flushes, which only flush the **CommitBatchSize** each time. This would lead to you continually being "backed up". + - For this reason you must set a **CommitBatchSize** that keeps pace with the throughput of your consumer. Use a larger **CommitBatchSize** for higher throughput consumers, smaller for lower. +- We sweep uncommitted offsets at an interval. This triggers a flush if no flush has run since the last flush plus the *Subscription's* **SweepUncommittedOffsetsIntervalMs**. + - A sweep will not run if a flush is currently running (and will in turn block a flush). + - A sweep flushes a **CommitBatchSize** worth of commits. + - It is intended for low-throughput consumers where commits might otherwise languish waiting for a batch-size increment. + - It is *not* intended to flush a buffer that backs up because the **CommitBatchSize** is too low, and won't function for that. Fix the **CommitBatchSize** instead. +- On a re-balance where we stop processing a **partition** on an individual consumer, we flush the remaining **offsets** for the revoked **partitions**. + - We configure the consumer to use sticky assignment strategy to avoid unnecessary re-assignments (see the [Confluent documentation](https://www.confluent.io/blog/cooperative-rebalancing-in-kafka-streams-consumer-ksqldb/)). +- On a consumer shutdown we flush the buffer to commit all **offsets**. + + +## Working with Schema Registry + +If you want to use tools within the Kafka ecosystem such as Kafka Connect or KQSL you will almost certainly need to use Confluent Schema Registry to provide the schema of your message. + +You will need to pull in the following package: + +* Confluent.SchemaRegistry + +and a package for the serialization of your choice. Here we are using JSON, so we use + +* Confluent.SchemaRegistry.Serdes.Json + +When working with Brighter, to use Confluent Schema Registry you will need to take a dependency on ISchemaRegistry in the constructor of your message mapper. To fulfill this constructor, in your application setup you will need to register an instance of schema registry. You should configure the schema registry config url to be the url of you schema registry. (Here we just use localhost for a development instance running in docker as an example). + +``` csharp +var schemaRegistryConfig = new SchemaRegistryConfig { Url = "http://localhost:8081"}; +var cachedSchemaRegistryClient = new CachedSchemaRegistryClient(schemaRegistryConfig); +services.AddSingleton(cachedSchemaRegistryClient); +``` + +Once you can satisfy the dependency, you will want to use the serializer from the Serdes package to serialize the body of your message, instead of System.Text.Json. Note that 'under-the-hood' the Serdes serializer uses [Json.NET](https://www.newtonsoft.com/json) and [NJsonSchema](https://github.com/RicoSuter/NJsonSchema), so you may need to mark up your code with attributes from these packages to create the schema you want and serialize a valid message to it. (Note that, at this time, the Serdes package does not support System.Text.Json so you will need to take a dependency on Json.NET if you want to use the schema registry). + +It is worth noting the following aspects of the code sample below: + +* We need to set up a SerializationContext and tell Serdes that we are serializing the message body using their serializer +* We provide two helpers, though you can pass your own settings if you prefer: + * **ConfluentJsonSerializationConfig.SerdesJsonSerializerConfig()** offers default settings for JSON serialization (many of these are passed through to Json.NET). + * **ConfluentJsonSerializationConfig.NJsonSchemaGeneratorSettings()** offers default settings for JSON Schema generation (such as using camelCase). + + +``` csharp +public class GreetingEventMessageMapper : IAmAMessageMapper +{ +private readonly ISchemaRegistryClient _schemaRegistryClient; +private readonly string _partitionKey = "KafkaTestQueueExample_Partition_One"; +private SerializationContext _serializationContext; +private const string Topic = "greeting.event"; + +public GreetingEventMessageMapper(ISchemaRegistryClient schemaRegistryClient) +{ + _schemaRegistryClient = schemaRegistryClient; + //We care about ensuring that we serialize the body using the Confluent tooling, as it registers and validates schema + _serializationContext = new SerializationContext(MessageComponentType.Value, Topic); +} + +public Message MapToMessage(GreetingEvent request) +{ + var header = new MessageHeader(messageId: request.Id, topic: Topic, messageType: MessageType.MT_EVENT); + //This uses the Confluent JSON serializer, which wraps Newtonsoft but also performs schema registration and validation + var serializer = new JsonSerializer(_schemaRegistryClient, ConfluentJsonSerializationConfig.SerdesJsonSerializerConfig(), ConfluentJsonSerializationConfig.NJsonSchemaGeneratorSettings()).AsSyncOverAsync(); + var s = serializer.Serialize(request, _serializationContext); + var body = new MessageBody(s, "JSON"); + header.PartitionKey = _partitionKey; + + var message = new Message(header, body); + return message; +} + +public GreetingEvent MapToRequest(Message message) +{ + var deserializer = new JsonDeserializer().AsSyncOverAsync(); + //This uses the Confluent JSON serializer, which wraps Newtonsoft but also performs schema registration and validation + var greetingCommand = deserializer.Deserialize(message.Body.Bytes, message.Body.Bytes is null, _serializationContext); + + return greetingCommand; +} +} + +``` + + +## Requeue with Delay + +We don't currently support requeue with delay for Kafka. It might be added in a future release, where the strategy would be to: + +- Publish the requeued message to a new stream +- Commit the offset +- Poll that stream with a new subscription but at a greater interval between polling (i.e. the delay) + +In the interim you can manually implement that approach if required. + + + + + + + diff --git a/contents/9/Logging.md b/contents/9/Logging.md new file mode 100644 index 0000000..522b18e --- /dev/null +++ b/contents/9/Logging.md @@ -0,0 +1,3 @@ +# Logging + +TODO \ No newline at end of file diff --git a/contents/9/MSSQLInbox.md b/contents/9/MSSQLInbox.md new file mode 100644 index 0000000..0b50aed --- /dev/null +++ b/contents/9/MSSQLInbox.md @@ -0,0 +1,35 @@ +# MSSQL Inbox + +## Usage +The MSSQL Inbox allows use of MSSQL for [Brighter's inbox support](/contents/BrighterInboxSupport.md). The configuration is described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#inbox). + +For this we will need the *Inbox* packages for the MsSQL *Inbox*. + +* **Paramore.Brighter.Inbox.MsSql** + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + services.AddServiceActivator(options => + { ... }) + .UseExternalInbox( + new MsSqlInbox(new MsSqlInboxConfiguration("server=localhost; port=3306; uid=root; pwd=root; database=Salutations", "Inbox"); + new InboxConfiguration( + scope: InboxScope.Commands, + onceOnly: true, + actionOnExists: OnceOnlyAction.Throw + ) + ); +} + +... + +``` + diff --git a/contents/9/MessageMappers.md b/contents/9/MessageMappers.md new file mode 100644 index 0000000..ddc65d6 --- /dev/null +++ b/contents/9/MessageMappers.md @@ -0,0 +1,264 @@ +# Message Mappers + +A message mapper turns domain code into a Brighter **Message**. A Brighter **Message** has a **MessageHeader** for information about the message. Key properties are: **TimeStamp**, **Topic**, and **Id**. The **Message** also has a **MessageBody**, which contains the payload. + +The messageType parameter tells the Dispatcher that listens to this message, how to treat it, as a Command or an Event. Brighter's *Dispatcher* dispatches a **Message** using either **commandProcessor.Send()** for **MT_COMMAND** or **commandProcessor.Publish()** for **MT_EVENT**. + +Typically, you serialize your request as the **MessageBody** for in **MapToMessage** and serialize your **MessageBody** into a request in **MapToRequest**. + +The body is a byte[] and as such we can support any format that can be converted into a byte[] as the message body. + +Because [message oriented middleware](#message-oriented-middleware-mom) typically looks in a header for routing information, you add your routing information in the **MessageHeader**. + +Each individual transport has code to turn a Brighter format message into a message oriented middleware compatible message, and vice versa, so your code only needs to translate to and from the Brighter format. + +## Writing A Message Mapper + +We use **IAmAMessageMapper\** to map between messages in the External Bus and a **Message**. + +You create a **Message Mapper** by deriving from **IAmAMessageMapper\** and implementing the **MapToMessage()** and **MapToRequest** methods. + +An example follows: + +``` csharp +public class GreetingMadeMessageMapper : IAmAMessageMapper +{ + public Message MapToMessage(GreetingMade request) + { + var header = new MessageHeader(messageId: request.Id, topic: "GreetingMade", messageType: MessageType.MT_EVENT); + var payload = System.Text.Json.JsonSerializer.Serialize(request, new JsonSerializerOptions(JsonSerializerDefaults.General)); + var body = new MessageBody(payload, ApplicationJson, CharacterEncoding.UTF8); + var message = new Message(header, body); + return message; + } + + public GreetingMade MapToRequest(Message message) + { + return JsonSerializer.Deserialize(message.Body.Value, JsonSerialisationOptions.Options); + } +} +``` + +## Brighter Message Structure + +Brighter divides a message into two parts: + +* Header: The header contains metadata (data about the message). It is typically used to control how we process the payload or provide additional context about it. +* Body: The body contains the payload, which is usually the **Command** or **Event** being raised for the consumer to action + +### The Message Header + +The Message Header has a number of Brighter defined properties and a bag that can be used for user-defined properties. + +#### Common Properties + +* **Id**(GUID): The identifier for this message +* **Topic**(string): The topic this message should be sent to, used to route the message in most transports +* **MessageType** (enum): The type of message: (Unacceptable (not translated), None (null object), Command, Event, Document, Quit (terminats a pump)) +* **CorrelationId** (GUID): Is this message a response to another message (usually an event reply to a command), if so this is the id that links them +* **ReplyTo** (string): A topic to reply to. In a request-reply set this to tell the receiver where to send replies +* **ContentType** (string): Normally, allow the **MessageBody** (below) to set this. +* **PartitionKey** (string): Where consistent hashing is used to partition a stream, what is the value to partition on + +#### Brighter Properties + +* **DelayedMilliseconds** (int): If we chose to retry with a delay, how long for? +* **HandledCount** (int): How many times have we tried to handle this message +* **Telemetry** (MessageTelemetry): Open Telemetry information for the message + +#### Routing + +In **MapToMessage**, the **topic** parameter on the **MessageHeader** controls the topic (or routing key) which we use when publishing a message to the external bus. We use this value when using the SDK for the message oriented middleware transport to publish a message on that transport. + +For this reason it is the **MessageMapper** that controls how messages published to the external bus are routed. + + +### The Message Body + +The Message Body stores the content for transmission over a transport as a byte[]. This supports both plain text and binary payloads. Your choice of payload type is constrained by what the transport requires or supports. + +In many cases the easiest option is to send the payload as plain text, as this is the easiest to inspect if you need to debug your messages. In this case the simplest path is to serialize the **Command** or **Event** as JSON and deserialize from that JSON. MessageBody contains a constructor that takes a string with two optional parameters, a media type (which defaults to **application/json**) and a character encoding type for the string (which defaults to **CharacterEncoding.UTF8**), + +```csharp +public MessageBody(string body, string contentType = ApplicationJson, CharacterEncoding characterEncoding = CharacterEncoding.UTF8) +{ + ... +``` + +which can be used as follows (or omitting the default parameters) + +```csharp + +var payload = System.Text.Json.JsonSerializer.Serialize(request, new JsonSerializerOptions(JsonSerializerDefaults.General)); +var body = new MessageBody(payload, ApplicationJson, CharacterEncoding.UTF8); + +``` + +If your payload is binary, then we provide two constructors that can be used to write bytes. For backwards compatibility these constructors also default to application/json and UTF-8. However, if you have binary content we recommend setting the media type to application/octet-stream and the character encoding to either **CharacterEncoding.Base64** if it needs transmission as a string, or **CharacterEncoding.Raw** if not). + +```csharp + +public MessageBody(byte[] bytes, string contentType = ApplicationJson, CharacterEncoding characterEncoding = CharacterEncoding.UTF8) +{ + ... + +public MessageBody(in ReadOnlyMemory body, string contentType = ApplicationJson, CharacterEncoding characterEncoding = CharacterEncoding.UTF8) +{ + ... + +``` + +For example, when writing a Kafka payload with leading bytes indicating the schema id, you would want to use a binary payload because conversion to and from a UTF8 string is lossy. Here we serialize the payload with the Kafka header (Magic Byte (0) + Schema Id Bytes) and a JSON payload using the Confluent Serdes serializer. Even though we serialize to JSON, because of the header bytes we treat the payload as binary: + +```csharp + +public Message MapToMessage(GreetingEvent request) +{ + var header = new MessageHeader(messageId: request.Id, topic: Topic, messageType: MessageType.MT_EVENT); + //This uses the Confluent JSON serializer, which wraps Newtonsoft but also performs schema registration and validation + var serializer = new JsonSerializer(_schemaRegistryClient, ConfluentJsonSerializationConfig.SerdesJsonSerializerConfig(), ConfluentJsonSerializationConfig.NJsonSchemaGeneratorSettings()).AsSyncOverAsync(); + var s = serializer.Serialize(request, _serializationContext); + var body = new MessageBody(s, MediaTypeNames.Application.Octet, CharacterEncoding.Raw); + header.PartitionKey = _partitionKey; + + var message = new Message(header, body); + return message; +} + +``` + +The **Value** property of the **MessageBody** returns a string depending on the character encoding type of the body. If you do not set a character encoding then we assume a standard UTF8 **string**; if you set the character encoding to base64 or raw, we return a base64 string; if you set the character encoding to ascii we will return an ascii string. + + +### Options for System.Text.Json Serialization + +The most common solution to serialization of the message payload is to use System.Text.Json to convert the message's metadata to JSON for sending over a messaging middleware transport. You can adjust the behavior of this serialization through our **JsonSerialisationOptions**. See [Brighter Configuration](/contents/BrighterBasicConfiguration.md#configuring-json-serialization) for more on how to set your options. + +You can then use this, when you want to set options consistently for message serialization. + +``` csharp + public GreetingMade MapToRequest(Message message) + { + return JsonSerializer.Deserialize(message.Body.Value, JsonSerialisationOptions.Options); + } +``` + +## Transformers + +Some concerns are orthogonal to how you map a **IRequest** into a **Message** or how you map a **Message** into an **IRequest**. Instead they concern how we process that Message. A typical list of such concerns might include: handling large message payloads (compression of moving to a distributed file store), encryption, registering or validating schema, and adding common metadata to headers. + +A *Transform* is a middleware that runs as part of the pipeline we use to map a **IRequest** into a **Message** or how you map a **Message** into an **IRequest**. A transform implements an **IMessageTransformAsync**. (All transforms are async). + +``` csharp +public interface IAmAMessageTransformAsync : IDisposable +{ + void InitializeWrapFromAttributeParams(params object[] initializerList); + void InitializeUnwrapFromAttributeParams(params object[] initializerList); + Task WrapAsync(Message message, CancellationToken cancellationToken); + Task UnwrapAsync(Message message, CancellationToken cancellationToken); +} +``` + +### Wrap + +When we *wrap* the source is the *Message Mapper* and the transform is applied to the **Message** that you generate from the **IRequest** in your **MapToMessage**. + +You indicate that you wish to *wrap* a *Message Mapper* with the **WrapWithAttribute** associated with the **IMessageTransformAsync** you want to apply to the **Message** you have created from the **IRequest**. In the example below we use a **ClaimCheck** to move large message payloads (those over the threshold) into a *luggage store* (for example an S3 bucket). + +``` csharp +[ClaimCheck(step:0, thresholdInKb: 256)] +public Message MapToMessage(GreetingEvent request) +{ + var header = new MessageHeader(messageId: request.Id, topic: typeof(GreetingEvent).FullName.ToValidSNSTopicName(), messageType: MessageType.MT_EVENT); + var body = new MessageBody(JsonSerializer.Serialize(request, JsonSerialisationOptions.Options)); + var message = new Message(header, body); + return message; +} +``` + +### Unwrap + +When we *unwrap* the sink is the *Message Mapper* and the transform is applied to the **Message** before you turn it into an **IRequest** in your **MapToRequest**. + +You indicate that you wish to *unwrap* a *Message Mapper* with the **UnwrapWithAttribute** associated with the **IMessageTransformAsync** you want to apply to the **Message** before you create your **IRequest**. In the example below we use a **RetrieveClaim** to retrieve a large message payload (most likely stored by a Claim Check in a *luggage store*) that will provide the body of our **Message** before we deserialize it to the **IRequest**. + +``` csharp +[RetrieveClaim(step:0)] +public GreetingEvent MapToRequest(Message message) +{ + var greetingCommand = JsonSerializer.Deserialize(message.Body.Value, JsonSerialisationOptions.Options); + + return greetingCommand; +} + +``` + +### Transform, Wrap and Unwrap + +Usually your **WrapWithAttribute** and **UnwrapWithAttribute** are paired and opposite. Usually they associate with a common **IMessageTransformAsync** that implements support for both transforms: the **WrapWithAttribute** results in the **WrapAsync** method of the transform being called (the **Message** is passed to it); the **UnwrapWithAttribute** results in the **UnwrapAsync** method being called (again the **Message** is passed to it). + +Both the **WrapWithAttribute** and the **UnwrapWithAttribute** are a type of **TransformAttribute** + +``` csharp + +public abstract class TransformAttribute : Attribute + { + public int Step { get; set; } + public abstract Type GetHandlerType(); + public virtual object[] InitializerParams() + { + return new object[0]; + } + +``` + +To implement a **TransformAttribute** you need to create a derived type that overrides the **GetHandlerType** to return the type of your **IMessageTransformAsync**. + +#### Step + +Step specifies the order in which a transform runs (attributes are not guaranteed to be made available in top-down order by reflection). This can be important in transforms. Imagine that you want to compress any message over 256Kb, but because a large enough message might still not be small enough after compression, a message that is *still* over 256Kb to distributed storage. In this case you would want to make sure that the step value for compression was lower than the step value to offload to distributed storage. + +#### Passing Parameters to a Transform + +If you want to pass parameters to your transform, they must be available at compile time as arguments to your derived **TransformAttribute**. The parameters of your attribute's constructor can be set from an attribute. Your attribute can then store these parameters in private fields. We call your derived attributes **InitializeParams** method after instantiating your **IMessageTransformAsync**, and pass the values to that object via either the **InitializeWrapFromAttributeParams** or **InitializeUnwrapFromAttributeParams** as appropriate for the type of **TransformAttribute** (either **WrapWithAttribute** or **UnwrapWithAttribute**). + +So in this example, the **ClaimCheck** takes a parameter for the *threshold* at which point we move the body of the message into distributed storage as opposed to serializing it in the message body. + +``` csharp +public class ClaimCheck : WrapWithAttribute +{ + private readonly int _thresholdInKb; + + public ClaimCheck(int step, int thresholdInKb = 0) : base(step) + { + _thresholdInKb = thresholdInKb; + } + + public override object[] InitializerParams() + { + return new object[] { _thresholdInKb }; + } + + public override Type GetHandlerType() + { + return typeof(ClaimCheckTransformer); + } +} +``` + +### Message Transformer Factory + +Because we do not know how to construct user-defined types, you have to pass us a **IAmAMessageTransformerFactory** that constructs instances of your **IMessageTransformAsync**. + +Normally, you implement this using your Inversion of Control container. We provide an implementation for the .NET Inversion of Control container **ServiceCollection** with **ServiceProviderTransformerFactory**. You need a reference to the following NuGet package: + +* **Paramore.Brighter.Extensions.DependencyInjection** + + +If you are using HostBuilder, our extension methods mean that you benefit from automatic inclusion of the **ServiceProviderTransformerFactory** and registration of your **IMessageTransformAsync**. + + + + + + diff --git a/contents/9/Microservices.md b/contents/9/Microservices.md new file mode 100644 index 0000000..d1922a7 --- /dev/null +++ b/contents/9/Microservices.md @@ -0,0 +1,47 @@ +# Microservices + +It is possible to think of microservices as 3rd generation SOA. First generation SOA was SOAP based web services. 2nd generation SOA was messaging, sometimes over SOAP, but also over middleware, often an +Enterpise Service Bus (ESB). The third generation emphasizes \"smart endpoints, dumb pipes\" over the use of an ESB, either via REST or a lightweight broker such as RMQ. + +But much of what applied to SOA,. still applies to microservices. + +\"SOA is focused on business processes. These processes are performed in different steps (also called activities or tasks) on different systems. The primary goal of a service is to represent a "natural" step of +business functionality. That is, according to the domain for which it's provided, a service should represent a self-contained functionality that corresponds to a real-world business activity.\" + +Josuttis, Nicolai M.. SOA in Practice: The Art of Distributed System Design. + +Don Box, the creator of SOAP, defined 4 tenets for a SOA service. These +rules are still useful for Microservices. + +1. Boundaries are explicit +2. Services are Autonomous +3. Share schema not type +4. Compatibility is based on policy + +![Microservices](_static/images/Microservices.png) + +## Boundaries are explicit + +We must have an API (it may be HTTP, gRPC, AMQP, Kafka etc.). The API hides our implementation details. We allow consumers to couple to this API, but not to the contents. The API should be a stable abstraction, it is our contract with our consumers. The implementation details can be unstable. + +## Services are autonomous + +We want to be able to release this microservice and this microservice alone. The implication of this includes the idea that because no one couples to our implementation details, then provided we do not alter the +contract expressed by the API, we can re-release easily. But this also implies that we are the single writer to any backing storage that keeps our state. Otherwise we would couple the schema of that backing store to another service and would not be able to release independently of other services if they had coupled to those details. + +Microservices are a logical, not a physical boundary, and they might consist of more than one container, such as web container and a console container, provided all the containers are considered to be part of the +release boundary for CI or CD. This is common where we have different scaling requirements for say the API served from the web container and a worker process reading from a task queue served from a console container. + +The key idea here is Independent Deployability. + +## Share Schema not type + +Our software system may not be homogeneous, we may have services developed in multiple languages. As a result we must not prevent interoperability between microservices by use of the type system from language in the API. Instead we should use platform neutral alternatives such as plain text formats (JSON, XML, YAML) or binary ones (Avro, ProtoBuf). + +## Compatibility is based on policy + +For our microservices to communicate we need to agree on the protocols we will use. In the SOAP era this led to the growth of WS- Specifications that described policies for a wide range of service capabilities. Under microservices there is no similar standards movement, but organizations still need to make assertions about the protocols that the will use in order to provide interoperability. + +## Next + +See [Event Driven Collaboration](EventDrivenCollaboration.html) for guidance on how to integrate microservices using events. diff --git a/contents/9/Monitoring.md b/contents/9/Monitoring.md new file mode 100644 index 0000000..57c8f8c --- /dev/null +++ b/contents/9/Monitoring.md @@ -0,0 +1,77 @@ +# Monitoring + +Brighter emits monitoring information from an External Bus using a configured [Control Bus](https://brightercommand.github.io/Brighter/ControlBus.html) + +## Configuring Monitoring + +Firstly [configure a Control +Bus](https://brightercommand.github.io/Brighter/ControlBus.html#configure) in the brighter application to emit monitoring messages + +## Config file + +Monitoring requires a new section to be added to the application config file: + +``` xml + +
+ +``` + +The monitoring config can then be speicified later in the file: + +``` xml + + + +``` + +This enables runtime changes to enable/disable emitting of monitoring messages. + +## Handler Configuration + +Each handler that requires monitoring must be configured in two stages, a Handler attribute and container registration of a MonitorHandler for the given request: + +For example, given: + +- TRequest - a Brighter Request, inheriting from IRequest +- TRequestHandler - handles the TRequest, inheriting IHandleRequest + \ + +### Attribute + +The following attribute must be added to the Handle method in the handler, TRequestHandler: + +``` csharp +[Monitor(step:1, timing:HandlerTiming.Before, handlerType:typeof(TRequestHandler))] +``` + +Please note the step and timing can vary if monitoring should be after another attribute step, or timing should be emitted after. + +### Container registration + +The following additional handler must be registered in the application container (where `MonitorHandler` is a built-in Brighter handler): + +``` csharp +container.Register> +``` + +## Monitor message format + +A message is emitted from the Control Bus on Handler Entry and Handler Exit. The following is the form of the message: + +``` javascript +{ + "Exception": null, // or Exception message + "EventType": "EnterHandler or ExitHandler", + "EventTime": "2016-06-21T15:48:26.1390192Z", + "TimeElapsedMs": 0 or Duration, + "HandlerName": "...", + "HandlerFullAssemblyName": "...", + "InstanceName": "ManagementAndMonitoring", + "RequestBody": "{\"Id\":\"dc32b35f-bc75-4197-9178-c8310a63e4fb\", ... }", + "Id": "048cc207-e820-40fa-b931-55b60203fbc2" +} +``` + +Messages can be processed from the queue and interated with your monitoring tool of choice, for example Live python consumers emitting to console or logstash consumption to the ELK stack using relevant plugins +to provide performance raditators or dashboards. diff --git a/contents/9/MySQLInbox.md b/contents/9/MySQLInbox.md new file mode 100644 index 0000000..9123289 --- /dev/null +++ b/contents/9/MySQLInbox.md @@ -0,0 +1,37 @@ +# MySQL Inbox + +## Usage +The MySQL Inbox allows use of MySQL for [Brighter's inbox support](/contents/BrighterInboxSupport.md). The configuration is described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#inbox). + +For this we will need the *Inbox* packages for the MySQL *Inbox*. + +* **Paramore.Brighter.Inbox.MySql** + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + services.AddServiceActivator(options => + { ... }) + .UseExternalInbox( + new MySqlInbox(new MySqlInboxConfiguration("server=localhost; port=3306; uid=root; pwd=root; database=Salutations", "Inbox"); + new InboxConfiguration( + scope: InboxScope.Commands, + onceOnly: true, + actionOnExists: OnceOnlyAction.Throw + ) + ); +} + +... + +``` + + + diff --git a/contents/9/OutboxPattern.md b/contents/9/OutboxPattern.md new file mode 100644 index 0000000..22be7f2 --- /dev/null +++ b/contents/9/OutboxPattern.md @@ -0,0 +1,43 @@ +# Outbox Pattern Support + +## Producer Correctness + +When a microservice changes the state for which it is the system of record, and then signals to subscribers via an event that it has changed its state, how do we ensure that subscribers receive the event and are +therefore consistent with the producer? + +The system of record may become inconsistent with downstream consumers because after writing changes to an entity, we may fail to publish the corresponding message. Our connection to the message-oriented middleware (MoM) may fail, or MoM may fail. + +The new record is saved to the backing store, but the event is not raised so subscribing systems become inconsistent. We have a lost send. + +![CorrectnessProblem](_static/images/CorrectnessProblem.png) + +Distributed Transactions may seem like an answer, but possess two issues. First, we are probably using a backing store and message-oriented middleware from different vendors or OSS projects that don\'t support the same distributed transaction protocol. Second, distributed transactions don\'t scale well. + +We might naively try to fix this by sending the message first, then updating the backing store if that succeeds. But this won\'t necessarily work either, as we might fail to write to the database. + +The new record is posted to downstream systems, but the local database call is rejected, and so the upstream system is now inconsistent. A phantom send. + +In either solution we might simply decide that the best option is to ensure that we can retry what is hopefully a transient error. This may solve the problem in many instances, and is a good first step. But an +endless retry loop has its own dangers, consuming resources and reducing throughput, and if the app crashes we will still only be partially complete. So it cannot guarantee delivery of the message that matches +the write. + +## The Outbox Pattern + +In the Outbox pattern, we use the ACID properties of an RDBMS. We write not only to the table that stores the entity that we are inserting, updating, or deleting, but also we write the message we intend to send +to an \'outbox table\' in the same Db. + +We mark the time that the message was written, as part of the transaction, on the record. + +Then when we send the message via the Broker, we mark the message as dispatched in the table. + +![OutboxPattern](_static/images/OutboxPattern.png) + +An out-of-band sweeper process can then run, and query for messages that have not been sent within a time window (their written date is over a threshold of milliseconds ago, and they have no dispatched time stamp). +It then resends those messages. If it sends them, it marks them as dispatched. As the sweeper process keeps polling for messages that have not yet been sent, we will eventually send all the messages. So we have +**guaranteed delivery** but eventual consistency. + +It is possible that the write to the row to update the dispatched status will fail. It is not in a transaction between broker and RDBMS either. If that happens, we may send the message twice. + +For this reason, the Outbox pattern offers us **guaranteed, at least once** delivery. Consumers must be prepared for this. Either they can use an *Inbox*, which records all the messages they have seen recently and discards duplicates, or they must be idempotent and the result of processing the message twice has no side-affects. + +See [Brighter Outbox Support](BrighterOutboxSupport.html) for more on how to ensure Producer-Consumer correctness in Brighter. diff --git a/contents/9/PolicyFallback.md b/contents/9/PolicyFallback.md new file mode 100644 index 0000000..1f480e6 --- /dev/null +++ b/contents/9/PolicyFallback.md @@ -0,0 +1,45 @@ +# Fallback + +You may want some sort of backstop exception handler, that allows you to take compensating action, such as undoing any partially committed work, issuing a compensating transaction, or queuing work for later delivery (perhaps using the [External Bus](/contents/ImplementingExternalBus.md)). + +To support this we provide a **IHandleRequests\Fallback** method. In the Fallback method you write your code to run in the event of failure. + +## Calling the Fallback Pipeline + +We provide a **FallbackPolicy** Attribute that you can use on your **IHandleRequests\.Handle()** method. The implementation of the **Fallback Policy Handler** is straightforward: it creates a backstop exception handler by encompassing later requests in the [Request Handling Pipeline](BuildingAPipeline.html) in a try\...catch block. You can configure it to catch all exceptions, or just [Broken Circuit Exceptions](PolicyRetryAndCircuitBreaker.html) when a Circuit +Breaker has tripped. + +When the **Fallback Policy Handler** catches an exception it calls the **IHandleRequests\.Fallback()** method of the next Handler in the pipeline, as determined by **IHandleRequests\.Successor** + +The implementation of **RequestHandler\.Fallback()** uses the same [Russian Doll](BuildingAPipeline.html) approach as it uses for **RequestHandler\.Handle()**. This means that the request to take compensating action for failure, flows through the same pipeline as the +request for service, allowing each Handler in the chain to contribute. + +In addition the **Fallback Policy Handler** makes the originating exception available to subsequent Handlers using the **Context Bag** with the key: **CAUSE_OF_FALLBACK_EXCEPTION** + +## Using the FallbackPolicy Attribute + +The following example shows a Handler with **Request Handler Attributes** for [Retry and Circuit Breaker policies](PolicyRetryAndCircuitBreaker.html) that is configured with a **Fallback Policy** which catches a **Broken Circuit Exception** (raised when the Circuit Breaker is tripped) and initiates the Fallback chain. + +``` csharp +public class MyFallbackProtectedHandler: RequestHandler +{ + [FallbackPolicy(backstop: false, circuitBreaker: true, step: 1)] + [UsePolicy(new [] {}"MyCircuitBreakerStrategy", "MyRetryStrategy"}, step: 2)] + public override MyCommand Handle(MyCommand command) + { + /*Do some work that can fail*/ + } + + public override MyCommand Fallback(MyCommand command) + { + if (Context.Bag.ContainsKey(FallbackPolicyHandler.CAUSE_OF_FALLBACK_EXCEPTION)) + { + /*Use fallback information to determine what action to take*/ + } + return base.Fallback(command); + } +} +``` +## Scope of a Fallback + +Where you put any **FallbackPolicy** attribute determines what exceptions it will call your Fallback method to guard against. This is controlled by the **Step** parameter. Remember that you encapsulate anything with a higher **Step** and can react to an exception thrown there. diff --git a/contents/9/PolicyRetryAndCircuitBreaker.md b/contents/9/PolicyRetryAndCircuitBreaker.md new file mode 100644 index 0000000..e29f0ac --- /dev/null +++ b/contents/9/PolicyRetryAndCircuitBreaker.md @@ -0,0 +1,129 @@ +# Supporting Retry and Circuit Breaker + +Brighter is a [Command Processor](https://www.goparamore.io/control-bus-and-data-bus/) and +supports a [pipeline of Handlers to handle orthogonal requests](BuildingAPipeline.html). + +Amongst the valuable uses of orthogonal requests is patterns to support Quality of Service in a distributed environment: [Timeout, Retry, and Circuit Breaker](PolicyRetryAndCircuitBreaker.html#using-brighter-s-usepolicy-attribute). + +Even if you don't believe that you are writing a distributed system that needs this protection, consider that as soon as you have multiple processes, such as a database server, you are distributed. + +Brighter uses [Polly](https://github.com/App-vNext/Polly) to support Retry and Circuit-Breaker. Through our [Russian Doll Model](BuildingAPipeline.html) we are able to run the target handler in +the context of a Policy Handler, that catches exceptions, and applies a Policy on how to deal with them. + +## Using Brighter's UsePolicy Attribute + +By adding the **UsePolicy** attribute, you instruct the Command Processor to insert a handler (filter) into the pipeline that runs all later steps using that Polly policy. + +``` csharp +internal class MyQoSProtectedHandler : RequestHandler +{ + static MyQoSProtectedHandler() + { + ReceivedCommand = false; + } + + [UsePolicy(policy: "MyExceptionPolicy", step: 1)] + public override MyCommand Handle(MyCommand command) + { + /*Do work that could throw error because of distributed computing reliability*/ + } +} +``` + +To configure the Polly policy you use the PolicyRegistry to register the Polly Policy with a name. At runtime we look up that Policy by name. + +``` csharp +var policyRegistry = new PolicyRegistry(); + +var policy = Policy + .Handle() + .WaitAndRetry(new[] + { + 1.Seconds(), + 2.Seconds(), + 3.Seconds() + }, (exception, timeSpan) => + { + s_retryCount++; + }); + +policyRegistry.Add("MyExceptionPolicy", policy); +``` + +You can use multiple policies with a handler, instead of passing in a single policy identifier, you can pass in an array of policy identifiers: + +So if in addition to the above policy we have: + +``` csharp +var circuitBreakerPolicy = Policy.Handle().CircuitBreaker( + 1, TimeSpan.FromMilliseconds(500)); + +policyRegistry.Add("MyCircuitBreakerPolicy", policy); +``` + +then you can add them both to your handler as follows: + +``` csharp +internal class MyQoSProtectedHandler : RequestHandler +{ + static MyQoSProtectedHandler() + { + ReceivedCommand = false; + } + + [UsePolicy(new [] {"MyCircuitBreakerPolicy", "MyExceptionPolicy"} , step: 1)] + public override MyCommand Handle(MyCommand command) + { + /*Do work that could throw error because of distributed computing reliability*/ + } +} +``` + +Where we have multiple policies they are evaluated left to right, so in this case "MyCircuitBreakerPolicy" wraps "MyExceptionPolicy". + +When creating policies, refer to the [Polly](https://github.com/App-vNext/Polly) documentation. + +Whilst [Polly](https://github.com/App-vNext/Polly) does not support a Policy that is both Circuit Breaker and Retry i.e. retry n times with an interval between each retry, and then break circuit, to implement that simply put a Circuit Breaker UsePolicy attribute as an earlier step than the Retry UsePolicy attribute. If retries expire, the exception will bubble out to the Circuit Breaker. + +## Timeout + +You should not allow a handler that calls out to another process (e.g. a call to a Database, queue, or an API) to run without a timeout. If the process has failed, you will consumer a resource in your application +polling that resource. This can cause your application to fail because another process failed. + +Usually the client library you are using will have a timeout value that you can set. + +In some scenarios the client library does not provide a timeout, so you have no way to abort. + +We provide the Timeout attribute for that circumstance. You can apply it to a Handler to force that Handler into a thread which we will timeout, if it does not complete within the required time period. + +``` csharp +public class EditTaskCommandHandler : RequestHandler +{ + private readonly ITasksDAO _tasksDAO; + + public EditTaskCommandHandler(ITasksDAO tasksDAO) + { + _tasksDAO = tasksDAO; + } + + [RequestLogging(step: 1, timing: HandlerTiming.Before)] + [Validation(step: 2, timing: HandlerTiming.Before)] + [TimeoutPolicy(step: 3, milliseconds: 300)] + public override EditTaskCommand Handle(EditTaskCommand editTaskCommand) + { + using (var scope = _tasksDAO.BeginTransaction()) + { + Task task = _tasksDAO.FindById(editTaskCommand.TaskId); + + task.TaskName = editTaskCommand.TaskName; + task.TaskDescription = editTaskCommand.TaskDescription; + task.DueDate = editTaskCommand.TaskDueDate; + + _tasksDAO.Update(task); + scope.Commit(); + } + + return editTaskCommand; + } +} +``` diff --git a/contents/9/PostgresInbox.md b/contents/9/PostgresInbox.md new file mode 100644 index 0000000..ce295b2 --- /dev/null +++ b/contents/9/PostgresInbox.md @@ -0,0 +1,37 @@ +# Postgres Inbox + +## Usage +The Postgres Inbox allows use of Postgres for [Brighter's inbox support](/contents/BrighterInboxSupport.md). The configuration is described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#inbox). + +For this we will need the *Inbox* packages for the Postgres *Inbox*. + +* **Paramore.Brighter.Inbox.Postgres** + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + services.AddServiceActivator(options => + { ... }) + .UseExternalInbox( + new PostgresSqlInbox(new PostgresSqlInboxConfiguration("Host=localhost; Username=root; Password=root; Database=Salutations", "Inbox"); + new InboxConfiguration( + scope: InboxScope.Commands, + onceOnly: true, + actionOnExists: OnceOnlyAction.Throw + ) + ); +} + +... + +``` + + + diff --git a/contents/9/RabbitMQConfiguration.md b/contents/9/RabbitMQConfiguration.md new file mode 100644 index 0000000..b01d5a5 --- /dev/null +++ b/contents/9/RabbitMQConfiguration.md @@ -0,0 +1,152 @@ +# RabbitMQ Configuration + +## General + +RabbitMQ is OSS message-oriented-middleware and is [well documented](https://www.rabbitmq.com/documentation.html). Brighter handles the details of sending to or receiving from RabbitMQ. You may find it useful to understand the [building blocks](https://www.rabbitmq.com/tutorials/amqp-concepts.html) of the protocol. You might find the [documentation for the .NET SDK](https://www.rabbitmq.com/dotnet-api-guide.html) helpful when debugging, but you should not have to interact with it directly to use Brighter. + +RabbitMQ offers an API that defines primitives used to configure the middleware used for messaging: + +- **Exchange**: A routing table. Different types of exchanges route messages differently. An entry in the table is a **Routing Key**. +- **Queue**: A store-and-forward queue over which a consumer receives messages. A message is locked whilst a consumer has read it, until they ack it, upon which it is deleted from the queue, or nack it, upon which it is requeued or sent to a DLQ. +- **Binding**: Adds a queue as a target for a routing rule on an exchange. The routing key is used for this on a direct exchange (on the default exchange the routing key is the queue name). + +We connect to RabbitMQ via a multiplexed TCP/IP connection - RabbitMQ calls these channels. Brighter uses a push consumer, so it has an open channel and can be seen on the consumers list in the management console. Brighter maintains a pool of connections and when asked for a new connection will take one from it's pool in preference to creating a new one. + +## Connection + +The Connection to RabbitMQ is provided by an **RmqMessagingGatewayConnection** which allows you to configure the following: + +* **Name**: A unique name for the connection, for diagnostic purposes +* **AmqpUri**: A connection to AMQP in the form of an [RabbitMQ Uri](https://www.rabbitmq.com/uri-spec.html) **Uri** with reliability options for a retry count (defaults to 3), **ConnectionRetryCount**, retry interval (defaults to 1000ms) **RetryWaitInMilliseconds** and a circuit breaker retry timeout (defaults to 60000ms), **CircuitBreakTimeInMilliseconds**, which introduces a delay when connections exceed the retry count. +* **Exchange**: The definition of the exchange. **Name** is the identifier for the exchange. All exchanges have a [**Type**](https://www.rabbitmq.com/tutorials/amqp-concepts.html), and the default is **ExchangeType.Direct**, but it is a string value that supports all RabbitMQ exchange types on the .NET SDK. The **Durable** flag is used to indicate if the exchange definition survives node failure or restart of the broker which defaults to *false*. **SupportDelay** indicates if the Exchange supports retry with delay, which defaults to *false*. +* **DeadLetterExchange**: Another exchange definition, but this one is used to host any Dead Letter Queues (DLQ). This could be the same exchange, but normal practice is to use a different exchange. +* **Heartbeat**: RabbitMQ uses a heartbeat to determine if a connection has died. This sets the interval for that heartbeat. Defaults to 20s. +* **PersistMessages**: Should messages be saved to disk? Saving messages to disk allows them to be recovered if a node fails, defaults to *false*. + +In RabbitMQ, recreating an exiting primitive is a no-op provided the definition does not change. + +The following code creates a typical RabbitMQ connection (here shown as part of configuring an External Bus): + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .UseExternalBus(new RmqProducerRegistryFactory( + new RmqMessagingGatewayConnection + { + Name = "MyCommandConnection", + AmpqUri = new AmqpUriSpecification( + new Uri("amqp://guest:guest@localhost:5672") + connectionRetryCount: 5, + retryWaitInMilliseconds: 250, + circuitBreakerTimeInMilliseconds = 30000 + ), + Exchange = new Exchange("paramore.brighter.exchange", durable: true, supportDelay: true), + DeadLetterExchange = new Exchange("paramore.brighter.exchange.dlq", durable: true, supportDelay: false), + Heartbeat = 15, + PersistMessages = true + }, + ... //publication, see below + ).Create() +} +``` + +## Publication + +For more on a *Publication* see the material on an *External Bus* in [Basic Configuration](/contents/BrighterBasicConfiguration.md#using-an-external-bus). + +We only support one custom property on RabbitMQ which configures shutdown delay to await pending confirmations. + +* **WaitForConfirmsTimeOutInMilliseconds** + +Under the hood, Brighter uses [Publisher Confirms](https://www.rabbitmq.com/confirms.html) to update its Outbox for the dispatch time. This means that when publishing a message we allow RabbitMQ to confirm delivery of a message to all available nodes asynchronously, and then call us back, over blocking. This allows for higher throughput. But it means that we cannot update the Outbox to show a message as dispatched, until we receive the callback, which may occur after your handler pipeline for that message has completed and the message has been acknowledged. + +When shutting down a producer, it is possible that not all confirms have yet been received from RabbitMQ. The delay instructs Brighter to wait for a period of time, in order to allow the confirms to arrive. + +Missing a confirm will cause the *Outbox Sweeper* to resend a message, as it will not be marked as dispatched. (This is why we refer to Guaranteed *At Least Once* because there are many opportunities where messages may be duplicated in order to guarantee they were sent). + +The following code creates a *Publication* for RabbitMQ when configuring an *External Bus* + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .UseExternalBus({ + ...//connection information, see above + }, + new RmqPublication[]{ + new RmqPublication + { + Topic = new RoutingKey("GreetingMade"), + MaxOutStandingMessages = 5, + MaxOutStandingCheckIntervalMilliSeconds = 500, + WaitForConfirmsTimeOutInMilliseconds = 1000, + MakeChannels = OnMissingChannel.Create + }} + ).Create() +} +``` + + +## Subscription + +For more on a *Subscription* see the material on configuring *Service Activator* in [Basic Configuration](/contents/BrighterBasicConfiguration.md#configuring-the-service-activator). + +We support a number of RabbitMQ specific *Subscription* options: + +* **DeadLetterChannelName**: The name of the queue to subscribe to DLQ notifications for this subscription (without a queue, the messages sent to the Dead Letter Exchange (DLX) will not be stored) +* **DeadLetterRoutingKey**: The routing key that binds the DLQ to the DLX +* **HighAvailability**: [Deprecated] Not used on versions of RabbitMQ 3+. Prior to this, configuring that a queue should be mirrored was an API option, now it is a configuration management option on the broker. +* **IsDurable**: Should subscription definitions survive a restart of nodes in the broker. +* **MaxQueueLength**: [Deprecated] Prefer to use policy to set this instead (see [RabbitMQ docs](https://www.rabbitmq.com/maxlength.html)). The maximum length a RabbitMQ queue can grow to, before new messages are rejected (and sent to a DLQ if there is one). + +This is a typical *Subscription* configuration in a Consumer application: + +``` csharp +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + var subscriptions = new Subscription[] + { + new RmqSubscription( + new SubscriptionName("paramore.sample.salutationanalytics"), + new ChannelName("SalutationAnalytics"), + new RoutingKey("GreetingMade"), + runAsync: false, + timeoutInMilliseconds: 200, + isDurable: true, + makeChannels: OnMissingChannel.Create), //change to OnMissingChannel.Validate if you have infrastructure declared elsewhere + }; + + var rmqConnection = new RmqMessagingGatewayConnection + { + AmpqUri = new AmqpUriSpecification( + new Uri("amqp://guest:guest@localhost:5672") + connectionRetryCount: 5, + retryWaitInMilliseconds: 250, + circuitBreakerTimeInMilliseconds = 30000 + ), + Exchange = new Exchange("paramore.brighter.exchange") + }; + + var rmqMessageConsumerFactory = new RmqMessageConsumerFactory(rmqConnection); + + services.AddServiceActivator(options => + { + options.Subscriptions = subscriptions; + options.ChannelFactory = new ChannelFactory(rmqMessageConsumerFactory); + ... //see Basic Configuration + }) +``` + +### Ack and Nack + +We use RabbitMQ's queues to subscribe to a routing key on an exchange. + +When we Accept/Ack a message, in response to a handler chain completing, we Ack the message to RabbitMQ using **Channel.BasicAck**. Note that we only Ack a message once we have completed running the chain. + +When we Reject/Nack a message (see [Handler Failure](/contents/HandlerFailure.md) for more on failure) then we use **Channel.Reject** to delete the message, and move it to a DLQ if there is one. + +Brighter has an internal buffer for messages pushed to a *Performer* (a thread running a message pump). This buffer has thread affinity (in RabbitMQ we have to Ack or Nack from the thread that received the message). When a consumer closes its connection to RabbitMQ, messages in the buffer that have not been Ack'd or Nack'd will be returned to the queue. + + + diff --git a/contents/9/Requests, Commands and Events.md b/contents/9/Requests, Commands and Events.md new file mode 100644 index 0000000..70c61df --- /dev/null +++ b/contents/9/Requests, Commands and Events.md @@ -0,0 +1,36 @@ +# Requests, Commands and Events +## The IRequest Interface + +We use the term **Request** for a data object containing parameters that you want to dispatch to a handler. Brighter uses the interface **IRequest** for this concept. + +We do not recommend deriving from **IRequest** but instead from the classes **Command** and **Event** which represent types of **Request**. + + +## What is the difference between a Command and an Event? + + Confusingly, both **Command** or **Event** which implement **IRequest** are examples of the [Command Pattern](https://brightercommand.github.io/Brighter/CommandsCommandDispatcherandProcessor.html). It is easiest to say that within Brighter **IRequest** is the abstraction that represents the Command from the [Command Pattern](https://brightercommand.github.io/Brighter/CommandsCommandDispatcherandProcessor.html). + +Why have both **Command** and **Event**? The difference is in how the **Command Dispatcher** dispatches them to handlers. + +- A **Command** is an imperative instruction to do something; it only has one handler. We will throw an error for multiple registered handlers of a command. +- An **Event** is a notification that something has happened; it has zero or more handlers. + +The difference is best explained by the following analogy. If I say \"Bob, make me a cup of coffee,\" I am giving a Command, an imperative instruction. My expectation is that Bob will make me coffee. If Bob does +not, then we have a failure condition (and I am thirsty and cranky). If I say \"I could do with a cup of coffee,\" then I am indicating a state of thirst and caffeine-withdrawal. If Bob or Alice make me a coffee I will be very grateful, but there is no expectation that they will. + +So choosing between **Command** or **Event** effects how the **Command Dispatcher** routes requests. + +See [Dispatching a Request](DispatchingARequest.html) for more on how to dispatch **Requests** to handlers. + +## Message Definitions and Independent Deployability + +Some messaging frameworks encourage you to share an assembly containing your message definitions between autonomous components, often as interfaces. Occasionally we see users trying to use **IRequest** for this purpose. + +We do not recommend this, instead preferring to keep to the Service Oriented Architecture (SOA) *tenet* of **Share schema not type**. + +Between components that we wish to be independently deployable - which might after all be in different languages, or use different frameworks - you should share a schema that defines the shape of the message (for example [AsyncAPI](https://www.asyncapi.com/). + +The only exception is where you have two apps that form part of a single service - such as a Task Queue that supports offloading work from a web API - as these tend to be a unit for Continuous Integration and not independently deployable, then sharing types may be appropriate. + +Many of our samples share types for convenience, but this is not advice to do that outside of a Task Queue. + diff --git a/contents/9/ReturningResultsFromAHandler.md b/contents/9/ReturningResultsFromAHandler.md new file mode 100644 index 0000000..01b1a91 --- /dev/null +++ b/contents/9/ReturningResultsFromAHandler.md @@ -0,0 +1,53 @@ + +# Returning Results from a Handler + +We use [Command-Query +separation](https://martinfowler.com/bliki/CommandQuerySeparation.html) so a Command does not have return value and **CommandDispatcher.Send()** does not return anything. Our project Darker provides a Query Processor that can be used to return results in response to queries. You can use both together to provide CQRS. + +This in turn leads to a set of questions that we need to answer about common scenarios: + +- How do I handle failure? With no return value, what do I do if my handler fails? +- How do I communicate the outcome of a command? + +## Handling Failure + +If we don\'t allow return values, what do you do on failure? + +- The basic failure strategy is to throw an exception. This will terminate the request handling pipeline. +- If you want *Internal Bus* support for [Retry, and Circuit Breaker](PolicyRetryAndCircuitBreaker.html) you can use our support for [Polly](https://github.com/App-vNext/Polly) Policies +- If you want to Requeue (with Delay) to an *External Bus*, you should throw a **DeferMessageAction** exception. +- Finally you can use our support for a [Fallback](PolicyFallback.html) handler to provide backstop exception handling. +- You can also build your own exception handling into your [Pipeline](BuildingAPipeline.html). + +We discuss these options in more detail in [Handler Failure](/contents/HandlerFailure.md). + +## Communicating the Outcome of a Command + +Sometimes you need to provide information to the caller about the outcome of a *Command*, instead of listening for an *Event* an. + +How do you communicate the outcome of handling a *Command*? There are two options, which depend on circumstance: + +* Raise an *Event* +* Update a field on the *Command* + +### Raising an Event + +This approach let's you take action in response to a *Command* by raising an *Event* within your handler using **CommandProcessor.Publish** or via an *External Bus* using **CommandProcessor.Post/CommandProcessor.DepositPost**. + +If you use an **Internal Bus** these handlers will run immediately, in their own pipeline, before your handler exits. If you use an **External Bus** you offload the work to another process. + +### Update a field on the Command + +If you are using an *Internal Bus* and need a return value from a *Command* you will note that **CommandProcessor.Send** has a void return value, so you cannot return a value from the handler. + +What happens if the caller needs to know the outcome, and can't be signalled via an *Event*? + +In that case add a property to the **Command** that you can initialize from the Handler. As an example, what happens if you need to return the identity of a newly created entity, so that you can use **Darker** to retrieve its details? In this case you can create a **NewEntityIdentity** property in your command that you write a newly created entity\'s identity to in the Handler, and then inspect the property in your **Command** in the calling code after the call to **commandProcessor.Send(command)** completes. + +You can think of these as *out* parameters. + +``` csharp +var createTaskCommand = new CreateTaskCommand(); +commandProcessor.Send(createTaskCommand); +var newTaskId = createTaskCommand.TaskId; +``` diff --git a/contents/9/Routing.md b/contents/9/Routing.md new file mode 100644 index 0000000..7d59d1a --- /dev/null +++ b/contents/9/Routing.md @@ -0,0 +1,141 @@ +# Routing + +### Routing Messages + +A producer routes messages to subscribers by setting a **Topic** on the **MessageHeader**. A **Topic** is just a string that you intend to use as a unique identifier for this message. A simple scheme can be the +typename of the event for the Producer. + +When implementing an **IAmAMessageMapper\** you set the **Topic** in the **MessageHeader** when serializing your **Command** or **Event** to disk. In the following example we set the **Topic** to *Task.Completed*. + +``` csharp +public class TaskCompletedEventMapper : IAmAMessageMapper +{ + public Message MapToMessage(TaskCompletedEvent request) + { + var header = new MessageHeader(messageId: request.Id, topic: "Task.Completed", messageType: MessageType.MT_EVENT); + var body = new MessageBody(JsonConvert.SerializeObject(request)); + var message = new Message(header, body); + return message; + } +} +``` + +The **routingKey** property of the connection must be the same key as used in the message mapper. This is passed to the broker to inform it that we want to subscribe to messages with that routing key on this channel. + +## Events + +Brighter has a default Publish-Subscribe approach to events. + +A broker provides an intermediary between the producer of a message and a consumer. A consumer registers interest in messages that have a key or topic. A producer sends messages with a key or topic to a broker, and the broker sends a copy of that message to every subscribing consumer. + + +## Commands + +For a command the producer knows its consumer. A command may be fire and forget, which means it does not expect a reply, or request-reply which means that it does. In request-reply the receiver knows its sender as well. + +The reason you might choose a command over an event is causality. + +Consider an application that needs to bill a customer's credit card. + +In an event driven approach, we could make the assumption that the transaction will succeed, raise an event to bill the customer and process the payment asynchronously. The producer of the billing request continues as though the transaction had succeeded. Eventually the customer is billed, and we are consistent. Our reason for taking this approach may be that our payment provider is often slow to respond and we do not want to make the customer wait whilst we handle details of their payment. If we fail to bill the customer we have to take compensating action - raising a billing failed event, which may alert an operator and email the customer. + +The problem is that this compensating action has to "chase" the success path, which may have already taken actions, such as shipping to the customer, that become expensive to reverse. + +With a command, we decide that as a payment transaction can fail we do not want to process the order until the payment has been received. In this case, our requirement is that we receive a response to our **Command** to bill. To route a command the Producer may send a reply-address to the Consumer so that it can send a response back on a 'private' channel. In our case, that reply-address is a topic that the sender subscribes to, in order to receive the response. + +Usually the Producer creates a topic for all of its replies, and matches request to response via a correlation id. This is simply a unique identifier that the Producer adds to the outgoing message. + +(Because the customer has to wait, in case we want to signal an error we are probably returning a 202 Accepted from our HTTP API with a link to a resource to monitor for the results of the transaction. In our client we display a progress indicator until we have completed the transaction.) + +To help route direct messages we provide two classes, **Request** and **Reply** but the real work occurs within the message mapper itself. + +Note also the correlation id that is added to the **ReplyAddress**. + +``` csharp + +public class MyRequest : Request +{ + public MyRequest(ReplyAddress sendersAddress) : base(sendersAddress) + { + } +} + +public class MyReply : Reply +{ + public MyReply(ReplyAddress sendersAddress) : base(sendersAddress) + { + } +} +``` + +When we convert this request into a **Message** via an **IAmAMessageMapper** we set the **MessageHeader** with the topic the Consumer should reply to. We also set the correlation id of the sender\'s message on the header. + +In the following code we also serialize the message back to a **Command** which is then routed by Brighter to a handler. When we serialize back to a **Command** we set the **ReplyAddress** with the Topic and Correlation Id. + +``` csharp +public class MyRequestMessageMapper : IAmAMessageMapper +{ + public Message MapToMessage(MyRequest request) + { + var header = new MessageHeader( + messageId: request.Id, + topic: "MyRequest", + messageType: MessageType.MT_COMMAND, + correlationId: request.ReplyAddress.CorrelationId, + replyTo: request.ReplyAddress.Topic); + + var json = new JObject(new JProperty("Id", request.Id)); + var body = new MessageBody(json.ToString()); + var message = new Message(header, body); + return message; + } + + public MyRequest MapToRequest(Message message) + { + var replyAddress = new ReplyAddress(topic: message.Header.ReplyTo, correlationId: message.Header.CorrelationId); + var request = new MyRequest(replyAddress); + var messageBody = JObject.Parse(message.Body.Value); + request.Id = Guid.Parse((string) messageBody["Id"]); + return request; + } +} +``` + +When we reply, we again use the message mapper to ensure that we route correctly. Again the key to responding is the **IAmAMessageMapper** implementation which uses the **ReplyAddress** to route the **Message** via its **MessageHeader** back to the caller. Note that whilst the response could be considered an event - a fact raised in response to a command - because it only has one Consumer, the sender, we route it as a command. If you want to broadcast the outcome, treat it as an event, but add **ReplyAddress** to your class derived from **Event** to correlate with the command. + +``` csharp +internal class MyReplyMessageMapper : IAmAMessageMapper +{ + public Message MapToMessage(MyReply request) + { + var header = new MessageHeader( + messageId:request.Id, + topic: request.SendersAddress.Topic, + messageType: MessageType.MT_COMMAND, + timeStamp: DateTime.UtcNow, + correlationId: request.SendersAddress.CorrelationId + ); + + var json = ...//serialize the reply body + + var body = new MessageBody(json.ToString()); + var message = new Message(header, body); + return message; + } + + public MyReply MapToRequest(Message message) + { + var replyAddress = new ReplyAddress(message.Header.Topic, message.Header.CorrelationId); + + var reply = new MyReply(replyAddress); + + ...//deserialize the body + + return reply; + } +} +``` + +## Summary + +The key to understanding routing in Brighter is that the **IAmAMessageMapper** implementation provides the point at which you control routing by setting the **MessageHeader**. diff --git a/contents/9/S3LuggageStore.md b/contents/9/S3LuggageStore.md new file mode 100644 index 0000000..5d2416e --- /dev/null +++ b/contents/9/S3LuggageStore.md @@ -0,0 +1,54 @@ +# S3 Luggage Store + +The **S3LuggageStore** is an implementation of **IAmAStorageProviderAsync** for AWS S3 Object Storage. It allows use of the [Claim Check](/contents/ClaimCheck.md) *Transformer* with S3 Object Storage as the *Luggage Store*. + +To use the **S3LuggageStore** you need to include the following NuGet package: + +* **Paramore.Brighter.Transformers.AWS** + +We then need to configure our **S3LuggageStore** and register it with our IoC container. Our **ClaimCheckTransformer** has a dependency on **IAmAStorageProviderAsync** and at runtime, when our [** **IAmAMessageTransformerFactory**](/contents/MessageMappers.md#message-transformer-factory) creates an instance it needs to be able to resolve that dependency. For this reason you need to register the implementation, in this case **S3LuggageStore** with the IoC container to allow it to resolve the dependency. + +We provide an extension method to **ServiceCollection** to help with this: + +``` csharp +serviceCollection.AddS3LuggageStore((options) => +{ + options.Connection = new AWSS3Connection(credentials, RegionEndpoint.EUWest1); + options.BucketName = "brightersamplebucketb0561a06-70ec-11ed-a1eb-0242ac120002"; + options.BucketRegion = S3Region.EUW1; + options.StoreCreation = S3LuggageStoreCreation.CreateIfMissing; +}); +``` + +You configure an **S3LuggageStore** using the **S3LuggateOptions** provided to the callback in **AddS3LuggageStore**. You MUST set the following options: + +* **Connection**: The **AWSS3Connection** that allows us to connect to your account. Used to create an **S3Client** and an **STSClient** +* **BucketName**: The name of the S3 bucket that backs the luggage store. We use one bucket for the luggage store. You may re-use a bucket that you already have. +* **BucketRegion**: Where is the bucket? Bucket names must be unique within a region. +* **StoreCreation**: What should we do when determining if there is a bucket for the store? + * **CreateIfMissing**: We will create the bucket in the requested region (provided the credentials provided have rights to do this.) + * **ValidateExists**: We will check if the bucket exists in the requested region. We throw an **InvalidOperationException** if it does not. + * **AssumeExists**: We do not check for the bucket, but just assume it exists + +If you choose **CreateIfMissing** or **ValidateExists** then you must register an **IHTTPClientFactory** as we will use this to obtain an HTTP Client for use with the AWS REST API to make a check for the bucket's existence. The simplest way to do this is to use the ServiceCollection extension provided for creating an **IHTTPClientFactory**: + +```csharp + serviceCollection.AddHttpClient(); +``` + +### Bucket Creation + +If we create the bucket we do so with the following properties: + +* Block public PUT access +* Object ownership transferred to bucket owner + +In addition we set the following properties on the bucket, which can be controlled: + +* We delete aborted uploads after the time given by **TimeToAbortFailedUploads**. Defaults to 1 day. +* We delete successful uploads after the time given by **TimeToDeleteGoodUploads**. Defaults to 7 days. + +We set *Tags* on the bucket if they are provided in the **Tags** property. + +We default the **ACLs** for the bucket to **S3CannedACL.Private, but you can choose to override this with another policy as described in [**S3CannedACL**](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-access-control.html#RESTCannedAccessPolicies). + diff --git a/contents/9/ShowMeTheCode.md b/contents/9/ShowMeTheCode.md new file mode 100644 index 0000000..a44a74b --- /dev/null +++ b/contents/9/ShowMeTheCode.md @@ -0,0 +1,178 @@ +# Show me the code! + +There is an old principle: show don't tell, and this introduction is about showing you what you can do with Brighter and Darker. It's not about how - more detailed documentation elsewhere shows you how to write this code. It's not about why - articles elsewhere discuss some of the reasons behind this approach. It is just, let me see how Brighter works. + +## Brighter and Darker + +### Brighter is about Requests + +A *Request* is a message sent over a bus. A request may update state. + +A *Command* is an instruction to execute some behavior. An *Event* is a notification. + +You use the *Command Processor* to separate the sender from the receiver, and to provide middleware functionality like a retry. + +### Darker is about Queries + +A *Query* is a message executed via a bus that returns a *Result*. A query does not update state. + +You use the *Query Processor* to separate the requester from the replier, and to provide middleware functionality like a retry. + +### Middleware + +Both Brighter and Darker allow you to provide middleware that runs between a request or query being made and being handled. The middleware used by a handler is configured by attributes. + +### Sending and Querying Example + +In this example, we show sending a command, and querying for the results of issuing it, from within an ASP.NET WebAPI controller method. + +``` csharp +[Route("{name}/new")] +[HttpPost] +public async Task> Post(string name, NewGreeting newGreeting) +{ + await _commandProcessor.SendAsync(new AddGreeting(name, newGreeting.Greeting)); + + var personsGreetings = await _queryProcessor.ExecuteAsync(new FindGreetingsForPerson(name)); + + if (personsGreetings == null) return new NotFoundResult(); + + return Ok(personsGreetings); +} +``` + +### Handling Examples + +Handler code listens for and responds to requests or queries. The handler for the above request and query are: + +``` csharp +[RequestLogging(0, HandlerTiming.Before)] +[UsePolicyAsync(step:1, policy: Policies.Retry.EXPONENTIAL_RETRYPOLICYASYNC)] +public override async Task HandleAsync(AddPerson addPerson, CancellationToken cancellationToken = default(CancellationToken)) +{ + await _uow.Database.InsertAsync(new Person(addPerson.Name)); + + return await base.HandleAsync(addPerson, cancellationToken); +} +``` + +``` csharp +[QueryLogging(0)] +[RetryableQuery(1, Retry.EXPONENTIAL_RETRYPOLICYASYNC)] +public override async Task ExecuteAsync(FindGreetingsForPerson query, CancellationToken cancellationToken = new CancellationToken()) +{ + var sql = @"select p.Id, p.Name, g.Id, g.Message + from Person p + inner join Greeting g on g.Recipient_Id = p.Id"; + + var people = await _uow.Database.QueryAsync(sql, (person, greeting) => + { + person.Greetings.Add(greeting); + return person; + }, splitOn: "Id"); + + var peopleGreetings = people.GroupBy(p => p.Id).Select(grp => + { + var groupedPerson = grp.First(); + groupedPerson.Greetings = grp.Select(p => p.Greetings.Single()).ToList(); + return groupedPerson; + }); + + var person = peopleGreetings.Single(); + + return new FindPersonsGreetings + { + Name = person.Name, + Greetings = person.Greetings.Select(g => new Salutation(g.Greet())) + }; + +} +``` + +## Using an External Bus + +As well as using an Internal Bus, in Brighter you can use an External Bus - middleware such as RabbitMQ or Kafka - to send a request between processes. Brighter supports both sending a request, and provides a *Dispatcher* than can listen for requests on middleware and forward it to a handler. + +The following code sends a request to another process. + +``` csharp +[RequestLogging(0, HandlerTiming.Before)] +[UsePolicyAsync(step:1, policy: Policies.Retry.EXPONENTIAL_RETRYPOLICYASYNC)] +public override async Task HandleAsync(AddGreeting addGreeting, CancellationToken cancellationToken = default(CancellationToken)) +{ + var posts = new List(); + + //We use the unit of work to grab connection and transaction, because Outbox needs + //to share them 'behind the scenes' + + var tx = await _uow.BeginOrGetTransactionAsync(cancellationToken); + try + { + var searchbyName = Predicates.Field(p => p.Name, Operator.Eq, addGreeting.Name); + var people = await _uow.Database.GetListAsync(searchbyName, transaction: tx); + var person = people.Single(); + + var greeting = new Greeting(addGreeting.Greeting, person); + + //write the added child entity to the Db + await _uow.Database.InsertAsync(greeting, tx); + + //Now write the message we want to send to the Db in the same transaction. + posts.Add(await _postBox.DepositPostAsync(new GreetingMade(greeting.Greet()), cancellationToken: cancellationToken)); + + //commit both new greeting and outgoing message + await tx.CommitAsync(cancellationToken); + } + catch (Exception e) + { + _logger.LogError(e, "Exception thrown handling Add Greeting request"); + //it went wrong, rollback the entity change and the downstream message + await tx.RollbackAsync(cancellationToken); + return await base.HandleAsync(addGreeting, cancellationToken); + } + + //Send this message via a transport. We need the ids to send just the messages here, not all outstanding ones. + //Alternatively, you can let the Sweeper do this, but at the cost of increased latency + await _postBox.ClearOutboxAsync(posts, cancellationToken:cancellationToken); + + return await base.HandleAsync(addGreeting, cancellationToken); +} +``` + +The following code receives a message, sent from another process, via a dispatcher. It uses an Inbox to ensure that it does not process duplicate messages + +``` csharp +[UseInboxAsync(step:0, contextKey: typeof(GreetingMadeHandlerAsync), onceOnly: true )] +[RequestLoggingAsync(step: 1, timing: HandlerTiming.Before)] +[UsePolicyAsync(step:2, policy: Policies.Retry.EXPONENTIAL_RETRYPOLICYASYNC)] +public override async Task HandleAsync(GreetingMade @event, CancellationToken cancellationToken = default(CancellationToken)) +{ + var posts = new List(); + + var tx = await _uow.BeginOrGetTransactionAsync(cancellationToken); + try + { + var salutation = new Salutation(@event.Greeting); + + await _uow.Database.InsertAsync(salutation, tx); + + posts.Add(await _postBox.DepositPostAsync(new SalutationReceived(DateTimeOffset.Now), cancellationToken: cancellationToken)); + + await tx.CommitAsync(cancellationToken); + } + catch (Exception e) + { + _logger.LogError(e, "Could not save salutation"); + + //if it went wrong rollback entity write and Outbox write + await tx.RollbackAsync(cancellationToken); + + return await base.HandleAsync(@event, cancellationToken); + } + + await _postBox.ClearOutboxAsync(posts, cancellationToken: cancellationToken); + + return await base.HandleAsync(@event, cancellationToken); +} +``` + diff --git a/contents/9/SqliteInbox.md b/contents/9/SqliteInbox.md new file mode 100644 index 0000000..7b3772d --- /dev/null +++ b/contents/9/SqliteInbox.md @@ -0,0 +1,37 @@ +# Sqlite Inbox + +## Usage +The Sqlite Inbox allows use of Sqlite for [Brighter's inbox support](/contents/BrighterInboxSupport.md). The configuration is described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#inbox). + +For this we will need the *Inbox* packages for the Sqlite *Inbox*. + +* **Paramore.Brighter.Inbox.Sqlite** + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + services.AddServiceActivator(options => + { ... }) + .UseExternalInbox( + new SqliteInbox(new SqliteInboxConfiguration("DataSource=test.db", "Inbox"); + new InboxConfiguration( + scope: InboxScope.Commands, + onceOnly: true, + actionOnExists: OnceOnlyAction.Throw + ) + ); +} + +... + +``` + + + diff --git a/contents/9/TaskQueuePattern.md b/contents/9/TaskQueuePattern.md new file mode 100644 index 0000000..583ca5d --- /dev/null +++ b/contents/9/TaskQueuePattern.md @@ -0,0 +1,21 @@ +# The Task Queue Pattern + +The Task Queue Pattern let's you use an External Bus to handle work asynchronously. It is a common use of an External Bus outside of using it for an [Event Driven Architecture](/contents/EventDrivenCollaboration.md). + +## Doing Work Asynchronously +You might have an HTTP API with a rule that any given request to that API must execute in under 100ms. On measuring the performance of a key POST or PUT operation to your API you find that you exceed this +value. Upon realizing that much of your time is spent I/O you consider two options: + +- Use the TPL (Task parallel library) to perform the work concurrently - Offload the work to a distributed task queue, ack the message, and allow the work to complete asynchronously + +Either way you probably return a 202 Accepted to the caller, with a Link header that points to an endpoint where the caller can poll for completion and/or monitor progress. This might be a resource you are +creating that will return a 404 until it exists, or a progress indicator that indicates how far through the work you are and redirects to the resource once it is complete. (You can store progress in a backing store, perhaps using a distributed cache such as Redis). + +There is a problem with the TPL approach is that your operation can only meet the 100ms threshold if your work can be parallelised such that no sub-task takes longer than 100ms. Your speed is always constrained by the slowest operation that you need to parallelize. If you are I/O bound on a resource experiencing contention beyond 100ms, you will not meet your goal by introducing more threads. Your minimum time is your minimum time. + +You might try to fix this by acking (acknowledging) the request, and completing the work asynchronously. This option is particularly attractive if the work is I/O bound as you can process other requests whilst you wait for the I/O to complete. + +The downside of the async approach is that you risk that the work will be lost if the server fails prior to completion of the work, or the app simply recycles. + +These requirements tend to push you in the direction of [Guaranteed Delivery](http://www.eaipatterns.com/GuaranteedMessaging.html) to ensure that work you ack will eventually be handled. + diff --git a/contents/9/Telemetry.md b/contents/9/Telemetry.md new file mode 100644 index 0000000..243a538 --- /dev/null +++ b/contents/9/Telemetry.md @@ -0,0 +1,53 @@ +# Telemetry + +Starting in version 9.2.1 Brighter now supports Open Telemetry Tracing + +## Configuring Open Telemetry + +The OpenTelemetry SDK can be configured to listen to Activities inside of Brighter for more information [OpenTelemetry Tracing](https://opentelemetry.io/docs/instrumentation/net/getting-started/) + + +The below code will +* Enable OpenTelemetry tracing +* Set the service name to "ProducerService" +* Set OpenTelemetry traving to listen to all Brighter and Microsoft sources +* Export the telemetry tracts to Jaeger + +```csharp +//The name of the service +const myServiceName = "ProducerService" + +var jaegerEndpoint = new Uri("http://localhost:9411/api/v2/spans"); + +using var tracerProvider = + Sdk.CreateTracerProviderBuilder() + .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(myServiceName)) + .AddSource("Paramore.*", "Microsoft.*") + .AddJaegerExporter(o => + { + o.Endpoint = jaegerEndpoint; + }) + .Build(); +``` + +## Activity Sources + +The activity sources that Brighter emits from are: + * Paramore.Brighter - Traces started in the **Command Processor** will be given this (Including **Outbox Sweeper**) + * Paramore.Brighter.ServiceActivator - Traces Started in the **Service Activator** + +Please note that Brighter will honor existing spans, i.e. When using ASPNet a Request will start a trace, it is for this reason that the sample above also includes "Microsft.*" as Bighter will participate in an active trace. + +![Distributed Trace](_static/images/DistributedTracingFromASP.png) +This distributed traces shows a message that is produced from an ASP.Net request and the consumed by Service Activator + +## Reported Events + +At this time Brighter records the following events + +| Event | Description | +| ------------------------------------------ | ----------- | +| Add message to outbox | When a message is added to the outbox | +| Get outstanding messages from the outbox | During an implicit clear when retrieving undispatched messages from the outbox | +| Dispatching message | When a message is being dispatched to a message transport | +| Bulk dispatching messages | When a batch of messages are being dispathced to a message transport | diff --git a/contents/9/UsingTheContextBag.md b/contents/9/UsingTheContextBag.md new file mode 100644 index 0000000..6fc909a --- /dev/null +++ b/contents/9/UsingTheContextBag.md @@ -0,0 +1,33 @@ +# Passing information between Handlers in the Pipeline + +A key constraint of the Pipes and Filters architectural style is that Filters do not share state. One reason is that this limits your ability to recompose the pipeline as steps must follow other steps. + +However, when dealing with Handlers that implement orthogonal concerns it can be useful to pass context along the chain. Given that many orthogonal concerns have constraints about ordering anyway, we can live +with the ordering constraints imposed by passing context. So how do you approach passing context from one Handler to another when it is necessary? + +The first thing is to avoid adding extra properties to the Command to support handling state for these orthogonal Filter steps in your pipeline. This couples your **Command** to orthogonal concerns and you +really only want to bind it to your **Target Handler**. + +Instead we provide a **Context Bag** as part of the Command Dispatcher which is injected into each Handler in the Pipeline. The lifetime of this **Context Bag** is the lifetime of the Request (although you will +need to take responsibility for freeing any unmanaged resources you place into the **Context Bag** for example when code called after the Handler that inserts the resource into the Bag returns to the Handler). + +``` csharp +public class MyContextAwareCommandHandler : RequestHandler +{ + public static string TestString { get; set; } + + public override MyCommand Handle(MyCommand command) + { + LogContext(); + return base.Handle(command); + } + + private void LogContext() + { + TestString = (string)Context.Bag["TestString"]; + Context.Bag["MyContextAwareCommandHandler"] = "I was called and set the context"; + } +} +``` + +Internally we use the **Context Bag** in a number of the Quality of Service supporting Attributes we provide. See [Fallback](PolicyFallback.html) for example. diff --git a/source/.gitbook.yaml b/source/.gitbook.yaml new file mode 100644 index 0000000..22ed1e8 --- /dev/null +++ b/source/.gitbook.yaml @@ -0,0 +1,5 @@ +root: ./ + +​structure: + readme: README.md + summary: SUMMARY.md​ diff --git a/source/10/.toc.yaml b/source/10/.toc.yaml new file mode 100644 index 0000000..e69de29 diff --git a/source/10/BrighterBasicConfiguration.md b/source/10/BrighterBasicConfiguration.md new file mode 100644 index 0000000..e36537a --- /dev/null +++ b/source/10/BrighterBasicConfiguration.md @@ -0,0 +1,742 @@ +# **Basic Configuration** + +Configuration is the most labor-intensive part of using Brighter.Once you have configured Brighter, using its model of requests and handlers is straightforward + +## **Using .NET Core Dependency Injection** + +This section covers using .NET Core Dependency Injection to configure Brighter. If you want to use an alternative DI container then see the section [How Configuration Works](/contents/HowConfigurationWorks.md) + +We divide configuration into two sections, depending on your requirements: + +* [**Configuring The Command Processor**](#configuring-the-command-processor): This section covers configuring the **Command Processor**. Use this if you want to dispatch requests to handlers, or publish messages from your application on an external bus +* [**Configuring The Service Activator**](#configuring-the-service-activator): This section covers configuring the **Service Activator**. Use this if you want to read messages from a transport (and then dispatch to handlers). + + +## **Configuring The Command Processor** + +### **Command Processor Service Collection Extensions** + +Brighter's package: + +* **Paramore.Brighter.Extensions.DependencyInjection** + + provides extension methods for **ServiceCollection** that can be used to add Brighter to the .NET Core DI Framework. + +By adding the package you can call the **AddBrighter()** extension method. + +If you are using a **Startup** class's **ConfigureServices** method call the following: + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) +} + +``` + +if you are using .NET 6 you can make the call direction on your **HostBuilder**'s Services property. + +The **AddBrighter()** method takes an **`Action`** delegate. The extension method supplies the delegate with a **BrighterOptions** object that allows you to configure how Brighter runs. + +The **AddBrighter()** method returns an **IBrighterBuilder** interface. **IBrighterBuilder** is a [fluent interface](https://en.wikipedia.org/wiki/Fluent_interface) that you can use to configure additional Brighter properties (see [Brighter Builder Fluent Interface](#brighter-builder-fluent-interface)). + +#### **Adding Polly Policies** + +Brighter uses Polly policies for both internal reliability, and to support adding a custom policy to a handler for reliability. + +To use a Polly policy with Brighter you need to register it first with a Polly **PolicyRegistry**. In this example we register both Synchronous and Asynchronous Polly policies with the registry. + +``` csharp + var retryPolicy = Policy.Handle().WaitAndRetry(new[] + { + TimeSpan.FromMilliseconds(50), + TimeSpan.FromMilliseconds(100), + TimeSpan.FromMilliseconds(150) }); + + var circuitBreakerPolicy = Policy.Handle().CircuitBreaker(1, + TimeSpan.FromMilliseconds(500)); + + var retryPolicyAsync = Policy.Handle() + .WaitAndRetryAsync(new[] { TimeSpan.FromMilliseconds(50), TimeSpan.FromMilliseconds(100), TimeSpan.FromMilliseconds(150) }); + + var circuitBreakerPolicyAsync = Policy.Handle().CircuitBreakerAsync(1, TimeSpan.FromMilliseconds(500)); + + var policyRegistry = new PolicyRegistry() + { + { "SyncRetryPolicy", retryPolicy }, + { "SyncCircuitBreakerPolicy", circuitBreakerPolicy }, + { "AsyncRetryPolicy", retryPolicyAsync }, + { "AsyncCircuitBreakerPolicy", circuitBreakerPolicyAsync } + }; + +``` + +And you can use them in you own handler like this: + +``` csharp +internal class MyQoSProtectedHandler : RequestHandler +{ + static MyQoSProtectedHandler() + { + ReceivedCommand = false; + } + + [UsePolicy(policy: "SyncRetryPolicy", step: 1)] + public override MyCommand Handle(MyCommand command) + { + /*Do work that could throw error because of distributed computing reliability*/ + } +} +``` + +See the section [Policy Retry and Circuit Breaker](/contents/PolicyRetryAndCircuitBreaker.md) for more on using Polly policies with handlers. + +With the Polly Policy Registry filled, you need to tell Brighter where to find the Policy Registry: + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(options => + options.PolicyRegistry = policyRegistry + ) +} + +``` + +#### **Configuring Lifetimes** + +Brighter can register your *Request Handlers* and *Message Mappers* for you (see [IBrighter Builder Fluent Interface](#ibrighterbuilder-fluent-interface)). When we register types for you with ServiceCollection, we need to register them with a given lifetime (see [Dependency Injection Service Lifetimes](https://docs.microsoft.com/en-us/dotnet/core/extensions/dependency-injection#service-lifetimes)). + +We also allow you to set the lifetime for the CommandProcessor. + +We recommend the following lifetimes: + +* If you are using *Scoped* lifetimes, for example with EF Core, make your *Request Handlers* and your *Command Processor* Scoped as well. +* If you are not using *Scoped* lifetimes you can use *Transient* lifetimes for *Request Handlers* and a *Singleton* lifetime for the *Command Processor*. +* Your *Message Mappers* should not have state and can be *Singletons*. + +(Be cautious about using *Singleton* lifetimes for *Request Handlers*. Even if your *Request Handler* is stateless today, and so does not risk carrying state across requests, a common bug is that state is added to an existing *Request Handler* which has previously been registered as a *Singleton*.) + +You configure the lifetimes for the different types that Brighter can create at run-time as follows: + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(options => + options.HandlerLifetime = ServiceLifetime.Scoped; + options.CommandProcessorLifetime = ServiceLifetime.Scoped; + options.MapperLifetime = ServiceLifetime.Singleton; + ); +} + +``` + +### **Brighter Builder Fluent Interface** + +#### **Type Registration** +The **IBrighterBuilder** fluent interface can scan your assemblies for your *Request Handlers* (inherit from **IHandleRequests<>** or **IHandleRequestsAsync<>**) and *Message Mappers* (inherit from **IAmAMessageMapper<>**) and register then with the **ServiceCollection**. This is the most common way to register your code. + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .AutoFromAssemblies(); +} + +``` + +The code scans any loaded assemblies. If you need to register types from assemblies that are not yet loaded, you can provide a list of additional assemblies to scan as an argument to the call to **AutoFromAssemblies()**. + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .AutoFromAssemblies(typeof(MyRequestHandlerAsync).Assembly); +} + +``` + +Instead of using **AutoFromAssemblies** you can exert more fine-grained control over the registration, by explicitly registering your *Request Handlers* and *Message Mappers*. We don't recommend this, but make it available for cases where the automatic registration does not meet your needs. + +* **MapperRegistryFromAssemblies()**, **HandlersFromAssemblies()** and **AsyncHandlersFromAssemblies** are the methods called by **AutoFromAssemblies()** and can be called explicitly. +* **Handlers()**, **AsyncHandlers()** and **MapperRegistry()** accept an **Action<>** delegate that respectively provide you with **IAmASubscriberRegistry** or **IAmAnAsyncSubscriberRegistry** to register your RequestHandlers explicitly or a **ServiceCollectionMapperRegistry** to register your mappers. This gives you explicit control over what you register. + +#### **Using an External Bus** + +Using an *External Bus* allows you to send messages between processes using a message-oriented middleware transport (such as RabbitMQ or Kafka). (For symmetry, we refer to the usage of the *Command Processor* without an external bus as using an *Internal Bus*). + +When raising a message on the *Internal Bus*, you use one of the following methods on the *Command Processor*: + +* **Send()** and **SendAsync()** - Sends a *Command* to one *Request Handler*. +* **Publish()** and **PublishAsync()** - Broadcasts an *Event* to zero or more *Request Handlers*. + +When raising a message on an *External Bus*, you use the following methods on the *CommandProcessor*: + +* **Post()** and **PostAsync()** - Immediately posts a *Command* or *Event* to another process via the external Bus +* **DepositPost()** and **DepositPostAsync()** - Puts on or many *Command*(s) or *Event*(s) in the *Outbox* for later delivery +* **ClearOutbox()** and **ClearOutboxAsync()** - Clears the *Outbox*, posting un-dispatched messages to another process via the *External Bus*. +* **ClearAsyncOutbox()** - Implicitly clears the **Outbox**, similar to above however allows bulk dispatching of messages onto a **Transport**. + +The major difference here is whether or not you wish to use an *Outbox* for Transactional Messaging. (See [Outbox Pattern](/contents/OutboxPattern.md) and [Brighter Outbox Support](/contents/BrighterOutboxSupport.md) for more on Brighter and the Outbox Pattern). + +To use an *External Bus*, you need to supply Brighter with configuration information that tells Brighter what middleware you are using and how to find it. (You don't need to do anything to configure an *Internal Bus*, it is always available.) + +In order to provide Brighter with this information we need to provide it with an implementation of **IAmAProducerRegistry** for the middleware you intend to use for the *External Bus*. + +#### **Transports and Gateways** + +*Transports* are how Brighter supports specific Message-Oriented-Middleware (MoM). *Transports* are provided in separate NuGet packages so that you can take a dependency only on the transport that you need. Brighter supports a number of different *transports*. + +A *Gateway Connection* is how you configure connection to MoM within a *transport*. As an example, the *Gateway Connection* **RMqGatewayConnection** is used to connect to RabbitMQ. Internally the *Gateway Connection* is used to create a *Gateway* object which wraps the client SDK for the MoM. + +We go into more depth on the fields you set here in sections dealing with specific transports. + +#### **Publications** + +A *Publication* configures a transport for sending a message to it's associated MoM. So an **RmqPublication** configures how we publish a message to RabbitMQ. There are a number of common properties to all publications. + +* **MakeChannels**: Do you want Brighter to create the infrastructure? Brighter can create infrastructure that it needs, and is aware of: **OnMissingChannel.Create**. So a publication can create the topic to send messages to. Alternatively if you create the channel by another method, such as IaaC, we can verify the infrastructure on startup: **OnMissingChannel.Validate**. Finally, you can avoid the performance cost of runtime checks by assuming your infrastructure exists: **OnMissingChannel.Assume**. +* **MaxOutstandingMessages**: How large can the number of messages in the Outbox grow before we stop allowing new messages to be published and raise an **OutboxLimitReachedException**. +* **MaxOutStandingCheckIntervalMilliSeconds**: How often do we check to see if the Outbox is full. +* **Topic**: A Topic is the key used within the MoM to route messages. Publishers publish to a topic and subscribers, subscribe to it. We use a class **RoutingKey** to encapsulate the identifier used for a topic. The name the MoM uses for a topic may vary. Kafka & SNS use *topic* whilst RMQ uses *routingkey* + +#### **Transport NuGet Packages** + +We use the naming convention **Paramore.Brighter.MessagingGateway.{TRANSPORT}** for *transports* where {TRANSPORT} is the name of the middleware. + +In this example we will show using an implementation of **IAmAProducerRegistry** for RabbitMQ, provided by the NuGet package: + +* **Paramore.Brighter.MessagingGateway.RMQ** + +See the documentation for detail on specific *transports* on how to configure them for use with Brighter, for now it is enough to know that you need to provide a *Messaging Gateway* which tells us how to reach the middleware and a *Publication* which tells us how to configure the middleware. + +*Transports* provide an **IAmAProducerRegistryFactory()** to allow you to create multiple *Publications* connected to the same middleware. + +#### Retry and Circuit Breaker with an External Bus + +When posting a request to the External Bus we use a Polly policy internally to control Retry and Circuit Breaker in case the External Bus is not available. These policies have defaults but you can configure the behavior using the policy keys: + +* **Paramore.RETRYPOLICY** +* **Paramore.CIRCUITBREAKER** + +#### **Bus Example** + +Putting this together, an example configuration for an External Bus for a local RabbitMQ instance could look like this: + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .UseExternalBus(new RmqProducerRegistryFactory( + new RmqMessagingGatewayConnection + { + AmpqUri = new AmqpUriSpecification(new Uri("amqp://guest:guest@localhost:5672")), + Exchange = new Exchange("paramore.brighter.exchange"), + }, + new RmqPublication[]{ + new RmqPublication + { + Topic = new RoutingKey("GreetingMade"), + MaxOutStandingMessages = 5, + MaxOutStandingCheckIntervalMilliSeconds = 500, + WaitForConfirmsTimeOutInMilliseconds = 1000, + MakeChannels = OnMissingChannel.Create + }} + ).Create() + ) + + ... +} +``` + +#### **Outbox Support** + +TODO: V10 Changes + +If you intend to use Brighter's *Outbox* support for Transactional Messaging then you need to provide us with details of your *Outbox*. + +Brighter provides a number of *Outbox* implementations for common Dbs (and you can write your own for a Db that we do not support). For this discussion we will look at Brighter's support for working with EF Core. See the documentation for working with specific *Outbox* implementations. + +EF Core supports a number of databases and you should pick the packages that match the Dy you want to use with EF Core. In this case we will choose MySQL. + +For this we will need the *Outbox* packages for the MySQL *Outbox*. + +* **Paramore.Brighter.MySql** +* **Paramore.Brighter.Outbox.MySql** + +For a given backing store the pattern should be Paramore.Brighter.{DATABASE} and Paramore.Brighter.Outbox.{DATABASE} where {DATABASE} is the name of the Db that you are using. + +In addition for an ORM you will need to add the package that supports the ORM, in this case EF Core: + +* **Paramore.Brighter.MySql.EntityFrameworkCore** + +For a given ORM the pattern should be Paramore.Brighter.{ORM}.{DATABASE} where {ORM} is the ORM you are choosing and {DATABASE} is the Db you are using with the ORM. + +To configure our *Outbox* we then need to use the Use{DATABASE}Outbox method call, where {DATABASE} is the {DATABASE} that we want, passing in the configuration for our Db so that we can access it. In our case this will be **UseMySqlOutbox()**. + +As we want to use an ORM, in our case EF Core, we have to tell the Outbox how to access EF Core transactions - as we need to participate in a transaction with the ORM. We call a method for the Db, Use{DATABASE}TransactionConnectionProvider, where {DATABASE} is our Db, so in our case **UseMySqlTransactionConnectionProvider()**. + +As a parameter to Use{DATABASE}TransactionConnectionProvider we need to provide a *Transaction Provider* for the ORM we are using, in our case this is *MySqlEntityFrameworkConnectionProvider<>). + +Finally, if we want the *Outbox* to use a background thread to clear un-dispatched items from the *Outbox*, and we do in most circumstances, otherwise they will not be dispatched, we need to run an *Outbox Sweeper* to do this work. + +To add the *Outbox Sweeper* you will need to take a dependency on another NuGet package: + +* **Paramore.Brighter.Extensions.Hosting** + +This results in: + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .UseExternalBus(...) + .UseMySqlOutbox(new MySqlConfiguration(DbConnectionString(), _outBoxTableName), typeof(MySqlConnectionProvider), ServiceLifetime.Singleton) + .UseMySqTransactionConnectionProvider(typeof(MySqlEntityFrameworkConnectionProvider), ServiceLifetime.Scoped) + .UseOutboxSweeper() + + ... +} + +``` + +(**UseExternalBus()** has optional parameters for use with Request-Reply support for some transports. We don't cover that here, instead see [Direct Messaging](/contents/Routing.md#direct-messaging) for more). + +#### **Configuring JSON Serialization** + +Brighter defines a set of serialization options for use when it needs to serialize messages to JSON. Internally we use these options in our transports, when serializing messages to an external bus and deserializing from an external bus. You may wish to use these options in your own [*Message Mapper*](/contents/MessageMappers.md) implementation. + +By default our JSONSerialization Options are configured as follows: + +``` csharp +static JsonSerialisationOptions() +{ + var opts = new JsonSerializerOptions + { + PropertyNameCaseInsensitive = true, + PropertyNamingPolicy = JsonNamingPolicy.CamelCase, + NumberHandling = JsonNumberHandling.AllowReadingFromString, + AllowTrailingCommas = true + }; + + opts.Converters.Add(new JsonStringConverter()); + opts.Converters.Add(new DictionaryStringObjectJsonConverter()); + opts.Converters.Add(new ObjectToInferredTypesConverter()); + opts.Converters.Add(new JsonStringEnumConverter()); + + Options = opts; +} +``` + +You can use the **IBrighterBuilder** extension **ConfigureJsonSerialisation** to override these values. The method takes an **Action\** lambda expression that allows you to override these defaults. For example: + +```csharp + +.ConfigureJsonSerialisation((options) => +{ + options.PropertyNameCaseInsensitive = true; +}) + +``` + +If you want to use this configured set of JSON Serialization options in your own code, you can, by using the static property JsonSerialisationOptions.Options. For example: + +```csharp +public GreetingMade MapToRequest(Message message) +{ + return JsonSerializer.Deserialize(message.Body.Value, JsonSerialisationOptions.Options); +} +``` + +### **Putting It All Together** + +Putting all this together, a typical configuration might looks as follows: + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(options => + { + options.HandlerLifetime = ServiceLifetime.Scoped; + options.MapperLifetime = ServiceLifetime.Singleton; + options.PolicyRegistry = policyRegistry; + }) + .ConfigureJsonSerialisation((options) => + { + options.PropertyNameCaseInsensitive = true; + }) + .UseExternalBus(new RmqProducerRegistryFactory( + new RmqMessagingGatewayConnection + { + AmpqUri = new AmqpUriSpecification(new Uri("amqp://guest:guest@rabbitmq:5672")), + Exchange = new Exchange("paramore.brighter.exchange"), + }, + new RmqPublication[] { + new RmqPublication + { + Topic = new RoutingKey("GreetingMade"), + MaxOutStandingMessages = 5, + MaxOutStandingCheckIntervalMilliSeconds = 500, + WaitForConfirmsTimeOutInMilliseconds = 1000, + MakeChannels = OnMissingChannel.Create + }} + ).Create() + ) + .UseMySqlOutbox(new MySqlConfiguration(DbConnectionString(), _outBoxTableName), typeof(MySqlConnectionProvider), ServiceLifetime.Singleton) + .UseMySqTransactionConnectionProvider(typeof(MySqlEntityFrameworkConnectionProvider), ServiceLifetime.Scoped) + .UseOutboxSweeper() + .AutoFromAssemblies(); +} + +``` + +## **Configuring The Service Activator** + +A *consumer* reads messages from Message-Oriented Middleware (MoM), and a *producer* puts messages onto the MoM for the *consumer* to read. + +A *consumer* waits for messages to appear on the queue, reads them, and then calls your *Request Handler* code to react. Because the •consumer* runs your code in response to an external request, a message being placed on the External Bus, we call the component that listens for messages and dispatches them a [*Service Activator*](https://www.enterpriseintegrationpatterns.com/patterns/messaging/MessagingAdapter.html) + +To use Brighter's Service Activator you will need to take a dependency on the NuGet package: + +* **Paramore.Brighter.ServiceActivator** + +### **ServiceActivator Service Collection Extensions** + +We provide support for configuring .NET Core's **HostBuilder** as a *ServiceActivator* for use with MoM. We use Brighter's Command Processor to dispatch the messages read by a *Dipatcher*. If you are not using **HostBuilder** then you will need to configure the Dispatcher yourself. See [How Configuring the Dispatcher Works](/contents/HowConfiguringTheDispatcherWorks.md) for more. + +To use Brighter's *Service Activator* with **HostBuilder** you will need to take a dependency on the following NuGet packages: + +* **Paramore.Brighter.ServiceActivator.Extensions.Hosting** +* **Paramore.Brighter.ServiceActivator.Extensions.DependencyInjection** + +These provide an extension method **AddServiceActivator()** that can be used to add Brighter to the .NET Core DI Framework. + +By adding the package you can call the **AddServiceActivator()** extension method. + +If you are using a **HostBuilder** class's **ConfigureServices** method call the following: + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + services.AddServiceActivator(...) + } + +``` + +if you are using .NET 6 you can make the call direction on your **HostBuilder**'s Services property. + +The **AddServiceActivator()** method takes an **`Action`** delegate. The extension method supplies the delegate with a **ServiceActivatorOptions** object that allows you to configure how Brighter runs. + +The **AddServiceActivator()** method returns an **IBrighterBuilder** interface. **IBrighterBuilder** is a [fluent interface](https://en.wikipedia.org/wiki/Fluent_interface) that you can use to configure Brighter *Command Processor* properties. It is discussed above at [Brighter Builder Fluent Interface](#brighter-builder-fluent-interface)) and the same options apply. We discuss one additional option that becomes important when receiving requests the *Inbox* in [Additional Brighter Builder Options](/contents/BasicConfiguration.md#the-inbox). + +#### **Subscriptions** + +When configuring your application's *Service Activator*, your *Subscriptions* indicate configure how your application will receive messages from the associated MoM queues or streams. + +All *Subscriptions* lets you configure the following common properties. + +* **Buffer Size**: The number of messages to hold in memory. Where the buffer is not shared, a single thread or Performer can access these; where the buffer is shared, multiple threads can access the same buffer of work. Work in a buffer is locked on queue based middleware, and thus not available to other consumers (threads or process depending if the buffer is shared or not) until *Acknowledged* or *Rejected*. +* **Channel Factory**: Creates or finds the necessary infrastructure for messaging on the MoM and wraps it in an object. +* **Channel *Name**: If queues are primitives in the MoM this names the queue, otherwise just used for diagnostics. +* **Channel Failure Delay**: How long should we delay if a channel fails before trying again, to give problems time to clear. +* **Data Type**: We use a [Datatype Channel](https://www.enterpriseintegrationpatterns.com/DatatypeChannel.html). What is the type of this channel? +* **Empty Channel Delay**: If there are no messages in the queue or stream when we read, how long should we pause before reading again? +* **MakeChannels**: Do you want Brighter to create the infrastructure? Brighter can create infrastructure that it needs, and is aware of: **OnMissingChannel.Create**. So a subscription can create the topic to send messages to, and any subscription to that topic required by the MoM, including a queue (which uses the *Channel Name*). Alternatively if you create the channel by another method, such as IaaC, we can verify the infrastructure on startup: **OnMissingChannel.Validate**. Finally, you can avoid the performance cost of runtime checks by assuming your infrastructure exists: **OnMissingChannel.Assume**. +* **Name**: What do we call this subscription for diagnostic purposes. +* **NoOfPerformers**: Effectively, how many threads do we use to read messages from the queue. As Brighter uses a Single-Threaded Apartment model, each thread has it's own message pump and is thus an in-process implementation of the [Competing Consumers](https://www.enterpriseintegrationpatterns.com/CompetingConsumers.html) pattern. +* **RequeueCount**: How many times can you retry a message before we declare it a poison pill message? +* **RequeueDelayInMilliseconds**: When we requeue a message how long should we delay it by? +* **RoutingKey**: The identifier used to routed messages to subscribers on MoM. You publish to this, and subscriber from this. This has different names; in Kafka or SNS this is a Topic, in RMQ this is the routing key. +* **RunAsync**: Is this an async pipeline? Your pipeline must be sync or async. An async pipeline can increase throughput where a handler is I/O bound by allowing the message pump to read another message whilst we await I/O completion. The cost of this is that strict ordering of messages will now be lost as processing of I/O bound requests may complete out-of-sequence. Brighter provides its own synchronization context for async operations. We recommend scaling via increasing the number of performers, unless you know that I/O is your bottleneck. +* **TimeoutInMilliseconds**: How long does a read 'wait' before assuming there are no pending messages. +* **UnaceptableMessageLimit**: Brighter will ack a message that throws an unhandled exception, thus removing it from a queue. + +For a more detailed discussion of using Requeue (with Delay) for Handler failure, (**RequeueCount** and **RequeueDelayInMilliseconds**) along with termination of a consumer due to message failure (**UnacceptableMessageLimit**) see [Handler Failure](/contents/HandlerFailure.md) + +In addition, individual transports that provide access to specific MoM sub-class *Subscription* to provide properties unique to the chosen middleware. We discuss those under a section for that transport. + +For RabbitMQ for example, this would look like this: + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + var subscriptions = new Subscription[] + { + new RmqSubscription( + new SubscriptionName("paramore.sample.salutationanalytics"), + new ChannelName("SalutationAnalytics"), + new RoutingKey("GreetingMade"), + runAsync: true, + timeoutInMilliseconds: 200, + isDurable: true, + makeChannels: OnMissingChannel.Create), //change to OnMissingChannel.Validate if you have infrastructure declared elsewhere + }; + + services.AddServiceActivator(options => + { + options.Subscriptions = subscriptions; + }) +} + +... + +``` + +#### **Gateway Connections & Channel Factories** + +A *Gateway Connection* tells Brighter how to connect to MoM for a particular transport. The transport package will contain a *Gateway Connection*, you need to provide the information to connect to your middleware (URIs, ports, credentials etc.) Your transport package provides a *Gateway Connection* + +A *Channel Factory* connects Brighter to MoM. Depending on the configuration settings for your *Subscription* it may create the required primitives (topics/routing keys, queues, streams) on MoM or simply attach to ones that you have created via Infrastructure as Code (IaC). Your transport provides a *Channel Factory* and you need to pass it a *Gateway Connection*. + +For RabbitMQ, this would look like: + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + + var rmqConnection = new RmqMessagingGatewayConnection + { + AmpqUri = new AmqpUriSpecification(new Uri($"amqp://guest:guest@local:5672")), + Exchange = new Exchange("paramore.brighter.exchange") + }; + + var rmqMessageConsumerFactory = new RmqMessageConsumerFactory(rmqConnection); + + services.AddServiceActivator(options => + { + options.ChannelFactory = new ChannelFactory(rmqMessageConsumerFactory); + }) +} + +... + +``` + +#### **Configuring Service Activator Lifetimes** + +Under the hood your *Service Activator* uses a *Command Processor* and you will need to configure lifetimes [as discussed above](#configuring-lifetimes). + +An additional requirement is configuring the lifetime of the *Command Processor* itself. Within the context of an ASP.NET application, configuring the lifetime of the **Command Processor** relies on ASP.NET creating an instance of the *Command Processor* in a request pipeline. When you are using *Service Activator* there is no ASP.NET pipeline, instead Brighter's *Dispatcher* manages the lifetime of the *Command Processor* that we pass a request to. By setting the **ServiceActivatorOptions.UseScoped** field to true, you instruct *Brighter* to use a new *Command Processor* instance for each request. This is important if you take the *Command Processor* as a dependency in any of your *Request Handlers* with a **Scoped** lifetime. If in doubt, just set **ServiceActivatorOptions.UseScoped** field to true. + + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + services.AddServiceActivator(options => + { + options.UseScoped = true; + options.HandlerLifetime = ServiceLifetime.Scoped; + options.MapperLifetime = ServiceLifetime.Singleton; + options.CommandProcessorLifetime = ServiceLifetime.Scoped; + }) +} + +... + +``` + + +### **Service Activator Brighter Builder Fluent Interface** + +The call to **AddServiceActivator()** returns an **IBrighterBuilder** fluent interface. This means that you can use any of the options described in [Brighter Build Fluent Interfaces](#brighter-builder-fluent-interface) to configure the associated *Command Processor* such as scanning assemblies for *Request Handlers* and adding an *External Bus* and *Outbox*. + +An option is intended for the context of a Service Activator is described below. + +#### **Inbox** + +As described in the [Outbox Pattern](/contents/OutboxPattern.md) an *Outbox* offers **Guaranteed, At Least Once** delivery. It explicitly may result in you sending duplicate messages. In addition, MoM tends to offer "At Least Once" guarantees only, further creating the risk that you will receive a duplicate message. + +If the request is not idempotent, you can use an Inbox to de-duplicate it. See [Inbox Support](/contents/BrighterInboxSupport.md) for more. + +Configuring an *Inbox* has two elements. The first is the type of *Inbox*, the second configuration for the *Inbox* behavior. + +Brighter provides a number of *Inbox* implementations for common Dbs (and you can write your own for a Db that we do not support). For this discussion we will look at Brighter's support for working with MySQL. See the documentation for working with specific *Inbox* implementations. + +For this we will need the *Inbox* packages for the MySQL *Inbox*. + +* **Paramore.Brighter.Inbox.MySql** + +For a given backing store the pattern should be Paramore.Brighter.Inbox.{DATABASE} where {DATABASE} is the name of the Db that you are using. + +To configure our *Inbox* we then need to use the UseExternalInbox method call and pass in an instance of a class that implements **IAmAnInbox**, taken from our package, and an instance of **InboxConfiguration** that tells Brighter how we want to use the Inbox. + +For *Inbox Configuration* you set the following properties: + +* **ActionOnExists**: What do we do if the request has been handled? The default,**OnceOnlyAction.Throw** is to throw a **OnceOnlyException**. If you take no other action this will cause the message to be rejected and sent to a DLQ if one is configured (See [Handler Failure](/contents/HandlerFailure.md)). The alternative is **OnceOnlyAction.Warn** simply logs that the request is a duplicate, but takes no other action. +* **OnceOnly**: This defaults to *true* and will check for a duplicate and take the action indicated by **ActionOnExists**. If *false* the *Inbox* will record the request, but will take no further action. (This tends to be set to *false* if you are using the *Inbox* to record what requests caused current state only and not de-duplicate). +* **Scope**: This indicates the type of request (*Command* or *Event*) to store in the *Inbox*. By default this is set to **InboxScope.All** and captures everything but you can be explicit and just capture **InboxScope.Commands** or **InboxScope.Events**. (This tends to be set to **InboxScope.Commands** when only commands cause changes to state that are not idempotent). +* **Context**: Used to uniquely identify receipt of this request via this handler. If you are recording *Events* and have multiple handlers, then the first event handler to receive the message will block the others from doing so, unless you disambiguate the handler identity by supplying a context method. + +A typical *Inbox* configuration for MySQL would be: + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + services.AddServiceActivator(options => + { ... }) + .UseExternalInbox( + new MySqlInbox(new MySqlInboxConfiguration("server=localhost; port=3306; uid=root; pwd=root; database=Salutations", "Inbox"); + new InboxConfiguration( + scope: InboxScope.Commands, + onceOnly: true, + actionOnExists: OnceOnlyAction.Throw + ) + ); +} + +... + +``` +Typically you would obtain the connection string for the Db from configuration (as opposed to hard coding the string), likewise for the table name for your *Inbox*. + + +### Running Service Activator + +To run *Service Activator* we add it as a [Hosted Service](https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-6.0&tabs=visual-studio). + +We provide the class **ServiceActivatorHostedService** for this in the NuGet package: + +* **Paramore.Brighter.ServiceActivator.Extensions.Hosting** + +The **ServiceActivatorHostedService** calls the **Dispatcher.Receive** method which starts message pumps for the configured *Subscriptions*. + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + ... + + services.AddHostedService(); +} + +... + +``` + +On shutdown Brighter will allow the current *Request Handler* to complete, then end the message pump loop and exit. If you have long-running handlers it is possible that they will not complete in the default 5s for graceful shutdown of the MS Generic Host. In this case, you need to [increase the timeout](https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/generic-host?view=aspnetcore-6.0#shutdowntimeout) of the host shutdown. + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + services.Configure(options => + { + options.ShutdownTimeout = TimeSpan.FromSeconds(20); + }); + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + ... + + services.AddHostedService(); +} + +``` + +### A Complete Service Activator Example + +When all of the relevant configuration sections are added together, your code will look something like this, with variations for your transport and stores. + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + services.Configure(options => + { + options.ShutdownTimeout = TimeSpan.FromSeconds(20); + }); + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + var subscriptions = new Subscription[] + { + new RmqSubscription( + new SubscriptionName("paramore.sample.salutationanalytics"), + new ChannelName("SalutationAnalytics"), + new RoutingKey("GreetingMade"), + runAsync: true, + timeoutInMilliseconds: 200, + isDurable: true, + makeChannels: OnMissingChannel.Create), //change to OnMissingChannel.Validate if you have infrastructure declared elsewhere + }; + + var rmqConnection = new RmqMessagingGatewayConnection + { + AmpqUri = new AmqpUriSpecification(new Uri($"amqp://guest:guest@localhost:5672")), + Exchange = new Exchange("paramore.brighter.exchange") + }; + + var rmqMessageConsumerFactory = new RmqMessageConsumerFactory(rmqConnection); + + services.AddServiceActivator(options => + { + options.Subscriptions = subscriptions; + options.ChannelFactory = new ChannelFactory(rmqMessageConsumerFactory); + options.UseScoped = true; + options.HandlerLifetime = ServiceLifetime.Scoped; + options.MapperLifetime = ServiceLifetime.Singleton; + options.CommandProcessorLifetime = ServiceLifetime.Scoped; + }) + .AutoFromAssemblies() + .UseExternalInbox( + new MySqlInbox(new MySqlInboxConfiguration("server=localhost; port=3306; uid=root; pwd=root; database=Salutations", "Inbox"); + new InboxConfiguration( + scope: InboxScope.Commands, + onceOnly: true, + actionOnExists: OnceOnlyAction.Throw + ) + ) + + services.AddHostedService(); + +} + +``` + +## Samples + +Brighter includes a comprehensive set of [Samples](https://github.com/BrighterCommand/Brighter/tree/master/samples) in its main repo that you can review for clarity on how Brighter works and should be configured. + + diff --git a/source/9/.toc.yaml b/source/9/.toc.yaml new file mode 100644 index 0000000..e69de29 diff --git a/source/README.md b/source/README.md new file mode 100644 index 0000000..4d8b12d --- /dev/null +++ b/source/README.md @@ -0,0 +1,5 @@ +# Brighter and Darker Documentation + +Documentation for the [Brighter](https://github.com/BrighterCommand/Brighter) and [Darker](https://github.com/BrighterCommand/Darker) projects. + +The documentation can be used as is, by cloning this [repository](https://github.com/BrighterCommand/Docs) and viewing the markdown files, however for convenience the documentation is made available via [GitBook](https://brightercommand.gitbook.io/paramore-brighter-documentation/). diff --git a/source/shared/.toc.yaml b/source/shared/.toc.yaml new file mode 100644 index 0000000..a0c0409 --- /dev/null +++ b/source/shared/.toc.yaml @@ -0,0 +1,201 @@ +--- +Sections: + Overview: + order: 10 + entries: + - name : Show me the code! + file : howMeTheCode.md + order : 10 + - name : Basic Concepts + file : BasicConcepts.md + order : 20 + + Brighter Configuration: + order: 20 + entries: + - name : Basic Configuration + file : BrighterBasicConfiguration.md + order : 10 + - name : How Configuring the Command Processor Works + file : HowConfiguringTheCommandProcessorWorks.md + order : 20 + - name : How Configuring a Dispatcher for an External Bus Works + file : HowConfiguringTheDispatcherWorks.md + order : 30 + - name : RabbitMQ Configuration + file : RabbitMQConfiguration.md + order : 40 + - name : AWS SNS Configuration + file : AWSSQSConfiguration.md + order : 50 + - name : Kafka Configuration + file : KafkaConfiguration.md + order : 60 + - name : Azure Service Bus Configuration + file: AzureServiceBusConfiguration.md + order : 70 + - name : Azure Archive Provider Configuration + order : 80 + + Darker Configuration: + order: 30 + entries: + - name : Basic Configuration + file : DarkerBasicConfiguration.md + order : 10 + + Brighter Request Handlers and Middleware Pipelines: + order: 40 + entries: + - name : Basic Configuration + file : DarkerBasicConfiguration.md + order : 10 + - name : How to Implement an Async Request Handler + file : ImplementingAsyncHandler.md + order : 20 + - name : Requests, Commands and an Events + file : Requests%2C%20Commands%20and%20Events.md + order : 30 + - name : Dispatching Requests + file : DispatchingARequest.md + order : 40 + - name : Dispatching An Async Request + file : AsyncDispatchARequest.md + order : 50 + - name : Returning results from a Handler + file : ReturningResultsFromAHandler.md + order : 60 + - name : Using an External Bus + file : ImplementingExternalBus.md + order : 70 + - name : Message Mappers + file : MessageMappers.md + order : 80 + - name : Routing + file : Routing.md + order : 90 + - name : Building a Pipeline of Request Handlers + file : BuildingAPipeline.md + order : 100 + - name : Building an Async Pipeline of Request Handlers + file : BuildingAnAsyncPipeline.md + - name : Passing information between Handlers in the Pipeline + file : UsingTheContextBag.md + order : 110 + - name : Failure and Dead Letter Queues + file : HandlerFailure.md + order : 120 + - name : Supporting Retry and Circuit Breaker + file : PolicyRetryAndCircuitBreaker.md + order : 130 + - name : Failure and Fallback + file : PolicyFallback.md + order : 140 + - name : Feature Switches + file : FeatureSwitches.md + order : 150 + + Guaranteed At Least Once: + order: 50 + entries: + - name : Outbox Support + file : BrighterOutboxSupport.md + order : 10 + - name : Inbox Support + file : BrighterInboxSupport.md + order : 20 + - name: EFCore Outbox + file: EFCoreOutbox.md + order : 30 + - name : Dapper Outbox + file : DapperOutbox.md + order : 40 + - name : Dynamo Outbox + file : DynamoOutbox.md + order : 50 + - name : MSSQL Inbox + file : MSSQLInbox.md + order : 60 + - name : MySQL Inbox + file : MySQLInbox.md + order : 70 + - name : Postgres Inbox + file : PostgresInbox.md + order : 80 + - name : Sqlite Inbox + file : SqliteInbox.md + order : 90 + - name : Dynamo Inbox + file : DynamoInbox.md + order : 100 + + Darker Query Handlers and Middleware Pipelines: + order: 60 + entries: + - name : How to Implement a Query Handler + file : ImplementAQueryHandler.md + order : 10 + + Health Checks and Observability: + order: 70 + entries: + - name : Logging + file : Logging.md + order : 10 + - name : Monitoring + file : Monitoring.md + order : 20 + - name : Health Checks + file : HealthChecks.md + order : 30 + - name : Telemetry + file : Telemetry.md + order : 40 + + Command, Processors and Dispatchers: + order: 80 + entries: + - name : Command, Processor and Dispatcher Patterns + file : CommandsCommandDispatcherandProcessor.md + order : 10 + + Under the Hood: + order: 90 + entries: + - name : How The Command Processor Works + file : HowBrighterWorks.md + order : 10 + - name : How Service Activator Works + file : HowServiceActivatorWorks.md + order : 20 + + Event Driven Architectures: + order: 100 + entries: + - name : Microservices + file : Microservices.md + order : 10 + - name : Event Driven Collaboration + file : EventDrivenCollaboration.md + order : 20 + - name : Event Carried State Transfer + file : EventCarriedStateTransfer.md + order : 30 + - name : Outbox Pattern + file : OutboxPattern.md + order : 40 + + Task Queues: + order: 110 + entries: + - name : Using a Task Queue + file : TaskQueuePattern.md + order : 10 + + FAQ: + order: 120 + entries: + - name : FAQ + file : FAQ.md + order : 1 +... \ No newline at end of file diff --git a/source/shared/AWSSQSConfiguration.md b/source/shared/AWSSQSConfiguration.md new file mode 100644 index 0000000..05f99bc --- /dev/null +++ b/source/shared/AWSSQSConfiguration.md @@ -0,0 +1,168 @@ +# AWS SQS Configuration + +## General + +SNS and SQS are proprietary message-oriented-middleware available on the AWS platform. Both are well documented: see [SNS](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) and [SQS](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html). Brighter handles the details of sending to SNS using an SQS queue for the consumer. You might find the [documentation for the AWS .NET SDK](https://docs.aws.amazon.com/sdk-for-net/) helpful when debugging, but you should not have to interact with it directly to use Brighter. + +It is useful to understand the relationship between these two components: + +- **SNS**: A routing table, [SNS](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) provides routing for messages to subscribers. Subscribers include, but are not limited to, SQS [see SNS Subscribe Protocol](https://docs.aws.amazon.com/sns/latest/api/API_Subscribe.html). An entry in the table is a **Topic**. +- **SQS**: A store-and-forward queue over which a consumer receives messages. A message is locked whilst a consumer has read it, until they ack it, upon which it is deleted from the queue, or nack it, upon which it is unlocked. A policy controls movement of messages that cannot be delivered to a DLQ. [SQS](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html) may be used for point-to-point integration, and does not require SNS. + +Brighter only supports the scenario where SNS is used as a routing table, and an SQS subscribes to a **topic**. It does not support stand-alone SQS queues. Point-to-point scenarios can be modelled as an SNS **topic** with one subscribing queue. + +## Connection + +The Connection to AWS is provided by an **AWSMessagingGatewayConnection**. This is a wrapper around AWS credentials and region, that allows us to create the .NET clients that abstract various AWS HTTP APIs. We require the following parameters: + +- **Credentials**: An instance of *AWSCredentials*. Storing and retrieving the credentials is a detail for your application and may vary by environment. There is AWS discussion of credentials resolution [here](https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/creds-assign.html) +- **Region**: The *RegionEndpoint* to use. SNS is a regional service, so we need to know which region to provision infrastructure in, or find it from. + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + if (!new CredentialProfileStoreChain().TryGetAWSCredentials("default", out var credentials) + throw InvalidOperationException("Missing AWS Credentials); + + services.AddBrighter(...) + .UseExternalBus(new SnsProducerRegistryFactory + new AwsMessagingGatewayConnection(credentials, Environment.GetEnvironmentVariable("AWS_REGION")) + , + ... //publication, see below + ).Create() +} +``` +## Publication + +For more on a *Publication* see the material on an *External Bus* in [Basic Configuration](/contents/BrighterBasicConfiguration.md#using-an-external-bus). + +Brighter's **Routing Key** represents the [SNS Topic Name](https://docs.aws.amazon.com/sns/latest/api/API_CreateTopic.html). + +### Finding and Creating Topics +Depending on the option you choose for how we handle required messaging infrastructure (Create, Validate, Assume), we will need to determine if a **Topic** already exists, when we want to create it if missing, or validate it. + +Naively using the AWS SDK's **FindTopic** method is an expensive operation. This enumerates all the **Topics** in that region, looking for those that have a matching name. Under-the-hood the client SDK pages through your topics. If you have a significant number of topics, this is expensive and subject to rate limiting. + +As creating a **Topic** is an *idempotent* operation in SNS, if asked to Create we do so without first searching to see if it already exists because of the cost of validation. + +If you create your infrastructure out-of-band, and ask us validate it exists, to mitigate the cost of searching for topics, we provide several options under **FindTopicBy**. + +- **FindTopicBy**: How do we find the topic: + - **TopicFindBy.Arn** -> On a *Publication*, the routing key is the **Topic** name, but you explicitly supply the ARN in another field: **TopicArn**. On a *Subscription* the routing key is the **Topic** ARN. + - **TopicFindBy.Convention** -> The routing key is the **Topic** name, and we use convention to construct the ARN from it + - **TopicFindBy.Name** -> The routing key is the **Topic** name & we use ListTopics to find it (rate limited 30/s) + +#### TopicFindBy.Arn +We use **GetTopicAttributesAsync** SDK method to request attributes of a Topic with the ARN supplied in **TopicArn**. If this call fails with a NotFoundException, we know that the Topic does not exist. This is a *hack*, but is much more efficient than enumeration as a way of determining if the ARN exists. + +#### TopicFindBy.Convention +If you supply only the **Topic** name via the routing key, we construct the ARN by convention as follows: + +``` csharp +var arn = new Arn + { + Partition = //derived from the partition of the region you supplied to us, + Service = "sns", + Region = //derived from the system name of the region you supplied to us, + AccountId = //your account id - derived from the credentials you supplied, + Resource = topicName + } +``` + +These assumptions work, if the topic is created by the account your credentials belong to. If not, you can't use by convention. + +Once we obtain an ARN by convention, we can then use the optimized approach described under [TopicFindBy.Arn](#topicfindbyarn) to confirm that your topic exists. + +#### TopicFindBy.Name +If you supply a name, but we can't construct the ARN via the above conventions, we have to fall back to the **SDKs** **FindTopic** approach. + +Because creation is idempotent, and **FindTopic** is expensive, you are almost always better off choosing to create over validating a topic by name. + +If you are creating the topics out-of-band, by CloudFormation for example, and so do not want Brighter the risk that Brighter will create them, then you will have an ARN. In that case you should use [TopicFindBy.Arn](#topicfindbyarn) or assume that any required infrastructure exists. + +### Other Attributes +- **SNSAttributes**: This property lets you pass through an instance of **SNSAttributes** which has properties representing the attributes used when creating a **Topic**. These are only used if you are creating a **Topic**. + - **DeliveryPolicy**: The policy that defines how Amazon SNS retries failed deliveries to HTTP/S endpoints. + - **Policy**: The policy that defines who can access your topic. By default, only the topic owner can publish or subscribe to the topic. + - **Tags**: A list of resource tags to use. + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + if (!new CredentialProfileStoreChain().TryGetAWSCredentials("default", out var credentials) + throw InvalidOperationException("Missing AWS Credentials); + + services.AddBrighter(...) + .UseExternalBus( + ...,//connection, see above + new SnsPublication[] + { + new SnsPublication() + { + Topic = new RoutingKey("GreetingEvent"), + FindTopicBy = TopicFindBy.Convention + } + } + ).Create() +} +``` + +## Subscription + +As normal with Brighter, we allow **Topic** creation from the *Subscription*. Because this works in the same way as the *Publication* see the notes under [Publication](#publication) for further detail on the options that you can configure around creation or validation. + +In SNS you need to [subscribe](https://docs.aws.amazon.com/sns/latest/api/API_Subscribe.html) to a **Topic** to receive messages from that **Topic**. Brighter subscribes using an SQS queue (there are other options for SNS, but Brighter does not use those). Much of the *Subscription* configuration allows you to control the parameters of that *Subscription*. + +We support the following properties on an *SQS Subscription* most of which relate to the creation of the [SQS Queue](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_CreateQueue.html) with which we subscribe: + +- **LockTimeout**: How long, in seconds, a 'lock' is held on a message for one consumer before it times out(*VisibilityTimeout*). Default is 10s. +- **DelaySeconds**: The length of time, in seconds, for which the delivery of all messages in the queue is delayed. Default is 0. +- **MessageRetentionPeriod**: The length of time, in seconds, for which Amazon SQS retains a message on a queue before deleting it. Default is 4 days. +- **IAMPolicy**: The queue's policy. A valid [AWS policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html). +- **RawMessageDelivery**: Indicate that the Raw Message Delivery setting is enabled or disabled. Defaults to true. +- **RedrivePolicy**: The parameters for the dead-letter queue functionality of the source queue. An instance of the **RedrivePolicy** class, which has the following parameters: + - **MaxReceiveCount**: The maximum number of requeues for a message before we push it to the DLQ instead + - **DeadlLetterQueueName**: The name of the dead letter queue we want to associate with any redrive policy. + + +``` csharp +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + (AWSCredentials credentials, RegionEndpoint region) = CredentialsChain.GetAwsCredentials(); + var awsConnection = new AWSMessagingGatewayConnection(credentials, region); + + var subscriptions = new Subscription[] + { + new SqsSubscription( + name: new SubscriptionName("Subscription-Name), + channelName: new ChannelName("Channel-Name"), + routingKey: new RoutingKey("arn:aws:sns:us-east-2:444455556666:MyTopic"), + findTopicBy: TopicFindBy.Arn, + makeChannels: OnMissingChannel.Validate + ); + } + + var sqsMessageConsumerFactory = new SqsMessageConsumerFactory(awsConnection); + + services.AddServiceActivator(options => + { + options.Subscriptions = subscriptions; + options.ChannelFactory = new ChannelFactory(sqsMessageConsumerFactory); + ... //see Basic Configuration + }) +``` + +### Ack and Nack + +As elsewhere, Brighter only Acks after your handler has run to process the message. We will Ack unless you throw a **DeferMessageAction**. See [Handler Failure](/contents/HandlerFailure.md) for more. + +An Ack will delete the message from the SQS queue using the SDK's **DeleteMessageAsync**. + +In response to a DeferMessageAction we will requeue, using the SDK's **ChangeMessageVisibilityAsync** to make the message available again to other consumers. + +On a Nack, we will move the message to a DLQ, if there is one. We Nack when we exceed the requeue count for a message, or we raise a ConfigurationException. + + + + + diff --git a/source/shared/AsyncDispatchARequest.md b/source/shared/AsyncDispatchARequest.md new file mode 100644 index 0000000..6673712 --- /dev/null +++ b/source/shared/AsyncDispatchARequest.md @@ -0,0 +1,98 @@ +# Dispatching Requests Asynchronously + +Once you have [implemented your Request Handler](ImplementingAHandler.html), you will want to dispatch **Commands** or **Events** to that Handler. + +## Usage + +In the following example code we register a handler, create a command processor, and then use that command processor to send a command to the handler asynchronously. + + +``` csharp + public class Program + { + private static async Task Main(string[] args) + { + var host = Host.CreateDefaultBuilder() + .ConfigureServices((hostContext, services) => + + { + services.AddBrighter() + .AutoFromAssemblies(); + } + ) + .UseConsoleLifetime() + .Build(); + + var commandProcessor = host.Services.GetService(); + + await commandProcessor.SendAsync(new GreetingCommand("Ian")); + + await host.RunAsync(); + } +``` + +## Registering a Handler + +In order for a **Command Dispatcher** to find a Handler for your **Command** or **Event** you need to register the association between that **Command** or **Event** and your Handler. + +Brighter's **HostBuilder** support provides **AutoFromAssemblies** to register any *Request Handlers* in the project. See [Basic Configuration](/contents/BrighterBasicConfiguration.md) for more. + +### Pipelines Must be Homogeneous + +Brighter only supports pipelines that are solely **IHandleRequestsAsync** or **IHandleRequests**. In particular, note that middleware (attributes on your handler) must be of the same type as the rest of your pipeline. A common mistake is to **UsePolicy** when you mean **UsePolicyAsync**. + +## Dispatching Requests + +Once you have registered your Handlers, you can dispatch requests to them. To do that you simply use the **commandProcessor.SendAsync()** (or **commandProcessor.PublishAsync()**) method passing in an instance of your command. *Send* expects one handler, *Publish* expects zero or more. (You can use **commandProcessor.DepositPostAsync** and **commandProcessor.ClearOutboxAsync** with an External Bus). + +``` csharp +await commandProcessor.SendAsync(new GreetingCommand("Ian")); +``` + +### Returning results to the caller. + +A Command does not have return value and **CommandDispatcher.Send()** does not return anything. Please see a discussion on how to handle this in [Returning Results from a Handler](/contents/ReturningResultsFromAHandler.md). + +### Cancellation + +Brighter supports the cancellation of asynchronous operations. + +The asynchronous methods: **SendAsync** and **PublishAsync** accept a **CancellationToken** and pass this token down the pipeline. The parameter defaults to default(CancellationToken) where the call does not intend to cancel. + +The responsibility for checking for a cancellation request lies with the individual handlers, which must determine what action to take if cancellation had been signalled. + +### Async Callback Context + +When an awaited method completes, what thread runs any completion code? There are two options: + +- The original thread that was running when the await began +- A new thread allocated from the thread pool + +Why does this matter? Because if you needed to access anything that is thread local, being called back on a new thread means you will not have access to those variables. + +As a result, when awaiting it is possible to configure how the continuation runs. + +- To run on the original thread, requires the CLR to capture information on the thread you were using. This is the SynchronizationContext; because the CLR must record this information, we refer to it as a captured context. Your execution will be queued back on to the original context, which has a performance cost. +- To run on a new thread, using the Task Scheduler to allocate from the thread pool. + +You can use ConfigureAwait to control this. This article explains why you might wish to use [ConfigureAwait](https://devblogs.microsoft.com/dotnet/configureawait-faq/), in more depth. + +As a library, we need to allow you to make this choice for your handler chain. For this reason, our *Async methods support the parameter **continueOnCapturedContext**. + +Library writers are encouraged to default to false i.e. use the Task Scheduler instead of the SychronizationContext. Brighter adopts this default, but it might not be what you want if your handler needs to run in the context of the original thread. As a result we let you use this parameter on the **\*Async** calls to change the behaviour throughout your pipeline. + +``` csharp +await commandProcessor.SendAsync(new GreetingCommand("Ian"), continueOnCapturedContext: true); +``` + +A handler exposes the parameter you supply via the property **ContinueOnCapturedContext**. + +You should pass this value via **ConfigureAwait** if you need to be able to support making this choice at the call site. For example, when you call the base handler in your return statement, to ensure that the decision as to whether to use the scheduler or the context flows down the pipeline. + +``` csharp +return await base.HandleAsync(command, ct).ConfigureAwait(ContinueOnCapturedContext); +``` + +You can ignore this, if you want to default to using the Task Scheduler. + + diff --git a/source/shared/AzureBlobArchiveProvider.md b/source/shared/AzureBlobArchiveProvider.md new file mode 100644 index 0000000..1fb9810 --- /dev/null +++ b/source/shared/AzureBlobArchiveProvider.md @@ -0,0 +1,40 @@ +# Azure Blob Archive Provider + +## Usage +The Azure Blob Archive Provider is a provider for [Outbox Archiver](/contents/BrighterOutboxSupport.md#outbox-archiver). + +For this we will need the *Archive* packages for the Azure *Archive Provider*. + +* **Paramore.Brighter.Archive.Azure** + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + services.AddBrighter(options => + { ... }) + .UseOutboxArchiver( + new AzureBlobArchiveProvider(new AzureBlobArchiveProviderOptions() + { + BlobContainerUri = "https://brighterarchivertest.blob.core.windows.net/messagearchive", + TokenCredential = New AzCliCredential(); + } + ), + options => { + TimerInterval = 5; // Every 5 seconds + BatchSize = 500; // 500 messages at a time + MinimumAge = 744; // 1 month + } + ); +} + +... + +``` + diff --git a/source/shared/AzureBlobConfiguration.md b/source/shared/AzureBlobConfiguration.md new file mode 100644 index 0000000..3af6ed9 --- /dev/null +++ b/source/shared/AzureBlobConfiguration.md @@ -0,0 +1,24 @@ +# Azure Archive Provider Configuration + +## General +Azure Service Bus (ASB) is a fully managed enterprise message broker and is [well documented](https://docs.microsoft.com/en-us/azure/service-bus-messaging/) Brighter handles the details of sending to or receiving from ASB. You may find it useful to understand the [concepts](https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-queues-topics-subscriptions) of the ASB. + +## Connection +At this time Azure Blob Archive Provider only supports Token Credential for authentication + +## Permissions +For the archiver to work the calling credential will require the role **Storage Blob Data Owner** however if **TagBlobs** is set to False then **Storage Blob Data Contributor** will be adequate. If you feel that Data Owner is too high you can create a custom role encompasing Contributor and 'Microsoft.Storage/storageAccounts/blobServices/containers/blobs/tags/write' + +## Options + +* **BlobContainerUri** : The URI of the Blob container to store messages in (i.e. "https://BlobTest.blob.core.windows.net/messagearchive) +* **TokenCredential** : The Credential to use when writing the Blob +* **AccessTier** : The Access Tier to write to the blob +* **TagBlobs** : if this is set to True the defined in **TagsFunc** will be written to the blobs +* **TagsFunc** : The function to arrange the tags to add when storing, please note that **TagBlobs** must be True for these to be used, default Tags : + - topic + - correlationId + - message_type + - timestamp + - content_type +* **StorageLocationFunc** : The function to provide the location to store the message inside of the Blob container, default location : The Id of the message at the root of the **BlobContainerUri** \ No newline at end of file diff --git a/source/shared/AzureServiceBusConfiguration.md b/source/shared/AzureServiceBusConfiguration.md new file mode 100644 index 0000000..77fadd5 --- /dev/null +++ b/source/shared/AzureServiceBusConfiguration.md @@ -0,0 +1,121 @@ +# Azure Service Bus Configuration + +## General +Azure Service Bus (ASB) is a fully managed enterprise message broker and is [well documented](https://docs.microsoft.com/en-us/azure/service-bus-messaging/) Brighter handles the details of sending to or receiving from ASB. You may find it useful to understand the [concepts](https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-queues-topics-subscriptions) of the ASB. + +## Connection +The connection to ASB id defined by an **IServiceBusClientProvider**, Brighter proviedes the following Implimentations + +* **ServiceBusChainedClientProvider**: A client provider that allows you to specific a chain of **TokenCredentials** to authenticate with. + +* **ServiceBusConnectionStringClientProvider**: A client provider that accepts a connection string (containg Authentication information) + +* **ServiceBusDefaultAzureClientProvider**: A client provider that uses the Default Azure Credential to authenticate. + +* **ServiceBusManagedIdentityClientProvider**: A client provider that uses Azure Managed Identity to authenticate. + +* **ServiceBusVisualStudioCredentialClientProvider**: A client provider that uses Visual Studio Credential to authenticate. + +In Brighter's implementation of the Messaging Gateway *Publications* and *Subscriptions* have their own Individual configuration. + +## Publication + +No custom properties are supported for ASB + +Basic Brighter configutarion publications is as follows + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .UseExternalBus( + new AzureServiceBusProducerRegistryFactory( + asbConnection, + new AzureServiceBusPublication[] + { + new() { Topic = new RoutingKey("greeting.event") }, + new() { Topic = new RoutingKey("greeting.addGreetingCommand") }, + new() { Topic = new RoutingKey("greeting.Asyncevent") } + } + ) + .Create() + ) +} +``` + +For more on a *Publication* see the material on an *External Bus* in [Basic Configuration](/contents/BrighterBasicConfiguration.md#using-an-external-bus). + +## Subscription + +For more on a *Subscription* see the material on configuring *Service Activator* in [Basic Configuration](/contents/BrighterBasicConfiguration.md#configuring-the-service-activator). + +When + +We support a number of ASB specific *Subscription* options: + +* **MaxDeliveryCount**: The Maximum amount of times that a Message can be delivered before it is dead Lettered. This differs from **requeue count** as this is used by the transport in the event of lock expiry (in the event of process failure or processing taking too long) **default:** 5 + +* **DeadLetteringOnMessageExpiration**: Dead letter a message when it expires **default:** true + +* **LockDuration**: How long message locks are held for **default:** true + +* **DefaultMessageTimeToLive**: How long messages sit in the queue before they expire **default:** 1 minute + +* **SqlFilter**: A Sql Filter to apply to the *subscription* see [Topic Filters](https://docs.microsoft.com/en-us/azure/service-bus-messaging/topic-filters) **default:** none + + +This is a typical *Subscription* configuration in a Consumer application: + +``` csharp +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + var subscriptions = new Subscription[] + { + new AzureServiceBusSubscription( + new SubscriptionName(GreetingEventAsyncMessageMapper.Topic), + new ChannelName(subscriptionName), + new RoutingKey(GreetingEventAsyncMessageMapper.Topic), + timeoutInMilliseconds: 400, + makeChannels: OnMissingChannel.Create, + requeueCount: 3, + isAsync: true, + noOfPerformers: 2, unacceptableMessageLimit: 1), + new AzureServiceBusSubscription( + new SubscriptionName(GreetingEventMessageMapper.Topic), + new ChannelName(subscriptionName), + new RoutingKey(GreetingEventMessageMapper.Topic), + timeoutInMilliseconds: 400, + makeChannels: OnMissingChannel.Create, + requeueCount: 3, + isAsync: false, + noOfPerformers: 2), + new AzureServiceBusSubscription( + new SubscriptionName(AddGreetingMessageMapper.Topic), + new ChannelName(subscriptionName), + new RoutingKey(AddGreetingMessageMapper.Topic), + timeoutInMilliseconds: 400, + makeChannels: OnMissingChannel.Create, + requeueCount: 3, + isAsync: true, + noOfPerformers: 2) + }; + + var clientProvider = new ServiceBusVisualStudioCredentialClientProvider("my-awesome-asb.servicebus.windows.net"); + + var asbConsumerFactory = new AzureServiceBusConsumerFactory(clientProvider); + + builder.Services.AddServiceActivator(options => + { + options.Subscriptions = subscriptions; + options.ChannelFactory = new AzureServiceBusChannelFactory(asbConsumerFactory); + + } +``` + +## Complete Reject + +We use ASB's *Subscription* to surscribe to a Topic on a namespace. + +When we Complete a message, in response to a handler chain completing, we Complete the message on ASB using **messageReceiver.CompleteMessageAsync**. Note that we only Complete a message once we have completed running the chain and only if AckOnRead is set to false (as the messages is removed from the queue otherwise). + +When we Dead Letter a message (see [Handler Failure](/contents/HandlerFailure.md) for more on failure) then we use **messageReceiver.DeadLetterMessageAsync** to delete the message, and move it to a DLQ. \ No newline at end of file diff --git a/source/shared/BasicConcepts.md b/source/shared/BasicConcepts.md new file mode 100644 index 0000000..88267c0 --- /dev/null +++ b/source/shared/BasicConcepts.md @@ -0,0 +1,121 @@ +# Basic Concepts + +## Command + +A command is an instruction to carry out work. It exercises the domain and results in a change of state. It expects a single handler. + +An [event](#event) may be used to indicate the outcome of a command. + +## Command Processor + +In Brighter, a command processor allows you to use the *Command Pattern* to separate caller from the executor, typically when separating I/O from domain code. It acts both as a *Command Dispatcher* which allows the separation of the parameters of a [request](#request) from the [handler](#request-handler) that executes that request and a *Command Processor* that allows you to use a middleware [pipeline](#pipeline) to provide additional and re-usable behaviors when processing that request. + +The Command Processor may dispatch to an [Internal Bus](#internal-bus) or an [External Bus](#external-bus). + +## Command-Query Separation (CQS) + +Command-Query separation is the principle that because a [query](#query) should never have the unexpected *side-effect* of updating state, a query should clearly be distinguished from a [command](#request). A query reports on the state of a domain, a command changes it. + +## Event + +An event is a fact. The domain may be updated to reflect the fact represented by the event. There may be no subscribers to an event. It may be skinny, a notification, where the fact is the event itself, or fat, a document, where the event provides facts describing a change. + +An event may be used to indicate the outcome of a [command](#command). + +## Event Stream + +In [message oriented middleware](#message-oriented-middleware-mom), an event stream delivers [messages](#message) (or records) via a steam. A consumer reads the stream at a specific offset from the start. Consumers can store their offsets to resume reading the stream for where they left off, or reset their offset to re-read a stream. Consumers neither lock, nor delete messages from the stream. For consuming apps to scale, the stream can be partitioned, allowing offsets to be maintained of a partition of the stream. By using separate consumer threads or processes to read a partition, an application can ensure that it is able to reduce the latency of reading the stream. + +Examples: Kafka, Kinesis, Redis Streams + +## External Bus + +An external bus allows a [command](#command) or [event](#event) to be turned into a [message](#message) and sent over message-oriented-middleware via broker to a [message queue](#message-queue) or [event stream](#event-stream). + +Brighter also offers a [service activator](#service-activator) to listen for messages published to a queue or stream and forward them to an [internal bus](#internal-bus) within another process. + +## Internal Bus + +A [command](#command), [event](#event) or [query](#query) is executed in-process, passed from the [command processor](#command-processor) or [query processor](#) to a [handler](#request-handler) [pipeline](#pipeline). + +## Message + +A message is a packet of data sent over message-oriented-middleware. It's on-the-wire representation is defined by the protocol used by [message-oriented-middleware](#message-oriented-middleware-mom). + +## Message Oriented Middleware (MoM) + +The class of applications that deliver a [message](#message) from one process to another. MoM may send messages either point-to-point (with just a [message queue](#message-queue)) between sender and receiver, or via a broker, which acts as a dynamic router for messages between sender and receiver. With a broker, the receiver often establishes a subscription to a routing table entry (a *routing key* or *topic*) via a [message queue](#message-queue) or an [event stream](#event-stream). + +Brighter abstracts a specific type of message-oriented middleware by a *Transport*. + +For simplicity, Brighter only supports transports that have a broker configuration, not point-to-point. If you need point-to-point semantics, configure your routing table entry so that it only delivers to one consuming queue or stream. + +## Message Mappers + +A message mapper turns domain code into a message: a header and a body, or turns a message into domain code. Because [message oriented middleware](#message-oriented-middleware-mom) typically looks in a header for routing information, it is also where you add routing information via the header. + +Each individual transport has code to turn a Brighter format message into a message oriented middleware compatible message, and vice versa, so your code only needs to translate to and from the Brighter format. + +## Message Queue + +In [message oriented middleware](#message-oriented-middleware-mom), a message queue delivers [messages](#message) via a queue. A consumer locks a message, processes it, and when it acknowledges it, it is deleted from the queue. Other consumers can process the same queue, and read past any locked messages. This allows scaling via the competing consumers pattern. A nack will release the lock and make a message visible in the queue again, sometimes with a delay. A dead-letter-queue (DLQ) can be used with a nack, to limit the number of retries before a message is considered to be "posion pill" and moved to another queue for undeliverable messages. + +Examples: SQS, AMQP 0-9-1 (Rabbit MQ), AMQP 1-0 (Azure Service Bus). + +## Pipeline + +A pipeline is a sequence of [handlers](#request-handler) that respond to a [request](#request) or [query](#query). The last handler in the sequence is the "target" handler, which forms the pipeline sink. Handlers prior to that form "middleware" that can transform or respond to the request before it reaches the target handler. + +Brighter and Darker's pipelines use a "Russian Doll Model" that is, each handler in the pipeline encompasses the call to the next handler, allowing the handler chain to behave like a call stack. + +## Query + +A query asks the domain for facts. The [result](#result) of the query reports these facts - the state of the domain. A query does not change the state of the domain, for that use a [request](#request). + +## Query Handler + +A handler is the entry point to domain code. It receives a query and returns a [result](#result) to the caller. A handler is always part of an [internal bus](#internal-bus). As such a handler forms part of a [pipeline](#pipeline). + +It is analogous to a method on an ASP.NET Controller. + +## Query Processor + +In Darker, a query processor allows you to use the *Query Object Pattern* to separate caller from the executor, typically when separating the code required to execute a query on a specific database/backing store from the parameters of that query. It acts both as a *Query Dispatcher* which allows the separation of the parameters of a [query](#query) from the [handler](#query-handler) that executes that query and a *Query Processor* that allows you to use a middleware [pipeline](#pipeline) to provide additional and re-usable behaviors when processing that query. + +The Query Processor dispatches to an [Internal Bus](#internal-bus). + +The Query Processor returns a [result](#result). + +## Result + +The return value from a [query](#query). The result is returned from a [query handler](#query-handler) and exposed to the caller via the [QueryProcessor](#query-processor). + +## Request + +In Brighter, either a [command](#command) or an [event](#event), a request for the domain to (potentially) change state in response to an instruction or new facts. + +## Request Handler + +A handler is the entry point to domain code. It receives a request, which may be a [command](#command) or an [event](#event). A handler is always part of an [internal bus](#internal-bus) even when the call to the handler was triggered by a [service activator](#service-activator) receiving a [message](#message) sent by another process to an [external bus](#external-bus). As such a handler forms part of a [pipeline](#pipeline). + +It is analogous to a method on an ASP.NET Controller. + +## Request-Reply + +Request-Reply is a pattern in which there is a request for work and a response. + +To enforce [Command-Query Separation](#command-query-separation-cqs) Brighter handles commands/events and Darker handles queries. + +Where the request changes state, Brighter models this as a [command](#command) and a matching [event](#event) which describes the change. (See [Returning Results from a Handler](/contents/ReturningResultsFromAHandler.md) for a discussion of returning a response directly to the sender of a Command). + +Where the request queries for state, Darker models this as a [query](#query), which returns a [result](#result) directly to the caller. + +A common approach is to change state via Brighter and query for the results of that state change via Darker (and return those results to the caller). + +If the call to Brighter results in a new entity, and the id for the new entity was not given to the command (for example it relies on the Database generating the id), a common problem is how to then request the details of that newly created entity via Darker. A simple solution is to update the command with the id (as a conceptual *out* parameter), and then retrieve it from there to use in the Darker query. See [update a field on a command](/contents/ReturningResultsFromAHandler.md#update-a-field-on-the-command) for more. + +## Service Activator + +A Service Activator triggers execution of your code due to an external input, such as an HTTP call, or a [message](#message) sent over middleware. + +In Brighter, the *Dispatcher* acts as a Service Activator, listening for a message from middleware, which it delivers via the [command processor](#command-processor) to a [handler](#request-handler). As such, it turns messages sent over middleware to a call on your [internal bus](#internal-bus). diff --git a/source/shared/BrighterBasicConfiguration.md b/source/shared/BrighterBasicConfiguration.md new file mode 100644 index 0000000..6acec44 --- /dev/null +++ b/source/shared/BrighterBasicConfiguration.md @@ -0,0 +1,740 @@ +# **Basic Configuration** + +Configuration is the most labor-intensive part of using Brighter.Once you have configured Brighter, using its model of requests and handlers is straightforward + +## **Using .NET Core Dependency Injection** + +This section covers using .NET Core Dependency Injection to configure Brighter. If you want to use an alternative DI container then see the section [How Configuration Works](/contents/HowConfigurationWorks.md) + +We divide configuration into two sections, depending on your requirements: + +* [**Configuring The Command Processor**](#configuring-the-command-processor): This section covers configuring the **Command Processor**. Use this if you want to dispatch requests to handlers, or publish messages from your application on an external bus +* [**Configuring The Service Activator**](#configuring-the-service-activator): This section covers configuring the **Service Activator**. Use this if you want to read messages from a transport (and then dispatch to handlers). + + +## **Configuring The Command Processor** + +### **Command Processor Service Collection Extensions** + +Brighter's package: + +* **Paramore.Brighter.Extensions.DependencyInjection** + + provides extension methods for **ServiceCollection** that can be used to add Brighter to the .NET Core DI Framework. + +By adding the package you can call the **AddBrighter()** extension method. + +If you are using a **Startup** class's **ConfigureServices** method call the following: + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) +} + +``` + +if you are using .NET 6 you can make the call direction on your **HostBuilder**'s Services property. + +The **AddBrighter()** method takes an **`Action`** delegate. The extension method supplies the delegate with a **BrighterOptions** object that allows you to configure how Brighter runs. + +The **AddBrighter()** method returns an **IBrighterBuilder** interface. **IBrighterBuilder** is a [fluent interface](https://en.wikipedia.org/wiki/Fluent_interface) that you can use to configure additional Brighter properties (see [Brighter Builder Fluent Interface](#brighter-builder-fluent-interface)). + +#### **Adding Polly Policies** + +Brighter uses Polly policies for both internal reliability, and to support adding a custom policy to a handler for reliability. + +To use a Polly policy with Brighter you need to register it first with a Polly **PolicyRegistry**. In this example we register both Synchronous and Asynchronous Polly policies with the registry. + +``` csharp + var retryPolicy = Policy.Handle().WaitAndRetry(new[] + { + TimeSpan.FromMilliseconds(50), + TimeSpan.FromMilliseconds(100), + TimeSpan.FromMilliseconds(150) }); + + var circuitBreakerPolicy = Policy.Handle().CircuitBreaker(1, + TimeSpan.FromMilliseconds(500)); + + var retryPolicyAsync = Policy.Handle() + .WaitAndRetryAsync(new[] { TimeSpan.FromMilliseconds(50), TimeSpan.FromMilliseconds(100), TimeSpan.FromMilliseconds(150) }); + + var circuitBreakerPolicyAsync = Policy.Handle().CircuitBreakerAsync(1, TimeSpan.FromMilliseconds(500)); + + var policyRegistry = new PolicyRegistry() + { + { "SyncRetryPolicy", retryPolicy }, + { "SyncCircuitBreakerPolicy", circuitBreakerPolicy }, + { "AsyncRetryPolicy", retryPolicyAsync }, + { "AsyncCircuitBreakerPolicy", circuitBreakerPolicyAsync } + }; + +``` + +And you can use them in you own handler like this: + +``` csharp +internal class MyQoSProtectedHandler : RequestHandler +{ + static MyQoSProtectedHandler() + { + ReceivedCommand = false; + } + + [UsePolicy(policy: "SyncRetryPolicy", step: 1)] + public override MyCommand Handle(MyCommand command) + { + /*Do work that could throw error because of distributed computing reliability*/ + } +} +``` + +See the section [Policy Retry and Circuit Breaker](/contents/PolicyRetryAndCircuitBreaker.md) for more on using Polly policies with handlers. + +With the Polly Policy Registry filled, you need to tell Brighter where to find the Policy Registry: + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(options => + options.PolicyRegistry = policyRegistry + ) +} + +``` + +#### **Configuring Lifetimes** + +Brighter can register your *Request Handlers* and *Message Mappers* for you (see [IBrighter Builder Fluent Interface](#ibrighterbuilder-fluent-interface)). When we register types for you with ServiceCollection, we need to register them with a given lifetime (see [Dependency Injection Service Lifetimes](https://docs.microsoft.com/en-us/dotnet/core/extensions/dependency-injection#service-lifetimes)). + +We also allow you to set the lifetime for the CommandProcessor. + +We recommend the following lifetimes: + +* If you are using *Scoped* lifetimes, for example with EF Core, make your *Request Handlers* and your *Command Processor* Scoped as well. +* If you are not using *Scoped* lifetimes you can use *Transient* lifetimes for *Request Handlers* and a *Singleton* lifetime for the *Command Processor*. +* Your *Message Mappers* should not have state and can be *Singletons*. + +(Be cautious about using *Singleton* lifetimes for *Request Handlers*. Even if your *Request Handler* is stateless today, and so does not risk carrying state across requests, a common bug is that state is added to an existing *Request Handler* which has previously been registered as a *Singleton*.) + +You configure the lifetimes for the different types that Brighter can create at run-time as follows: + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(options => + options.HandlerLifetime = ServiceLifetime.Scoped; + options.CommandProcessorLifetime = ServiceLifetime.Scoped; + options.MapperLifetime = ServiceLifetime.Singleton; + ); +} + +``` + +### **Brighter Builder Fluent Interface** + +#### **Type Registration** +The **IBrighterBuilder** fluent interface can scan your assemblies for your *Request Handlers* (inherit from **IHandleRequests<>** or **IHandleRequestsAsync<>**) and *Message Mappers* (inherit from **IAmAMessageMapper<>**) and register then with the **ServiceCollection**. This is the most common way to register your code. + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .AutoFromAssemblies(); +} + +``` + +The code scans any loaded assemblies. If you need to register types from assemblies that are not yet loaded, you can provide a list of additional assemblies to scan as an argument to the call to **AutoFromAssemblies()**. + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .AutoFromAssemblies(typeof(MyRequestHandlerAsync).Assembly); +} + +``` + +Instead of using **AutoFromAssemblies** you can exert more fine-grained control over the registration, by explicitly registering your *Request Handlers* and *Message Mappers*. We don't recommend this, but make it available for cases where the automatic registration does not meet your needs. + +* **MapperRegistryFromAssemblies()**, **HandlersFromAssemblies()** and **AsyncHandlersFromAssemblies** are the methods called by **AutoFromAssemblies()** and can be called explicitly. +* **Handlers()**, **AsyncHandlers()** and **MapperRegistry()** accept an **Action<>** delegate that respectively provide you with **IAmASubscriberRegistry** or **IAmAnAsyncSubscriberRegistry** to register your RequestHandlers explicitly or a **ServiceCollectionMapperRegistry** to register your mappers. This gives you explicit control over what you register. + +#### **Using an External Bus** + +Using an *External Bus* allows you to send messages between processes using a message-oriented middleware transport (such as RabbitMQ or Kafka). (For symmetry, we refer to the usage of the *Command Processor* without an external bus as using an *Internal Bus*). + +When raising a message on the *Internal Bus*, you use one of the following methods on the *Command Processor*: + +* **Send()** and **SendAsync()** - Sends a *Command* to one *Request Handler*. +* **Publish()** and **PublishAsync()** - Broadcasts an *Event* to zero or more *Request Handlers*. + +When raising a message on an *External Bus*, you use the following methods on the *CommandProcessor*: + +* **Post()** and **PostAsync()** - Immediately posts a *Command* or *Event* to another process via the external Bus +* **DepositPost()** and **DepositPostAsync()** - Puts on or many *Command*(s) or *Event*(s) in the *Outbox* for later delivery +* **ClearOutbox()** and **ClearOutboxAsync()** - Clears the *Outbox*, posting un-dispatched messages to another process via the *External Bus*. +* **ClearAsyncOutbox()** - Implicitly clears the **Outbox**, similar to above however allows bulk dispatching of messages onto a **Transport**. + +The major difference here is whether or not you wish to use an *Outbox* for Transactional Messaging. (See [Outbox Pattern](/contents/OutboxPattern.md) and [Brighter Outbox Support](/contents/BrighterOutboxSupport.md) for more on Brighter and the Outbox Pattern). + +To use an *External Bus*, you need to supply Brighter with configuration information that tells Brighter what middleware you are using and how to find it. (You don't need to do anything to configure an *Internal Bus*, it is always available.) + +In order to provide Brighter with this information we need to provide it with an implementation of **IAmAProducerRegistry** for the middleware you intend to use for the *External Bus*. + +#### **Transports and Gateways** + +*Transports* are how Brighter supports specific Message-Oriented-Middleware (MoM). *Transports* are provided in separate NuGet packages so that you can take a dependency only on the transport that you need. Brighter supports a number of different *transports*. + +A *Gateway Connection* is how you configure connection to MoM within a *transport*. As an example, the *Gateway Connection* **RMqGatewayConnection** is used to connect to RabbitMQ. Internally the *Gateway Connection* is used to create a *Gateway* object which wraps the client SDK for the MoM. + +We go into more depth on the fields you set here in sections dealing with specific transports. + +#### **Publications** + +A *Publication* configures a transport for sending a message to it's associated MoM. So an **RmqPublication** configures how we publish a message to RabbitMQ. There are a number of common properties to all publications. + +* **MakeChannels**: Do you want Brighter to create the infrastructure? Brighter can create infrastructure that it needs, and is aware of: **OnMissingChannel.Create**. So a publication can create the topic to send messages to. Alternatively if you create the channel by another method, such as IaaC, we can verify the infrastructure on startup: **OnMissingChannel.Validate**. Finally, you can avoid the performance cost of runtime checks by assuming your infrastructure exists: **OnMissingChannel.Assume**. +* **MaxOutstandingMessages**: How large can the number of messages in the Outbox grow before we stop allowing new messages to be published and raise an **OutboxLimitReachedException**. +* **MaxOutStandingCheckIntervalMilliSeconds**: How often do we check to see if the Outbox is full. +* **Topic**: A Topic is the key used within the MoM to route messages. Publishers publish to a topic and subscribers, subscribe to it. We use a class **RoutingKey** to encapsulate the identifier used for a topic. The name the MoM uses for a topic may vary. Kafka & SNS use *topic* whilst RMQ uses *routingkey* + +#### **Transport NuGet Packages** + +We use the naming convention **Paramore.Brighter.MessagingGateway.{TRANSPORT}** for *transports* where {TRANSPORT} is the name of the middleware. + +In this example we will show using an implementation of **IAmAProducerRegistry** for RabbitMQ, provided by the NuGet package: + +* **Paramore.Brighter.MessagingGateway.RMQ** + +See the documentation for detail on specific *transports* on how to configure them for use with Brighter, for now it is enough to know that you need to provide a *Messaging Gateway* which tells us how to reach the middleware and a *Publication* which tells us how to configure the middleware. + +*Transports* provide an **IAmAProducerRegistryFactory()** to allow you to create multiple *Publications* connected to the same middleware. + +#### Retry and Circuit Breaker with an External Bus + +When posting a request to the External Bus we use a Polly policy internally to control Retry and Circuit Breaker in case the External Bus is not available. These policies have defaults but you can configure the behavior using the policy keys: + +* **Paramore.RETRYPOLICY** +* **Paramore.CIRCUITBREAKER** + +#### **Bus Example** + +Putting this together, an example configuration for an External Bus for a local RabbitMQ instance could look like this: + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .UseExternalBus(new RmqProducerRegistryFactory( + new RmqMessagingGatewayConnection + { + AmpqUri = new AmqpUriSpecification(new Uri("amqp://guest:guest@localhost:5672")), + Exchange = new Exchange("paramore.brighter.exchange"), + }, + new RmqPublication[]{ + new RmqPublication + { + Topic = new RoutingKey("GreetingMade"), + MaxOutStandingMessages = 5, + MaxOutStandingCheckIntervalMilliSeconds = 500, + WaitForConfirmsTimeOutInMilliseconds = 1000, + MakeChannels = OnMissingChannel.Create + }} + ).Create() + ) + + ... +} +``` + +#### **Outbox Support** + +If you intend to use Brighter's *Outbox* support for Transactional Messaging then you need to provide us with details of your *Outbox*. + +Brighter provides a number of *Outbox* implementations for common Dbs (and you can write your own for a Db that we do not support). For this discussion we will look at Brighter's support for working with EF Core. See the documentation for working with specific *Outbox* implementations. + +EF Core supports a number of databases and you should pick the packages that match the Dy you want to use with EF Core. In this case we will choose MySQL. + +For this we will need the *Outbox* packages for the MySQL *Outbox*. + +* **Paramore.Brighter.MySql** +* **Paramore.Brighter.Outbox.MySql** + +For a given backing store the pattern should be Paramore.Brighter.{DATABASE} and Paramore.Brighter.Outbox.{DATABASE} where {DATABASE} is the name of the Db that you are using. + +In addition for an ORM you will need to add the package that supports the ORM, in this case EF Core: + +* **Paramore.Brighter.MySql.EntityFrameworkCore** + +For a given ORM the pattern should be Paramore.Brighter.{ORM}.{DATABASE} where {ORM} is the ORM you are choosing and {DATABASE} is the Db you are using with the ORM. + +To configure our *Outbox* we then need to use the Use{DATABASE}Outbox method call, where {DATABASE} is the {DATABASE} that we want, passing in the configuration for our Db so that we can access it. In our case this will be **UseMySqlOutbox()**. + +As we want to use an ORM, in our case EF Core, we have to tell the Outbox how to access EF Core transactions - as we need to participate in a transaction with the ORM. We call a method for the Db, Use{DATABASE}TransactionConnectionProvider, where {DATABASE} is our Db, so in our case **UseMySqlTransactionConnectionProvider()**. + +As a parameter to Use{DATABASE}TransactionConnectionProvider we need to provide a *Transaction Provider* for the ORM we are using, in our case this is *MySqlEntityFrameworkConnectionProvider<>). + +Finally, if we want the *Outbox* to use a background thread to clear un-dispatched items from the *Outbox*, and we do in most circumstances, otherwise they will not be dispatched, we need to run an *Outbox Sweeper* to do this work. + +To add the *Outbox Sweeper* you will need to take a dependency on another NuGet package: + +* **Paramore.Brighter.Extensions.Hosting** + +This results in: + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .UseExternalBus(...) + .UseMySqlOutbox(new MySqlConfiguration(DbConnectionString(), _outBoxTableName), typeof(MySqlConnectionProvider), ServiceLifetime.Singleton) + .UseMySqTransactionConnectionProvider(typeof(MySqlEntityFrameworkConnectionProvider), ServiceLifetime.Scoped) + .UseOutboxSweeper() + + ... +} + +``` + +(**UseExternalBus()** has optional parameters for use with Request-Reply support for some transports. We don't cover that here, instead see [Direct Messaging](/contents/Routing.md#direct-messaging) for more). + +#### **Configuring JSON Serialization** + +Brighter defines a set of serialization options for use when it needs to serialize messages to JSON. Internally we use these options in our transports, when serializing messages to an external bus and deserializing from an external bus. You may wish to use these options in your own [*Message Mapper*](/contents/MessageMappers.md) implementation. + +By default our JSONSerialization Options are configured as follows: + +``` csharp +static JsonSerialisationOptions() +{ + var opts = new JsonSerializerOptions + { + PropertyNameCaseInsensitive = true, + PropertyNamingPolicy = JsonNamingPolicy.CamelCase, + NumberHandling = JsonNumberHandling.AllowReadingFromString, + AllowTrailingCommas = true + }; + + opts.Converters.Add(new JsonStringConverter()); + opts.Converters.Add(new DictionaryStringObjectJsonConverter()); + opts.Converters.Add(new ObjectToInferredTypesConverter()); + opts.Converters.Add(new JsonStringEnumConverter()); + + Options = opts; +} +``` + +You can use the **IBrighterBuilder** extension **ConfigureJsonSerialisation** to override these values. The method takes an **Action\** lambda expression that allows you to override these defaults. For example: + +```csharp + +.ConfigureJsonSerialisation((options) => +{ + options.PropertyNameCaseInsensitive = true; +}) + +``` + +If you want to use this configured set of JSON Serialization options in your own code, you can, by using the static property JsonSerialisationOptions.Options. For example: + +```csharp +public GreetingMade MapToRequest(Message message) +{ + return JsonSerializer.Deserialize(message.Body.Value, JsonSerialisationOptions.Options); +} +``` + +### **Putting It All Together** + +Putting all this together, a typical configuration might looks as follows: + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(options => + { + options.HandlerLifetime = ServiceLifetime.Scoped; + options.MapperLifetime = ServiceLifetime.Singleton; + options.PolicyRegistry = policyRegistry; + }) + .ConfigureJsonSerialisation((options) => + { + options.PropertyNameCaseInsensitive = true; + }) + .UseExternalBus(new RmqProducerRegistryFactory( + new RmqMessagingGatewayConnection + { + AmpqUri = new AmqpUriSpecification(new Uri("amqp://guest:guest@rabbitmq:5672")), + Exchange = new Exchange("paramore.brighter.exchange"), + }, + new RmqPublication[] { + new RmqPublication + { + Topic = new RoutingKey("GreetingMade"), + MaxOutStandingMessages = 5, + MaxOutStandingCheckIntervalMilliSeconds = 500, + WaitForConfirmsTimeOutInMilliseconds = 1000, + MakeChannels = OnMissingChannel.Create + }} + ).Create() + ) + .UseMySqlOutbox(new MySqlConfiguration(DbConnectionString(), _outBoxTableName), typeof(MySqlConnectionProvider), ServiceLifetime.Singleton) + .UseMySqTransactionConnectionProvider(typeof(MySqlEntityFrameworkConnectionProvider), ServiceLifetime.Scoped) + .UseOutboxSweeper() + .AutoFromAssemblies(); +} + +``` + +## **Configuring The Service Activator** + +A *consumer* reads messages from Message-Oriented Middleware (MoM), and a *producer* puts messages onto the MoM for the *consumer* to read. + +A *consumer* waits for messages to appear on the queue, reads them, and then calls your *Request Handler* code to react. Because the •consumer* runs your code in response to an external request, a message being placed on the External Bus, we call the component that listens for messages and dispatches them a [*Service Activator*](https://www.enterpriseintegrationpatterns.com/patterns/messaging/MessagingAdapter.html) + +To use Brighter's Service Activator you will need to take a dependency on the NuGet package: + +* **Paramore.Brighter.ServiceActivator** + +### **ServiceActivator Service Collection Extensions** + +We provide support for configuring .NET Core's **HostBuilder** as a *ServiceActivator* for use with MoM. We use Brighter's Command Processor to dispatch the messages read by a *Dipatcher*. If you are not using **HostBuilder** then you will need to configure the Dispatcher yourself. See [How Configuring the Dispatcher Works](/contents/HowConfiguringTheDispatcherWorks.md) for more. + +To use Brighter's *Service Activator* with **HostBuilder** you will need to take a dependency on the following NuGet packages: + +* **Paramore.Brighter.ServiceActivator.Extensions.Hosting** +* **Paramore.Brighter.ServiceActivator.Extensions.DependencyInjection** + +These provide an extension method **AddServiceActivator()** that can be used to add Brighter to the .NET Core DI Framework. + +By adding the package you can call the **AddServiceActivator()** extension method. + +If you are using a **HostBuilder** class's **ConfigureServices** method call the following: + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + services.AddServiceActivator(...) + } + +``` + +if you are using .NET 6 you can make the call direction on your **HostBuilder**'s Services property. + +The **AddServiceActivator()** method takes an **`Action`** delegate. The extension method supplies the delegate with a **ServiceActivatorOptions** object that allows you to configure how Brighter runs. + +The **AddServiceActivator()** method returns an **IBrighterBuilder** interface. **IBrighterBuilder** is a [fluent interface](https://en.wikipedia.org/wiki/Fluent_interface) that you can use to configure Brighter *Command Processor* properties. It is discussed above at [Brighter Builder Fluent Interface](#brighter-builder-fluent-interface)) and the same options apply. We discuss one additional option that becomes important when receiving requests the *Inbox* in [Additional Brighter Builder Options](/contents/BasicConfiguration.md#the-inbox). + +#### **Subscriptions** + +When configuring your application's *Service Activator*, your *Subscriptions* indicate configure how your application will receive messages from the associated MoM queues or streams. + +All *Subscriptions* lets you configure the following common properties. + +* **Buffer Size**: The number of messages to hold in memory. Where the buffer is not shared, a single thread or Performer can access these; where the buffer is shared, multiple threads can access the same buffer of work. Work in a buffer is locked on queue based middleware, and thus not available to other consumers (threads or process depending if the buffer is shared or not) until *Acknowledged* or *Rejected*. +* **Channel Factory**: Creates or finds the necessary infrastructure for messaging on the MoM and wraps it in an object. +* **Channel *Name**: If queues are primitives in the MoM this names the queue, otherwise just used for diagnostics. +* **Channel Failure Delay**: How long should we delay if a channel fails before trying again, to give problems time to clear. +* **Data Type**: We use a [Datatype Channel](https://www.enterpriseintegrationpatterns.com/DatatypeChannel.html). What is the type of this channel? +* **Empty Channel Delay**: If there are no messages in the queue or stream when we read, how long should we pause before reading again? +* **MakeChannels**: Do you want Brighter to create the infrastructure? Brighter can create infrastructure that it needs, and is aware of: **OnMissingChannel.Create**. So a subscription can create the topic to send messages to, and any subscription to that topic required by the MoM, including a queue (which uses the *Channel Name*). Alternatively if you create the channel by another method, such as IaaC, we can verify the infrastructure on startup: **OnMissingChannel.Validate**. Finally, you can avoid the performance cost of runtime checks by assuming your infrastructure exists: **OnMissingChannel.Assume**. +* **Name**: What do we call this subscription for diagnostic purposes. +* **NoOfPerformers**: Effectively, how many threads do we use to read messages from the queue. As Brighter uses a Single-Threaded Apartment model, each thread has it's own message pump and is thus an in-process implementation of the [Competing Consumers](https://www.enterpriseintegrationpatterns.com/CompetingConsumers.html) pattern. +* **RequeueCount**: How many times can you retry a message before we declare it a poison pill message? +* **RequeueDelayInMilliseconds**: When we requeue a message how long should we delay it by? +* **RoutingKey**: The identifier used to routed messages to subscribers on MoM. You publish to this, and subscriber from this. This has different names; in Kafka or SNS this is a Topic, in RMQ this is the routing key. +* **RunAsync**: Is this an async pipeline? Your pipeline must be sync or async. An async pipeline can increase throughput where a handler is I/O bound by allowing the message pump to read another message whilst we await I/O completion. The cost of this is that strict ordering of messages will now be lost as processing of I/O bound requests may complete out-of-sequence. Brighter provides its own synchronization context for async operations. We recommend scaling via increasing the number of performers, unless you know that I/O is your bottleneck. +* **TimeoutInMilliseconds**: How long does a read 'wait' before assuming there are no pending messages. +* **UnaceptableMessageLimit**: Brighter will ack a message that throws an unhandled exception, thus removing it from a queue. + +For a more detailed discussion of using Requeue (with Delay) for Handler failure, (**RequeueCount** and **RequeueDelayInMilliseconds**) along with termination of a consumer due to message failure (**UnacceptableMessageLimit**) see [Handler Failure](/contents/HandlerFailure.md) + +In addition, individual transports that provide access to specific MoM sub-class *Subscription* to provide properties unique to the chosen middleware. We discuss those under a section for that transport. + +For RabbitMQ for example, this would look like this: + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + var subscriptions = new Subscription[] + { + new RmqSubscription( + new SubscriptionName("paramore.sample.salutationanalytics"), + new ChannelName("SalutationAnalytics"), + new RoutingKey("GreetingMade"), + runAsync: true, + timeoutInMilliseconds: 200, + isDurable: true, + makeChannels: OnMissingChannel.Create), //change to OnMissingChannel.Validate if you have infrastructure declared elsewhere + }; + + services.AddServiceActivator(options => + { + options.Subscriptions = subscriptions; + }) +} + +... + +``` + +#### **Gateway Connections & Channel Factories** + +A *Gateway Connection* tells Brighter how to connect to MoM for a particular transport. The transport package will contain a *Gateway Connection*, you need to provide the information to connect to your middleware (URIs, ports, credentials etc.) Your transport package provides a *Gateway Connection* + +A *Channel Factory* connects Brighter to MoM. Depending on the configuration settings for your *Subscription* it may create the required primitives (topics/routing keys, queues, streams) on MoM or simply attach to ones that you have created via Infrastructure as Code (IaC). Your transport provides a *Channel Factory* and you need to pass it a *Gateway Connection*. + +For RabbitMQ, this would look like: + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + + var rmqConnection = new RmqMessagingGatewayConnection + { + AmpqUri = new AmqpUriSpecification(new Uri($"amqp://guest:guest@local:5672")), + Exchange = new Exchange("paramore.brighter.exchange") + }; + + var rmqMessageConsumerFactory = new RmqMessageConsumerFactory(rmqConnection); + + services.AddServiceActivator(options => + { + options.ChannelFactory = new ChannelFactory(rmqMessageConsumerFactory); + }) +} + +... + +``` + +#### **Configuring Service Activator Lifetimes** + +Under the hood your *Service Activator* uses a *Command Processor* and you will need to configure lifetimes [as discussed above](#configuring-lifetimes). + +An additional requirement is configuring the lifetime of the *Command Processor* itself. Within the context of an ASP.NET application, configuring the lifetime of the **Command Processor** relies on ASP.NET creating an instance of the *Command Processor* in a request pipeline. When you are using *Service Activator* there is no ASP.NET pipeline, instead Brighter's *Dispatcher* manages the lifetime of the *Command Processor* that we pass a request to. By setting the **ServiceActivatorOptions.UseScoped** field to true, you instruct *Brighter* to use a new *Command Processor* instance for each request. This is important if you take the *Command Processor* as a dependency in any of your *Request Handlers* with a **Scoped** lifetime. If in doubt, just set **ServiceActivatorOptions.UseScoped** field to true. + + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + services.AddServiceActivator(options => + { + options.UseScoped = true; + options.HandlerLifetime = ServiceLifetime.Scoped; + options.MapperLifetime = ServiceLifetime.Singleton; + options.CommandProcessorLifetime = ServiceLifetime.Scoped; + }) +} + +... + +``` + + +### **Service Activator Brighter Builder Fluent Interface** + +The call to **AddServiceActivator()** returns an **IBrighterBuilder** fluent interface. This means that you can use any of the options described in [Brighter Build Fluent Interfaces](#brighter-builder-fluent-interface) to configure the associated *Command Processor* such as scanning assemblies for *Request Handlers* and adding an *External Bus* and *Outbox*. + +An option is intended for the context of a Service Activator is described below. + +#### **Inbox** + +As described in the [Outbox Pattern](/contents/OutboxPattern.md) an *Outbox* offers **Guaranteed, At Least Once** delivery. It explicitly may result in you sending duplicate messages. In addition, MoM tends to offer "At Least Once" guarantees only, further creating the risk that you will receive a duplicate message. + +If the request is not idempotent, you can use an Inbox to de-duplicate it. See [Inbox Support](/contents/BrighterInboxSupport.md) for more. + +Configuring an *Inbox* has two elements. The first is the type of *Inbox*, the second configuration for the *Inbox* behavior. + +Brighter provides a number of *Inbox* implementations for common Dbs (and you can write your own for a Db that we do not support). For this discussion we will look at Brighter's support for working with MySQL. See the documentation for working with specific *Inbox* implementations. + +For this we will need the *Inbox* packages for the MySQL *Inbox*. + +* **Paramore.Brighter.Inbox.MySql** + +For a given backing store the pattern should be Paramore.Brighter.Inbox.{DATABASE} where {DATABASE} is the name of the Db that you are using. + +To configure our *Inbox* we then need to use the UseExternalInbox method call and pass in an instance of a class that implements **IAmAnInbox**, taken from our package, and an instance of **InboxConfiguration** that tells Brighter how we want to use the Inbox. + +For *Inbox Configuration* you set the following properties: + +* **ActionOnExists**: What do we do if the request has been handled? The default,**OnceOnlyAction.Throw** is to throw a **OnceOnlyException**. If you take no other action this will cause the message to be rejected and sent to a DLQ if one is configured (See [Handler Failure](/contents/HandlerFailure.md)). The alternative is **OnceOnlyAction.Warn** simply logs that the request is a duplicate, but takes no other action. +* **OnceOnly**: This defaults to *true* and will check for a duplicate and take the action indicated by **ActionOnExists**. If *false* the *Inbox* will record the request, but will take no further action. (This tends to be set to *false* if you are using the *Inbox* to record what requests caused current state only and not de-duplicate). +* **Scope**: This indicates the type of request (*Command* or *Event*) to store in the *Inbox*. By default this is set to **InboxScope.All** and captures everything but you can be explicit and just capture **InboxScope.Commands** or **InboxScope.Events**. (This tends to be set to **InboxScope.Commands** when only commands cause changes to state that are not idempotent). +* **Context**: Used to uniquely identify receipt of this request via this handler. If you are recording *Events* and have multiple handlers, then the first event handler to receive the message will block the others from doing so, unless you disambiguate the handler identity by supplying a context method. + +A typical *Inbox* configuration for MySQL would be: + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + services.AddServiceActivator(options => + { ... }) + .UseExternalInbox( + new MySqlInbox(new MySqlInboxConfiguration("server=localhost; port=3306; uid=root; pwd=root; database=Salutations", "Inbox"); + new InboxConfiguration( + scope: InboxScope.Commands, + onceOnly: true, + actionOnExists: OnceOnlyAction.Throw + ) + ); +} + +... + +``` +Typically you would obtain the connection string for the Db from configuration (as opposed to hard coding the string), likewise for the table name for your *Inbox*. + + +### Running Service Activator + +To run *Service Activator* we add it as a [Hosted Service](https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-6.0&tabs=visual-studio). + +We provide the class **ServiceActivatorHostedService** for this in the NuGet package: + +* **Paramore.Brighter.ServiceActivator.Extensions.Hosting** + +The **ServiceActivatorHostedService** calls the **Dispatcher.Receive** method which starts message pumps for the configured *Subscriptions*. + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + ... + + services.AddHostedService(); +} + +... + +``` + +On shutdown Brighter will allow the current *Request Handler* to complete, then end the message pump loop and exit. If you have long-running handlers it is possible that they will not complete in the default 5s for graceful shutdown of the MS Generic Host. In this case, you need to [increase the timeout](https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/generic-host?view=aspnetcore-6.0#shutdowntimeout) of the host shutdown. + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + services.Configure(options => + { + options.ShutdownTimeout = TimeSpan.FromSeconds(20); + }); + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + ... + + services.AddHostedService(); +} + +``` + +### A Complete Service Activator Example + +When all of the relevant configuration sections are added together, your code will look something like this, with variations for your transport and stores. + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + services.Configure(options => + { + options.ShutdownTimeout = TimeSpan.FromSeconds(20); + }); + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + var subscriptions = new Subscription[] + { + new RmqSubscription( + new SubscriptionName("paramore.sample.salutationanalytics"), + new ChannelName("SalutationAnalytics"), + new RoutingKey("GreetingMade"), + runAsync: true, + timeoutInMilliseconds: 200, + isDurable: true, + makeChannels: OnMissingChannel.Create), //change to OnMissingChannel.Validate if you have infrastructure declared elsewhere + }; + + var rmqConnection = new RmqMessagingGatewayConnection + { + AmpqUri = new AmqpUriSpecification(new Uri($"amqp://guest:guest@localhost:5672")), + Exchange = new Exchange("paramore.brighter.exchange") + }; + + var rmqMessageConsumerFactory = new RmqMessageConsumerFactory(rmqConnection); + + services.AddServiceActivator(options => + { + options.Subscriptions = subscriptions; + options.ChannelFactory = new ChannelFactory(rmqMessageConsumerFactory); + options.UseScoped = true; + options.HandlerLifetime = ServiceLifetime.Scoped; + options.MapperLifetime = ServiceLifetime.Singleton; + options.CommandProcessorLifetime = ServiceLifetime.Scoped; + }) + .AutoFromAssemblies() + .UseExternalInbox( + new MySqlInbox(new MySqlInboxConfiguration("server=localhost; port=3306; uid=root; pwd=root; database=Salutations", "Inbox"); + new InboxConfiguration( + scope: InboxScope.Commands, + onceOnly: true, + actionOnExists: OnceOnlyAction.Throw + ) + ) + + services.AddHostedService(); + +} + +``` + +## Samples + +Brighter includes a comprehensive set of [Samples](https://github.com/BrighterCommand/Brighter/tree/master/samples) in its main repo that you can review for clarity on how Brighter works and should be configured. + + diff --git a/source/shared/BrighterInboxSupport.md b/source/shared/BrighterInboxSupport.md new file mode 100644 index 0000000..0b04294 --- /dev/null +++ b/source/shared/BrighterInboxSupport.md @@ -0,0 +1,116 @@ +# Brighter Inbox Support + +## Guaranteed, At Least Once + +Messaging makes the *guaranteed, at least once* promise. + +- Guaranteed: A broker writes a copy of your message to disk, so that it is not lost. An [Outbox](/contents/OutboxPattern.md) writes the message to the application's database to ensure it is not lost. +- At Least Once: In a distributed system you cannot guarantee that a writer will receive a response that it has successfully persisted a message, it must choose to retry the persistence if it does not receive an acknowledgement. This means that duplicates will occur, hence *at least once*. + +## Guaranteed, Once Only + +There are two possible reactions to the *at least once* problem. + +- Idempotency: Ensure that when receiving a message, handling it multiple times will not have side-effects +- De-duplication: Ensure that when receiving a message, you check whether you have handled it before, and discard if you have already processed it. + +Events are often idempotent, whilst commands often require de-duplication. + +## Inbox + +An inbox records messages that you have received and processed. Brighter provides inbox implementations built over a range of databases. If your preferred database is not included, see [implementing an inbox](#implementing-an-inbox). Brighter also makes available an in-memory inbox which is intended for development only as it will not work across multiple consumers, or survive process restarts. + +You can configure the usage of an inbox for de-duplication on a per-handler, or per command processor basis. + +### Adding an Inbox to a Handler + +The inbox is middleware and forms a part of the internal bus pipeline. When added to the pipeline, it will run before your handler, and after subsequent handlers (see the [Russian Doll Model](/contents/BuildingAnAsyncPipeline.md)). + +- Before: Check to see if we have already seen this request +- After: Add this request to those that we have already seen. + +You also need to configure what action the inbox takes when it has seen a request: + +- OnceOnly: Does the inbox reject duplicates, or does it simply record requests. WARNING: Defaults to false - so it won't reject duplicates unless you set this parameter to true, just log requests that pass through this handler. +- OnceOnlyAction: What does the inbox do, when a duplicate is encountered. Defaults to Throw. + - Warn: Just log that a duplicate was received + - Throw: Throws a OnceOnlyException + +**If you wish to terminate processing on a duplicate, you should set OnceOnly to true, which is what you need to terminate processing; the OnceOnlyAction will default to Throw** + +In the context of the Service Activator (listening to messages over middleware) throwing a OnceOnlyException will result in the message being acked (because it has already been processed). + +The inbox is global to your application and uses the request id; you will want to distinguish requests in the inbox if you need to store the same request id for different pipelines. For example, if you deliver an event to multiple handlers, each handler has a request with the same request id. + +Use the **contextKey** parameter to the attribute to disambiguate the request id. We recommend using the type of the handler, as the id usually needs to be unique via pipeline. + +There are two versions of the attribute: sync and async. Ensure that you choose the correct version, which should [match your handler](/contents/DispatchingARequest.md#pipelines-must-be-homogeneous). + +``` csharp + [UseInboxAsync(step:0, contextKey: typeof(GreetingMadeHandlerAsync), onceOnly: true )] + public override async Task HandleAsync(GreetingMade @event, CancellationToken cancellationToken = default(CancellationToken)) + { + Console.WriteLine($"Greeting Received: {@event.Greeting}"); + + return await base.HandleAsync(@event, cancellationToken); + } +``` + +### Inbox Configuration + +Your inbox is configured as part of the Brighter extensions to ServiceCollection. See [Inbox Configuration](/contents/BrighterBasicConfiguration.md#inbox) for more. + +### Inbox Builder + +Brighter contains DDL to configure your Inbox. For each supported database we include an **InboxBuilder**. The Inbox Builder **GetDDL** which allows you to obtain the DDL statements required to create an Inbox. You can use this as part of your application start up to configure the Inbox if it does not already exist. + +The following example shows creation of a MySql inbox. + +We assume that INBOX_TABLE_NAME is a constant, shared with the code that configures your inbox. + +``` csharp + +private static void CreateInbox(IConfiguration config, IHostEnvironment env) +{ + try + { + var connectionString = config.GetConnectionString("Salutations") + + using var sqlConnection = new MySqlConnection(connectionString); + sqlConnection.Open(); + + using var existsQuery = sqlConnection.CreateCommand(); + existsQuery.CommandText = MySqlInboxBuilder.GetExistsQuery(INBOX_TABLE_NAME); + bool exists = existsQuery.ExecuteScalar() != null; + + if (exists) return; + + using var command = sqlConnection.CreateCommand(); + command.CommandText = MySqlInboxBuilder.GetDDL(INBOX_TABLE_NAME); + command.ExecuteScalar(); + } + catch (System.Exception e) + { + Console.WriteLine($"Issue with creating Inbox table, {e.Message}"); + throw; + } +} + +``` + +## Clearing the Inbox + +As of V9, clearing the inbox is deferred to the implementer i.e. Brighter will not do this for you. Typically this involves creating a cron job, or agent, that clears inbox entries that are outside of the window during which they may be resent. + +Later versions of Brighter may include data retention policy options that let you configure clearing an inbox. + +## Non-Transactional Inbox + +As of V9 Brighter's inbox is not transactional, that is it does not participate in the transaction that may write to disk as a result of processing a message. This means that the inbox could fail if your changes to state as a result of processing a request are made, but the inbox is not updated. + +Later versions of Brighter may address including the inbox within a transaction, as outbox does today. + +## Implementing an Inbox + +You can refer to existing inbox implementations if you need to implement an inbox that Brighter does not support. + diff --git a/source/shared/BrighterOutboxSupport.md b/source/shared/BrighterOutboxSupport.md new file mode 100644 index 0000000..febc3d4 --- /dev/null +++ b/source/shared/BrighterOutboxSupport.md @@ -0,0 +1,181 @@ +# Outbox Support + +Brighter supports storing messages that are sent via an External Bus in an Outbox, as per the [Outbox Pattern](/contents/OutboxPattern.md) + +This allows you to determine that a change to an entity owned by your application should always result in a message being sent i.e. you have Transactional Messaging. + +There are two approaches to using Brighter's Outbox: + +* Post: This does not offer Transactional Messaging, but does offer replay +* Deposit and Clear: This approach offers Transactional Messaging. + +The **Post** method on the CommandProcessor in Brighter writes first to the **Outbox** and if that succeeds to the Message-Oriented Middleware. If you use Post, then your correctness options are **Ignore/Retry** or **Compensation**. You can use **Post** with **Log Tailing** or **Event Change Capture** but you have to implement those yourself. + +The **DepositPost** and **ClearOutbox** methods allow you to use the **Outbox** pattern instead. + +## Post + +In this approach you choose to **CommandProcessor.Post** a message after your Db transaction writes entity state to the Db. You intend to rely on the *retrying* the call to the broker if it fails. You should make sure that you have setup your **CommandProcessor.RETRYPOLICY** policy with this in mind. + +One caveat here is to look at the interaction of the retry on Post and any **UsePolicy** attribute for the handler. If your **CommandProcessor.RETRYPOLICY** policy bubbles up an exception following the last Retry attempt, and your **UsePolicy** attribute for the handler then catches that exception for your handler and forces a Retry, you will end up re-running the database transaction, which may result in duplicate entries. Your **UsePolicy** attribute for the handler needs to explicitly catch the Db errors you wish to retry, and not errors Posting to the message queue in this case. + +(As an aside, you should generally write Retry policies to catch specific errors that you know you can retry, not all errors anyway). + +In this case, you might also need to consider using a **Fallback** method via the FallbackPolicy attribute to catch **CommandProcessor.Post** exceptions that bubble out and issue a reversing transaction to kill any Db entries made +in error, or raise a log to ensure that there will be manual compensation. + +**CommandProcessor.Post** still uses the **Outbox** to store messages you send, but you are not including them in the Db transaction scope, so you have no **guarantees**. + +If the failure was on the call to the transport, and not the write to the **Outbox**, you will still have a **Outbox** entry that you can resend via manual compensation later. If the message is posted to the +broker, it **must** have already been written to the **Outbox**. + +In you fail to write to the **Outbox**, but have successfully written the entity to the Db, you would need to compensate by reversing the write to the Db in a **Fallback** handler. + +## Deposit and Clear + +Brighter allows the write to the **Outbox** and the write to the Broker to be separated. This form or Brighter allows you to support Producer-Consumer correctness via the **Outbox Pattern**. + +Metaphorically, you can think of this as a post box. You deposit a letter in a post box. Later the postal service clears the post box of letters and delivers them to their recipients. + +Within your database transaction you write the message to the Outbox with **CommandProcessor.DepositPost**. This means that if the entity write succeeds, the corresponding write to the **Outbox** will have +taken place. This method returns the Id for that message. + +(Note that we use **CommandProcessor.RETRYPOLICY** on the write, but this will only impact the attempt to write within the transaction, not the success or failure of the overall Db transaction, which is under +your control. You can safely ignore Db errors on this policy within this approach for this reason.) + +You can then call **CommandProcessor.ClearPostBox** to flush one or more messages from the **Outbox** to the broker. We support multiple messages as your entity write might possibly involve sending multiple downstream messages, which you want to include in the transaction. + +It provides a stronger guarantee than the **CommandProcessor.Post** outside Db transaction with Retry approach as the write to the **Outbox** shares a transaction with the persistence of entity state. + + +## Bulk Deposit + +Starting in v9.2.1 Brighter allows a batch of Messages to be written to the **Outbox**. If your outbox suoports Bulk (This will become a requirement in v10) **CommandProcessor.DepositPost** can be used to deposit a large number of messages in much quicker than individually. + +When creating your **CommandProcessor** you can set an outbox bulk chunk size, if the amount of mesages to be deposited into the **Outbox** is greater than this number it will be broken up into chunks of no more than this size. + +## Participating in Transactions + +Brighter has the functionality to allow the **Outbox** to participate in the database transactions of your application so that you can ensure that distributed requests will be persisted (or fail to persist) inline with application changes. + +To have the Brighter **Outbox** participate in Database transactions the command process must be built specifying a **IAmABoxTransactionConnectionProvider**, this connection provider will be used when **CommandProcessor.DepositPost** is called and if there is an active transactions the **Outbox** will participate in the active transaction provider by the specified **IAmABoxTransactionConnectionProvider**. + +It is important to note that **CommandProcessor.Clear** and **CommandPorcessor.Post** will never participate in transactions as the purpose transaction participation is to ensure that **Outbox** messages are committed (or fail to commit) in the same transaction as application entity changes. + +Below is an example using a UnitOfWork that wraps the database connection for your application +``` csharp +//Begin Database transaction +unitOfWork.BeginTransaction(); + +try +{ + //Update applicationEntities + var updatedContact = contactsService.UpdateContact(contact); + + //Deposit the message in the outbox + commandProcess.DepositPost(updatedContact.ToBrighterMessage()); + + //Commit Transaction + unitOfWork.CommitTransaction(); +} +catch(Exception e) +{ + // If there was an error during processing, rollback all changes + unitOfWork.RollbackTransaction(); +} +``` + +## Implicit or Explicit Clearing of Messages from the Outbox + +There are two approaches to dispatching messages from Brighter's **Outbox** + * Implicitly: This relies on a **Sweeper** to dispatch messages out of process + * Explicitly: This ensures that your message is sent sooner but will processing time to your application code. + +To explicitly clear a message you can call **CommandProcessor.ClearOutbox** directly in your handler, after the Db transaction completes. This has the lowest latency. You are responsible for tracking the ids of messages that you wish to send in **CommandProcessor.ClearOutbox**, we do not maintain this state for you. Note that you cannnot guarantee that this will succeed, although you can Retry. We use **CommandProcessor.RETRYPOLICY** on the write to the Broker, and you should retry errors writing to the Broker in that policy. However, as the message is now in the **Outbox** you can compensate for eventual failure to write to the Broker by replaying the message from the **MessageStore** at a later time. + +To implicitly clear messages from your outbox, configure a **Outbox Sweeper** to listen to your **Outbox** and dispatch messages for you. Once an **Outbox Sweeper** is running you no longer need to call **CommandProcessor.ClearOutbox** however you still have the choice to if you feel a specific message is time sensitive. + +## Outbox Sweeper + +The **Outbox Sweeper** is an out of process service that monitors an **Outbox** and dispatches messages that have yet to be dispatches. Using **Outbox Sweeper** has a lower latency impact for your application, but because it keeps trying to send the messages until it succeeds is the recommended approach to *Guranteed, At Least Once, Delivery*. + +The benefits of using an **Outbox Sweeper** are: + * If there is a failure dispatch a message after it is committed to the **Outbox** it will be retried until it is dispatches + * The ability to choose between the implicit and explicit clearing of messages + +The **Timed Outbox Sweeper** has the following configurables + * TimerInterval: The amount of seconds to wait between checks for undispatches messages (default: 5) + * MinimumMessageAge: The age a message (in miliseconds) that a messages should be before the **OutboxSweeper** should attempt to dispatch it. (default: 5000) + * BatchSize: The number of messages to attempt to dispatch in each check (default: 100) + * UseBulk: Use Bulk dispatching of messages on your **Messaging Gateway** (default: false), note: not all **messaging Gateway**s support Bulk dispatching. + +It is important to note that the lower the Minimum Message age is the more likely it is that your message will be dispatches more than once (as if you are explicitly clearing messages your application may have instructed the clearing of a message at the same time as the **Outbox Sweeper**) + +## Outbox Archiver + +The **Outbox Archiver** is an out of process services that monitors an **Outbox** and will archive messages of older than a certain age. + +The **Timed Outbox Archiver** has the following configurables + * TimerInterval: The number of seconds to wait between checked for messages eligable for archival (default: 15) + * BatchSize: The maximum number of messages to archive for each check (default: 100) + * MinimunAge: The time ellapsed since a message was dispated in hours before it is eligable for archival (default: 24) + +### Outbox Configuration + +Your outbox is configured as part of the Brighter extensions to ServiceCollection. See [Outbox Configuration](/contents/BrighterBasicConfiguration.md#outbox-support) for more. + +### Outbox Builder + +Brighter contains DDL to configure your Outbox. For each supported database we include an **OutboxBuilder**. The Inbox Builder **GetDDL** which allows you to obtain the DDL statements required to create an Outbox. You can use this as part of your application start up to configure the Outbox if it does not already exist. + +The following example shows creation of a MySql outbox. + +We assume that OUTBOX_TABLE_NAME is a constant, shared with the code that configures your inbox. + +``` csharp + +public static IHost CreateOutbox(this IHost webHost) +{ + using (var scope = webHost.Services.CreateScope()) + { + var services = scope.ServiceProvider; + var env = services.GetService(); + var config = services.GetService(); + + CreateOutbox(config, env); + } + + return webHost; +} + +private static void CreateOutbox(IConfiguration config, IWebHostEnvironment env) +{ + try + { + var connectionString = config.GetConnectionString("Greetings"); + + using var sqlConnection = new MySqlConnection(connectionString); + sqlConnection.Open(); + + using var existsQuery = sqlConnection.CreateCommand(); + existsQuery.CommandText = MySqlOutboxBuilder.GetExistsQuery(OUTBOX_TABLE_NAME); + bool exists = existsQuery.ExecuteScalar() != null; + + if (exists) return; + + using var command = sqlConnection.CreateCommand(); + command.CommandText = MySqlOutboxBuilder.GetDDL(OUTBOX_TABLE_NAME); + command.ExecuteScalar(); + + } + catch (System.Exception e) + { + Console.WriteLine($"Issue with creating Outbox table, {e.Message}"); + //Rethrow, if we can't create the Outbox, shut down + throw; + } +} + +``` + + diff --git a/source/shared/BuildingAPipeline.md b/source/shared/BuildingAPipeline.md new file mode 100644 index 0000000..31ee49b --- /dev/null +++ b/source/shared/BuildingAPipeline.md @@ -0,0 +1,175 @@ +# Building a Pipeline of Request Handlers + +Once you are using the features of Brighter to act as a [command dispatcher](CommandsCommandDispatcherAndProcessor.html#command-dispatcher) and send or publish messages to a target handler, you may want to use +its [command processor](CommandsCommandDispatcherAndProcessor.html#command-processor) features to handle orthogonal operations. + +Common examples of orthogonal operations include: + +- Logging the Command +- Providing integration with tools for monitoring performance and + availability +- Validating the Command +- Supporting idempotency of messages +- Supporting re-sequencing of messages +- Handling exceptions +- [Providing Timeout, Retry, and Circuit Breaker + support](QualityOfServicePatterns.html) +- Providing undo support, or rollback + +## The Pipes and Filters Architectural Style + +To handle these orthogonal concerns our [command processor](CommandsCommandDispatcherAndProcessor.html#command-processor) uses a pipes and filters architectural style: the filters are where +processing occurs, they do not share state with other filters, nor do they know about adjacent filters. The pipe is the connector between the filters in our case this is provided by the +**IHandleRequests\** interface which has a method **IHandleRequests\ Successor** that allows us to chain filters together. + +![PipesAndFilters](_static/images/PipesAndFilters.png) + +The sink handler is handler that is the receiver you wish to invoke the action on. The pump is the **Command Dispatcher**. We occasionally use *target handler* as a synonym for *sink handler* + +## The Russian Doll Model + +Our pipes and filters approach supports the *Russian Doll Model* of calling the handler pipeline, a context bag for the pipeline, and support for generating a request path description out-of-the-box. + +The *Russian Doll Model* is names for the [Matryoshka](https://en.wikipedia.org/wiki/Matryoshka_doll) wooden dolls, in which dolls of decreasing sizes are nested one inside another. The importance of this for a [pipes and filters pattern](https://msdn.microsoft.com/en-us/library/dn589788.aspx) style is that each filter in the pipeline is called within the scope of a previous filter in the pipeline. + +![RussianDoll](_static/images/RussianDoll.png) + +This is significant because you may desire to act before and after a subsequent filter step. One particular use case is exception handling: a try-catch block that wraps the call to a subsequent step can react to +exceptions raised by subsequent steps. This allows us to create policy decisions around exceptions using a library such as [Polly](https://github.com/App-vNext/Polly) and thus support [Retry](https://msdn.microsoft.com/en-us/library/dn589788.aspx) and [Circuit Breaker](https://msdn.microsoft.com/en-gb/library/dn589784.aspx?f=255&MSPPError=-2147217396) + +Our usage of the Russian Doll Model was inspired by [FubuMVC](http://codebetter.com/jeremymiller/2011/01/09/fubumvcs-internal-runtime-the-russian-doll-model-and-how-it-compares-to-asp-net-mvc-and-openrasta/) + +## Implementing a Pipeline + +The first step in building a pipeline is to decide that we want an orthogonal operation in our pipeline. Let us assume that we want to do basic request logging. + +Because you do not want to write an orthogonal handler for every Command or Event type, these handlers should remain generic types. At runtime the HandlerFactory creates an instance of the generic type specialized for the type parameter of the Command or Event being passed along the pipeline. + +The limitation here is that you can only make assumptions about the type you receive into the pipeline from the constraints on the generic type. + +Although it is possible to implement the [IHandleRequests](https://github.com/BrighterCommand/Brighter/blob/master/src/Paramore.Brighter/IHandleRequests.cs) interface directly, we recommend deriving your handler from [RequestHandler](https://github.com/BrighterCommand/Brighter/blob/master/src/Paramore.Brighter/RequestHandler.cs\). + +Let us assume that we want to log all requests travelling through the pipeline. (We provide this for you in the Brighter.CommandProcessor packages so this for illustration only). We could implement a generic +handler as follows: + +``` csharp +using System; +using Newtonsoft.Json; +using Brighter.commandprocessor.Logging; + +namespace Brighter.commandprocessor +{ + public class RequestLoggingHandler + : RequestHandler where TRequest : class, IRequest + { + private HandlerTiming _timing; + + public override void InitializeFromAttributeParams( + params object[] initializerList + ) + { + _timing = (HandlerTiming)initializerList[0]; + } + + public override TRequest Handle(TRequest command) + { + LogCommand(command); + return base.Handle(command); + } + + private void LogCommand(TRequest request) + { + logger.InfoFormat("Logging handler pipeline call. Pipeline timing {0} target, for {1} with values of {2} at: {3}", + _timing.ToString(), + typeof(TRequest), + JsonConvert.SerializeObject(request), + DateTime.UtcNow); + } + } +} +``` + +Our Handle method is the method which will be called by the pipeline to service the request. After we log we call **return base.Handle(command)** to ensure that the next handler in the chain is +called. If we failed to do this, the *target handler* would not be called nor any subsequent handlers in the chain. This call to the next item in the chain is how we support the \'Russian Doll\' model - because +the next handler is called within the scope of this handler, we can manage when it is called handle exceptions, units of work, etc. + +It is worth remembering that handlers may be called after the target handler (in essence you can designate an orthogonal handler as the sink handler when configuring your pipeline). For this reason **all** handlers should remember to call their successor, **even your target handler**. + +We now need to tell our pipeline to call this orthogonal handler before our target handler. To do this we use attributes. The code we want to write looks like this: + +``` csharp +class GreetingCommandHandler : RequestHandler +{ + [RequestLogging(step: 1, timing: HandlerTiming.Before)] + public override GreetingCommand Handle(GreetingCommand command) + { + Console.WriteLine("Hello {0}", command.Name); + return base.Handle(command); + } +} +``` + +The **RequestLogging** Attribute tells the Command Processor to insert a Logging handler into the request handling pipeline before (**HandlerTiming.Before**) we run the target handler. It tells the Command Processor that we want it to be the first handler to run if we have multiple orthogonal handlers i.e. attributes (**step: 1**). + +We implement the **RequestLoggingAttribute** by creating our own Attribute class, derived from **RequestHandlerAttribute**. + +``` csharp +public class RequestLoggingAttribute : RequestHandlerAttribute +{ + public RequestLoggingAttribute(int step, HandlerTiming timing) + : base(step, timing) + { } + + public override object[] InitializerParams() + { + return new object[] { Timing }; + } + + public override Type GetHandlerType() + { + return typeof(RequestLoggingHandler<>); + } +} +``` + +The most important part of this implementation is the GetHandlerType() method, where we return the type of our handler. At runtime the Command Processor uses reflection to determine what attributes are on the target handler and requests an instance of that type from the user-supplied **Handler Factory**. + +Your Handler Factory needs to respond to requests for instances of a **RequestHandler\** specialized for a concrete type. For example, if you create a **RequestLoggingHandler\** we will ask you for a **RequestLoggingHandler\** etc. Depending on your implementation of HandlerFactory, you may need to register an implementation for every concrete instance of your handler with your +underlying IoC container etc. + +Note that as we rely on an user supplied implementation of **IAmAHandlerFactory** to instantiate Handlers, you can have any dependencies in the constructor of your handler that you can resolve at +runtime. In this case we pass in an ILog reference to actually log to. + +You may wish to pass parameter from your Attribute to the handler. Attributes can have constructor parameters or public members that you can set when adding the Attribute to a target method. These can only be +compile time constants, see the documentation [here](https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/attributes/). After the Command Processor calls your Handler Factory to create an +instance of your type it calls the **RequestHandler.InitializeFromAttributeParams** method on that created type and passes it the object array defined in the **RequestHandlerAttribute.InitializerParams**. By this approach, you can pass parameters to the handler, for example the Timing parameter is passed to the handler above. + +It is worth noting that you are limited when using Attributes to provide constructor values that are compile time constants, you cannot pass dynamic information. To put it another way you are limited to value set +at design time not at run time. + +In fact, you can use this approach to pass any data to the handler on initialization, not just attribute constructor or property values, but you are constrained to what you can access from the context of the +Attribute at run time. it can be tempting to set retrieve global state via the [Service Locator](https://en.wikipedia.org/wiki/Service_locator_pattern) pattern at this point. Avoid that temptation as it creates coupling between your Attribute and global state reducing modifiability. + +## Using a Manual Approach + +Using an attribute based approach is not an approach favoured by everyone. Some people prefer a more explicit approach to configuring the pipeline. + +The trick is to remember that any handler that derives from **IHandleRequests\** has a **Successor** and you can build a chain by having the first handler call the second handler\'s +**Handle()** method i.e. **Successor.Handle()**. You can derive from **RequestHandler\** and call **base.Handle()** for this, even if you don\'t want to use the Attribute based pipelines. + +In the SubscriberRegistry you just register the first Handler in your pipeline. When we lookup the Handler for the Command in the SubscriberRegistry we will call it\'s Handle method. It can execute your +code, and then call it\'s Successor (using the Russian Doll approach). + +``` csharp +var myCommandHandler = new MyCommandHandler(); +var myLoggingHandler = new MyLoggingHandler(log); + +myLoggingHandler.Successor = myCommandHandler; + +var subscriberRegistry = new SubscriberRegistry(); +subscriberRegistry.Register(); +``` + +It is worth noting that as you control the HandlerFactory, you could also register the sink handler, but when instantiating an instance of it on request, build the pipeline of handlers yourself. + +We think it is easier to use attributes, but there may be circumstances where that approach does not work, and so this option is supported as well. diff --git a/source/shared/BuildingAnAsyncPipeline.md b/source/shared/BuildingAnAsyncPipeline.md new file mode 100644 index 0000000..0a5fedd --- /dev/null +++ b/source/shared/BuildingAnAsyncPipeline.md @@ -0,0 +1,107 @@ +# Building a Pipeline of Async Request Handlers + +Once you are using the features of Brighter to act as a [command dispatcher](CommandsCommandDispatcherAndProcessor.html#command-dispatcher) and send or publish messages to a target handler, you may want to use +its [command processor](CommandsCommandDispatcherAndProcessor.html#command-processor) features to handle orthogonal operations. + +# Implementing a Pipeline + +The first step in building a pipeline is to decide that we want an orthogonal operation in our pipeline. Let us assume that we want to do command sourcing. + +Because you do not want to write an orthogonal handler for every Command or Event type, these handlers should remain generic types. At runtime the framework will request HandlerFactory creates an instance of the +generic type specialized for the type parameter of the Command or Event being passed along the pipeline. + +The limitation here is that you can only make assumptions about the type you receive into the pipeline from the constraints on the generic type. + +Although it is possible to implement the +[IHandleRequestsAsync](https://github.com/BrighterCommand/Brighter/blob/master/src/Paramore.Brighter/IHandleRequestsAsync.cs) +interface directly, we recommend deriving your handler from +[RequestHandlerAsync\ +\]{.title-ref}\_\_. + +Let us assume that we want to log all requests travelling through the pipeline. (We provide this for you in the Brighter.CommandProcessor packages so this for illustration only). We could implement a generic +handler as follows: + +``` csharp +public class CommandSourcingHandlerAsync : RequestHandlerAsync where T : class, IRequest +{ + private readonly IAmACommandStoreAsync _commandStore; + + public CommandSourcingHandlerAsync(IAmACommandStoreAsync commandStore) + { + _commandStore = commandStore; + } + + public override async Task HandleAsync(T command, CancellationToken? ct = null) + { + await _commandStore.AddAsync(command, -1, ct).ConfigureAwait(ContinueOnCapturedContext); + } +} +``` + +Our HandleAsync method is the method which will be called by the pipeline to service the request. After we log we call **return await base.HandleAsync(command, ct)** to ensure that the next handler in the +chain is called. + +If we failed to do this, the *target handler* would not be called nor any subsequent handlers in the chain. This call to the next item in the chain is how we support the \'Russian Doll\' model - because the next +handler is called within the scope of this handler, we can manage when it is called handle exceptions, units of work, etc. + +It is worth remembering that handlers may be called after the target handler (in essence you can designate an orthogonal handler as the sink handler when configuring your pipeline). For this reason **all** handlers should remember to call their successor, **even your target handler**. + +We now need to tell our pipeline to call this orthogonal handler before our target handler. To do this we use attributes. The code we want to write looks like this: + +``` csharp +internal class GreetingCommandRequestHandlerAsync : RequestHandlerAsync +{ + [UseCommandSourcingAsync(step: 1, timing: HandlerTiming.Before)] + public override async Task HandleAsync(GreetingCommand command, CancellationToken? ct = null) + { + var api = new IpFyApi(new Uri("https://api.ipify.org")); + + var result = await api.GetAsync(ct); + + Console.WriteLine("Hello {0}", command.Name); + Console.WriteLine(result.Success ? "Your public IP addres is {0}" : "Call to IpFy API failed : {0}", result.Message); + return await base.HandleAsync(command, ct).ConfigureAwait(base.ContinueOnCapturedContext); + } +} +``` + +The **UseCommandSourcingAsync** Attribute tells the Command Processor to insert a Logging handler into the request handling pipeline before (**HandlerTiming.Before**) we run the target handler. It tells the +Command Processor that we want it to be the first handler to run if we have multiple orthogonal handlers i.e. attributes (**step: 1**). + +We implement the **UseCommandSourcingAsyncAttribute** by creating our own Attribute class, derived from **RequestHandlerAttribute**. + +``` csharp +public class UseCommandSourcingAsyncAttribute : RequestHandlerAttribute +{ + + public UseCommandSourcingAsyncAttribute(int step, HandlerTiming timing = HandlerTiming.Before) + : base(step, timing) + { } + + + public override Type GetHandlerType() + { + return typeof (CommandSourcingHandlerAsync<>); + } +} +``` + +The most important part of this implementation is the GetHandlerType() method, where we return the type of our handler. At runtime the Command Processor uses reflection to determine what attributes are on the target handler and requests an instance of that type from the user-supplied **Handler Factory**. + +Your Handler Factory needs to respond to requests for instances of a **RequestHandlerAsync\** specialized for a concrete type. For example, if you create a **CommandSourcingHandlerAsync\** we +will ask you for a **CommandSourcingHandlerAsync\** etc. Depending on your implementation of HandlerFactory, you may need to register an implementation for every concrete instance of your handler +with your underlying IoC container etc. + +Note that as we rely on an user supplied implementation of **IAmAHandlerFactoryAsync** to instantiate Handlers, you can have any dependencies in the constructor of your handler that you can resolve at +runtime. In this case we pass in an ILog reference to actually log to. + +You may wish to pass parameter from your Attribute to the handler. Attributes can have constructor parameters or public members that you can set when adding the Attribute to a target method. These can only be +compile time constants, see the documentation [here](https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/language-specification/attributes). + +After the Command Processor calls your Handler Factory to create an instance of your type it calls the **RequestHandler.InitializeFromAttributeParams** method on that created type and passes it the object array defined in the **RequestHandlerAttribute.InitializerParams**. By this approach, you can pass parameters to the handler, for example the Timing parameter is passed to the handler above. + +It is worth noting that you are limited when using Attributes to provide constructor values that are compile time constants, you cannot pass dynamic information. To put it another way you are limited to value set +at design time not at run time. + +In fact, you can use this approach to pass any data to the handler on initialization, not just attribute constructor or property values, but you are constrained to what you can access from the context of the +Attribute at run time. It can be tempting to set retrieve global state via the [Service Locator](https://en.wikipedia.org/wiki/Service_locator_pattern) pattern at this point. Avoid that temptation as it creates coupling between your Attribute and global state reducing modifiability. diff --git a/source/shared/ClaimCheck.md b/source/shared/ClaimCheck.md new file mode 100644 index 0000000..912dbe2 --- /dev/null +++ b/source/shared/ClaimCheck.md @@ -0,0 +1,66 @@ +# Claim Check + +The [Claim Check](https://www.enterpriseintegrationpatterns.com/patterns/messaging/StoreInLibrary.html) pattern helps us reduce the size of our messages, without losing information that we need to exchange. + +Instead of being transmitted in the body of the message, the payload is written to a distributed file storage and a token to retrieve the payload is sent instead. The receiver can read the payload by taking the reference and requesting it from the distributed file storage. The metaphor here is a luggage check. Instead of carrying large items of luggage aboard an aircraft we check them into the hold of the aircraft. The airline gives us a claim check for our luggage, that matches a tag on the bag. This pattern is sometimes called Reference Based Messaging. + +## Claim Check and Retrieve Claim + +We treat the Claim Check pattern as [Transformer](/contents/MessageMappers.md#message-transformer-factory) middleware. + +We provide a **WrapWithAttribute** of **ClaimCheck** that will use the **ClaimCheckTransformer** to upload the body of your **Message** to a *luggage store* replacing it with a body that contains an claim check for the body, as well as setting a message header of "claim_check_header" with the claim. The trigger for this behavior can be controlled by a threshold parameter that sets the size above which the message body should be moved to the *luggage store*. + +In the following example we add the **ClaimCheck** attribute to the *Message Mapper* with a trigger at 256Kb + +``` csharp +[ClaimCheck(step:0, thresholdInKb: 256)] +public Message MapToMessage(GreetingEvent request) +{ + var header = new MessageHeader(messageId: request.Id, topic: typeof(GreetingEvent).FullName.ToValidSNSTopicName(), messageType: MessageType.MT_EVENT); + var body = new MessageBody(JsonSerializer.Serialize(request, JsonSerialisationOptions.Options)); + var message = new Message(header, body); + return message; +} +``` + +We provide a matching **UnwrapWithAttribute** of **RetrieveClaim** that will use the **ClaimCheckTransformer** to download the body of your **Message** from a luggage store and replace the existing body (likely a claim check reference) with the downloaded content. + +``` csharp +[RetrieveClaim(0, retain:false)] +public GreetingEvent MapToRequest(Message message) +{ + var greetingCommand = JsonSerializer.Deserialize(message.Body.Value, JsonSerialisationOptions.Options); + + return greetingCommand; +} + +``` + +An optional parameter 'retain' determines if we keep the body in storage after it is retrieved or delete it. The default is to delete it. + +The outcome of these attributes is that the uploading of the body to the *luggage store* and downloading from it is transparent to your code. You serialize your **IRequest** to a **Message** as normal, or serialize your **Message** to an **IRequest** as normal - everything happens in the middleware pipeline. + +## The Luggage Store + +The *luggage store* is where we store the body of the message for later retrieval. We provide implementations of the Luggage Store interface for popular distributed stores, but you can implement the interface for any that we do not provide. + +```csharp + + public interface IAmAStorageProviderAsync + { + Task DeleteAsync(string claimCheck, CancellationToken cancellationToken); + Task DownloadAsync(string claimCheck, CancellationToken cancellationToken); + Task HasClaimAsync(string claimCheck, CancellationToken cancellationToken); + Task UploadAsync(Stream stream, CancellationToken cancellationToken); + } + +``` + +* DeleteAsync: Deletes a item from the store +* DownloadAsync: Creates a stream for a download from the store +* HasClaimAsync: Does the claim check exist in the store +* UploadAsync: Uploads a stream to the store and returns a claim, an identifier that can later be used to delete, download or check for the existence of the file uploaded to the store. + +We provide the following implementations of **IAmAStorageProviderAsync: + +* [S3LuggageStore](/contents/S3LuggageStore.md) \ No newline at end of file diff --git a/source/shared/CommandsCommandDispatcherandProcessor.md b/source/shared/CommandsCommandDispatcherandProcessor.md new file mode 100644 index 0000000..950ca4b --- /dev/null +++ b/source/shared/CommandsCommandDispatcherandProcessor.md @@ -0,0 +1,109 @@ +# Command Patterns + +## Command + +The **Command** design pattern encapsulates a request as an object, allowing reuse, queuing or logging of requests, or undoable operations. It also serves to decouple the implementation of the request from the +requestor. The caller of a Command object does not need to understand how the Command is implemented, only that the Command exists. When the caller and the implementer are decoupled it becomes easy to replace or +refactor the implementation of the request, without impacting the caller - our system is more modifiable. Our ability to test the Command in isolation of the caller - allows us to implement the ports and +adapters model easily - we can instantiate the Command, provide \'fake\' parameters to it and confirm the results. We can also use the command from multiple callers, although this is not a differentiator from the +service class approach. + +![Command](_static/images/Command.png) + +**Command** - Declares an interface for executing an operation. + +**ConcreteCommand** --Defines a binding between a Receiver object and an action. Implements Execute by invoking the corresponding operation(s) on the Receiver. + +**Client** -- creates a ConcreteCommand object and sets its receiver. + +**Invoker** - asks the command to carry out the request. + +![CommandWorkflow](_static/images/CommandWorkflow.png) + +An **Invoker** object knows about the **Concrete Command** object. The Invoker issues a request by calling Execute on the **Command**. When commands are un-doable, the Command stores state for undoing the command +prior to invoking Execute.The Command object invokes operations on its **Receiver** to carry out the request + +In addition we can structure a system transactionally using Commands. A Command is a transactional boundary. Because a Command is a transactional boundary, when using the [Domain Driven Design](https://en.wikipedia.org/wiki/Domain-driven_design) technique of an aggregate there is a natural affinity between the Command, which operates on a transactional boundary and the Aggregate which is a transactional boundary within the domain model. The Aggregate is the Receiver stereotype within the Command Design pattern. Because we want to separate use of outgoing Adapters via a secondary Port, such as a Repository +[Repository](https://martinfowler.com/eaaCatalog/repository.html) in the DDD case, this can lead to a pattern for implementation of a Command: + +> 1. Begin Transaction +> 2. Load from Repository +> 3. Operate on Aggregate +> 4. Flush to Repository +> 5. Commit Transaction + +In the Repository pattern we may need to notify other Aggregates that can be eventually consistent of the change within the transactionaly consistent boundary. The pattern suggested there is a notification. Because the handling of that notification is in itself likely to be a transactional boundary for a different aggregate we can encapsulate this domain event with the Command design pattern as well, which gives rise to the following additional step to the sequence,outside the original transactional boundary: + +> 6. Invoke Command Encapsulating Notification + +This has obvious similarities to the [actor model](https://en.wikipedia.org/wiki/Actor_model), particularly if you use an External Bus. + +The problems with the Command pattern are that the caller is coupled to a specific Command at the call site - which undermines the promise of being extensible through use of Commands. To change that Command, or +call orthogonal services before calling the command requires us to amend the calling code, wherever the Command is used. To decouple a higher and lower layers we want to be able alter the implementation of the commands that we call on the lower layer without altering the calling layer. + +## Command Dispatcher + +Brighter is a .NET implementation of the **Command Dispatcher** pattern. + +*This pattern increases the flexibility of applications by enabling their services to be changed, by adding, replacing or removing any command handlers at any point in time without having to modify, recompile or statically relink the application. By simulating the command-evaluation feature common in interpreted languages, this pattern supports the need for continual, incremental evolution of applications.* + +A [Command Dispatcher](https://en.wikipedia.org/wiki/Command_pattern) is often used with a hierarchical architecture to avoid the [Fat Controller problem](https://github.com/BrighterCommand/Brighter/wiki/Fat-Controllers) and allow us to [decouple from the caller](https://github.com/BrighterCommand/Brighter/wiki/Why-use-a-Command-Processor). + +An Action-Request object is an object that both encapsulates the identity of the action we want to fire and the parameters for this action, i.e. the extrinsic state of the action to undertake. In other words, an Action-Request object is a representation of the action to undertake, which is identified using a key, possibly a string such as \'set_depth\'. An Action-Handler is the object that knows how to perform a particular action, and is passed the parameters at run-time. It is therefore a shared object that can be used in multiple contexts simultaneously. The Command-Dispatcher is the object that links the Action-Request to the appropriate Action Handler object. It has a dictionary that contains a reference to all the registered Action-Handlers. The Command-Dispatcher uses the Action-Request\'s key to find the right entry and dispatches the appropriate Action-Handler. The Action Handler can then perform the requested action. + +We want to separate an Action-Request object that contains the identity of the action we want to perform, and the parameter for that action from the Action-Handler which knows how to perform that action. + +A Command Dispatcher is an object that links the Action-Request with the appropriate Action-Handler. + +We may distinguish between a Command Action-Request that has one Action Handler and an Event Action-Request that has many + +The Command Dispatcher allows dynamic registration and removal of Command Handlers, it is an administrative entity that manages linking of commands to the appropriate command handlers. + +It relates to the Observer pattern in that hooks together publishers and subscribers. + +Command Dispatcher registration requires a key -- provided by the Command Dispatcher for the Commands it can service, using getKey(). \[In practice we often use RTTI for this\]. + +The Command Handler is fired, when a command with the same name (key) is sent to the Command Dispatcher. + +The Command Dispatcher is a repository of key-value pairs (key., Command Handler) and when the Command Dispatcher is called it looks up the command's key in the repository. If there is a match it calls the +appropriate method(s) on the handler to process the Command. + +![CommandDispatcher](_static/images/CommandDispatcher.png) + +**Invoker** - has a lit of Commands that are to be executed + +**Command** - represents the request to be processed, encapsulating the parameters to be passed to the command-handler to perform the request + +**Command Handler** - specifies the interface that any command handler must implement + +**Concrete Command Handler** -- implements the request + +**Command Dispatcher** -- Allows dynamic registration of Command Handlers and looks up handlers for commands, by matching command and handler key. + +**Client** -- registers Commands with the Command Dispatcher. + +![CommandExtendedWorkflow](_static/images/CommandExtendedWorkflow.png) + +A Command Dispatcher can also act as the port layer in a [Ports & Adapters architecture](http://alistair.cockburn.us/Hexagonal+architecture). + +## Command Processor + +Brighter is a .NET implementation of the [Command Processor pattern](https://wiki.hsr.ch/APF/files/CommandProcessor.pdf). + +The Command Processor pattern separates the request for a service from its execution. A Command Processor component manages requests as separate objects, schedules their execution, and provides additional +services such as the storing of request objects for later undo. + +A Command Dispatcher and a Command Processor are similar in that both divorce the caller of a Command from invoker of that Command. However, the motivation is different. A Dispatcher seeks to decouple the caller from the invoker to allow us to easily extend the system without modification to the caller. Conversely the motivation behind a Command Processor is to allows us to implement orthogonal operations such as logging, or scheduling without forcing the sender or receiver to be aware of them. It does this by giving those responsibilities to the invoker. + +Of course as both patterns separate the invoker from sender and receiver, it is possible for us to combine them by having the Command Dispatcher\'s invoker support executing orthogonal concerns when it invokes the Command. + +![CommandProcessor](_static/images/CommandProcessor.png) + +The central command processor easily allows the addition of services related to command execution. An advanced command processor can log or store commands to a file for later examination or replay. A command +processor can queue commands and schedule them at a later time. This is useful if commands should execute at a specified time, if they are handled according to priority, or if they will execute in a separate +thread of control. An additional example is a single command processor shared by several concurrent applications that provides a transaction control mechanism with logging and rollback of commands. + +A Command Processor enforces quality of service and maximizes throughput. A Command Processor forms a juncture at which concerns like: [retry, timeout and circuit breaker](PolicyRetryAndCircuitBreaker.html) +can be implemented for all commands. + +![CommandProcesorCapitalize](_static/images/CommandProcesorCapitalize.png) diff --git a/source/shared/Compression.md b/source/shared/Compression.md new file mode 100644 index 0000000..650e6cc --- /dev/null +++ b/source/shared/Compression.md @@ -0,0 +1,51 @@ +# Compression + +The Compression transform helps us reduce the size of a message using a compression algorithm. It is an efficient approach to reducing the size of a payload. + +We offer [gzip](https://en.wikipedia.org/wiki/Gzip) on netstandard2.0 and add [deflate](https://en.wikipedia.org/wiki/Deflate) and [brotli](https://en.wikipedia.org/wiki/Brotli) on net6+. + +## Compress and Decompress + +We treat Compress and Decompress as [Transformer](/contents/MessageMappers.md#message-transformer-factory) middleware. + +We provide a **WrapWithAttribute** of **Compress** that will use the **CompressPayloadTransformer** to compress the body of your message using your choice of compression algorithm. The claim check has a threshold, over which messages will be compressed. + +In the following example we compress any string larger than 150K + +``` csharp +[Compress(0, CompressionMethod.GZip, CompressionLevel.Optimal, 150)] +public Message MapToMessage(GreetingEvent request) +{ + var header = new MessageHeader(messageId: request.Id, topic: typeof(GreetingEvent).FullName.ToValidSNSTopicName(), messageType: MessageType.MT_EVENT); + var body = new MessageBody(JsonSerializer.Serialize(request, JsonSerialisationOptions.Options)); + var message = new Message(header, body); + return message; +} +``` + +We provide a matching **UnwrapWithAttribute** of **Decompress** that will use the **CompressPayloadTransformer** to decompress the body of your message using the algorithm the message body was compressed with. If the string is not compressed, we take no action. This supports the scenario where some messages on a channel are small enough not to cross the threshold for compression, but others will be large and require compression. (If you want compress all messages on a channel, regardless of individual size, just set your threshold to zero). + +In this example, we look for a GZip compressed string and if we find it, decompress the body. + +```csharp +[Decompress(0, CompressionMethod.GZip)] +public GreetingEvent MapToRequest(Message message) +{ + var greetingCommand = JsonSerializer.Deserialize(message.Body.Value, JsonSerialisationOptions.Options); + + return greetingCommand; +} +``` + + +### Impact of Compression + +When we compress a message we change the *Content Type* header (content-type) for the message to reflect the compressed type: "application/gzip" for GZip, "application/deflate" for Deflate and "application/br" for Brotli. We store the pre-compression content type, in the *Original Content Type* (originalContentType) header. + +Compression produces binary content. Where middleware requires that we transmit the message as text (for example over HTTPs such as SNS) we use a base64 string to ensure that the translation to and from text does not corrupt the data. Because turning binary data into a base64 string inflates it, you may need to adjust for that. As an example, if the limit of the middleware is 256K, a string that compresses to more than 192K will breach your limit. This is particularly useful to note if your strategy is to compress a string, and then use a [Claim Check](ClaimCheck.md) to offload any payloads that remain too large. In the example case your claim check would need to be at 192K and not 256K. + + + + + + diff --git a/source/shared/DapperOutbox.md b/source/shared/DapperOutbox.md new file mode 100644 index 0000000..a7f573b --- /dev/null +++ b/source/shared/DapperOutbox.md @@ -0,0 +1,93 @@ +# Dapper Outbox + +## Usage +The Dapper Outbox allows integration between Dapper and [Brighter's outbox support](/contents/BrighterOutboxSupport.md). The configuration is described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#outbox-support). + +For this we will need the *Outbox* package for Dapper. Packages for Dapper exist for the following RDBMS: MSSQL, MYSQL, and Sqlite. Packages have the naming convention: + +* **Paramore.Brighter.{DB}.Dapper** + +In addition, you will need the Outbox package for the relevant RDBMS: + +* **Paramore.Brighter.Outbox.{DB}** + +Obviously, {DB} should match. In the example below we use MySql, so we would need the following packages: + +* **Paramore.Brighter.MySql.Dapper** +* **Paramore.Brighter.Outbox.MySql** + +**Paramore.Brighter.MySql.Dapper** will pull in another two packages: + +* **Paramore.Brighter.MySql** +* **Paramore.Brighter.Dapper** + +As described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#outbox-support), we configure Brighter to use an outbox with the Use{DB}Outbox method call. + +As we want to use Dapper, we also call: Use{DB}TransactionConnectionProvider so that we can share your transaction scope when persisting messages to the outbox. + + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .UseExternalBus(...) + .UseMySqlOutbox(new MySqlConfiguration(DbConnectionString(), _outBoxTableName), typeof(MySqlConnectionProvider), ServiceLifetime.Singleton) + .UseMySqTransactionConnectionProvider(typeof(Paramore.Brighter.MySql.Dapper.UnitOfWork), ServiceLifetime.Scoped) + .UseOutboxSweeper() + + ... +} + +``` + +In our handler we take a dependency on Brighter's Dapper Unit of Work. We explicitly start a transaction within the handler on the Database within the Unit of Work. Dapper provides extension methods on a DbConnection for typical CRUD operations. Our Unit of Work wraps that DbConnection, and allows you to create a DB transaction associated with that DbConnection. You must use our method, and not create the transaction directly via the connection, because we cannot obtain that transaction. Sharing that transaction allows us to insert a message into the Outbox within the same transaction. + +We call **DepositPostAsync** within that transaction to write the message to the Outbox. Once the transaction has closed we can call **ClearOutboxAsync** to immediately clear, or we can rely on the Outbox Sweeper, if we have configured one to clear for us. (There are equivalent synchronous versions of these APIs).x + +``` csharp + public override async Task HandleAsync(AddGreeting addGreeting, CancellationToken cancellationToken = default(CancellationToken)) +{ + var posts = new List(); + + //We use the unit of work to grab connection and transaction, because Outbox needs + //to share them 'behind the scenes' + + var tx = await _uow.BeginOrGetTransactionAsync(cancellationToken); + try + { + var searchbyName = Predicates.Field(p => p.Name, Operator.Eq, addGreeting.Name); + var people = await _uow.Database.GetListAsync(searchbyName, transaction: tx); + var person = people.Single(); + + var greeting = new Greeting(addGreeting.Greeting, person); + + //write the added child entity to the Db + await _uow.Database.InsertAsync(greeting, tx); + + //Now write the message we want to send to the Db in the same transaction. + posts.Add(await _postBox.DepositPostAsync(new GreetingMade(greeting.Greet()), cancellationToken: cancellationToken)); + + //commit both new greeting and outgoing message + await tx.CommitAsync(cancellationToken); + } + catch (Exception e) + { + _logger.LogError(e, "Exception thrown handling Add Greeting request"); + //it went wrong, rollback the entity change and the downstream message + await tx.RollbackAsync(cancellationToken); + return await base.HandleAsync(addGreeting, cancellationToken); + } + + //Send this message via a transport. We need the ids to send just the messages here, not all outstanding ones. + //Alternatively, you can let the Sweeper do this, but at the cost of increased latency + await _postBox.ClearOutboxAsync(posts, cancellationToken:cancellationToken); + + return await base.HandleAsync(addGreeting, cancellationToken); +} +``` + +## Brighter Unit of Work without Dapper + +Because the Brighter Unit of Work just wraps a DbConnection and is's associated transaction, it can be used to provide a DbTransaction that works with the outbox whenever you want to use DbConnection to interface with a database. Whilst Dapper adds value on top of DbConnection, it just a set of extension methods, and our unit of work does not depend upon Dapper itself. + + diff --git a/source/shared/DarkerBasicConfiguration.md b/source/shared/DarkerBasicConfiguration.md new file mode 100644 index 0000000..464505b --- /dev/null +++ b/source/shared/DarkerBasicConfiguration.md @@ -0,0 +1 @@ +# Basic Configuration \ No newline at end of file diff --git a/source/shared/DispatchingARequest.md b/source/shared/DispatchingARequest.md new file mode 100644 index 0000000..aa12211 --- /dev/null +++ b/source/shared/DispatchingARequest.md @@ -0,0 +1,168 @@ +# Dispatching Requests + +Once you have [implemented your Request Handler](ImplementingAHandler.html), you will want to dispatch **Commands** or **Events** to that Handler. + +## Usage + +In the following example code we register a handler, create a *Command Processor*, and then use that *Command Processor* to dispatch a request to the handler. + + +``` csharp + public class Program + { + private static void Main() + { + var host = Host.CreateDefaultBuilder() + .ConfigureServices((context, collection) => + { + collection.AddBrighter().AutoFromAssemblies(); + }) + .UseConsoleLifetime() + .Build(); + + var commandProcessor = host.Services.GetService(); + + commandProcessor.Send(new GreetingCommand("Ian")); + + host.WaitForShutdown(); + } + } +``` + +## Registering a Handler + +In order for the *Command Processor* to find a Handler for your **Command** or **Event** you need to register the association between that **Command** or **Event** and your Handler. + +Brighter's **HostBuilder** support provides **AutoFromAssemblies** to register any *Request Handlers* in the project. See [Basic Configuration](/contents/BrighterBasicConfiguration.md) for more. If you are not using **HostBuilder** and or **ServiceCollection** you will need to register your handlers yourself. See [How Configuring the Command Processor Works](/contents/HowConfiguringTheCommandProcessorWorks.md). + +### Taking a Dependency on a Command Processor + +#### Producers + +Typically, a producer is an ASP.NET WebAPI or MVC app. In this case you take a dependency in your *Controller* on the **IAmACommandProcessor** interface, which is satisfied via *ServiceCollection*. + +If you intend to dispatch messages to another app, via message oriented middleware, your Brighter configuration will need a **Publication** which identifies how to to do that. + +#### Consumer + +An *Internal Bus* consumer is just a handler, typically registered through Brighter's ServiceCollection integration via our HostBuilder extension. It can thus take dependencies on other registered services within your app. + +An *External Bus* consumer is just a handler, but typically you host it using Brighter's *Service Activator*. You configure *Service Activator* to listens to messages flowing over message oriented middleware through a **Subscription**. *Service Activator* takes care of listening to messages arriving via the middleware, and delivering them to your handler code. In this way the complexity of using middleware is abstracted away from you, and you can focus on the business logic in your handler that you want to run in response to a message. + +### Pipelines Must be Homogeneous + +Brighter only supports pipelines that are solely **IHandleRequestsAsync** or **IHandleRequests**. In particular, note that middleware (attributes on your handler) must be of the same type as the rest of your pipeline. A common mistake is to **UsePolicy** when you mean **UsePolicyAsync**. + +## Dispatching Requests + +Once you have registered your Handlers, you can dispatch requests to them. + +### Internal Bus: Send & Publish + +When using an *Internal Bus*, the *Command Processor* has two options for dispatching messages: + +* **Send**: Used with a **Command**, send expects one, and only one, receiver. +* **Publish**: used with an **Event**, publish expects zero or more receivers. + +All methods have versions that support async...await. + +#### Internal Bus: Sending a Command + +A **Command** is an instruction to do work. We only expect one recipient to do the work, and side-effects mean that we want to ensure that only one receiver actions it as it typically mutates state. + +To send a **Command** you simply use **CommandProcessor.Send()** + +``` csharp +commandProcessor.Send(new GreetingCommand("Ian")); +``` + +NOTE: On a call to **CommandProcessor.Send()** the execution path flows to the handler. The Internal Bus is not buffered. + +#### Internal Bus: Returning results of a Command to the caller. + +Brighter follows Command-Query separation, and a Command does not have return value. So **CommandDispatcher.Send()** does not return anything. Please see a discussion on how to handle this in [Returning Results from a Handler](/contents/ReturningResultsFromAHandler.md). Also note that **Darker** provides our support for a **Query** over an Internal Bus. + +#### Internal Bus: Publishing an Event + +An **Event** is a fact, often the results of work that has been done. It is not atypical to raise an event to indicate the results of a **Command** having been actioned. + +``` csharp +commandProcessor.Publish(new GreetingEvent("Ian has been greeted")); +``` + +NOTE: On a call to **CommandProcessor.Publish()** the execution path flows to all handlers in a loop. The Internal Bus is not buffered. + +### External Bus: Post, Deposit and Clear + +When using an [External Bus](/contents/ImplementingExternalBus.md) the *Command Processor* has two options for dispatching a message: + +* **DepositPost** and **ClearOutbox**: This is a two-step approach to dispatching a message via middleware. It allows you to include the **DepositPost** call that puts the message in your [Outbox](/contents/BrighterOutboxSupport.md) within a database transation, so that you can achieve transactional messaging (either the message is placed in the Outbox and the change is made to any entities, or nothing is written to either). +* **Post**: This is a one-step approach to dispatching a message via middleware. Use it if you do not need transactional messaging, as described above. + +All methods have versions that support async...await. + +In both cases, if you use an Outbox with external storage, the message will be eventually delivered if it is written to the Outbox, provided that you run an *Outbox Sweeper* to dispatch any messages in the Outbox that have not been marked as dispatched. + +In this example we use **CommandProcessor.Post()** to dispatch a message over middleware. + +``` csharp +commandProcessor.Post(new GreetingCommand("Ian")); +``` + +In this exaple, we use **CommandProcessor.DepositPost()** and **CommandProcessor.ClearOutbox** to raise a transactional message. We then immediately clear it to lower latency. (We could have relied on an Outbox Sweeper and you should have an Outbox Sweeper in case this was to fail). + +In this example we are using Dapper as the library for writing our entities to the Db, and have used Brighter's Unit of Work support for that (passed into the handler constructor). + +```csharp +public override async Task HandleAsync(AddGreeting addGreeting, CancellationToken cancellationToken = default(CancellationToken)) +{ + var posts = new List(); + + //We use the unit of work to grab connection and transaction, because Outbox needs + //to share them 'behind the scenes' + + var conn = await _uow.GetConnectionAsync(cancellationToken); + await conn.OpenAsync(cancellationToken); + var tx = _uow.GetTransaction(); + try + { + var searchbyName = Predicates.Field(p => p.Name, Operator.Eq, addGreeting.Name); + var people = await conn.GetListAsync(searchbyName, transaction: tx); + var person = people.Single(); + + var greeting = new Greeting(addGreeting.Greeting, person); + + //write the added child entity to the Db + await conn.InsertAsync(greeting, tx); + + //Now write the message we want to send to the Db in the same transaction. + posts.Add(await _postBox.DepositPostAsync(new GreetingMade(greeting.Greet()), cancellationToken: cancellationToken)); + + //commit both new greeting and outgoing message + await tx.CommitAsync(cancellationToken); + } + catch (Exception e) + { + _logger.LogError(e, "Exception thrown handling Add Greeting request"); + //it went wrong, rollback the entity change and the downstream message + await tx.RollbackAsync(cancellationToken); + return await base.HandleAsync(addGreeting, cancellationToken); + } + + //Send this message via a transport. We need the ids to send just the messages here, not all outstanding ones. + //Alternatively, you can let the Sweeper do this, but at the cost of increased latency + await _postBox.ClearOutboxAsync(posts, cancellationToken:cancellationToken); + + return await base.HandleAsync(addGreeting, cancellationToken); +} +``` + +#### Message Mapper, MT_COMMAND and MT_EVENT + +When sending a message, the [Message Mapper](/contents/MessageMappers.md) is invoked to map your request to a Brighter **Message** which can be sent over message oriented middleware. + +Given you may have both a **Command** and an **Event** how do we preserve that behavior (a command expects one handler, an event zero or more) in listening applications? + +By setting the **Message.MessageType** to **MT_COMMAND** or **MT_EVENT** you indicate whether you expect this message to be treated as a **Command** or an **Event**. We flow that information in the message headers when sending over middleware. + +When *Service Activator* listens to messages it expects that the **MessageType** matches the type of **IRequest**, either **Command** or **Event** that your message mapper code transforms the message into. It will then use **CommandProcessor.Send()** to dispatch messages to a single handler, or **CommandProcessor.Publish** to dispatch messages to zero or more handlers, as appropriate. diff --git a/source/shared/DynamoInbox.md b/source/shared/DynamoInbox.md new file mode 100644 index 0000000..f2c6c27 --- /dev/null +++ b/source/shared/DynamoInbox.md @@ -0,0 +1,36 @@ +# Dynamo Inbox + +## Usage +The DynamoDb Inbox allows use of DynamoDb for [Brighter's inbox support](/contents/BrighterInboxSupport.md). The configuration is described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#inbox). + +For this we will need the *Inbox* packages for the DynamoDb *Inbox*. + +* **Paramore.Brighter.Inbox.DynamoDb** + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + var dynamoDb = new AmazonDynamoDBClient(credentials, new AmazonDynamoDBConfig { ServiceURL = "http://dynamodb.us-east-1.amazonaws.com"; }); + + services.AddServiceActivator(options => + { ... }) + .UseExternalInbox( + new DynamoDbInbox(dynamoDb); + new InboxConfiguration( + scope: InboxScope.Commands, + onceOnly: true, + actionOnExists: OnceOnlyAction.Throw + ) + ); +} + +... + +``` \ No newline at end of file diff --git a/source/shared/DynamoOutbox.md b/source/shared/DynamoOutbox.md new file mode 100644 index 0000000..d10e30b --- /dev/null +++ b/source/shared/DynamoOutbox.md @@ -0,0 +1,81 @@ +# DynamoDb Outbox + +## Usage +The DynamoDb Outbox allows integration between DynamoDb and [Brighter's outbox support](/contents/BrighterOutboxSupport.md). The configuration is described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#outbox-support). + +To support transactional messaging when using DynamoDb requires us to use DynamoDb's support for ACID transactions. You should understand best practices for using transactions with DynamoDb. + +For this we will need the *Outbox* package for DynamoDb: + +* **Paramore.Brighter.Outbox.DynamoDB** + +**Paramore.Brighter.Outbox.DynamoDb** will pull in another package: + +* **Paramore.Brighter.DynamoDb** + +As described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#outbox-support), we configure Brighter to use an outbox with the Use{DB}Outbox method call. + +As we want to use DynamoDb with the outbox, we also call: Use{DB}TransactionConnectionProvider so that we can share your transaction scope when persisting messages to the outbox. + + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .UseExternalBus(...) + .UseDynamoDbOutbox(ServiceLifetime.Singleton) + .UseDynamoDbTransactionConnectionProvider(typeof(DynamoDbUnitOfWork), ServiceLifetime.Scoped) + .UseOutboxSweeper() + + ... +} + +``` + +In our handler we take a dependency on Brighter's **IAmABoxTransactionConnectionProvider** interface and convert it to a **DynamoDbUnitofWork**. We explicitly start a transaction within the handler on the Database within the Unit of Work. + +We call **DepositPostAsync** within that transaction to write the message to the Outbox. Once the transaction has closed we can call **ClearOutboxAsync** to immediately clear, or we can rely on the Outbox Sweeper, if we have configured one to clear for us. (There are equivalent synchronous versions of these APIs).x + +``` csharp +public override async Task HandleAsync(AddGreeting addGreeting, CancellationToken cancellationToken = default(CancellationToken)) +{ + var posts = new List(); + + //We use the unit of work to grab connection and transaction, because Outbox needs + //to share them 'behind the scenes' + var context = new DynamoDBContext(_unitOfWork.DynamoDb); + var transaction = _unitOfWork.BeginOrGetTransaction(); + try + { + var person = await context.LoadAsync(addGreeting.Name); + + person.Greetings.Add(addGreeting.Greeting); + + var document = context.ToDocument(person); + var attributeValues = document.ToAttributeMap(); + + //write the added child entity to the Db - just replace the whole entity as we grabbed the original + //in production code, an update expression would be faster + transaction.TransactItems.Add(new TransactWriteItem{Put = new Put{TableName = "People", Item = attributeValues}}); + + //Now write the message we want to send to the Db in the same transaction. + posts.Add(await _postBox.DepositPostAsync(new GreetingMade(addGreeting.Greeting), cancellationToken: cancellationToken)); + + //commit both new greeting and outgoing message + await _unitOfWork.CommitAsync(cancellationToken); + } + catch (Exception e) + { + _logger.LogError(e, "Exception thrown handling Add Greeting request"); + //it went wrong, rollback the entity change and the downstream message + _unitOfWork.Rollback(); + return await base.HandleAsync(addGreeting, cancellationToken); + } + + //Send this message via a transport. We need the ids to send just the messages here, not all outstanding ones. + //Alternatively, you can let the Sweeper do this, but at the cost of increased latency + await _postBox.ClearOutboxAsync(posts, cancellationToken:cancellationToken); + + return await base.HandleAsync(addGreeting, cancellationToken); +} +``` diff --git a/source/shared/EFCoreOutbox.md b/source/shared/EFCoreOutbox.md new file mode 100644 index 0000000..97b6c30 --- /dev/null +++ b/source/shared/EFCoreOutbox.md @@ -0,0 +1,87 @@ +# EF Core Outbox + +## Usage +The EFCore Outbox allows integration between EF Core and [Brighter's outbox support](/contents/BrighterOutboxSupport.md). The configuration is described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#outbox-support). + +For this we will need the *Outbox* package for EF Core. Packages for EF Core exist for the following RDBMS: MSSQL, MYSQL, Postgres, and Sqlite. Packages have the naming convention: + +* **Paramore.Brighter.{DB}.EntityFrameworkCore** + +In addition, you will need the Outbox package for the relevant RDBMS: + +* **Paramore.Brighter.Outbox.{DB}** + +Obviously, {DB} should match. In the example below we use MySql, so we would need the following packages: + +* **Paramore.Brighter.MySql.EntityFrameworkCore** +* **Paramore.Brighter.Outbox.MySql** + +**Paramore.Brighter.MySql.EntityFrameworkCore** will pull in another package + +* **Paramore.Brighter.MySql** + +As described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#outbox-support), we configure Brighter to use an outbox with the Use{DB}Outbox method call. + +As we want to use EF Core, we also call: Use{DB}TransactionConnectionProvider so that we can share your transaction scope when persisting messages to the outbox. + + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .UseExternalBus(...) + .UseMySqlOutbox(new MySqlConfiguration(DbConnectionString(), _outBoxTableName), typeof(MySqlConnectionProvider), ServiceLifetime.Singleton) + .UseMySqTransactionConnectionProvider(typeof(MySqlEntityFrameworkConnectionProvider), ServiceLifetime.Scoped) + .UseOutboxSweeper() + + ... +} + +``` + +In our handler we take a dependency on our EF Core Context (derived from Db context). We explicitly start a transaction within the handler, because the Outbox is not within the Db Context we cannot rely on the DBContext's implicit transaction. + +We call **DepositPostAsync** within that transaction to write the message to the Outbox. Once the transaction has closed we can call **ClearOutboxAsync** to immediately clear, or we can rely on the Outbox Sweeper, if we have configured one to clear for us. (There are equivalent synchronous versions of these APIs).x + +``` csharp + public override async Task HandleAsync(AddGreeting addGreeting, CancellationToken cancellationToken = default(CancellationToken)) +{ + var posts = new List(); + + //We span a Db outside of EF's control, so start an explicit transactional scope + var tx = await _uow.Database.BeginTransactionAsync(cancellationToken); + try + { + var person = await _uow.People + .Where(p => p.Name == addGreeting.Name) + .SingleAsync(cancellationToken); + + var greeting = new Greeting(addGreeting.Greeting); + + person.AddGreeting(greeting); + + //Now write the message we want to send to the Db in the same transaction. + posts.Add(await _postBox.DepositPostAsync(new GreetingMade(greeting.Greet()), cancellationToken: cancellationToken)); + + //write the changed entity to the Db + await _uow.SaveChangesAsync(cancellationToken); + + //write new person and the associated message to the Db + await tx.CommitAsync(cancellationToken); + } + catch (Exception) + { + //it went wrong, rollback the entity change and the downstream message + await tx.RollbackAsync(cancellationToken); + return await base.HandleAsync(addGreeting, cancellationToken); + } + + //Send this message via a transport. We need the ids to send just the messages here, not all outstanding ones. + //Alternatively, you can let the Sweeper do this, but at the cost of increased latency + await _postBox.ClearOutboxAsync(posts, cancellationToken:cancellationToken); + + return await base.HandleAsync(addGreeting, cancellationToken); +} + +``` + diff --git a/source/shared/EventCarriedStateTransfer.md b/source/shared/EventCarriedStateTransfer.md new file mode 100644 index 0000000..46a9154 --- /dev/null +++ b/source/shared/EventCarriedStateTransfer.md @@ -0,0 +1,114 @@ +# Event Carried State Transfer (ECST) + +## Outside and Inside Data + +In his white paper \"Data on the Outside vs. Data on the Inside\", Pat Helland classifies data according to whether it exists inside a service boundary or outside that boundary. He calls the former Inside Data and the latter Outside Data. + +![ReferenceData](_static/images/ReferenceData.png) + +Inside Data is the data inside the service boundary. No one outside the service boundary can take a dependency on Inside Data. The service is the single writer of this data, the system of record, and is at liberty to change its schema. The backing store for a service holds Inside Data. + +Outside Data is what the service communicates to consumers at its boundary. It has three interesting properties: + +1: **It is immutable.** Changing this data does not impact the Inside Data of the originating service. That data; can only change from outside by using a provided API, if any, at the boundary. + +2: **It is stale.** As soon as data leaves our service it risks being stale. Any update to the data in the originating service will not be reflected in that data. Consider that if we use an HTTP API and GET the +state of a resource, a PUT that arrives to change that resource immediately after will mean the copy that we have is now stale. + +3: **It should be versioned.** Because the data risks being stale, we need to version it, so that we can compare it against potentially fresher versions. For example in HTTP we can use the If-None-Match +header with an ETag to determine if our data is stale, or if the resource would be the same if we retrieved it again. + +(It\'s worth noting that data supplied by a client as part of a command or request sent to the service is also Outside Data.) + +### Reference Data + +Pat Helland uses the term Reference Data to describe the types of data suitable for sharing as outside data. + +1\. Shared Collections. This is ubiquitous data that everyone needs to use to do work, such as a list of users, products, suppliers, brokers etc. It is so common for code to need to join this data that it makes +sense to copy it to each service that needs it. + +2\. Operand Data. Constructing requests to other microservices may require a service to understand a set of available options such as customer billing plans, or product categories. Operand data is where we +share the range of available options we can use to construct requests. + +3\. Snapshots. Where we want to query across multiple microservices we can end up with chatty solutions making requests to other microservices which we then need to join in the caller. An alternative is to listen to events so as to build an model that we can query. This is the model used by many Big Data pipelines or by Composite View Models. + +### Caching + +Outside Data is amenable to caching. Indeed this is how the web scales, we expose Outside Data from the Origin Server that is immutable (changing the response body has no impact on Inside Data), stale (post our GET a subsequent POST may modify the Inside Data our response was built from, perhaps before we have even parsed the result, and versioned (use of a Last Modified or ETag header allows the Origin Server to version the result that it responds with). + +## Event Carried State Transfer + +It\'s not just HTTP APIs whose results can be cached, we could also cache an event from the origin server, raised via AMQP, Kafka, ATOM or some other protocol. + +If the origin server raises events whenever a change occurs to the resources it manages, then consuming services downstream can cache those results, thus preventing the need to make a request to the origin server +to GET the current state of the entity. We can think of this as a push based cache instead of a pull based one. + +![ECST](_static/images/EventCarriedStateTransfer.png) + +The name given to building a cache upstream of the origin server from events is Event Carried State Transfer or ECST. + +There are some valuable aspects to this kind of cache. + +When we need to combine the state from two or more microservices to answer a query or respond to a command, we want to avoid making requests to that other service. This is because we temporally couple the two +services together - for our service to work, the other services must also be available. + +To solve this problem, we could work with a cache of the data we need to enrich our service built by ECST. By listening to the microservice we need the data from, we are able to join to the cache of data we require without making a request to that service. + +Usually a worker process listens for events from the other service and populates our local cache from the other microservice\'s events. + +Note that what we are putting in the cache is Outside Data, not Inside Data. We do not want to couple our consuming microservice to the the internals of the producing microservices model. We want to store the +equivalent to what we would recieve if we queried for it. This is why we prefer ECST over simple replication of data between services which would couple us to the details. + +### Alternatives to Event Carried State Transfer + +We could also meet the constraint that our service needs data from another to respond by ensuring that the request has all the data we require to process the event. If we think of the sender as the data +source, and our service as the data sink, we can build a pipeline where the original request is enriched with the required data by the microservices that own that data as a filter step. + +Where no central process controls this pipeline we refer to it as choreography. + +![Choreography](_static/images/Choreography.png) + +And we refer to it as orchestration when a process controls the pipeline. Orchestration uses commands, whereas choreography uses events. For this reason we may prefer choreography, as it has lower behavioural +coupling, unless we need confirmed rollback, or use of the reservation pattern which are easier with a Process Manager controlling the workflow. + +![Orchestration](_static/images/Orchestration.png) + +Whilst this could also work for a query, it is less common to take this approach to populating a response due to the likely latency of the response. + + +## Worked Scenario + +Imagine that we are writing software for a hotel. We have identified a number of microservices for our hotel: + +![HotelMicroservices](_static/images/HotelMicroservices.png) + +DirectBooking: Lets a customer reserve a room. May be a customer with an account or a guest. Credit Card Payments: Handles taking payments from a customer. Accounts: Holds information on account holders, including card details Housekeeping: Prepares rooms for a guest\'s stay and provides upkeep of the room during the stay Channel Manager: Markets our hotel rooms via various aggregator sites. + +When an account holder books a room they use the DirectBooking API to POST a booking. DirectBooking validates the booking and then raises an event to indicate that there has been a BookingMadeOnAccount. A number +of services listen for this message: + +Channel Manager: Decrements the rooms available on aggregator sites. Housekeeping: Schedules occupancy, cleaning of the room prior to occupancy, during and after. Credit Card Payments: Takes a payment from the Account holder. + +How does the Credit Card Payments system take the payment, when Accounts holds the account holders credit card details? We don\'t want to call a credit card details HTTP directly as this moves us back to a request driven architecture. + +We have two options. + +### A Pipeline + +Accounts listens for DirectBookingMadeOnAccount. It adds the credit card details to the booking and raises a DirectBookingMadeOnAccountWithCardDetails message. It is this message that Credit Card Payments listens to and then takes the card payment +via. + +![Choreography](_static/images/Choreography.png) + +### ECST + +Accounts publishes an event whenever an account holder changes name, address, or credit card details, called AccountDetailsChanged. Credit Card Payments subscribes to this event and caches the data in its own +backing store. Then when a payment request comes in via BookingMadeOnAccount it is able to look up the credit card details and take the payment. When we cross-check we can see that account details would seem to be a clear case of Shared Collection Reference Data and suitable for use in ECST. + +![ECST](_static/images/EventCarriedStateTransfer.png) + +Our preference for the two may depend on the extent to which we want to allow Credit Card Payments to take a payment even if Accounts is down, as Credit Card Payments is working with a cache. we may decide that a bulkhead is valuable enough to us to use ECST over choreography via a pipeline. + +## Next + +See [Correctness in Brighter](BrighterOutboxSupport.html) for guidance on how to use Brighter's support for the Outbox pattern to ensure producer-consumer correctness. diff --git a/source/shared/EventDrivenCollaboration.md b/source/shared/EventDrivenCollaboration.md new file mode 100644 index 0000000..05a293e --- /dev/null +++ b/source/shared/EventDrivenCollaboration.md @@ -0,0 +1,101 @@ +# Event Driven Collaboration + +Event Driven Architectures (EDA) are a major use case for Brighter's External Bus-you want processes to collaborate via messaging. + +(For another use case, offloading work to be performed asynchronously, see [Task Queues](/contents/TaskQueuePattern.md)) + +## Messaging + +Messages are packets of data, sent asynchronously over middleware + +- Commands, Documents, and Events are types of messages. +- Commands expect another service to handle the request, and possibly to respond. +- Documents represent the transfer of data; a query is a command, with a document response. +- Events are a notification that something happened. + +Commands can use a point-to-point channel i.e. only the sender and receiver are aware of the channel. Events can either use a publish-subscribe channel, or raise the event via a router that manages +dynamic subscriptions. The significant difference is that a sender of a command knows who the receiver is, the raiser of an event does not. + +## Temporal Coupling + +Why is it important that this integration is asynchronous? The answer is that we want to avoid Temporal Coupling. + +When we move from a monolithic to a microservices architecture, functionality and data becomes distributed across multiple microservices. In order to satisfy a given use case of our software, we +may need to multiple microservices to collaborate. + +Temporal Coupling occurs when in order to satisfy this use case, a set of microservices must all be available at the same time. + +Commonly, this occurs in systems that use synchronous communication protocols to integrate, such as HTTP + JSON, or gRPC. We refer to this as a Request Driven Architecture (or perhaps even Request Driven Collaboration). + +Let\'s take an example. In the illustration below we imagine hotel software and the use case of booking a room at the hotel. + +![RequestDrivenArchitecture](_static/images/RequestDrivenArchitecture.png) + +When a booking is made by an actor, an HTTP POST is made to our Direct Booking microservice with the details in the body of the request. The Direct Booking microservice validates the supplied details, and then +validates the request for missing information, room availability and so on. + +Once the booking is made the workflow for the operation suggests that we need to take a payment. In this use case the payment is being made by an existing account holder, and so the POST body does not contain the +payment details, instead, details already held on account are being used. + +In addition to taking a payment, we want to inform housekeeping of the booking, and use our Channel Manager to lower the availability of rooms on aggregator sites. + +How does the Direct Booking microservice communicate these steps to other microservices? + +In a Request Driven Architecture we would use a synchronous protocol such as HTTP+JSON or gRPC to call the API exposed by the other microservices. The issue here is that for our service to work, all these other services must also be available. + +A useful metaphor here is a phone call. If we make a phone call, the other party must be available i.e. present and not busy with another call. + +If the Channel Manager is not available, as in the diagram, does our transaction, the room booking, fail. + +Now, we can try to mitigate the risks of a Request Driven Architecture. We can call through a proxy that load balances across a pool of upstream instances of the service. The proxy can retry failed requests to +alleviate transient availability issues and take bad instances out of the pool. + +This is what a \'service mesh\' does - it improves the availability of a Request Driven Architecture by seeking to lower Temporal Coupling. + +## Behavioral Coupling + +Why does it matter whether we know about the receiver (a command) or are ignorant as to receivers? The answer is behavioral coupling. + +Let\'s look at our hotel example again. + +What happens if we decide that we no longer want a single housekeeping service, but a set of services such a laundry, room cleaning etc.? In a Request Driven Architecture the caller knows about the sequence of steps to complete a booking. This means that any change to the sequence, to call the new Room Cleaning microservice instead of the housekeeping service requires a change to Direct Booking. + +![BehavioralCoupling](_static/images/BehavioralCoupling.png) + +This coupling, through knowledge of other services, if a form of behavioral coupling. It hampers our goal of independent deployability because downstream components are impacted by changes to the partitions of upstream components, or to the shape of their APIs. + +## Event Driven Collaboration + +Event Driven Collaboration helps us solve the problems of temporal and behavioural coupling. + +When we publish an event, a subscriber uses a queue to receive events asynchronously. Because messages are held in a queue, the subscriber does not have to be available when the publisher produces the message, +only the queue does. If the subscriber is not available, the queue holds the message until the subcriber can process it. + +This removes temporal coupling - we do not need both services to be available at the same time. + +Let\'s look at the hotel example again. When the direct booking is made, the Payments microservice the Housekeeping service and the Channel Manager service can be unavailable. Their subscription just queues the +message until the service is available. + +![EventDrivenArchitecture1](_static/images/EventDrivenArchitecture1.png) + +![EventDrivenArchitecture2](_static/images/EventDrivenArchitecture2.png) + +This allows us to take the booking, even if these services are not available - we have given our microservices \'bulkheads\' against failure. Our system can keep offering service, even if parts of it are +not available. + +We do have to design our workflow, such that the customer expects an asynchronous operation i.e. \"We will mail you to confirm your booking\". This may seem like a limitation, but many workflows were +traditionally asynchronous before widespread automation, so processes exist for this approach, and customers expect \'tasks\', such as making a booking may need confirmation. + +If the metaphor for Request Driven Architectures is a phone call, for Event Drive Architectures it is SMS or a messaging app such as WhatsApp or Slack (or even a phone answering service). You don\'t need both +parties to be available, when you use messaging. + +In addition, a publisher does not know who it\'s subscribers are. That is the function of message oriented middleware - a broker. The broker routes messages from a publisher to subscribers. Because of this, +changes to the subscribers don\'t impact the publisher. The publisher remains independently deployable of its subscribers. + +So an Event Driven Architecture benefits from a lack of behavioral coupling too. + +(Note that if we use commands, and not events, between microservices i.e. the sender knows who should receive this instruction, we do not have temporal coupling, but we do have behavioral coupling). + +## Next + +See [Event Carried State Transfer](EventCarriedStateTransfer.html) for guidance on how to \'join\' data between two microservices, when you need data from more than one service to carry out an operation. diff --git a/source/shared/FAQ.md b/source/shared/FAQ.md new file mode 100644 index 0000000..186e900 --- /dev/null +++ b/source/shared/FAQ.md @@ -0,0 +1,51 @@ +# FAQ + +## Asynchronous or External Bus + +When should you use an asynchronous pipeline to handle work and when should you use an External Bus. + +Using an asynchronous handler allows you to avoid blocking I/O. This can increase your throughput by allowing you to re-use threads to service new requests. Using this approach, even a single-threaded application can +achieve high throughput, if it is not CPU-bound. + +Using an External Bus allows you to hand-off work to another process, to be executed at some point in the future. This also allows you to improve throughput by freeing up the thread to service new requests. We assume that we can accept dealing with that work at some point in the future i.e. we can be eventually consistent. + +One disadvantage of an External Bus is that the pattern - ack to callers, and then do the work, can create additional complexity because we must deal with notifying the user of completion, or errors. Because an async operation simply has the caller wait, the programming model is simpler. The trade-off here is that the client of our process is still using resources awaiting for the request with the async operation. If the operation takes time to complete the client may not know if the operation failed and should be timed out, or is still running. + +Where work is long-running there is a risk that the server faults, and we lose the long-running work. An External Bus provides reliability here, through guaranteed delivery. The queue keeps the work until it is +successfully processed and acknowledged. + +Our recommendation is to use the async pattern to improve throughput where the framework supports async, such as ASP.NET WebAPI but to continue to hand-off work that takes a long time to complete to a work +queue. You may choose to define your own thresholds but we recommend that operations that take longer than 200ms to complete be handed-off. We also recommend that operations that are CPU bound be handed-off as +they diminish the throughput of your application. + +## Iterating over a list of requests to dispatch them + +All **Command** or **Event** messages derive from **IRequest** and **ICommand** and **IEvent** respectively. So it may seem natural to create a collection of them, for example **List\**, and then +process a set of messages by enumerating over them. + +When you try this, you will encounter the issue that we dispatch based on the concrete type of the **Command** or **Event**. In other words the type you register via the **SubscriberRegistry.** Because +**CommandProcessor.Send()** is actually **CommandProcessor.Send\()** you need to provide the concrete type in the call for the compiler to determine the type to use with the cool as the concrete type. + +If you try this: + +``` csharp +ICommand command = new GreetingCommand("Ian"); +commandProcessor.Send(command); +``` + +Then you will get this error: *\"ArgumentException \"No command handler was found for the typeof command Brighter.commandprocessor.ICommand - a command should have exactly one handler.\"\"* + +Now, you don\'t see this issue if you pass the concrete type in, so the compiler can correctly resolve the run-time type. + +``` csharp +commandProcessor.Send(new GreetingCommand("Ian")); +``` + +So what can you do if you must pass the base class to the **Command Processor** i.e. because you are using a list. + +The workaround is to use the dynamic keyword. Using the dynamic keyword means that the type will be evaluated using RTTI, which will successfully pick up the type that you need. + +``` csharp +ICommand command = new GreetingCommand("Ian"); +commandProcessor.Send((dynamic)command); +``` diff --git a/source/shared/FeatureSwitches.md b/source/shared/FeatureSwitches.md new file mode 100644 index 0000000..18f6955 --- /dev/null +++ b/source/shared/FeatureSwitches.md @@ -0,0 +1,123 @@ +# Feature Switches + +We provide a **FeatureSwitch** Attribute and **FeatureSwitchAsync** Attribute that you can use on your **IHandleRequests\.Handle()** method, and **IHandleRequests\.HandleAysnc()** method. The **FeatureSwitch** Attribute and **FeatureSwitchAsync** Attribute that you have configured will determine whether or not the +**IHandleRequests\.Handle()** and **IHandleRequests\.HandleAsync()** will be executed. + +## Using the Feature Switch Attribute + +By adding the **FeatureSwitch** Attribute or **FeatureSwitchAsync** Attribute, you instruct the Command Processor to do one of the following: + +- run the handler as normal, this is **FeatureSwitchStatus.On**. +- not execute the handler, this is **FeatureSwitchStatus.Off**. +- detemine whether to run the handler based on a **Feature Switch + Registry**, [creating of which is described + later](FeatureSwitches.html#building-a-config-for-feature-switches-with-fluentconfigregistrybuilder). + +In the following example, **MyFeatureSwitchedHandler** will only be run if it has been configured in the **Feature Switch Registry** and set to **FeatureSwitchStatus.On**. + +``` csharp +class MyFeatureSwitchedHandler : RequestHandler +{ + [FeatureSwitch(typeof(MyFeatureSwitchedHandler), FeatureSwitchStatus.Config, step: 1)] + public override MyCommand Handle (MyCommand command) + { + /* Do work */ + return base.Handle(command); + } +} +``` + +In the second example, **MyIncompleteHandlerAsync** will not be run in the pipeline. + +``` csharp +class MyIncompleteHandlerAsync : RequestHandlerAsync +{ + [FeatureSwitchAsync(typeof(MyIncompleteHandlerAsync), FeatureSwitchStatus.Off, step: 1)] + public override Task HandleAsync(MyCommand command, CancellationToken cancellationToken = default) + { + /* Nothing implmented so we're skipping this handler */ + return await base.HandleAsync(command, cancellationToken); + } +} +``` + +## Building a config for Feature Switches with FluentConfigRegistryBuilder + +We provide a **FluentConfigRegistryBuilder** to build a mapping of request handlers to **FeatureSwitchStatus**. For each Handler that you wish to feature switch you supply a type and a status using a fluent +API. The valid statuses used in the builder are **FeatureSwitchStatus.On** and **FeatureSwitchStatus.Off**. + +``` csharp +var featureSwitchRegistry = FluentConfigRegistryBuilder + .With() + .StatusOf().Is(FeatureSwitchStatus.On) + .StatusOf().Is(FeatureSwitchStatus.Off) + .Build(); +``` + +## Implementing a custom Feature Switch Registry + +The **FluentConfigRegistryBuilder** provides compile time configuration of **FeatureSwitch** Attributes. If this is not suitable to your needs then you can write you own Feature Switch Registry using the [IAmAFeatureSwitchRegistry](https://github.com/BrighterCommand/Brighter/blob/master/src/Paramore.Brighter/FeatureSwitch/IAmAFeatureSwitchRegistry.cs) interface. The two requirements of this interface is a [MissingConfigStrategy](https://github.com/BrighterCommand/Brighter/blob/master/src/Paramore.Brighter/FeatureSwitch/MissingConfigStrategy.cs), and an implementation of **StatusOf(Type type)** which returns a +**FeatureSwitchStatus**. + +The **MissingConfigStrategy** determines how the Command Processor should behave when a Handler is decorated with a **FeatureSwitch** Attribute that is set to **FeatureSwitchStatus.Config** does not exist +in the registry. + +Your implementation of the **StatusOf** method is used to determine the **FeatureSwitchStatus** of the Handler type that is passed in as a parameter. How you store and retrieve these configurations is then up to +you. + +In the following example there are two FeatureSwitches configured in the **AppSettings.config**. We then build an **AppSettingsConfigRegistry**. The **StatusOf** method is implemetned to read the config from the App Settings and return the status for the given type. + +``` xml + + + + +``` + +``` csharp +class AppSettingsConfigRegistry : IAmAFeatureSwitchRegistry +{ + public MissingConfigStrategy MissingConfigStrategy { get; set; } + + public FeatureSwitchStatus StatusOf(Type handler) + { + var configStatus = ConfigurationManager.AppSettings[$"FeatureSwitch::{handler}"].ToLower(); + + switch (configStatus) + { + case "on": + return FeatureSwitchStatus.On; + case "off": + return FeatureSwitchStatus.Off; + default: + return MissingConfigStrategy is MissingConfigStrategy.SilentOn + ? FeatureSwitchStatus.On + : MissingConfigStrategy is MissingConfigStrategy.SilentOff + ? FeatureSwitchStatus.Off + : throw new InvalidOperationException($"No Feature Switch configuration for {handler} specified"); + } + } +} +``` + +## Setting Feature Switching Registry + +We associate a **Feature Switch Registry** with a **Command Processor** by passing it into the constructor of the **Command Processor**. For convenience, we provide a **Commmand Processor Builder** that helps you +configure new instances of **Command Processor**. + +``` csharp +var featureSwitchRegistry = FluentConfigRegistryBuilder + .With() + .StatusOf().Is(FeatureSwitchStatus.Off) + .Build(); + +var builder = CommandProcessorBuilder + .With() + .Handlers(new HandlerConfiguration(_registry, _handlerFactory)) + .DefaultPolicy() + .NoTaskQueues() + .ConfigureFeatureSwitches(fluentConfig) + .RequestContextFactory(new InMemoryRequestContextFactory()); + +var commandProcessor = builder.Build(); +``` diff --git a/source/shared/HandlerFailure.md b/source/shared/HandlerFailure.md new file mode 100644 index 0000000..f36f8db --- /dev/null +++ b/source/shared/HandlerFailure.md @@ -0,0 +1,48 @@ +# Failure and Dead Letter Queues + +When a *Request* is passed to **RequestHandler.Handle()** it runs in your application code. If your application code fails, you have a number of options: + +- [Retry (and Circuit Break) the *Request* on the Internal Bus](#retry-and-circuit-break-the-request-on-the-internal-bus) +- [Retry (with Delay) the *Request* on the External Bus](#retry-with-delay-the-request-on-the-external-bus) +- [Terminate processing of that *Request*](#terminate-processing-of-that-request) +- [Run a Fallback](#run-a-fallback) +- [Use Custom Middleware](#use-custom-middleware) + +Any unhandled exception that leaves the *Request Handling Pipeline* (in other words is not intercepted by middleware) will [Terminate Processing of the Request](#terminate-processing-of-that-request). + +## **Retry (and Circuit Break) the *Request* on the Internal Bus** + +You can use Brighter's support for Polly policies to retry the operation on the Internal Bus. See [Retry and Circuit Breaker](/contents/PolicyRetryAndCircuitBreaker.md) + +A circuit breaker is triggered when all Retry attempts fail, and will prevent further requests from succeeding. + +Both the triggering of the circuit breaker, and requests passed to the *Request Handler Pipeline* while the circuit breaker is open will [Terminate processing of that *Request*](#terminate-processing-of-that-request) + +## Retry (with Delay) the *Request* on the External Bus + +If you the failure is potentially retriable, but you want to retry on the External Bus (by making the message available to be consume from the External Bus again) then you can throw a **DeferMessageAction** exception. Upon receipt of a **DeferMessageAction** the pump will Reject the message and Requeue it. with a delay. The delay is configured by the External Bus **Subscription.RequeueDelayInMilliseconds** property. + +You can configure a limit on the number of requeue attempts by setting the **Subscription.RequeueCount**. A value of -1 will allow infinite retries. + +## Terminate processing of that *Request* + +An unhandled exception leaving the pipeline results in us terminating processing of the *Request*. + +- On an Internal Bus that exception will bubble out to the caller. +- On an External Bus we will nack (or reject) the message. On a queue this will delete the message for all consumers, on a stream we will increment the offset past that message for a consumer group. + +We do this because you have responsibility to handle exceptions thrown in your code, not the framework and we assume that non-recovered errors are not potentially retriable. + +On and Exernal Bus if your middleware supports a Dead Letter Queue (DLQ), and it is configured in your subscription, when we reject a message it will be copied to the DLQ. + +On an External Bus, to prevent discarding too many messages, you can set an **Subscription.UnacceptableMessageLimit**. If the number of messages terminated due to unhandled exceptions equals or exceeds this limit, the message pump processing the External Bus will terminate. + +## Run a Fallback + +If you want to take action before exiting a handler, due to a failure you can use a Fallback policy. See [Fallback Policy](/contents/PolicyFallback.md) for more details + +## Use Custom Middleware + +If none of the above options meet your needs, you can define custom approaches to exception handling by building your own middleware, see [Pipeline](BuildingAPipeline.html). + + diff --git a/source/shared/HealthChecks.md b/source/shared/HealthChecks.md new file mode 100644 index 0000000..98f0860 --- /dev/null +++ b/source/shared/HealthChecks.md @@ -0,0 +1,75 @@ +# Health Checks + +Brighter provides an AspNet Core Health check for **Service Activator** + +## Configure Health Checks + +The below will configure ASP.Net Core Health checks for Brighter's **Service Activator**, for more information on [ASP.NET Core Health Check](https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/health-checks?view=aspnetcore-6.0) + +```csharp +// Web Application Builder code goes here + +builder.Services.AddHealthChecks() + .AddCheck("Brighter", HealthStatus.Unhealthy); + +var app = builder.Build(); + +app.UseEndpoints(endpoints => +{ + endpoints.MapHealthChecks("/health"); + endpoints.MapHealthChecks("/health/detail", new HealthCheckOptions + { + ResponseWriter = async (context, report) => + { + var content = new + { + Status = report.Status.ToString(), + Results = report.Entries.ToDictionary(e => e.Key, + e => new + { + Status = e.Value.Status.ToString(), + Description = e.Value.Description, + Duration = e.Value.Duration + }), + TotalDuration = report.TotalDuration + }; + + context.Response.ContentType = "application/json"; + await context.Response.WriteAsync(JsonSerializer.Serialize(content, JsonSerialisationOptions.Options)); + } + }); +}); + +app.Run(); + +``` + +The /health endpoing will return a Status 200 with the Body of the Health status (i.e. Healthy) + +The /health/detail endpoint will return a detailed response with all of your information for example: + +```json +{ + "status": "Healthy", + "results": { + "Brighter": { + "status": "Healthy", + "description": "21 healthy consumers.", + "duration": "00:00:00.0000132" + } + }, + "totalDuration": "00:00:00.0029747" +} +``` + +## Health Status + +The following will be produced + +| Scenario | Status | +| -------- | ------ | +| All Message Pumps are running | Healthy | +| Some Message Pumps are running | Degraded | +| No Message Pumps are running | Unhealthy | + +In the event on a Degraded status the /health/details page can be used to find out which Dispatchers have failed pumps \ No newline at end of file diff --git a/source/shared/HowBrighterWorks.md b/source/shared/HowBrighterWorks.md new file mode 100644 index 0000000..ecb8e10 --- /dev/null +++ b/source/shared/HowBrighterWorks.md @@ -0,0 +1,87 @@ +# How The Command Processor Works + +You don\'t need to understand how Brighter works under the hood to use it, but if you want to debug, or contribute to the project, it can help to know what is going on. + +## The Dispatcher + +Ignoring attributes, which create a pipeline, for now, all of CommandProcessor\'s dispatch methods: + +- Accept a IAmARequest derived class as an argument +- Find the registered handler(s) for the type of the command +- Ask the factory you provide to create an instance of that handler +- Call Handle() or HandleAsync() as appropriate on that handler, passing in the command + +Let\'s look at that sequence in more detail, for those of you who want to understand the detail, so that you can walk the code if required. We\'ll show the code looking for a SendAsync() but the Send() works the +same way. Publish() and PublishAsync() are a minor variation in that they may dispatch to multiple handler chains, not just the one. + +![SequenceDiagram](_static/images/Brighter_SendAsync_Pipeline.png) + +Let\'s model what happens, if you use Brighter as the Ports layer behind the an ASP.NET Core web controller. + +1: The client makes an HTTP POST request to ASP.NET Core which marshalls the parameters and calls your web controller\'s POST method. + +2: The web controller takes those parameters, along with any other relevant information, such as identity from the bearer token, and creates a Command. Let\'s call it MyCommand + +3: The web controller calls the SendAsync() method on CommandProcessor because it is a Command and we expect one handler. If this was an Event, perhaps raised by the command handler itself, we would call +PublishAsync() instead. + +4: The SendAsync() method creates a RequestContext, using the supplied RequestContextFactory. Here we use the supplied InMemoryRequestContextFactory, which does not try to persist the state of the request, and is adequate for most purposes. + +5: The CommandProcessor then creates a new PipelineBuilder. The PipelineBuilder is a generic type specialized to the type of the Request. (Both Command and Event inherit from Request.) The PipelineBuilder orchestrates building the chain of responsibility that will handle our request. + +6: The PipelineBuilder creates an Interpreter. Again this is a generic type, specialized to the type of the Request. The interpreter is going to find the \'target handlers\'. A \'target handler\' is your code that implements IAmARequestHandlerAsync (usually via RequestHandlerAsync\<\>) that you register via the SubscriberRegistry. it is where the code which exercises the entities of your domain should live. The PipelineBuilder also creates a LifeTimeScope It will track the handlers we have created as part of this request. + +7: The CommandProcessor calls Build() on the PipelineBuilder to create the pipeline. + +8: The PipelineBuilder asks the Interpreter for an instance of the registered RequestHandlerAsync\<\>. + +9: The Interpreter uses the SubscriberRegistry to lookup the RequestHandlerAsync\<\> that we have associated with this Command. We don\'t show it here, but we passed the SubscriberRegistry to the +CommandProcessor when we built it, so it\'s not created here. The SubscriberRegistry returns the type of RequestHandlerAsync\<\>. In our example MyCommandHandlerAsync. + +10: The Interpreter now tries to build an instance of the handler by calling the HandlerFactory. The HandlerFactory is supplied to the CommandProcessor when it is built. It is user implemented because we +don\'t know how to build your handler, which has its dependencies in your code. However, we do integrate with ServiceCollection if desired. In our case we build a MyHandlerAsync. + +11: Having constructed a handler, we now need to build the pipeline based on the attributes that you have decorated the handler with (and any global attributes). We call the BuildPipeline method to create the +pipeline. + +12: We start with the part of the chain called before the registered handler - attributes flagged HandlerTiming.Before. + +13: Because your pipeline\'s configuration can only change at design time (when you write code) and not at runtime (when you execute) we only want to figure out the attributes of the chain once. So we store the +attribute list, once it\'s been determined, in a memento collection. This collection is a static. The first thing we do when building is see if we have already determined the configuration. If we have, we will use that. + +14: If we have not got a pipeline configuration, then we need to build one. Our first step is to call FindHandlerMethod() to get the handler method from our target handler i.e. MyCommandHandler in this case. RequestHandlerAsync does the heavy lifting for you here. + +15: We then use RTTI to find the attributes you have tagged your handler method with. In this case, let\'s pretend we just have RequestLoggingAsync\<\>. We order them via the Step value on the attribute (note that step collision order behavior is undefined). + +16: Once we have the per-handler list of attributes, we check if the CommandProcessor has been configured to use a global inbox. If it already has a UseInboxAsync attribute, or has a NoGlobalInbox attribute in the preAttributes list, we are done. Otherwise, if we have set up a global inbox, we add UseInboxAsync into the list of handlers. + +17: We then add the preAttributes into the collection of preAttributes, so that we will not need to use RTTI to build them again. Let\'s assume that we have configured out CommandProcessor with a global inbox. + +18: Next we have to construct this preAttribute chain. We iterate over all of the attributes and create the handlers by calling the supplied HandlerFactory. Again, this is because only your code knows how to create your handlers. If you use our support for ServiceCollection, then we will search the assemblies in the project for classes that implement IAmARequestHandlerAsync (including those we supply) and register them for you. The hander we create is determined from the type information supplied by the Attribute. In our example we will construct a RequestLoggingHandlerAsync and a UseInboxHandlerAsync. + +19: We construct the pre-Attributes. once we have constructed a Handler, we set it\'s successor property to be the next handler in the chain. + +20: We then repeat this process for any post handler attributes i.e. those tagged as HandlerTiming.After. We don\'t have any of those here, so we don\'t show that again this time. This decision was mainly for +simplicity. + +21: We then add the handlers to the LifeTimeScope of the pipeline. + +22: We then return the handler chain that we just constructed to the CommandProcessor. + +23: The CommandProcessor checks that we have a valid pipeline; for a SendAsync we expect exactly one pipeline will handle the Command. For PublishAsync we allow zero or more. + +24: Now we call the pipeline by passing our MyCommand to the first handler in the chain, in our case the UseInboxHandlerAsync. + +25: The UseInboxHanlderAsync has an inbox as a private member, that was passed in via the constructor via the HandlerFactoryAsync. This is a data store specific implementation of IAmAnInbox. if the OnceOnly parameter is set on the attribute then we call the Inbox\'s ExistsAsync method to determine if we have already processed the command. If the command has not already been processed, we call the base class\'s HandleAsync method. + +26: The base class\'s HandleAsync() method we use the successor field (see 19 above) to determine the next handler in the chain and we call it\'s HandleAsync() method. In this case we call RequestLoggingAsync\<\>\'s HandleAsync method. + +27: The RequestLoggingAsync\<\>\'s HandleAsync method logs the call, and again calls the base class\'s HandleAsync() method to pass the call down the pipeline. + +28: Finally, we call MyCommandHandlerAsync whose HandleAsync() command runs our business logic. Again we call the base class\'s HandleAsync() method, but as there is no successor we return. + +29: We return from RequestLoggingAsync\<\> which has no work left to do. + +30: UseInboxHandlerAsync calls IAmAnIbox\'s AddAsync method to write the command to the Inbox. Then it returns. + +31: SendAsync returns, and we are done. diff --git a/source/shared/HowConfiguringTheCommandProcessorWorks.md b/source/shared/HowConfiguringTheCommandProcessorWorks.md new file mode 100644 index 0000000..2a5f014 --- /dev/null +++ b/source/shared/HowConfiguringTheCommandProcessorWorks.md @@ -0,0 +1,165 @@ +# How Configuring the Command Processor Works + +Brighter does not have a dependency on an Inversion Of Control (IoC) framework. This gives you freedom to choose the DI libraries you want for your project. + +We follow an approach outlined by Mark Seeman in his blog on a [DI Friendly +Framework](http://blog.ploeh.dk/2014/05/19/di-friendly-framework/) and +[Message Dispatching without Service +Location](http://blog.ploeh.dk/2011/09/19/MessageDispatchingwithoutServiceLocation/). + +This means that we can support any approach to DI that you choose, provided you implement a range of interfaces that we require to create instances of your classes at runtime. + +For .NET Core's DI framework we provide the implementation of these interfaces. If you are using that approach, just follow the outline in [Basic Configuration](/contents/BrighterBasicConfiguration.md). This chapter is 'interest only' at that point, and you don't need to read it. It may be helpful for debugging. + +If you choose another DI framework, this document explains what you need to do to support that DI framework. + +## CommandProcessor Configuration Dependencies + +- You need to provide a **Subscriber Registry** with all of the **Command**s or **Event**s you wish to handle, mapped to their **Request Handlers**. +- You need to provide a **Handler Factory** to create your Handlers +- You need to provide a **Policy Registry** if you intend to use [Polly](https://github.com/App-vNext/Polly) to support Retry and Circuit-Breaker. +- You need to provide a **Request Context Factory** + +## Subscriber Registry + +The Command Dispatcher needs to be able to map **Command**s or **Event**s to a **Request Handlers**. + +For a **Command** we expect one and only one **Request Handlers** for an event we expect many. + +YOu can use our **SubcriberRegistry** regardless of your DI framework. + +Register your handlers with your **Subscriber Registry** + +``` csharp +var registry = new SubscriberRegistry(); +registry.Register(); +``` + +We also support an initializer syntax + +``` csharp +var registry = new SubscriberRegistry() +{ + {typeof(GreetingCommand), typeof(GreetingCommandHandler)} +} +``` + +## Handler Factory + +We don't know how to construct your handler so we call a factory, that you provide, to build your handler (and its entire dependency chain). + +Instead, we take a dependency on an interface for a handler factory, and you implement that. Within the handler factory you need to construct instances of your types in response to our request to create one. + +For this you need to implement the interface: **IAmAHandlerFactory**. + +Brighter manages the lifetimes of handlers, as we consider the request pipeline to be a scope, and we will call your factory again informing that we have terminated the pipeline and finished processing the request. You should take any required action to clear up the handler and its dependencies in response to that call. + +You can implement the Handler Factory using an IoC container. This is what Brighter does with .NET Core + +For example using [TinyIoC Container](https://github.com/grumpydev/TinyIoC): + +``` csharp +internal class HandlerFactory : IAmAHandlerFactory +{ + private readonly TinyIoCContainer _container; + + public HandlerFactory(TinyIoCContainer container) + { + _container = container; + } + + public IHandleRequests Create(Type handlerType) + { + return (IHandleRequests)_container.GetInstance(handlerType); + } + + public void Release(IHandleRequests handler) + { + _container.Release(handler); + } +} +``` + +## Policy Registry + +If you intend to use a [Polly](https://github.com/App-vNext/Polly) Policy to support [Retry and Circuit-Breaker](PolicyRetryAndCircuitBreaker.html) then you will need to register the Policies in the **Policy Registry**. + +This is just the Polly **PolicyRegistry**. + +Registration requires a string as a key, that you will use in your [UsePolicy] attribute to choose the policy. + +The two keys: CommandProcessor.RETRYPOLICY and CommandProcessor.CIRCUITBREAKER are used within Brighter to control our response to broker issues. You can override them if you wish to change our behavior from the default. + +You can also use them for a generic retry policy, though we recommend building retry policies that handle the kind of exceptions that will be thrown from your handlers. + +In this example, we set up a policy. To make it easy to reference the string, instead of adding it everywhere, we use a global readonly reference, not shown here. + +``` csharp +var retryPolicy = + Policy.Handle().WaitAndRetry( + new[] { + TimeSpan.FromMilliseconds(50), + TimeSpan.FromMilliseconds(100), + TimeSpan.FromMilliseconds(150) }); + +var circuitBreakerPolicy = Policy.Handle().CircuitBreaker( + 1, TimeSpan.FromMilliseconds(500)); + +var policyRegistry = new PolicyRegistry() { + { Globals.MYRETRYPOLICY, retryPolicy }, + { Globals.MYCIRCUITBREAKER, circuitBreakerPolicy } + }; +``` + +When you attribute your code, you then use the key to attach a specific policy: + +``` csharp +[RequestLogging(step: 1, timing: HandlerTiming.Before)] +[UsePolicy(Globals.MYRETRYPOLICY, step: 2)] +public override TaskReminderCommand Handle(TaskReminderCommand command) +{ + _mailGateway.Send(new TaskReminder( + taskName: new TaskName(command.TaskName), + dueDate: command.DueDate, + reminderTo: new EmailAddress(command.Recipient), + copyReminderTo: new EmailAddress(command.CopyTo) + )); + + return base.Handle(command); +} +``` + +If you need multiple policies then you can pass them as an array. We evaluate them left to right. + +``` csharp +[RequestLogging(step: 1, timing: HandlerTiming.Before)] +[UsePolicy(new [] {Globals.MYRETRYPOLICY, Globals.MYCIRCUITBREAKER}, step: 2)] +public override TaskReminderCommand Handle(TaskReminderCommand command) +{ + _mailGateway.Send(new TaskReminder( + taskName: new TaskName(command.TaskName), + dueDate: command.DueDate, + reminderTo: new EmailAddress(command.Recipient), + copyReminderTo: new EmailAddress(command.CopyTo) + )); + + return base.Handle(command); +} +``` + +## Request Context Factory + +You need to provide a factory to give us instances of a [Context](UsingTheContextBag.html). If you have no implementation to use, just use the default **InMemoryRequestContextFactory**. Typically you would replace ours if you wanted to support initializing the context outside of our pipeline, for tracing for example. + +## Command Processor Builder + +All these individual elements can be passed to a **Command Processor Builder** to help build a **Command Processor**. This has a fluent interface to help guide you when configuring Brighter. The result looks like this: + +``` csharp +var commandProcessor = CommandProcessorBuilder.With() + .Handlers(new HandlerConfiguration(subscriberRegistry, handlerFactory)) + .Policies(policyRegistry) + .NoExternalBus() + .RequestContextFactory(new InMemoryRequestContextFactory()) + .Build(); +``` diff --git a/source/shared/HowConfiguringTheDispatcherWorks.md b/source/shared/HowConfiguringTheDispatcherWorks.md new file mode 100644 index 0000000..ce17b49 --- /dev/null +++ b/source/shared/HowConfiguringTheDispatcherWorks.md @@ -0,0 +1,111 @@ +# How Configuring a Dispatcher for an External Bus Works + +In order to receive messages from Message Oriented Middleware (MoM) such as RabbitMQ or Kafka you have to configure a *Dispatcher*. The *Dispatcher* works with a *Command Processor* to deliver messages read from a queue or stream to your *Request Handler*. You write a Request Handler as you would for a request sent over an Internal Bus, and hook it up to Message Oriented Middleware via a *Dispatcher*. + +For each message source (queue or stream) that you listen to, the Dispatcher lets you run one or more *Performers*. A *Performer* is a single-threaded message pump. As such, ordering is guaranteed on a *Peformer*. You can run multiple *Peformers* to utilize the [Competing Consumers](https://www.enterpriseintegrationpatterns.com/CompetingConsumers.html) pattern, at the cost of ordering. + +If you are using .NET Core Dependency Injection, we provide extension methods to **HostBuilder** to help you configure a Dispatcher. This information is then for background only, but may be useful when debugging. Just follow the steps outlined in [BasicConfiguration](/contents/BrighterBasicConfiguration.md). + +If you are not using **HostBuilder** you will need to perform the following steps explicitly in your code. + +## Configuring the Dispatcher + +We provide a Dispatch Builder that has a progressive interface to assist you in configuring a **Dispatcher** + +You need to consider the following when configuring the Dispatcher + +- Command Processor +- Message Mappers +- Channel Factory +- Connection List + +Configuring the **Command Processor** is covered in [How Configuring the Command Processor Works](/contents/HowConfiguringTheCommandProcessorWorks.md). + +### Message Mappers + +You need to register your [Message Mapper](/contents/MessageMappers.md) so that we can find it. The registry must implement **IAmAMessageMapperRegistry**. We recommend using Brighter's **MessageMapperRegistry** unless you have more specific requirements. + +``` csharp +var messageMapperRegistry = new MessageMapperRegistry(messageMapperFactory) +{ + { typeof(GreetingCommand), typeof(GreetingCommandMessageMapper) } +}; +``` + +### Channel Factory + +The Channel Factory is where we take a dependency on a specific Broker. We pass the **Dispatcher** an instances of **InputChannelFactory** which in turn has a dependency on implementation of **IAmAChannelFactory**. The channel factory is used to create channels that wrap the underlying Message-Oriented Middleware that you are using. + +### Creating a Builder + +This code fragment shows putting the whole thing together + +``` csharp +// create message mappers +var messageMapperRegistry = new MessageMapperRegistry(messageMapperFactory) +{ + { typeof(GreetingCommand), typeof(GreetingCommandMessageMapper) } +}; + +// create the gateway +var rmqMessageConsumerFactory = new RmqMessageConsumerFactory(logger); +_dispatcher = DispatchBuilder.With() + .CommandProcessor(CommandProcessorBuilder.With() + .Handlers(new HandlerConfiguration(subscriberRegistry, handlerFactory)) + .Policies(policyRegistry) + .NoExternalBus() + .RequestContextFactory(new InMemoryRequestContextFactory()) + .Build()) + .MessageMappers(messageMapperRegistry) + .ChannelFactory(new InputChannelFactory(rmqMessageConsumerFactory)) + .Subscribers(subscriptions) + .Build(); +``` + +## Running The Dispatcher + +To ensure that messages reach the handlers from the queue you have to run a **Dispatcher**. + +The Dispatcher reads messages of input channels. Internally it creates a message pump for each channel, and allocates a thread to run that message pump. The pump consumes messages from the channel, using the +**Message Mapper** to translate them into a **Message** and from there a **Command** or **Event**. It then dispatches those to handlers (using the Brighter **Command Processor**). + +To use the Dispatcher you need to host it in a consumer application. Usually a console application or Windows Service is appropriate. + +We recommend using HostBuilder, but if not you will need to use something like [Topshelf](http://topshelf-project.com/) to host your consumers. + +The following code shows an example of using the **Dispatcher** from Topshelf. The key methods are **Dispatcher.Receive()** to start the message pumps and **Dispatcher.End()** to shut them. + +We do allow you to start and stop individual channels, but this is an advanced feature for operating the services. + +``` csharp +internal class GreetingService : ServiceControl +{ + private Dispatcher _dispatcher; + + public GreetingService() + { + /* Configfuration Code Goes here*/ + } + + public bool Start(HostControl hostControl) + { + _dispatcher.Receive(); + return true; + } + + public bool Stop(HostControl hostControl) + { + _dispatcher.End().Wait(); + _dispatcher = null; + return false; + } + + public void Shutdown(HostControl hostcontrol) + { + if (_dispatcher != null) + _dispatcher.End(); + return; + } +} +``` + diff --git a/source/shared/HowServiceActivatorWorks.md b/source/shared/HowServiceActivatorWorks.md new file mode 100644 index 0000000..883d55c --- /dev/null +++ b/source/shared/HowServiceActivatorWorks.md @@ -0,0 +1,3 @@ +# How The Service Activator Works + +Coming Soon... \ No newline at end of file diff --git a/source/shared/ImplementAQueryHandler.md b/source/shared/ImplementAQueryHandler.md new file mode 100644 index 0000000..c858f1e --- /dev/null +++ b/source/shared/ImplementAQueryHandler.md @@ -0,0 +1,3 @@ +# Implementing a Query Handler + +TODO \ No newline at end of file diff --git a/source/shared/ImplementingAHandler.md b/source/shared/ImplementingAHandler.md new file mode 100644 index 0000000..c0cdb16 --- /dev/null +++ b/source/shared/ImplementingAHandler.md @@ -0,0 +1,32 @@ +# How to Implement a Request Handler + +To implement a handler, derive from RequestHandler\ where T should be the **Command** or **Event** derived type that you wish to handle. Then override the base class Handle() method to implement your handling +for the Command or Event. + +For example, assume that you want to handle the **Command** GreetingCommand + +``` csharp +public class GreetingCommand : Command +{ + public GreetingCommand(string name) + : base(Guid.NewGuid()) + { + Name = name; + } + + public string Name { get; private set; } +} +``` + +Then derive your handler from **RequestHandler\** and accept a parameter of that type on the overriden **Handle()** method. + +``` csharp +public class GreetingCommandHandler : RequestHandler +{ + public override GreetingCommand Handle(GreetingCommand command) + { + Console.WriteLine("Hello {0}", command.Name); + return base.Handle(command); + } +} +``` diff --git a/source/shared/ImplementingAsyncHandler.md b/source/shared/ImplementingAsyncHandler.md new file mode 100644 index 0000000..a7378bb --- /dev/null +++ b/source/shared/ImplementingAsyncHandler.md @@ -0,0 +1,52 @@ +# How to Implement an Asynchronous Request Handler + +To implement an asynchronous handler, derive from **RequestHandlerAsync\** where *T* should be the **Command** or **Event** derived type that you wish to handle. Then override the base +class **RequestHandlerAsync\.HandleAsync()** method to implement your handling for the Command or Event. + +For example, assume that you want to handle the **Command** GreetingCommand + +``` csharp +public class GreetingCommand : Command +{ + public GreetingCommand(string name) + : base(Guid.NewGuid()) + { + Name = name; + } + + public Guid Id { get; set; } + public string Name { get; private set; } +} +``` + +Then derive your handler from **RequestHandlerAsync\** and accept a parameter of that type on the overriden **HandleAsync()** method, along with a nullable cancellation token - which you should +default to null. + +To ensure that the pipeline runs, you should return the result of the next handler in the chain, by awaiting the base class **HandleAsync()**. + +(Because the next element in the pipeline should also be async, you should always await the result of this call.) + +``` csharp +public class GreetingCommandRequestHandlerAsync : RequestHandlerAsync +{ + public override async Task HandleAsync(GreetingCommand command, CancellationToken? ct = null) + { + var api = new IpFyApi(new Uri("https://api.ipify.org")); + + var result = await api.GetAsync(ct); + + Console.WriteLine("Hello {0}", command.Name); + Console.WriteLine(result.Success ? "Your public IP addres is {0}" : "Call to IpFy API failed : {0}", + result.Message); + return await base.HandleAsync(command, ct).ConfigureAwait(base.ContinueOnCapturedContext); + } +} +``` + +Note how we use **ConfigureAwait()** when calling the next handler in the chain, and set the value to the **RequestHandlerAsync\.ContinueOnCapturedContext** property. This ensures that we utilize any override of the default (which is to use the Task Scheduler) made when the call to **SendAsync**, **PublishAsync**, or **PostAsync** was made. + +It is worth noting that although the override forces you to return a **Task\** it does not force you to add the **async** keyword to the method to compile. This risks introducing a subtle bug. You can await a +method that returns a **Task\** but creation of the state machine in the caller depends on the presence of the **async** keyword. If your handler does not await anything, you will not be forced to add the +**async** keyword. Your handler will run sychronously in this context, which may not be what you expect. + +Remembering to always await the base class **HandleAsync()** mitigates against this as even if your handler does not do asynchronous work, you will be forced to add **async** to the signature. diff --git a/source/shared/ImplementingExternalBus.md b/source/shared/ImplementingExternalBus.md new file mode 100644 index 0000000..4790968 --- /dev/null +++ b/source/shared/ImplementingExternalBus.md @@ -0,0 +1,106 @@ +# Using an External Bus + +Brighter provides support for an External Bus. Instead of handling a command or event, synchronously and in-process, (an Internal Event Bus) work can be dispatched to a distributed queue to be handled +asynchronously and out-of-process. The trade-off here is between the cost of distribution (see [The Fallacies of Distributed Computing](https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing). + +An External Bus allows you offload work to another process, to be handled asynchronously (once you push the work onto the queue, you don\'t wait) and in parallel (you can use other cores to process the +work). It also allows you to ensure delivery of the message, eventually (the queue will hold the work until a consumer is available to read it). + +As part of the [Microservices](https://martinfowler.com/articles/microservices.html) architectural style an External Bus let's you implement an [Event Driven Architecture](/contents/EventDrivenCollaboration.md). + +In addition use of an External Bus allows you to throttle requests - you can hand work off from the web server to a queue that only needs to consume at the rate you have resources to support. This +allows you to scale to meet unexpected demand, at the price of [eventual consistency.](https://en.wikipedia.org/wiki/Eventual_consistency). See the [Task Queue Pattern](/contents/TaskQueuePattern.md) + +## Brighter\'s External Bus Architecture + +Brighter implements an External Bus using a [Message Broker](http://www.enterpriseintegrationpatterns.com/MessageBroker.html). The software that provides the Message Broker is referred to as Message-Oriented Middleware (MoM). Brighter calls its abstraction over MoM a *Transport*. + +The producer sends a **Command** or **Event** to a [Message Broker](http://www.enterpriseintegrationpatterns.com/MessageBroker.html) using **CommandProcessor.Post()** or **CommandProcessor.DepositPost** and **CommandProcessor.ClearOutBox** (or their \*Async equivalents). + +We use an **IAmAMessageMapper** to map the **Command** or **Event** to a **Message**. (Usually we just serialize the object to JSON and add to the **MessageBody**), but if you want to use higher performance +serialization approaches, such as [protobuf-net](https://github.com/mgravell/protobuf-net), the message mapper is agnostic to the way the body is formatted.) + +When we deserialize we set the **MessageHeader** which includes a topic (often we use a namespaced name for the **Command** or **Event**). + +We store the created **Message** in a [Outbox](https://microservices.io/patterns/data/transactional-outbox.html) for use by **CommandProcessor.ClearOutbox()** if we need to resend a failed message. + +The Message Broker manages a [Recipient List](http://www.enterpriseintegrationpatterns.com/RecipientList.html) of subscribers to a topic. When it receives a **Message** the Broker looks at the topic (in Brighter terms the *Routing Key*) in the **MessageHeader** and dispatches the **Message** to the [Recipient Channels](http://www.enterpriseintegrationpatterns.com/MessageChannel.html) identified by the Recipient List. + +The consumer registers a [Recipient Channel](http://www.enterpriseintegrationpatterns.com/MessageChannel.html) to receive messages on a given topic. In other words when the consumer\'s registered topic matches the producer\'s topic, the broker dispatches the message to the consumer when it receives it from the producer. + +A **Message** may be delivered to multiple Consumers, all of whom get their own copy. + +in addition, we can support a [Competing Consumers](http://www.enterpriseintegrationpatterns.com/CompetingConsumers.html) approach by having multiple consumers read from the same [Channel](http://www.enterpriseintegrationpatterns.com/MessageChannel.html) to allow us to scale out to meet load. + +![TaskQueues](_static/images/TaskQueues.png) + + +## Sending via the External Bus + +Instead of using **CommandProcessor.Send()** you use **CommandProcessor.Post()** or **CommandProcessor.DepositPost** and **CommandProcessor.ClearOutbox** to send the message + +``` csharp +var reminderCommand = new TaskReminderCommand( + taskName: reminder.TaskName, + dueDate: DateTime.Parse(reminder.DueDate), + recipient: reminder.Recipient, + copyTo: reminder.CopyTo); + + _commandProcessor.Post(reminderCommand); +``` + +You add a message mapper to tell Brighter how to serialize the message for sending to your consumers. + +``` csharp +public class TaskReminderCommandMessageMapper : IAmAMessageMapper +{ + public Message MapToMessage(TaskReminderCommand request) + { + var header = new MessageHeader(messageId: request.Id, topic: "Task.Reminder", messageType: MessageType.MT_COMMAND); + var body = new MessageBody(JsonConvert.SerializeObject(request)); + var message = new Message(header, body); + return message; + } + + public TaskReminderCommand MapToRequest(Message message) + { + return JsonConvert.DeserializeObject(message.Body.Value); + } +} +``` +## Receiving via the External Bus + +A consumer reads the **Message** using the [Service Activator](http://www.enterpriseintegrationpatterns.com/MessagingAdapter.html) pattern to map between an [Event Driven Consumer](http://www.enterpriseintegrationpatterns.com/EventDrivenConsumer.html) and a Handler. + +The use of the Service Activator pattern means the complexity of the distributed task queue is hidden from you. You just write a handler as normal, but call it via post and create a message mapper, the result is +that your command is handled reliably, asynchronously, and in parallel with little cognitive overhead. It just works! + +``` csharp +public class MailTaskReminderHandler : RequestHandler +{ + private readonly IAmAMailGateway _mailGateway; + + public MailTaskReminderHandler(IAmAMailGateway mailGateway, IAmACommandProcessor commandProcessor) + : this(mailGateway, commandProcessor, LogProvider.GetCurrentClassLogger()) + {} + + public MailTaskReminderHandler(IAmAMailGateway mailGateway, ILog logger) : base(logger) + { + _mailGateway = mailGateway; + } + + [RequestLogging(step: 1, timing: HandlerTiming.Before)] + [UsePolicy(new [] {CommandProcessor.CIRCUITBREAKER, CommandProcessor.RETRYPOLICY}, step: 2)] + public override TaskReminderCommand Handle(TaskReminderCommand command) + { + _mailGateway.Send(new TaskReminder( + taskName: new TaskName(command.TaskName), + dueDate: command.DueDate, + reminderTo: new EmailAddress(command.Recipient), + copyReminderTo: new EmailAddress(command.CopyTo) + )); + + return base.Handle(command); + } +} +``` \ No newline at end of file diff --git a/source/shared/KafkaConfiguration.md b/source/shared/KafkaConfiguration.md new file mode 100644 index 0000000..3626f78 --- /dev/null +++ b/source/shared/KafkaConfiguration.md @@ -0,0 +1,317 @@ +# Kafka Configuration + +## General + +Kafka is OSS message-oriented-middleware and is [well documented](https://kafka.apache.org/documentation/#gettingStarted). Brighter handles the details of sending to or receiving from Kafka. You may find it useful to understand the [building blocks](https://kafka.apache.org/documentation/#introduction) of the protocol. Brighter's Kafka support is implemented on top of the Confluent .NET client, and you might find the [documentation for the .NET client](https://docs.confluent.io/kafka-clients/dotnet/current/overview.html) helpful when debugging, but you should not have to interact with it directly to use Brighter (although we expose many of its configuration options). + +Kafka has two main roles: + +- **Producer**: A producer sends events to a **Topic** on a Kafka broker. +- **Consumer**: A consumer reads events from a **Topic** on a Kafka broker. + +**Topics** are append-only streams of events. Multiple producers can write to a topic, and multiple consumers can read from one. A **consumer** uses an **offset** into the stream to indicate the event it wants to read. Kafka does not delete an event from the stream when it is ack'd by the consumer; instead a **consumer** increments its **offset** once an item has been read so that it can avoid processing the same event twice. See [Offset Management](#offset-management) for more on how Brighter manages **consumer offsets**. As a result the lifetime of events on a stream is instead a configuration setting for the stream. + +As a **consumer** manages an **offset** to record events that is has read, you cannot scale an application that wishes to consume a **topic** by increasing the number of **consumers**--they don't share an offset--without partitioning the **topic**. If you supply a **partition key**, a **partition** uses consistent hashing to slice a **topic** into a number of streams; otherwise it will use round-robin. See [this documentation](https://jaceklaskowski.gitbooks.io/apache-kafka/content/kafka-producer-internals-DefaultPartitioner.html) for more. Each **partition** is only read by a single **consumer** within the application. All of the consumers for an application should share the same group id, called a **consumer group** in Kafka. As each **consumer** tracks the **offset** for the **partitions** it is reading, it is possible to have multiple **consumers** read and process the same **topic**. + +A **consumer** may read from *multiple* **partitions**, but only one **consumer** may read from a **partition** at one time in a given **consumer group**. Kafka will assign partitions across the pool of consumers for the **consumer group**. When the pool changes, a **rebalance** occurs, which may mean that a consumer changes the **partition** that it is assigned within the **consumer group**. Brighter favors *sticky assignment of partitions* to avoid unnecessary churn of partitions. + +In addition to the Producer API and Consumer API Kafka streams have features such as the Streams API and the Connect API. We do not use either of these from Brighter. + +## Connection + +The Connection to Kafka is provided by an **KafkaMessagingGatewayConnection** which allows you to configure the following: + +- **BootstrapServers**: A **bootstrap** server is a well-known broker through which we discover the servers in the Kafka cluster that we can connect to. You should supply a comma-separated list of host and port pairs. These are the addresses of the Kafka brokers in the "bootstrap" Kafka cluster. +- **Debug**: A comma-separated list of debug contexts to enable. Producer: broker, topic, msg. Consumer: consumer, cgrp, topic, fetch. +- **Name**: An identifier to use for the client. +- **SaslMechanisms**: If any, what is the protocol used for authenticated connection to the Kafka broker: plain, scram-sha-256, scram-sha-256, gssapi (kerberos), oauthbearer +- **SaslKerberosName**: If using kerberos, what is the connection name. +- **SaslUsername**: SASL username for use with PLAIN and SASL-SCRAM +- **SaslPassword**: SASL password for use with PLAIN and SASL-SCRAM +- **SecurityProtocol**: How are messages between client and server encrypted, if at all: plaintext, ssl, saslplaintext, saslssl +- **SslCaLocation**: Where is the CA certificate located (see [here](https://docs.confluent.io/platform/current/tutorials/examples/clients/docs/csharp.html) for guidance). +- **SslKeystoreLocation**: Path to the client's keystore +- **SslKeystorePassword**: Password for the client's keystore + +The following code connects to a local Kafka instance (for development): + +``` csharp + services.AddBrighter(...) + .UseExternalBus( + new KafkaProducerRegistryFactory( + new KafkaMessagingGatewayConfiguration() + { + Name = "paramore.brighter.greetingsender", + BootStrapServers = new[] {"localhost:9092"} + }, + ...//publication, see below + ) + .Create()) + ... + +``` + +The following code connects to a remote Kafka instance. The settings here will depend on how your production broker is configured for access. We show getting secrets from environment variables for simplicity, again you will need to adjust this for your approach to secrets management: + +``` csharp + services.AddBrighter(...) + .UseExternalBus( + new KafkaProducerRegistryFactory( + new KafkaMessagingGatewayConfiguration() + { + Name = "paramore.brighter.greetingsender", + BootStrapServers = new[] { Environment.GetEnvironmentVariable("BOOSTRAP_SERVER")}, + SecurityProtocol = Paramore.Brighter.MessagingGateway.Kafka.SecurityProtocol.SaslSsl, + SaslMechanisms = Paramore.Brighter.MessagingGateway.Kafka.SaslMechanism.Plain, + SaslUsername = Environment.GetEnvironmentVariable("SASL_USERNAME"), + SaslPassword = Environment.GetEnvironmentVariable("SASL_PASSWORD"), + SslCaLocation = RuntimeInformation.IsOSPlatform(OSPlatform.OSX) ? "/usr/local/etc/openssl@1.1/cert.pem" : null; + }, + ...//publication, see below + ) + .Create()) + ... + +``` + +## Publication + +For more on a *Publication* see the material on an *External Bus* in [Basic Configuration](/contents/BrighterBasicConfiguration.md#using-an-external-bus). + +We allow you to configure properties for both Brighter and the Confluent .NET client. Because there are many properties on the Confluent .NET Client we also configure a callback to let you inspect and modify the configuration that we will pass to the client if you so desire. This can be used to add properties we do not support or adjust how we set them. + +- **Replication**: how many ISR nodes must receive the record before the producer can consider the write successful. Default is Acks.All. +- **BatchNumberMessages**: Maximum number of messages batched in one MessageSet. Default is 10. +- **EnableIdempotence**: Messages are produced once only. Will adjust the following if not set: `max.in.flight.requests.per.connection=5` (must be less than or equal to 5), `retries=INT32_MAX` (must be greater than 0), `acks=all`, `queuing.strategy=fifo`. Default is true. +- **LingerMs**: Maximum time, in milliseconds, for buffering data on the producer queue. Default is 5. +- **MessageSendMaxRetries**: How many times to retry sending a failing MessageSet. Note: retrying may cause reordering, set the max in flight to 1 if you need ordering by when sent. Default is 3. +- **MessageTimeoutMs**: Local message timeout. This value is only enforced locally and limits the time a produced message waits for successful delivery. A time of 0 is infinite. Default is 5000. +- **MaxInFlightRequestsPerConnection**: Maximum number of in-flight requests the client will send. We default this to 1, so as to allow retries to not de-order the stream. +- **NumPartitions**: How many partitions for this topic. We default to 1. +- **Partitioner**: How do we partition? Defaults to Partitioner.ConsistentRandom. +- **QueueBufferingMaxMessages**: Maximum number of messages allowed on the producer queue. Defaults to 10. +- **QueueBufferingMaxKbytes**: Maximum total message size sum allowed on the producer queue. Defaults to 1048576 bytes (so for 10 messages about 104Kb per message). +- **ReplicationFactor**: What is the replication factor? How many nodes is the topic copied to on the broker? Defaults to 1. +- **RetryBackoff**: The backoff time before retrying a message send. Defaults to 100. +- **RequestTimeoutMs**: The ack timeout of the producer request. This value is only enforced by the broker and relies on Replication being != AcksEnum.None. Defaults to 500. +- **TopicFindTimeoutMs**: How long to wait when asking for topic metadata. Defaults to 5000. +- **TransactionalId**: The unique identifier for this producer, used with transactions + +The following example shows how a *Publication* might be configured: + +``` csharp + services.AddBrighter(...) + .UseExternalBus( + new KafkaProducerRegistryFactory( + ...,//connection see above + new KafkaPublication[] {new KafkaPublication() + { + Topic = new RoutingKey("MyTopicName"), + NumPartitions = 3, + ReplicationFactor = 3, + MessageTimeoutMs = 1000, + RequestTimeoutMs = 1000, + MakeChannels = OnMissingChannel.Create + } + ) + .Create()) + ... + +``` + +### Configuration Callback + +The Confluent .NET client has a range of configuration options. Some of those can be controlled through the publication. But, to allow you the full range of configuration options for the Confluent client, including new options that may appear, we provide a callback on the **KafkaProducerRegistryFactory**. The registry exposes a method, **SetConfigHook(Action hook)**. The method takes a *delegate* (you can pass a lambda). Your delegate will be called with the *proposed* ProducerConfig (taking into account the *Publication* settings). You can adjust additional parameters at this point. + +You can use it as follows: + +``` csharp + + var publication = new KafkaPublication() + { + Topic = new RoutingKey("MyTopicName"), + NumPartitions = 3, + ReplicationFactor = 3, + MessageTimeoutMs = 1000, + RequestTimeoutMs = 1000, + MakeChannels = OnMissingChannel.Create + }; + publication.SetConfigHook(config => config.EnableGaplessGuarantee = true) + + services.AddBrighter(...) + .UseExternalBus( + new KafkaProducerRegistryFactory( + ...,//connection see above + new KafkaPublication[] {publication}) + .Create()) + ... + +``` + +### Kafka Topic Auto Create + +Brighter uses the Kafka AdminClient for topic creation. For this to work as expected you should set the server property of **auto.create.topics.enable** to **false**; otherwise the topic will be auto-created with the values defined by your server for new topics, such as the number of partitions. This error can be insidious because your code will still work against this topic, but without inspection you will not observe that its properties do not match those requested. + +If you want to specify the topic through Brighter, or through your own IaaS code, we recommend always setting this setting to false; we recommend only setting it to true if you tell Brighter to assume that the infrastructure exists, as it will then be created on the first write. + +## Subscription + +For more on a *Subscription* see the material on configuring *Service Activator* in [Basic Configuration](/contents/BrighterBasicConfiguration.md#configuring-the-service-activator). + +We support a number of Kafka specific *Subscription* options: + +- **CommitBatchSize**: We commit processed work (marked as acked or rejected) when a batch size worth of work has been completed (see [below](#offset-management)). +- **GroupId**: Only one consumer in a group can read from a partition at any one time; this preserves ordering. We do not default this value, and expect you to set it. +- **IsolationLevel**: Default to read only committed messages, change if you want to read uncommitted messages. May cause duplicates. +- **MaxPollIntervalMs**: How often the consumer needs to poll for new messages to be considered alive, polling greater than this interval triggers a re-balance. Kafka default to 300000ms +- **NumPartitions**: How many partitions does the topic have? Used for topic creation, if required. +- **OffsetDefault**: What do we do if there is no offset stored in ZooKeeper for this consumer. Defaults to AutoOffsetReset.Earliest - Begin reading the stream from the start. Options include AutOffsetRest.Latest - Start from now i.e. only consume messages after we start and AutoOffsetReset.Error - which considers it an error if not reset is found +- **ReadCommittedOffsetsTimeOutMs**: How long before attempting to read back committed offsets (mainly used in debugging) is an error. Defaults to 5000. +- **ReplicationFactor**: What is the replication factor? How many nodes is the topic copied to on the broker? Defaults to 1. Used for topic creation if required. +- **SessionTimeoutMs**: If Kafka does not receive a heartbeat from the consumer within this time window, trigger a re-balance. Default is Kafka default of 10s. +- **SweepUncommittedOffsetsIntervalMs**: The interval at which we sweep, looking for offsets that have not been flushed (see [below](#offset-management)). + +The following example shows how a subscription might be configured: + +``` csharp + private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + var subscriptions = new KafkaSubscription[] + { + new KafkaSubscription( + new SubscriptionName("paramore.example.greeting"), + channelName: new ChannelName("greeting.event"), + routingKey: new RoutingKey("greeting.event"), + groupId: Environment.GetEnvironmentVariable("KAFKA_GROUPID"), + timeoutInMilliseconds: 100, + commitBatchSize: 5, + sweepUncommittedOffsetsIntervalMs: 3000 + ) + }; + + //create the gateway + var consumerFactory = new KafkaMessageConsumerFactory( + new KafkaMessagingGatewayConfiguration {...} // see connection information above + ); + + services.AddServiceActivator(options => + { + options.Subscriptions = subscriptions; + options.ChannelFactory = new ChannelFactory(consumerFactory); + }).AutoFromAssemblies(); + + + services.AddHostedService(); +} +``` + +## Offset Management + +It is important to understand how Brighter manages the **offset** of any **partitions** assigned to your **consumer**. + +- Brighter manages committing **offsets** to Kafka. This means we set the Confluent client's *auto store* and *auto commit* properties to *false*. +- The **CommitBatchSize** setting on the *Subscription* determines the size of your buffer. A smaller buffer is less efficient, but if your consumer crashes any **offsets** pending commit in the buffer will be lost, and you will be represented with those records when you next read from the **partition**. We default this value to 10. +- We do not add an offset commit to the buffer until you Ack the request. The message pump will Ack for you once you exit your handler (via return or [throwing an exception](/contents/HandlerFailure.md)). +- Flushing the commit buffer happens on a separate thread. We only run one flush at a time, and we flush a **CommitBatchSize** number of items from the buffer. + - A busy consumer may not flush on every increment of the **CommitBatchSize**, as it may need to wait for the last flush to finish. + - We won't flush again until we cross the next multiple of the **CommitBatchSize**. For example if the **CommitBatchSize** is 10, and the handler is busy so that by the time the buffer flushes there are 13 pending commits in the buffer, the buffer would only flush 10, and 3 would remain in the buffer; we would not flush the next 10 until the buffer hit 20. + - If your **CommitBatchSize** is too low for the throughput, you might find that you miss a flush interval, because you are already flushing. + - If you miss a flush on a busy consumer, your buffer will begin to back up. If this continues, you will not catch up with subsequent flushes, which only flush the **CommitBatchSize** each time. This would lead to you continually being "backed up". + - For this reason you must set a **CommitBatchSize** that keeps pace with the throughput of your consumer. Use a larger **CommitBatchSize** for higher throughput consumers, smaller for lower. +- We sweep uncommitted offsets at an interval. This triggers a flush if no flush has run since the last flush plus the *Subscription's* **SweepUncommittedOffsetsIntervalMs**. + - A sweep will not run if a flush is currently running (and will in turn block a flush). + - A sweep flushes a **CommitBatchSize** worth of commits. + - It is intended for low-throughput consumers where commits might otherwise languish waiting for a batch-size increment. + - It is *not* intended to flush a buffer that backs up because the **CommitBatchSize** is too low, and won't function for that. Fix the **CommitBatchSize** instead. +- On a re-balance where we stop processing a **partition** on an individual consumer, we flush the remaining **offsets** for the revoked **partitions**. + - We configure the consumer to use sticky assignment strategy to avoid unnecessary re-assignments (see the [Confluent documentation](https://www.confluent.io/blog/cooperative-rebalancing-in-kafka-streams-consumer-ksqldb/)). +- On a consumer shutdown we flush the buffer to commit all **offsets**. + + +## Working with Schema Registry + +If you want to use tools within the Kafka ecosystem such as Kafka Connect or KQSL you will almost certainly need to use Confluent Schema Registry to provide the schema of your message. + +You will need to pull in the following package: + +* Confluent.SchemaRegistry + +and a package for the serialization of your choice. Here we are using JSON, so we use + +* Confluent.SchemaRegistry.Serdes.Json + +When working with Brighter, to use Confluent Schema Registry you will need to take a dependency on ISchemaRegistry in the constructor of your message mapper. To fulfill this constructor, in your application setup you will need to register an instance of schema registry. You should configure the schema registry config url to be the url of you schema registry. (Here we just use localhost for a development instance running in docker as an example). + +``` csharp +var schemaRegistryConfig = new SchemaRegistryConfig { Url = "http://localhost:8081"}; +var cachedSchemaRegistryClient = new CachedSchemaRegistryClient(schemaRegistryConfig); +services.AddSingleton(cachedSchemaRegistryClient); +``` + +Once you can satisfy the dependency, you will want to use the serializer from the Serdes package to serialize the body of your message, instead of System.Text.Json. Note that 'under-the-hood' the Serdes serializer uses [Json.NET](https://www.newtonsoft.com/json) and [NJsonSchema](https://github.com/RicoSuter/NJsonSchema), so you may need to mark up your code with attributes from these packages to create the schema you want and serialize a valid message to it. (Note that, at this time, the Serdes package does not support System.Text.Json so you will need to take a dependency on Json.NET if you want to use the schema registry). + +It is worth noting the following aspects of the code sample below: + +* We need to set up a SerializationContext and tell Serdes that we are serializing the message body using their serializer +* We provide two helpers, though you can pass your own settings if you prefer: + * **ConfluentJsonSerializationConfig.SerdesJsonSerializerConfig()** offers default settings for JSON serialization (many of these are passed through to Json.NET). + * **ConfluentJsonSerializationConfig.NJsonSchemaGeneratorSettings()** offers default settings for JSON Schema generation (such as using camelCase). + + +``` csharp +public class GreetingEventMessageMapper : IAmAMessageMapper +{ +private readonly ISchemaRegistryClient _schemaRegistryClient; +private readonly string _partitionKey = "KafkaTestQueueExample_Partition_One"; +private SerializationContext _serializationContext; +private const string Topic = "greeting.event"; + +public GreetingEventMessageMapper(ISchemaRegistryClient schemaRegistryClient) +{ + _schemaRegistryClient = schemaRegistryClient; + //We care about ensuring that we serialize the body using the Confluent tooling, as it registers and validates schema + _serializationContext = new SerializationContext(MessageComponentType.Value, Topic); +} + +public Message MapToMessage(GreetingEvent request) +{ + var header = new MessageHeader(messageId: request.Id, topic: Topic, messageType: MessageType.MT_EVENT); + //This uses the Confluent JSON serializer, which wraps Newtonsoft but also performs schema registration and validation + var serializer = new JsonSerializer(_schemaRegistryClient, ConfluentJsonSerializationConfig.SerdesJsonSerializerConfig(), ConfluentJsonSerializationConfig.NJsonSchemaGeneratorSettings()).AsSyncOverAsync(); + var s = serializer.Serialize(request, _serializationContext); + var body = new MessageBody(s, "JSON"); + header.PartitionKey = _partitionKey; + + var message = new Message(header, body); + return message; +} + +public GreetingEvent MapToRequest(Message message) +{ + var deserializer = new JsonDeserializer().AsSyncOverAsync(); + //This uses the Confluent JSON serializer, which wraps Newtonsoft but also performs schema registration and validation + var greetingCommand = deserializer.Deserialize(message.Body.Bytes, message.Body.Bytes is null, _serializationContext); + + return greetingCommand; +} +} + +``` + + +## Requeue with Delay + +We don't currently support requeue with delay for Kafka. It might be added in a future release, where the strategy would be to: + +- Publish the requeued message to a new stream +- Commit the offset +- Poll that stream with a new subscription but at a greater interval between polling (i.e. the delay) + +In the interim you can manually implement that approach if required. + + + + + + + diff --git a/source/shared/Logging.md b/source/shared/Logging.md new file mode 100644 index 0000000..522b18e --- /dev/null +++ b/source/shared/Logging.md @@ -0,0 +1,3 @@ +# Logging + +TODO \ No newline at end of file diff --git a/source/shared/MSSQLInbox.md b/source/shared/MSSQLInbox.md new file mode 100644 index 0000000..0b50aed --- /dev/null +++ b/source/shared/MSSQLInbox.md @@ -0,0 +1,35 @@ +# MSSQL Inbox + +## Usage +The MSSQL Inbox allows use of MSSQL for [Brighter's inbox support](/contents/BrighterInboxSupport.md). The configuration is described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#inbox). + +For this we will need the *Inbox* packages for the MsSQL *Inbox*. + +* **Paramore.Brighter.Inbox.MsSql** + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + services.AddServiceActivator(options => + { ... }) + .UseExternalInbox( + new MsSqlInbox(new MsSqlInboxConfiguration("server=localhost; port=3306; uid=root; pwd=root; database=Salutations", "Inbox"); + new InboxConfiguration( + scope: InboxScope.Commands, + onceOnly: true, + actionOnExists: OnceOnlyAction.Throw + ) + ); +} + +... + +``` + diff --git a/source/shared/MessageMappers.md b/source/shared/MessageMappers.md new file mode 100644 index 0000000..ddc65d6 --- /dev/null +++ b/source/shared/MessageMappers.md @@ -0,0 +1,264 @@ +# Message Mappers + +A message mapper turns domain code into a Brighter **Message**. A Brighter **Message** has a **MessageHeader** for information about the message. Key properties are: **TimeStamp**, **Topic**, and **Id**. The **Message** also has a **MessageBody**, which contains the payload. + +The messageType parameter tells the Dispatcher that listens to this message, how to treat it, as a Command or an Event. Brighter's *Dispatcher* dispatches a **Message** using either **commandProcessor.Send()** for **MT_COMMAND** or **commandProcessor.Publish()** for **MT_EVENT**. + +Typically, you serialize your request as the **MessageBody** for in **MapToMessage** and serialize your **MessageBody** into a request in **MapToRequest**. + +The body is a byte[] and as such we can support any format that can be converted into a byte[] as the message body. + +Because [message oriented middleware](#message-oriented-middleware-mom) typically looks in a header for routing information, you add your routing information in the **MessageHeader**. + +Each individual transport has code to turn a Brighter format message into a message oriented middleware compatible message, and vice versa, so your code only needs to translate to and from the Brighter format. + +## Writing A Message Mapper + +We use **IAmAMessageMapper\** to map between messages in the External Bus and a **Message**. + +You create a **Message Mapper** by deriving from **IAmAMessageMapper\** and implementing the **MapToMessage()** and **MapToRequest** methods. + +An example follows: + +``` csharp +public class GreetingMadeMessageMapper : IAmAMessageMapper +{ + public Message MapToMessage(GreetingMade request) + { + var header = new MessageHeader(messageId: request.Id, topic: "GreetingMade", messageType: MessageType.MT_EVENT); + var payload = System.Text.Json.JsonSerializer.Serialize(request, new JsonSerializerOptions(JsonSerializerDefaults.General)); + var body = new MessageBody(payload, ApplicationJson, CharacterEncoding.UTF8); + var message = new Message(header, body); + return message; + } + + public GreetingMade MapToRequest(Message message) + { + return JsonSerializer.Deserialize(message.Body.Value, JsonSerialisationOptions.Options); + } +} +``` + +## Brighter Message Structure + +Brighter divides a message into two parts: + +* Header: The header contains metadata (data about the message). It is typically used to control how we process the payload or provide additional context about it. +* Body: The body contains the payload, which is usually the **Command** or **Event** being raised for the consumer to action + +### The Message Header + +The Message Header has a number of Brighter defined properties and a bag that can be used for user-defined properties. + +#### Common Properties + +* **Id**(GUID): The identifier for this message +* **Topic**(string): The topic this message should be sent to, used to route the message in most transports +* **MessageType** (enum): The type of message: (Unacceptable (not translated), None (null object), Command, Event, Document, Quit (terminats a pump)) +* **CorrelationId** (GUID): Is this message a response to another message (usually an event reply to a command), if so this is the id that links them +* **ReplyTo** (string): A topic to reply to. In a request-reply set this to tell the receiver where to send replies +* **ContentType** (string): Normally, allow the **MessageBody** (below) to set this. +* **PartitionKey** (string): Where consistent hashing is used to partition a stream, what is the value to partition on + +#### Brighter Properties + +* **DelayedMilliseconds** (int): If we chose to retry with a delay, how long for? +* **HandledCount** (int): How many times have we tried to handle this message +* **Telemetry** (MessageTelemetry): Open Telemetry information for the message + +#### Routing + +In **MapToMessage**, the **topic** parameter on the **MessageHeader** controls the topic (or routing key) which we use when publishing a message to the external bus. We use this value when using the SDK for the message oriented middleware transport to publish a message on that transport. + +For this reason it is the **MessageMapper** that controls how messages published to the external bus are routed. + + +### The Message Body + +The Message Body stores the content for transmission over a transport as a byte[]. This supports both plain text and binary payloads. Your choice of payload type is constrained by what the transport requires or supports. + +In many cases the easiest option is to send the payload as plain text, as this is the easiest to inspect if you need to debug your messages. In this case the simplest path is to serialize the **Command** or **Event** as JSON and deserialize from that JSON. MessageBody contains a constructor that takes a string with two optional parameters, a media type (which defaults to **application/json**) and a character encoding type for the string (which defaults to **CharacterEncoding.UTF8**), + +```csharp +public MessageBody(string body, string contentType = ApplicationJson, CharacterEncoding characterEncoding = CharacterEncoding.UTF8) +{ + ... +``` + +which can be used as follows (or omitting the default parameters) + +```csharp + +var payload = System.Text.Json.JsonSerializer.Serialize(request, new JsonSerializerOptions(JsonSerializerDefaults.General)); +var body = new MessageBody(payload, ApplicationJson, CharacterEncoding.UTF8); + +``` + +If your payload is binary, then we provide two constructors that can be used to write bytes. For backwards compatibility these constructors also default to application/json and UTF-8. However, if you have binary content we recommend setting the media type to application/octet-stream and the character encoding to either **CharacterEncoding.Base64** if it needs transmission as a string, or **CharacterEncoding.Raw** if not). + +```csharp + +public MessageBody(byte[] bytes, string contentType = ApplicationJson, CharacterEncoding characterEncoding = CharacterEncoding.UTF8) +{ + ... + +public MessageBody(in ReadOnlyMemory body, string contentType = ApplicationJson, CharacterEncoding characterEncoding = CharacterEncoding.UTF8) +{ + ... + +``` + +For example, when writing a Kafka payload with leading bytes indicating the schema id, you would want to use a binary payload because conversion to and from a UTF8 string is lossy. Here we serialize the payload with the Kafka header (Magic Byte (0) + Schema Id Bytes) and a JSON payload using the Confluent Serdes serializer. Even though we serialize to JSON, because of the header bytes we treat the payload as binary: + +```csharp + +public Message MapToMessage(GreetingEvent request) +{ + var header = new MessageHeader(messageId: request.Id, topic: Topic, messageType: MessageType.MT_EVENT); + //This uses the Confluent JSON serializer, which wraps Newtonsoft but also performs schema registration and validation + var serializer = new JsonSerializer(_schemaRegistryClient, ConfluentJsonSerializationConfig.SerdesJsonSerializerConfig(), ConfluentJsonSerializationConfig.NJsonSchemaGeneratorSettings()).AsSyncOverAsync(); + var s = serializer.Serialize(request, _serializationContext); + var body = new MessageBody(s, MediaTypeNames.Application.Octet, CharacterEncoding.Raw); + header.PartitionKey = _partitionKey; + + var message = new Message(header, body); + return message; +} + +``` + +The **Value** property of the **MessageBody** returns a string depending on the character encoding type of the body. If you do not set a character encoding then we assume a standard UTF8 **string**; if you set the character encoding to base64 or raw, we return a base64 string; if you set the character encoding to ascii we will return an ascii string. + + +### Options for System.Text.Json Serialization + +The most common solution to serialization of the message payload is to use System.Text.Json to convert the message's metadata to JSON for sending over a messaging middleware transport. You can adjust the behavior of this serialization through our **JsonSerialisationOptions**. See [Brighter Configuration](/contents/BrighterBasicConfiguration.md#configuring-json-serialization) for more on how to set your options. + +You can then use this, when you want to set options consistently for message serialization. + +``` csharp + public GreetingMade MapToRequest(Message message) + { + return JsonSerializer.Deserialize(message.Body.Value, JsonSerialisationOptions.Options); + } +``` + +## Transformers + +Some concerns are orthogonal to how you map a **IRequest** into a **Message** or how you map a **Message** into an **IRequest**. Instead they concern how we process that Message. A typical list of such concerns might include: handling large message payloads (compression of moving to a distributed file store), encryption, registering or validating schema, and adding common metadata to headers. + +A *Transform* is a middleware that runs as part of the pipeline we use to map a **IRequest** into a **Message** or how you map a **Message** into an **IRequest**. A transform implements an **IMessageTransformAsync**. (All transforms are async). + +``` csharp +public interface IAmAMessageTransformAsync : IDisposable +{ + void InitializeWrapFromAttributeParams(params object[] initializerList); + void InitializeUnwrapFromAttributeParams(params object[] initializerList); + Task WrapAsync(Message message, CancellationToken cancellationToken); + Task UnwrapAsync(Message message, CancellationToken cancellationToken); +} +``` + +### Wrap + +When we *wrap* the source is the *Message Mapper* and the transform is applied to the **Message** that you generate from the **IRequest** in your **MapToMessage**. + +You indicate that you wish to *wrap* a *Message Mapper* with the **WrapWithAttribute** associated with the **IMessageTransformAsync** you want to apply to the **Message** you have created from the **IRequest**. In the example below we use a **ClaimCheck** to move large message payloads (those over the threshold) into a *luggage store* (for example an S3 bucket). + +``` csharp +[ClaimCheck(step:0, thresholdInKb: 256)] +public Message MapToMessage(GreetingEvent request) +{ + var header = new MessageHeader(messageId: request.Id, topic: typeof(GreetingEvent).FullName.ToValidSNSTopicName(), messageType: MessageType.MT_EVENT); + var body = new MessageBody(JsonSerializer.Serialize(request, JsonSerialisationOptions.Options)); + var message = new Message(header, body); + return message; +} +``` + +### Unwrap + +When we *unwrap* the sink is the *Message Mapper* and the transform is applied to the **Message** before you turn it into an **IRequest** in your **MapToRequest**. + +You indicate that you wish to *unwrap* a *Message Mapper* with the **UnwrapWithAttribute** associated with the **IMessageTransformAsync** you want to apply to the **Message** before you create your **IRequest**. In the example below we use a **RetrieveClaim** to retrieve a large message payload (most likely stored by a Claim Check in a *luggage store*) that will provide the body of our **Message** before we deserialize it to the **IRequest**. + +``` csharp +[RetrieveClaim(step:0)] +public GreetingEvent MapToRequest(Message message) +{ + var greetingCommand = JsonSerializer.Deserialize(message.Body.Value, JsonSerialisationOptions.Options); + + return greetingCommand; +} + +``` + +### Transform, Wrap and Unwrap + +Usually your **WrapWithAttribute** and **UnwrapWithAttribute** are paired and opposite. Usually they associate with a common **IMessageTransformAsync** that implements support for both transforms: the **WrapWithAttribute** results in the **WrapAsync** method of the transform being called (the **Message** is passed to it); the **UnwrapWithAttribute** results in the **UnwrapAsync** method being called (again the **Message** is passed to it). + +Both the **WrapWithAttribute** and the **UnwrapWithAttribute** are a type of **TransformAttribute** + +``` csharp + +public abstract class TransformAttribute : Attribute + { + public int Step { get; set; } + public abstract Type GetHandlerType(); + public virtual object[] InitializerParams() + { + return new object[0]; + } + +``` + +To implement a **TransformAttribute** you need to create a derived type that overrides the **GetHandlerType** to return the type of your **IMessageTransformAsync**. + +#### Step + +Step specifies the order in which a transform runs (attributes are not guaranteed to be made available in top-down order by reflection). This can be important in transforms. Imagine that you want to compress any message over 256Kb, but because a large enough message might still not be small enough after compression, a message that is *still* over 256Kb to distributed storage. In this case you would want to make sure that the step value for compression was lower than the step value to offload to distributed storage. + +#### Passing Parameters to a Transform + +If you want to pass parameters to your transform, they must be available at compile time as arguments to your derived **TransformAttribute**. The parameters of your attribute's constructor can be set from an attribute. Your attribute can then store these parameters in private fields. We call your derived attributes **InitializeParams** method after instantiating your **IMessageTransformAsync**, and pass the values to that object via either the **InitializeWrapFromAttributeParams** or **InitializeUnwrapFromAttributeParams** as appropriate for the type of **TransformAttribute** (either **WrapWithAttribute** or **UnwrapWithAttribute**). + +So in this example, the **ClaimCheck** takes a parameter for the *threshold* at which point we move the body of the message into distributed storage as opposed to serializing it in the message body. + +``` csharp +public class ClaimCheck : WrapWithAttribute +{ + private readonly int _thresholdInKb; + + public ClaimCheck(int step, int thresholdInKb = 0) : base(step) + { + _thresholdInKb = thresholdInKb; + } + + public override object[] InitializerParams() + { + return new object[] { _thresholdInKb }; + } + + public override Type GetHandlerType() + { + return typeof(ClaimCheckTransformer); + } +} +``` + +### Message Transformer Factory + +Because we do not know how to construct user-defined types, you have to pass us a **IAmAMessageTransformerFactory** that constructs instances of your **IMessageTransformAsync**. + +Normally, you implement this using your Inversion of Control container. We provide an implementation for the .NET Inversion of Control container **ServiceCollection** with **ServiceProviderTransformerFactory**. You need a reference to the following NuGet package: + +* **Paramore.Brighter.Extensions.DependencyInjection** + + +If you are using HostBuilder, our extension methods mean that you benefit from automatic inclusion of the **ServiceProviderTransformerFactory** and registration of your **IMessageTransformAsync**. + + + + + + diff --git a/source/shared/Microservices.md b/source/shared/Microservices.md new file mode 100644 index 0000000..d1922a7 --- /dev/null +++ b/source/shared/Microservices.md @@ -0,0 +1,47 @@ +# Microservices + +It is possible to think of microservices as 3rd generation SOA. First generation SOA was SOAP based web services. 2nd generation SOA was messaging, sometimes over SOAP, but also over middleware, often an +Enterpise Service Bus (ESB). The third generation emphasizes \"smart endpoints, dumb pipes\" over the use of an ESB, either via REST or a lightweight broker such as RMQ. + +But much of what applied to SOA,. still applies to microservices. + +\"SOA is focused on business processes. These processes are performed in different steps (also called activities or tasks) on different systems. The primary goal of a service is to represent a "natural" step of +business functionality. That is, according to the domain for which it's provided, a service should represent a self-contained functionality that corresponds to a real-world business activity.\" + +Josuttis, Nicolai M.. SOA in Practice: The Art of Distributed System Design. + +Don Box, the creator of SOAP, defined 4 tenets for a SOA service. These +rules are still useful for Microservices. + +1. Boundaries are explicit +2. Services are Autonomous +3. Share schema not type +4. Compatibility is based on policy + +![Microservices](_static/images/Microservices.png) + +## Boundaries are explicit + +We must have an API (it may be HTTP, gRPC, AMQP, Kafka etc.). The API hides our implementation details. We allow consumers to couple to this API, but not to the contents. The API should be a stable abstraction, it is our contract with our consumers. The implementation details can be unstable. + +## Services are autonomous + +We want to be able to release this microservice and this microservice alone. The implication of this includes the idea that because no one couples to our implementation details, then provided we do not alter the +contract expressed by the API, we can re-release easily. But this also implies that we are the single writer to any backing storage that keeps our state. Otherwise we would couple the schema of that backing store to another service and would not be able to release independently of other services if they had coupled to those details. + +Microservices are a logical, not a physical boundary, and they might consist of more than one container, such as web container and a console container, provided all the containers are considered to be part of the +release boundary for CI or CD. This is common where we have different scaling requirements for say the API served from the web container and a worker process reading from a task queue served from a console container. + +The key idea here is Independent Deployability. + +## Share Schema not type + +Our software system may not be homogeneous, we may have services developed in multiple languages. As a result we must not prevent interoperability between microservices by use of the type system from language in the API. Instead we should use platform neutral alternatives such as plain text formats (JSON, XML, YAML) or binary ones (Avro, ProtoBuf). + +## Compatibility is based on policy + +For our microservices to communicate we need to agree on the protocols we will use. In the SOAP era this led to the growth of WS- Specifications that described policies for a wide range of service capabilities. Under microservices there is no similar standards movement, but organizations still need to make assertions about the protocols that the will use in order to provide interoperability. + +## Next + +See [Event Driven Collaboration](EventDrivenCollaboration.html) for guidance on how to integrate microservices using events. diff --git a/source/shared/Monitoring.md b/source/shared/Monitoring.md new file mode 100644 index 0000000..57c8f8c --- /dev/null +++ b/source/shared/Monitoring.md @@ -0,0 +1,77 @@ +# Monitoring + +Brighter emits monitoring information from an External Bus using a configured [Control Bus](https://brightercommand.github.io/Brighter/ControlBus.html) + +## Configuring Monitoring + +Firstly [configure a Control +Bus](https://brightercommand.github.io/Brighter/ControlBus.html#configure) in the brighter application to emit monitoring messages + +## Config file + +Monitoring requires a new section to be added to the application config file: + +``` xml + +
+ +``` + +The monitoring config can then be speicified later in the file: + +``` xml + + + +``` + +This enables runtime changes to enable/disable emitting of monitoring messages. + +## Handler Configuration + +Each handler that requires monitoring must be configured in two stages, a Handler attribute and container registration of a MonitorHandler for the given request: + +For example, given: + +- TRequest - a Brighter Request, inheriting from IRequest +- TRequestHandler - handles the TRequest, inheriting IHandleRequest + \ + +### Attribute + +The following attribute must be added to the Handle method in the handler, TRequestHandler: + +``` csharp +[Monitor(step:1, timing:HandlerTiming.Before, handlerType:typeof(TRequestHandler))] +``` + +Please note the step and timing can vary if monitoring should be after another attribute step, or timing should be emitted after. + +### Container registration + +The following additional handler must be registered in the application container (where `MonitorHandler` is a built-in Brighter handler): + +``` csharp +container.Register> +``` + +## Monitor message format + +A message is emitted from the Control Bus on Handler Entry and Handler Exit. The following is the form of the message: + +``` javascript +{ + "Exception": null, // or Exception message + "EventType": "EnterHandler or ExitHandler", + "EventTime": "2016-06-21T15:48:26.1390192Z", + "TimeElapsedMs": 0 or Duration, + "HandlerName": "...", + "HandlerFullAssemblyName": "...", + "InstanceName": "ManagementAndMonitoring", + "RequestBody": "{\"Id\":\"dc32b35f-bc75-4197-9178-c8310a63e4fb\", ... }", + "Id": "048cc207-e820-40fa-b931-55b60203fbc2" +} +``` + +Messages can be processed from the queue and interated with your monitoring tool of choice, for example Live python consumers emitting to console or logstash consumption to the ELK stack using relevant plugins +to provide performance raditators or dashboards. diff --git a/source/shared/MySQLInbox.md b/source/shared/MySQLInbox.md new file mode 100644 index 0000000..9123289 --- /dev/null +++ b/source/shared/MySQLInbox.md @@ -0,0 +1,37 @@ +# MySQL Inbox + +## Usage +The MySQL Inbox allows use of MySQL for [Brighter's inbox support](/contents/BrighterInboxSupport.md). The configuration is described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#inbox). + +For this we will need the *Inbox* packages for the MySQL *Inbox*. + +* **Paramore.Brighter.Inbox.MySql** + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + services.AddServiceActivator(options => + { ... }) + .UseExternalInbox( + new MySqlInbox(new MySqlInboxConfiguration("server=localhost; port=3306; uid=root; pwd=root; database=Salutations", "Inbox"); + new InboxConfiguration( + scope: InboxScope.Commands, + onceOnly: true, + actionOnExists: OnceOnlyAction.Throw + ) + ); +} + +... + +``` + + + diff --git a/source/shared/OutboxPattern.md b/source/shared/OutboxPattern.md new file mode 100644 index 0000000..22be7f2 --- /dev/null +++ b/source/shared/OutboxPattern.md @@ -0,0 +1,43 @@ +# Outbox Pattern Support + +## Producer Correctness + +When a microservice changes the state for which it is the system of record, and then signals to subscribers via an event that it has changed its state, how do we ensure that subscribers receive the event and are +therefore consistent with the producer? + +The system of record may become inconsistent with downstream consumers because after writing changes to an entity, we may fail to publish the corresponding message. Our connection to the message-oriented middleware (MoM) may fail, or MoM may fail. + +The new record is saved to the backing store, but the event is not raised so subscribing systems become inconsistent. We have a lost send. + +![CorrectnessProblem](_static/images/CorrectnessProblem.png) + +Distributed Transactions may seem like an answer, but possess two issues. First, we are probably using a backing store and message-oriented middleware from different vendors or OSS projects that don\'t support the same distributed transaction protocol. Second, distributed transactions don\'t scale well. + +We might naively try to fix this by sending the message first, then updating the backing store if that succeeds. But this won\'t necessarily work either, as we might fail to write to the database. + +The new record is posted to downstream systems, but the local database call is rejected, and so the upstream system is now inconsistent. A phantom send. + +In either solution we might simply decide that the best option is to ensure that we can retry what is hopefully a transient error. This may solve the problem in many instances, and is a good first step. But an +endless retry loop has its own dangers, consuming resources and reducing throughput, and if the app crashes we will still only be partially complete. So it cannot guarantee delivery of the message that matches +the write. + +## The Outbox Pattern + +In the Outbox pattern, we use the ACID properties of an RDBMS. We write not only to the table that stores the entity that we are inserting, updating, or deleting, but also we write the message we intend to send +to an \'outbox table\' in the same Db. + +We mark the time that the message was written, as part of the transaction, on the record. + +Then when we send the message via the Broker, we mark the message as dispatched in the table. + +![OutboxPattern](_static/images/OutboxPattern.png) + +An out-of-band sweeper process can then run, and query for messages that have not been sent within a time window (their written date is over a threshold of milliseconds ago, and they have no dispatched time stamp). +It then resends those messages. If it sends them, it marks them as dispatched. As the sweeper process keeps polling for messages that have not yet been sent, we will eventually send all the messages. So we have +**guaranteed delivery** but eventual consistency. + +It is possible that the write to the row to update the dispatched status will fail. It is not in a transaction between broker and RDBMS either. If that happens, we may send the message twice. + +For this reason, the Outbox pattern offers us **guaranteed, at least once** delivery. Consumers must be prepared for this. Either they can use an *Inbox*, which records all the messages they have seen recently and discards duplicates, or they must be idempotent and the result of processing the message twice has no side-affects. + +See [Brighter Outbox Support](BrighterOutboxSupport.html) for more on how to ensure Producer-Consumer correctness in Brighter. diff --git a/source/shared/PolicyFallback.md b/source/shared/PolicyFallback.md new file mode 100644 index 0000000..1f480e6 --- /dev/null +++ b/source/shared/PolicyFallback.md @@ -0,0 +1,45 @@ +# Fallback + +You may want some sort of backstop exception handler, that allows you to take compensating action, such as undoing any partially committed work, issuing a compensating transaction, or queuing work for later delivery (perhaps using the [External Bus](/contents/ImplementingExternalBus.md)). + +To support this we provide a **IHandleRequests\Fallback** method. In the Fallback method you write your code to run in the event of failure. + +## Calling the Fallback Pipeline + +We provide a **FallbackPolicy** Attribute that you can use on your **IHandleRequests\.Handle()** method. The implementation of the **Fallback Policy Handler** is straightforward: it creates a backstop exception handler by encompassing later requests in the [Request Handling Pipeline](BuildingAPipeline.html) in a try\...catch block. You can configure it to catch all exceptions, or just [Broken Circuit Exceptions](PolicyRetryAndCircuitBreaker.html) when a Circuit +Breaker has tripped. + +When the **Fallback Policy Handler** catches an exception it calls the **IHandleRequests\.Fallback()** method of the next Handler in the pipeline, as determined by **IHandleRequests\.Successor** + +The implementation of **RequestHandler\.Fallback()** uses the same [Russian Doll](BuildingAPipeline.html) approach as it uses for **RequestHandler\.Handle()**. This means that the request to take compensating action for failure, flows through the same pipeline as the +request for service, allowing each Handler in the chain to contribute. + +In addition the **Fallback Policy Handler** makes the originating exception available to subsequent Handlers using the **Context Bag** with the key: **CAUSE_OF_FALLBACK_EXCEPTION** + +## Using the FallbackPolicy Attribute + +The following example shows a Handler with **Request Handler Attributes** for [Retry and Circuit Breaker policies](PolicyRetryAndCircuitBreaker.html) that is configured with a **Fallback Policy** which catches a **Broken Circuit Exception** (raised when the Circuit Breaker is tripped) and initiates the Fallback chain. + +``` csharp +public class MyFallbackProtectedHandler: RequestHandler +{ + [FallbackPolicy(backstop: false, circuitBreaker: true, step: 1)] + [UsePolicy(new [] {}"MyCircuitBreakerStrategy", "MyRetryStrategy"}, step: 2)] + public override MyCommand Handle(MyCommand command) + { + /*Do some work that can fail*/ + } + + public override MyCommand Fallback(MyCommand command) + { + if (Context.Bag.ContainsKey(FallbackPolicyHandler.CAUSE_OF_FALLBACK_EXCEPTION)) + { + /*Use fallback information to determine what action to take*/ + } + return base.Fallback(command); + } +} +``` +## Scope of a Fallback + +Where you put any **FallbackPolicy** attribute determines what exceptions it will call your Fallback method to guard against. This is controlled by the **Step** parameter. Remember that you encapsulate anything with a higher **Step** and can react to an exception thrown there. diff --git a/source/shared/PolicyRetryAndCircuitBreaker.md b/source/shared/PolicyRetryAndCircuitBreaker.md new file mode 100644 index 0000000..e29f0ac --- /dev/null +++ b/source/shared/PolicyRetryAndCircuitBreaker.md @@ -0,0 +1,129 @@ +# Supporting Retry and Circuit Breaker + +Brighter is a [Command Processor](https://www.goparamore.io/control-bus-and-data-bus/) and +supports a [pipeline of Handlers to handle orthogonal requests](BuildingAPipeline.html). + +Amongst the valuable uses of orthogonal requests is patterns to support Quality of Service in a distributed environment: [Timeout, Retry, and Circuit Breaker](PolicyRetryAndCircuitBreaker.html#using-brighter-s-usepolicy-attribute). + +Even if you don't believe that you are writing a distributed system that needs this protection, consider that as soon as you have multiple processes, such as a database server, you are distributed. + +Brighter uses [Polly](https://github.com/App-vNext/Polly) to support Retry and Circuit-Breaker. Through our [Russian Doll Model](BuildingAPipeline.html) we are able to run the target handler in +the context of a Policy Handler, that catches exceptions, and applies a Policy on how to deal with them. + +## Using Brighter's UsePolicy Attribute + +By adding the **UsePolicy** attribute, you instruct the Command Processor to insert a handler (filter) into the pipeline that runs all later steps using that Polly policy. + +``` csharp +internal class MyQoSProtectedHandler : RequestHandler +{ + static MyQoSProtectedHandler() + { + ReceivedCommand = false; + } + + [UsePolicy(policy: "MyExceptionPolicy", step: 1)] + public override MyCommand Handle(MyCommand command) + { + /*Do work that could throw error because of distributed computing reliability*/ + } +} +``` + +To configure the Polly policy you use the PolicyRegistry to register the Polly Policy with a name. At runtime we look up that Policy by name. + +``` csharp +var policyRegistry = new PolicyRegistry(); + +var policy = Policy + .Handle() + .WaitAndRetry(new[] + { + 1.Seconds(), + 2.Seconds(), + 3.Seconds() + }, (exception, timeSpan) => + { + s_retryCount++; + }); + +policyRegistry.Add("MyExceptionPolicy", policy); +``` + +You can use multiple policies with a handler, instead of passing in a single policy identifier, you can pass in an array of policy identifiers: + +So if in addition to the above policy we have: + +``` csharp +var circuitBreakerPolicy = Policy.Handle().CircuitBreaker( + 1, TimeSpan.FromMilliseconds(500)); + +policyRegistry.Add("MyCircuitBreakerPolicy", policy); +``` + +then you can add them both to your handler as follows: + +``` csharp +internal class MyQoSProtectedHandler : RequestHandler +{ + static MyQoSProtectedHandler() + { + ReceivedCommand = false; + } + + [UsePolicy(new [] {"MyCircuitBreakerPolicy", "MyExceptionPolicy"} , step: 1)] + public override MyCommand Handle(MyCommand command) + { + /*Do work that could throw error because of distributed computing reliability*/ + } +} +``` + +Where we have multiple policies they are evaluated left to right, so in this case "MyCircuitBreakerPolicy" wraps "MyExceptionPolicy". + +When creating policies, refer to the [Polly](https://github.com/App-vNext/Polly) documentation. + +Whilst [Polly](https://github.com/App-vNext/Polly) does not support a Policy that is both Circuit Breaker and Retry i.e. retry n times with an interval between each retry, and then break circuit, to implement that simply put a Circuit Breaker UsePolicy attribute as an earlier step than the Retry UsePolicy attribute. If retries expire, the exception will bubble out to the Circuit Breaker. + +## Timeout + +You should not allow a handler that calls out to another process (e.g. a call to a Database, queue, or an API) to run without a timeout. If the process has failed, you will consumer a resource in your application +polling that resource. This can cause your application to fail because another process failed. + +Usually the client library you are using will have a timeout value that you can set. + +In some scenarios the client library does not provide a timeout, so you have no way to abort. + +We provide the Timeout attribute for that circumstance. You can apply it to a Handler to force that Handler into a thread which we will timeout, if it does not complete within the required time period. + +``` csharp +public class EditTaskCommandHandler : RequestHandler +{ + private readonly ITasksDAO _tasksDAO; + + public EditTaskCommandHandler(ITasksDAO tasksDAO) + { + _tasksDAO = tasksDAO; + } + + [RequestLogging(step: 1, timing: HandlerTiming.Before)] + [Validation(step: 2, timing: HandlerTiming.Before)] + [TimeoutPolicy(step: 3, milliseconds: 300)] + public override EditTaskCommand Handle(EditTaskCommand editTaskCommand) + { + using (var scope = _tasksDAO.BeginTransaction()) + { + Task task = _tasksDAO.FindById(editTaskCommand.TaskId); + + task.TaskName = editTaskCommand.TaskName; + task.TaskDescription = editTaskCommand.TaskDescription; + task.DueDate = editTaskCommand.TaskDueDate; + + _tasksDAO.Update(task); + scope.Commit(); + } + + return editTaskCommand; + } +} +``` diff --git a/source/shared/PostgresInbox.md b/source/shared/PostgresInbox.md new file mode 100644 index 0000000..ce295b2 --- /dev/null +++ b/source/shared/PostgresInbox.md @@ -0,0 +1,37 @@ +# Postgres Inbox + +## Usage +The Postgres Inbox allows use of Postgres for [Brighter's inbox support](/contents/BrighterInboxSupport.md). The configuration is described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#inbox). + +For this we will need the *Inbox* packages for the Postgres *Inbox*. + +* **Paramore.Brighter.Inbox.Postgres** + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + services.AddServiceActivator(options => + { ... }) + .UseExternalInbox( + new PostgresSqlInbox(new PostgresSqlInboxConfiguration("Host=localhost; Username=root; Password=root; Database=Salutations", "Inbox"); + new InboxConfiguration( + scope: InboxScope.Commands, + onceOnly: true, + actionOnExists: OnceOnlyAction.Throw + ) + ); +} + +... + +``` + + + diff --git a/source/shared/RabbitMQConfiguration.md b/source/shared/RabbitMQConfiguration.md new file mode 100644 index 0000000..b01d5a5 --- /dev/null +++ b/source/shared/RabbitMQConfiguration.md @@ -0,0 +1,152 @@ +# RabbitMQ Configuration + +## General + +RabbitMQ is OSS message-oriented-middleware and is [well documented](https://www.rabbitmq.com/documentation.html). Brighter handles the details of sending to or receiving from RabbitMQ. You may find it useful to understand the [building blocks](https://www.rabbitmq.com/tutorials/amqp-concepts.html) of the protocol. You might find the [documentation for the .NET SDK](https://www.rabbitmq.com/dotnet-api-guide.html) helpful when debugging, but you should not have to interact with it directly to use Brighter. + +RabbitMQ offers an API that defines primitives used to configure the middleware used for messaging: + +- **Exchange**: A routing table. Different types of exchanges route messages differently. An entry in the table is a **Routing Key**. +- **Queue**: A store-and-forward queue over which a consumer receives messages. A message is locked whilst a consumer has read it, until they ack it, upon which it is deleted from the queue, or nack it, upon which it is requeued or sent to a DLQ. +- **Binding**: Adds a queue as a target for a routing rule on an exchange. The routing key is used for this on a direct exchange (on the default exchange the routing key is the queue name). + +We connect to RabbitMQ via a multiplexed TCP/IP connection - RabbitMQ calls these channels. Brighter uses a push consumer, so it has an open channel and can be seen on the consumers list in the management console. Brighter maintains a pool of connections and when asked for a new connection will take one from it's pool in preference to creating a new one. + +## Connection + +The Connection to RabbitMQ is provided by an **RmqMessagingGatewayConnection** which allows you to configure the following: + +* **Name**: A unique name for the connection, for diagnostic purposes +* **AmqpUri**: A connection to AMQP in the form of an [RabbitMQ Uri](https://www.rabbitmq.com/uri-spec.html) **Uri** with reliability options for a retry count (defaults to 3), **ConnectionRetryCount**, retry interval (defaults to 1000ms) **RetryWaitInMilliseconds** and a circuit breaker retry timeout (defaults to 60000ms), **CircuitBreakTimeInMilliseconds**, which introduces a delay when connections exceed the retry count. +* **Exchange**: The definition of the exchange. **Name** is the identifier for the exchange. All exchanges have a [**Type**](https://www.rabbitmq.com/tutorials/amqp-concepts.html), and the default is **ExchangeType.Direct**, but it is a string value that supports all RabbitMQ exchange types on the .NET SDK. The **Durable** flag is used to indicate if the exchange definition survives node failure or restart of the broker which defaults to *false*. **SupportDelay** indicates if the Exchange supports retry with delay, which defaults to *false*. +* **DeadLetterExchange**: Another exchange definition, but this one is used to host any Dead Letter Queues (DLQ). This could be the same exchange, but normal practice is to use a different exchange. +* **Heartbeat**: RabbitMQ uses a heartbeat to determine if a connection has died. This sets the interval for that heartbeat. Defaults to 20s. +* **PersistMessages**: Should messages be saved to disk? Saving messages to disk allows them to be recovered if a node fails, defaults to *false*. + +In RabbitMQ, recreating an exiting primitive is a no-op provided the definition does not change. + +The following code creates a typical RabbitMQ connection (here shown as part of configuring an External Bus): + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .UseExternalBus(new RmqProducerRegistryFactory( + new RmqMessagingGatewayConnection + { + Name = "MyCommandConnection", + AmpqUri = new AmqpUriSpecification( + new Uri("amqp://guest:guest@localhost:5672") + connectionRetryCount: 5, + retryWaitInMilliseconds: 250, + circuitBreakerTimeInMilliseconds = 30000 + ), + Exchange = new Exchange("paramore.brighter.exchange", durable: true, supportDelay: true), + DeadLetterExchange = new Exchange("paramore.brighter.exchange.dlq", durable: true, supportDelay: false), + Heartbeat = 15, + PersistMessages = true + }, + ... //publication, see below + ).Create() +} +``` + +## Publication + +For more on a *Publication* see the material on an *External Bus* in [Basic Configuration](/contents/BrighterBasicConfiguration.md#using-an-external-bus). + +We only support one custom property on RabbitMQ which configures shutdown delay to await pending confirmations. + +* **WaitForConfirmsTimeOutInMilliseconds** + +Under the hood, Brighter uses [Publisher Confirms](https://www.rabbitmq.com/confirms.html) to update its Outbox for the dispatch time. This means that when publishing a message we allow RabbitMQ to confirm delivery of a message to all available nodes asynchronously, and then call us back, over blocking. This allows for higher throughput. But it means that we cannot update the Outbox to show a message as dispatched, until we receive the callback, which may occur after your handler pipeline for that message has completed and the message has been acknowledged. + +When shutting down a producer, it is possible that not all confirms have yet been received from RabbitMQ. The delay instructs Brighter to wait for a period of time, in order to allow the confirms to arrive. + +Missing a confirm will cause the *Outbox Sweeper* to resend a message, as it will not be marked as dispatched. (This is why we refer to Guaranteed *At Least Once* because there are many opportunities where messages may be duplicated in order to guarantee they were sent). + +The following code creates a *Publication* for RabbitMQ when configuring an *External Bus* + +``` csharp +public void ConfigureServices(IServiceCollection services) +{ + services.AddBrighter(...) + .UseExternalBus({ + ...//connection information, see above + }, + new RmqPublication[]{ + new RmqPublication + { + Topic = new RoutingKey("GreetingMade"), + MaxOutStandingMessages = 5, + MaxOutStandingCheckIntervalMilliSeconds = 500, + WaitForConfirmsTimeOutInMilliseconds = 1000, + MakeChannels = OnMissingChannel.Create + }} + ).Create() +} +``` + + +## Subscription + +For more on a *Subscription* see the material on configuring *Service Activator* in [Basic Configuration](/contents/BrighterBasicConfiguration.md#configuring-the-service-activator). + +We support a number of RabbitMQ specific *Subscription* options: + +* **DeadLetterChannelName**: The name of the queue to subscribe to DLQ notifications for this subscription (without a queue, the messages sent to the Dead Letter Exchange (DLX) will not be stored) +* **DeadLetterRoutingKey**: The routing key that binds the DLQ to the DLX +* **HighAvailability**: [Deprecated] Not used on versions of RabbitMQ 3+. Prior to this, configuring that a queue should be mirrored was an API option, now it is a configuration management option on the broker. +* **IsDurable**: Should subscription definitions survive a restart of nodes in the broker. +* **MaxQueueLength**: [Deprecated] Prefer to use policy to set this instead (see [RabbitMQ docs](https://www.rabbitmq.com/maxlength.html)). The maximum length a RabbitMQ queue can grow to, before new messages are rejected (and sent to a DLQ if there is one). + +This is a typical *Subscription* configuration in a Consumer application: + +``` csharp +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + var subscriptions = new Subscription[] + { + new RmqSubscription( + new SubscriptionName("paramore.sample.salutationanalytics"), + new ChannelName("SalutationAnalytics"), + new RoutingKey("GreetingMade"), + runAsync: false, + timeoutInMilliseconds: 200, + isDurable: true, + makeChannels: OnMissingChannel.Create), //change to OnMissingChannel.Validate if you have infrastructure declared elsewhere + }; + + var rmqConnection = new RmqMessagingGatewayConnection + { + AmpqUri = new AmqpUriSpecification( + new Uri("amqp://guest:guest@localhost:5672") + connectionRetryCount: 5, + retryWaitInMilliseconds: 250, + circuitBreakerTimeInMilliseconds = 30000 + ), + Exchange = new Exchange("paramore.brighter.exchange") + }; + + var rmqMessageConsumerFactory = new RmqMessageConsumerFactory(rmqConnection); + + services.AddServiceActivator(options => + { + options.Subscriptions = subscriptions; + options.ChannelFactory = new ChannelFactory(rmqMessageConsumerFactory); + ... //see Basic Configuration + }) +``` + +### Ack and Nack + +We use RabbitMQ's queues to subscribe to a routing key on an exchange. + +When we Accept/Ack a message, in response to a handler chain completing, we Ack the message to RabbitMQ using **Channel.BasicAck**. Note that we only Ack a message once we have completed running the chain. + +When we Reject/Nack a message (see [Handler Failure](/contents/HandlerFailure.md) for more on failure) then we use **Channel.Reject** to delete the message, and move it to a DLQ if there is one. + +Brighter has an internal buffer for messages pushed to a *Performer* (a thread running a message pump). This buffer has thread affinity (in RabbitMQ we have to Ack or Nack from the thread that received the message). When a consumer closes its connection to RabbitMQ, messages in the buffer that have not been Ack'd or Nack'd will be returned to the queue. + + + diff --git a/source/shared/Requests, Commands and Events.md b/source/shared/Requests, Commands and Events.md new file mode 100644 index 0000000..70c61df --- /dev/null +++ b/source/shared/Requests, Commands and Events.md @@ -0,0 +1,36 @@ +# Requests, Commands and Events +## The IRequest Interface + +We use the term **Request** for a data object containing parameters that you want to dispatch to a handler. Brighter uses the interface **IRequest** for this concept. + +We do not recommend deriving from **IRequest** but instead from the classes **Command** and **Event** which represent types of **Request**. + + +## What is the difference between a Command and an Event? + + Confusingly, both **Command** or **Event** which implement **IRequest** are examples of the [Command Pattern](https://brightercommand.github.io/Brighter/CommandsCommandDispatcherandProcessor.html). It is easiest to say that within Brighter **IRequest** is the abstraction that represents the Command from the [Command Pattern](https://brightercommand.github.io/Brighter/CommandsCommandDispatcherandProcessor.html). + +Why have both **Command** and **Event**? The difference is in how the **Command Dispatcher** dispatches them to handlers. + +- A **Command** is an imperative instruction to do something; it only has one handler. We will throw an error for multiple registered handlers of a command. +- An **Event** is a notification that something has happened; it has zero or more handlers. + +The difference is best explained by the following analogy. If I say \"Bob, make me a cup of coffee,\" I am giving a Command, an imperative instruction. My expectation is that Bob will make me coffee. If Bob does +not, then we have a failure condition (and I am thirsty and cranky). If I say \"I could do with a cup of coffee,\" then I am indicating a state of thirst and caffeine-withdrawal. If Bob or Alice make me a coffee I will be very grateful, but there is no expectation that they will. + +So choosing between **Command** or **Event** effects how the **Command Dispatcher** routes requests. + +See [Dispatching a Request](DispatchingARequest.html) for more on how to dispatch **Requests** to handlers. + +## Message Definitions and Independent Deployability + +Some messaging frameworks encourage you to share an assembly containing your message definitions between autonomous components, often as interfaces. Occasionally we see users trying to use **IRequest** for this purpose. + +We do not recommend this, instead preferring to keep to the Service Oriented Architecture (SOA) *tenet* of **Share schema not type**. + +Between components that we wish to be independently deployable - which might after all be in different languages, or use different frameworks - you should share a schema that defines the shape of the message (for example [AsyncAPI](https://www.asyncapi.com/). + +The only exception is where you have two apps that form part of a single service - such as a Task Queue that supports offloading work from a web API - as these tend to be a unit for Continuous Integration and not independently deployable, then sharing types may be appropriate. + +Many of our samples share types for convenience, but this is not advice to do that outside of a Task Queue. + diff --git a/source/shared/ReturningResultsFromAHandler.md b/source/shared/ReturningResultsFromAHandler.md new file mode 100644 index 0000000..01b1a91 --- /dev/null +++ b/source/shared/ReturningResultsFromAHandler.md @@ -0,0 +1,53 @@ + +# Returning Results from a Handler + +We use [Command-Query +separation](https://martinfowler.com/bliki/CommandQuerySeparation.html) so a Command does not have return value and **CommandDispatcher.Send()** does not return anything. Our project Darker provides a Query Processor that can be used to return results in response to queries. You can use both together to provide CQRS. + +This in turn leads to a set of questions that we need to answer about common scenarios: + +- How do I handle failure? With no return value, what do I do if my handler fails? +- How do I communicate the outcome of a command? + +## Handling Failure + +If we don\'t allow return values, what do you do on failure? + +- The basic failure strategy is to throw an exception. This will terminate the request handling pipeline. +- If you want *Internal Bus* support for [Retry, and Circuit Breaker](PolicyRetryAndCircuitBreaker.html) you can use our support for [Polly](https://github.com/App-vNext/Polly) Policies +- If you want to Requeue (with Delay) to an *External Bus*, you should throw a **DeferMessageAction** exception. +- Finally you can use our support for a [Fallback](PolicyFallback.html) handler to provide backstop exception handling. +- You can also build your own exception handling into your [Pipeline](BuildingAPipeline.html). + +We discuss these options in more detail in [Handler Failure](/contents/HandlerFailure.md). + +## Communicating the Outcome of a Command + +Sometimes you need to provide information to the caller about the outcome of a *Command*, instead of listening for an *Event* an. + +How do you communicate the outcome of handling a *Command*? There are two options, which depend on circumstance: + +* Raise an *Event* +* Update a field on the *Command* + +### Raising an Event + +This approach let's you take action in response to a *Command* by raising an *Event* within your handler using **CommandProcessor.Publish** or via an *External Bus* using **CommandProcessor.Post/CommandProcessor.DepositPost**. + +If you use an **Internal Bus** these handlers will run immediately, in their own pipeline, before your handler exits. If you use an **External Bus** you offload the work to another process. + +### Update a field on the Command + +If you are using an *Internal Bus* and need a return value from a *Command* you will note that **CommandProcessor.Send** has a void return value, so you cannot return a value from the handler. + +What happens if the caller needs to know the outcome, and can't be signalled via an *Event*? + +In that case add a property to the **Command** that you can initialize from the Handler. As an example, what happens if you need to return the identity of a newly created entity, so that you can use **Darker** to retrieve its details? In this case you can create a **NewEntityIdentity** property in your command that you write a newly created entity\'s identity to in the Handler, and then inspect the property in your **Command** in the calling code after the call to **commandProcessor.Send(command)** completes. + +You can think of these as *out* parameters. + +``` csharp +var createTaskCommand = new CreateTaskCommand(); +commandProcessor.Send(createTaskCommand); +var newTaskId = createTaskCommand.TaskId; +``` diff --git a/source/shared/Routing.md b/source/shared/Routing.md new file mode 100644 index 0000000..7d59d1a --- /dev/null +++ b/source/shared/Routing.md @@ -0,0 +1,141 @@ +# Routing + +### Routing Messages + +A producer routes messages to subscribers by setting a **Topic** on the **MessageHeader**. A **Topic** is just a string that you intend to use as a unique identifier for this message. A simple scheme can be the +typename of the event for the Producer. + +When implementing an **IAmAMessageMapper\** you set the **Topic** in the **MessageHeader** when serializing your **Command** or **Event** to disk. In the following example we set the **Topic** to *Task.Completed*. + +``` csharp +public class TaskCompletedEventMapper : IAmAMessageMapper +{ + public Message MapToMessage(TaskCompletedEvent request) + { + var header = new MessageHeader(messageId: request.Id, topic: "Task.Completed", messageType: MessageType.MT_EVENT); + var body = new MessageBody(JsonConvert.SerializeObject(request)); + var message = new Message(header, body); + return message; + } +} +``` + +The **routingKey** property of the connection must be the same key as used in the message mapper. This is passed to the broker to inform it that we want to subscribe to messages with that routing key on this channel. + +## Events + +Brighter has a default Publish-Subscribe approach to events. + +A broker provides an intermediary between the producer of a message and a consumer. A consumer registers interest in messages that have a key or topic. A producer sends messages with a key or topic to a broker, and the broker sends a copy of that message to every subscribing consumer. + + +## Commands + +For a command the producer knows its consumer. A command may be fire and forget, which means it does not expect a reply, or request-reply which means that it does. In request-reply the receiver knows its sender as well. + +The reason you might choose a command over an event is causality. + +Consider an application that needs to bill a customer's credit card. + +In an event driven approach, we could make the assumption that the transaction will succeed, raise an event to bill the customer and process the payment asynchronously. The producer of the billing request continues as though the transaction had succeeded. Eventually the customer is billed, and we are consistent. Our reason for taking this approach may be that our payment provider is often slow to respond and we do not want to make the customer wait whilst we handle details of their payment. If we fail to bill the customer we have to take compensating action - raising a billing failed event, which may alert an operator and email the customer. + +The problem is that this compensating action has to "chase" the success path, which may have already taken actions, such as shipping to the customer, that become expensive to reverse. + +With a command, we decide that as a payment transaction can fail we do not want to process the order until the payment has been received. In this case, our requirement is that we receive a response to our **Command** to bill. To route a command the Producer may send a reply-address to the Consumer so that it can send a response back on a 'private' channel. In our case, that reply-address is a topic that the sender subscribes to, in order to receive the response. + +Usually the Producer creates a topic for all of its replies, and matches request to response via a correlation id. This is simply a unique identifier that the Producer adds to the outgoing message. + +(Because the customer has to wait, in case we want to signal an error we are probably returning a 202 Accepted from our HTTP API with a link to a resource to monitor for the results of the transaction. In our client we display a progress indicator until we have completed the transaction.) + +To help route direct messages we provide two classes, **Request** and **Reply** but the real work occurs within the message mapper itself. + +Note also the correlation id that is added to the **ReplyAddress**. + +``` csharp + +public class MyRequest : Request +{ + public MyRequest(ReplyAddress sendersAddress) : base(sendersAddress) + { + } +} + +public class MyReply : Reply +{ + public MyReply(ReplyAddress sendersAddress) : base(sendersAddress) + { + } +} +``` + +When we convert this request into a **Message** via an **IAmAMessageMapper** we set the **MessageHeader** with the topic the Consumer should reply to. We also set the correlation id of the sender\'s message on the header. + +In the following code we also serialize the message back to a **Command** which is then routed by Brighter to a handler. When we serialize back to a **Command** we set the **ReplyAddress** with the Topic and Correlation Id. + +``` csharp +public class MyRequestMessageMapper : IAmAMessageMapper +{ + public Message MapToMessage(MyRequest request) + { + var header = new MessageHeader( + messageId: request.Id, + topic: "MyRequest", + messageType: MessageType.MT_COMMAND, + correlationId: request.ReplyAddress.CorrelationId, + replyTo: request.ReplyAddress.Topic); + + var json = new JObject(new JProperty("Id", request.Id)); + var body = new MessageBody(json.ToString()); + var message = new Message(header, body); + return message; + } + + public MyRequest MapToRequest(Message message) + { + var replyAddress = new ReplyAddress(topic: message.Header.ReplyTo, correlationId: message.Header.CorrelationId); + var request = new MyRequest(replyAddress); + var messageBody = JObject.Parse(message.Body.Value); + request.Id = Guid.Parse((string) messageBody["Id"]); + return request; + } +} +``` + +When we reply, we again use the message mapper to ensure that we route correctly. Again the key to responding is the **IAmAMessageMapper** implementation which uses the **ReplyAddress** to route the **Message** via its **MessageHeader** back to the caller. Note that whilst the response could be considered an event - a fact raised in response to a command - because it only has one Consumer, the sender, we route it as a command. If you want to broadcast the outcome, treat it as an event, but add **ReplyAddress** to your class derived from **Event** to correlate with the command. + +``` csharp +internal class MyReplyMessageMapper : IAmAMessageMapper +{ + public Message MapToMessage(MyReply request) + { + var header = new MessageHeader( + messageId:request.Id, + topic: request.SendersAddress.Topic, + messageType: MessageType.MT_COMMAND, + timeStamp: DateTime.UtcNow, + correlationId: request.SendersAddress.CorrelationId + ); + + var json = ...//serialize the reply body + + var body = new MessageBody(json.ToString()); + var message = new Message(header, body); + return message; + } + + public MyReply MapToRequest(Message message) + { + var replyAddress = new ReplyAddress(message.Header.Topic, message.Header.CorrelationId); + + var reply = new MyReply(replyAddress); + + ...//deserialize the body + + return reply; + } +} +``` + +## Summary + +The key to understanding routing in Brighter is that the **IAmAMessageMapper** implementation provides the point at which you control routing by setting the **MessageHeader**. diff --git a/source/shared/S3LuggageStore.md b/source/shared/S3LuggageStore.md new file mode 100644 index 0000000..5d2416e --- /dev/null +++ b/source/shared/S3LuggageStore.md @@ -0,0 +1,54 @@ +# S3 Luggage Store + +The **S3LuggageStore** is an implementation of **IAmAStorageProviderAsync** for AWS S3 Object Storage. It allows use of the [Claim Check](/contents/ClaimCheck.md) *Transformer* with S3 Object Storage as the *Luggage Store*. + +To use the **S3LuggageStore** you need to include the following NuGet package: + +* **Paramore.Brighter.Transformers.AWS** + +We then need to configure our **S3LuggageStore** and register it with our IoC container. Our **ClaimCheckTransformer** has a dependency on **IAmAStorageProviderAsync** and at runtime, when our [** **IAmAMessageTransformerFactory**](/contents/MessageMappers.md#message-transformer-factory) creates an instance it needs to be able to resolve that dependency. For this reason you need to register the implementation, in this case **S3LuggageStore** with the IoC container to allow it to resolve the dependency. + +We provide an extension method to **ServiceCollection** to help with this: + +``` csharp +serviceCollection.AddS3LuggageStore((options) => +{ + options.Connection = new AWSS3Connection(credentials, RegionEndpoint.EUWest1); + options.BucketName = "brightersamplebucketb0561a06-70ec-11ed-a1eb-0242ac120002"; + options.BucketRegion = S3Region.EUW1; + options.StoreCreation = S3LuggageStoreCreation.CreateIfMissing; +}); +``` + +You configure an **S3LuggageStore** using the **S3LuggateOptions** provided to the callback in **AddS3LuggageStore**. You MUST set the following options: + +* **Connection**: The **AWSS3Connection** that allows us to connect to your account. Used to create an **S3Client** and an **STSClient** +* **BucketName**: The name of the S3 bucket that backs the luggage store. We use one bucket for the luggage store. You may re-use a bucket that you already have. +* **BucketRegion**: Where is the bucket? Bucket names must be unique within a region. +* **StoreCreation**: What should we do when determining if there is a bucket for the store? + * **CreateIfMissing**: We will create the bucket in the requested region (provided the credentials provided have rights to do this.) + * **ValidateExists**: We will check if the bucket exists in the requested region. We throw an **InvalidOperationException** if it does not. + * **AssumeExists**: We do not check for the bucket, but just assume it exists + +If you choose **CreateIfMissing** or **ValidateExists** then you must register an **IHTTPClientFactory** as we will use this to obtain an HTTP Client for use with the AWS REST API to make a check for the bucket's existence. The simplest way to do this is to use the ServiceCollection extension provided for creating an **IHTTPClientFactory**: + +```csharp + serviceCollection.AddHttpClient(); +``` + +### Bucket Creation + +If we create the bucket we do so with the following properties: + +* Block public PUT access +* Object ownership transferred to bucket owner + +In addition we set the following properties on the bucket, which can be controlled: + +* We delete aborted uploads after the time given by **TimeToAbortFailedUploads**. Defaults to 1 day. +* We delete successful uploads after the time given by **TimeToDeleteGoodUploads**. Defaults to 7 days. + +We set *Tags* on the bucket if they are provided in the **Tags** property. + +We default the **ACLs** for the bucket to **S3CannedACL.Private, but you can choose to override this with another policy as described in [**S3CannedACL**](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-access-control.html#RESTCannedAccessPolicies). + diff --git a/source/shared/ShowMeTheCode.md b/source/shared/ShowMeTheCode.md new file mode 100644 index 0000000..a44a74b --- /dev/null +++ b/source/shared/ShowMeTheCode.md @@ -0,0 +1,178 @@ +# Show me the code! + +There is an old principle: show don't tell, and this introduction is about showing you what you can do with Brighter and Darker. It's not about how - more detailed documentation elsewhere shows you how to write this code. It's not about why - articles elsewhere discuss some of the reasons behind this approach. It is just, let me see how Brighter works. + +## Brighter and Darker + +### Brighter is about Requests + +A *Request* is a message sent over a bus. A request may update state. + +A *Command* is an instruction to execute some behavior. An *Event* is a notification. + +You use the *Command Processor* to separate the sender from the receiver, and to provide middleware functionality like a retry. + +### Darker is about Queries + +A *Query* is a message executed via a bus that returns a *Result*. A query does not update state. + +You use the *Query Processor* to separate the requester from the replier, and to provide middleware functionality like a retry. + +### Middleware + +Both Brighter and Darker allow you to provide middleware that runs between a request or query being made and being handled. The middleware used by a handler is configured by attributes. + +### Sending and Querying Example + +In this example, we show sending a command, and querying for the results of issuing it, from within an ASP.NET WebAPI controller method. + +``` csharp +[Route("{name}/new")] +[HttpPost] +public async Task> Post(string name, NewGreeting newGreeting) +{ + await _commandProcessor.SendAsync(new AddGreeting(name, newGreeting.Greeting)); + + var personsGreetings = await _queryProcessor.ExecuteAsync(new FindGreetingsForPerson(name)); + + if (personsGreetings == null) return new NotFoundResult(); + + return Ok(personsGreetings); +} +``` + +### Handling Examples + +Handler code listens for and responds to requests or queries. The handler for the above request and query are: + +``` csharp +[RequestLogging(0, HandlerTiming.Before)] +[UsePolicyAsync(step:1, policy: Policies.Retry.EXPONENTIAL_RETRYPOLICYASYNC)] +public override async Task HandleAsync(AddPerson addPerson, CancellationToken cancellationToken = default(CancellationToken)) +{ + await _uow.Database.InsertAsync(new Person(addPerson.Name)); + + return await base.HandleAsync(addPerson, cancellationToken); +} +``` + +``` csharp +[QueryLogging(0)] +[RetryableQuery(1, Retry.EXPONENTIAL_RETRYPOLICYASYNC)] +public override async Task ExecuteAsync(FindGreetingsForPerson query, CancellationToken cancellationToken = new CancellationToken()) +{ + var sql = @"select p.Id, p.Name, g.Id, g.Message + from Person p + inner join Greeting g on g.Recipient_Id = p.Id"; + + var people = await _uow.Database.QueryAsync(sql, (person, greeting) => + { + person.Greetings.Add(greeting); + return person; + }, splitOn: "Id"); + + var peopleGreetings = people.GroupBy(p => p.Id).Select(grp => + { + var groupedPerson = grp.First(); + groupedPerson.Greetings = grp.Select(p => p.Greetings.Single()).ToList(); + return groupedPerson; + }); + + var person = peopleGreetings.Single(); + + return new FindPersonsGreetings + { + Name = person.Name, + Greetings = person.Greetings.Select(g => new Salutation(g.Greet())) + }; + +} +``` + +## Using an External Bus + +As well as using an Internal Bus, in Brighter you can use an External Bus - middleware such as RabbitMQ or Kafka - to send a request between processes. Brighter supports both sending a request, and provides a *Dispatcher* than can listen for requests on middleware and forward it to a handler. + +The following code sends a request to another process. + +``` csharp +[RequestLogging(0, HandlerTiming.Before)] +[UsePolicyAsync(step:1, policy: Policies.Retry.EXPONENTIAL_RETRYPOLICYASYNC)] +public override async Task HandleAsync(AddGreeting addGreeting, CancellationToken cancellationToken = default(CancellationToken)) +{ + var posts = new List(); + + //We use the unit of work to grab connection and transaction, because Outbox needs + //to share them 'behind the scenes' + + var tx = await _uow.BeginOrGetTransactionAsync(cancellationToken); + try + { + var searchbyName = Predicates.Field(p => p.Name, Operator.Eq, addGreeting.Name); + var people = await _uow.Database.GetListAsync(searchbyName, transaction: tx); + var person = people.Single(); + + var greeting = new Greeting(addGreeting.Greeting, person); + + //write the added child entity to the Db + await _uow.Database.InsertAsync(greeting, tx); + + //Now write the message we want to send to the Db in the same transaction. + posts.Add(await _postBox.DepositPostAsync(new GreetingMade(greeting.Greet()), cancellationToken: cancellationToken)); + + //commit both new greeting and outgoing message + await tx.CommitAsync(cancellationToken); + } + catch (Exception e) + { + _logger.LogError(e, "Exception thrown handling Add Greeting request"); + //it went wrong, rollback the entity change and the downstream message + await tx.RollbackAsync(cancellationToken); + return await base.HandleAsync(addGreeting, cancellationToken); + } + + //Send this message via a transport. We need the ids to send just the messages here, not all outstanding ones. + //Alternatively, you can let the Sweeper do this, but at the cost of increased latency + await _postBox.ClearOutboxAsync(posts, cancellationToken:cancellationToken); + + return await base.HandleAsync(addGreeting, cancellationToken); +} +``` + +The following code receives a message, sent from another process, via a dispatcher. It uses an Inbox to ensure that it does not process duplicate messages + +``` csharp +[UseInboxAsync(step:0, contextKey: typeof(GreetingMadeHandlerAsync), onceOnly: true )] +[RequestLoggingAsync(step: 1, timing: HandlerTiming.Before)] +[UsePolicyAsync(step:2, policy: Policies.Retry.EXPONENTIAL_RETRYPOLICYASYNC)] +public override async Task HandleAsync(GreetingMade @event, CancellationToken cancellationToken = default(CancellationToken)) +{ + var posts = new List(); + + var tx = await _uow.BeginOrGetTransactionAsync(cancellationToken); + try + { + var salutation = new Salutation(@event.Greeting); + + await _uow.Database.InsertAsync(salutation, tx); + + posts.Add(await _postBox.DepositPostAsync(new SalutationReceived(DateTimeOffset.Now), cancellationToken: cancellationToken)); + + await tx.CommitAsync(cancellationToken); + } + catch (Exception e) + { + _logger.LogError(e, "Could not save salutation"); + + //if it went wrong rollback entity write and Outbox write + await tx.RollbackAsync(cancellationToken); + + return await base.HandleAsync(@event, cancellationToken); + } + + await _postBox.ClearOutboxAsync(posts, cancellationToken: cancellationToken); + + return await base.HandleAsync(@event, cancellationToken); +} +``` + diff --git a/source/shared/SqliteInbox.md b/source/shared/SqliteInbox.md new file mode 100644 index 0000000..7b3772d --- /dev/null +++ b/source/shared/SqliteInbox.md @@ -0,0 +1,37 @@ +# Sqlite Inbox + +## Usage +The Sqlite Inbox allows use of Sqlite for [Brighter's inbox support](/contents/BrighterInboxSupport.md). The configuration is described in [Basic Configuration](/contents/BrighterBasicConfiguration.md#inbox). + +For this we will need the *Inbox* packages for the Sqlite *Inbox*. + +* **Paramore.Brighter.Inbox.Sqlite** + +``` csharp +private static IHostBuilder CreateHostBuilder(string[] args) => + Host.CreateDefaultBuilder(args) + .ConfigureServices(hostContext, services) => + { + ConfigureBrighter(hostContext, services); + } + +private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services) +{ + services.AddServiceActivator(options => + { ... }) + .UseExternalInbox( + new SqliteInbox(new SqliteInboxConfiguration("DataSource=test.db", "Inbox"); + new InboxConfiguration( + scope: InboxScope.Commands, + onceOnly: true, + actionOnExists: OnceOnlyAction.Throw + ) + ); +} + +... + +``` + + + diff --git a/source/shared/TaskQueuePattern.md b/source/shared/TaskQueuePattern.md new file mode 100644 index 0000000..583ca5d --- /dev/null +++ b/source/shared/TaskQueuePattern.md @@ -0,0 +1,21 @@ +# The Task Queue Pattern + +The Task Queue Pattern let's you use an External Bus to handle work asynchronously. It is a common use of an External Bus outside of using it for an [Event Driven Architecture](/contents/EventDrivenCollaboration.md). + +## Doing Work Asynchronously +You might have an HTTP API with a rule that any given request to that API must execute in under 100ms. On measuring the performance of a key POST or PUT operation to your API you find that you exceed this +value. Upon realizing that much of your time is spent I/O you consider two options: + +- Use the TPL (Task parallel library) to perform the work concurrently - Offload the work to a distributed task queue, ack the message, and allow the work to complete asynchronously + +Either way you probably return a 202 Accepted to the caller, with a Link header that points to an endpoint where the caller can poll for completion and/or monitor progress. This might be a resource you are +creating that will return a 404 until it exists, or a progress indicator that indicates how far through the work you are and redirects to the resource once it is complete. (You can store progress in a backing store, perhaps using a distributed cache such as Redis). + +There is a problem with the TPL approach is that your operation can only meet the 100ms threshold if your work can be parallelised such that no sub-task takes longer than 100ms. Your speed is always constrained by the slowest operation that you need to parallelize. If you are I/O bound on a resource experiencing contention beyond 100ms, you will not meet your goal by introducing more threads. Your minimum time is your minimum time. + +You might try to fix this by acking (acknowledging) the request, and completing the work asynchronously. This option is particularly attractive if the work is I/O bound as you can process other requests whilst you wait for the I/O to complete. + +The downside of the async approach is that you risk that the work will be lost if the server fails prior to completion of the work, or the app simply recycles. + +These requirements tend to push you in the direction of [Guaranteed Delivery](http://www.eaipatterns.com/GuaranteedMessaging.html) to ensure that work you ack will eventually be handled. + diff --git a/source/shared/Telemetry.md b/source/shared/Telemetry.md new file mode 100644 index 0000000..243a538 --- /dev/null +++ b/source/shared/Telemetry.md @@ -0,0 +1,53 @@ +# Telemetry + +Starting in version 9.2.1 Brighter now supports Open Telemetry Tracing + +## Configuring Open Telemetry + +The OpenTelemetry SDK can be configured to listen to Activities inside of Brighter for more information [OpenTelemetry Tracing](https://opentelemetry.io/docs/instrumentation/net/getting-started/) + + +The below code will +* Enable OpenTelemetry tracing +* Set the service name to "ProducerService" +* Set OpenTelemetry traving to listen to all Brighter and Microsoft sources +* Export the telemetry tracts to Jaeger + +```csharp +//The name of the service +const myServiceName = "ProducerService" + +var jaegerEndpoint = new Uri("http://localhost:9411/api/v2/spans"); + +using var tracerProvider = + Sdk.CreateTracerProviderBuilder() + .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(myServiceName)) + .AddSource("Paramore.*", "Microsoft.*") + .AddJaegerExporter(o => + { + o.Endpoint = jaegerEndpoint; + }) + .Build(); +``` + +## Activity Sources + +The activity sources that Brighter emits from are: + * Paramore.Brighter - Traces started in the **Command Processor** will be given this (Including **Outbox Sweeper**) + * Paramore.Brighter.ServiceActivator - Traces Started in the **Service Activator** + +Please note that Brighter will honor existing spans, i.e. When using ASPNet a Request will start a trace, it is for this reason that the sample above also includes "Microsft.*" as Bighter will participate in an active trace. + +![Distributed Trace](_static/images/DistributedTracingFromASP.png) +This distributed traces shows a message that is produced from an ASP.Net request and the consumed by Service Activator + +## Reported Events + +At this time Brighter records the following events + +| Event | Description | +| ------------------------------------------ | ----------- | +| Add message to outbox | When a message is added to the outbox | +| Get outstanding messages from the outbox | During an implicit clear when retrieving undispatched messages from the outbox | +| Dispatching message | When a message is being dispatched to a message transport | +| Bulk dispatching messages | When a batch of messages are being dispathced to a message transport | diff --git a/source/shared/UsingTheContextBag.md b/source/shared/UsingTheContextBag.md new file mode 100644 index 0000000..6fc909a --- /dev/null +++ b/source/shared/UsingTheContextBag.md @@ -0,0 +1,33 @@ +# Passing information between Handlers in the Pipeline + +A key constraint of the Pipes and Filters architectural style is that Filters do not share state. One reason is that this limits your ability to recompose the pipeline as steps must follow other steps. + +However, when dealing with Handlers that implement orthogonal concerns it can be useful to pass context along the chain. Given that many orthogonal concerns have constraints about ordering anyway, we can live +with the ordering constraints imposed by passing context. So how do you approach passing context from one Handler to another when it is necessary? + +The first thing is to avoid adding extra properties to the Command to support handling state for these orthogonal Filter steps in your pipeline. This couples your **Command** to orthogonal concerns and you +really only want to bind it to your **Target Handler**. + +Instead we provide a **Context Bag** as part of the Command Dispatcher which is injected into each Handler in the Pipeline. The lifetime of this **Context Bag** is the lifetime of the Request (although you will +need to take responsibility for freeing any unmanaged resources you place into the **Context Bag** for example when code called after the Handler that inserts the resource into the Bag returns to the Handler). + +``` csharp +public class MyContextAwareCommandHandler : RequestHandler +{ + public static string TestString { get; set; } + + public override MyCommand Handle(MyCommand command) + { + LogContext(); + return base.Handle(command); + } + + private void LogContext() + { + TestString = (string)Context.Bag["TestString"]; + Context.Bag["MyContextAwareCommandHandler"] = "I was called and set the context"; + } +} +``` + +Internally we use the **Context Bag** in a number of the Quality of Service supporting Attributes we provide. See [Fallback](PolicyFallback.html) for example. diff --git a/contents/_static/.DS_Store b/source/shared/_static/.DS_Store similarity index 100% rename from contents/_static/.DS_Store rename to source/shared/_static/.DS_Store diff --git a/contents/_static/images/.DS_Store b/source/shared/_static/images/.DS_Store similarity index 100% rename from contents/_static/images/.DS_Store rename to source/shared/_static/images/.DS_Store diff --git a/contents/_static/images/BehavioralCoupling.png b/source/shared/_static/images/BehavioralCoupling.png similarity index 100% rename from contents/_static/images/BehavioralCoupling.png rename to source/shared/_static/images/BehavioralCoupling.png diff --git a/contents/_static/images/Brighter_SendAsync_Pipeline.png b/source/shared/_static/images/Brighter_SendAsync_Pipeline.png similarity index 100% rename from contents/_static/images/Brighter_SendAsync_Pipeline.png rename to source/shared/_static/images/Brighter_SendAsync_Pipeline.png diff --git a/contents/_static/images/Choreography.png b/source/shared/_static/images/Choreography.png similarity index 100% rename from contents/_static/images/Choreography.png rename to source/shared/_static/images/Choreography.png diff --git a/contents/_static/images/Command.png b/source/shared/_static/images/Command.png similarity index 100% rename from contents/_static/images/Command.png rename to source/shared/_static/images/Command.png diff --git a/contents/_static/images/CommandDispatcher.png b/source/shared/_static/images/CommandDispatcher.png similarity index 100% rename from contents/_static/images/CommandDispatcher.png rename to source/shared/_static/images/CommandDispatcher.png diff --git a/contents/_static/images/CommandExtendedWorkflow.png b/source/shared/_static/images/CommandExtendedWorkflow.png similarity index 100% rename from contents/_static/images/CommandExtendedWorkflow.png rename to source/shared/_static/images/CommandExtendedWorkflow.png diff --git a/contents/_static/images/CommandProcesorCapitalize.png b/source/shared/_static/images/CommandProcesorCapitalize.png similarity index 100% rename from contents/_static/images/CommandProcesorCapitalize.png rename to source/shared/_static/images/CommandProcesorCapitalize.png diff --git a/contents/_static/images/CommandProcessor.png b/source/shared/_static/images/CommandProcessor.png similarity index 100% rename from contents/_static/images/CommandProcessor.png rename to source/shared/_static/images/CommandProcessor.png diff --git a/contents/_static/images/CommandWorkflow.png b/source/shared/_static/images/CommandWorkflow.png similarity index 100% rename from contents/_static/images/CommandWorkflow.png rename to source/shared/_static/images/CommandWorkflow.png diff --git a/contents/_static/images/CorrectnessProblem.png b/source/shared/_static/images/CorrectnessProblem.png similarity index 100% rename from contents/_static/images/CorrectnessProblem.png rename to source/shared/_static/images/CorrectnessProblem.png diff --git a/contents/_static/images/DistributedTracingFromASP.png b/source/shared/_static/images/DistributedTracingFromASP.png similarity index 100% rename from contents/_static/images/DistributedTracingFromASP.png rename to source/shared/_static/images/DistributedTracingFromASP.png diff --git a/contents/_static/images/EventCarriedStateTransfer.png b/source/shared/_static/images/EventCarriedStateTransfer.png similarity index 100% rename from contents/_static/images/EventCarriedStateTransfer.png rename to source/shared/_static/images/EventCarriedStateTransfer.png diff --git a/contents/_static/images/EventDrivenArchitecture1.png b/source/shared/_static/images/EventDrivenArchitecture1.png similarity index 100% rename from contents/_static/images/EventDrivenArchitecture1.png rename to source/shared/_static/images/EventDrivenArchitecture1.png diff --git a/contents/_static/images/EventDrivenArchitecture2.png b/source/shared/_static/images/EventDrivenArchitecture2.png similarity index 100% rename from contents/_static/images/EventDrivenArchitecture2.png rename to source/shared/_static/images/EventDrivenArchitecture2.png diff --git a/contents/_static/images/EventStateCapture.png b/source/shared/_static/images/EventStateCapture.png similarity index 100% rename from contents/_static/images/EventStateCapture.png rename to source/shared/_static/images/EventStateCapture.png diff --git a/contents/_static/images/Events And Joins.png b/source/shared/_static/images/Events And Joins.png similarity index 100% rename from contents/_static/images/Events And Joins.png rename to source/shared/_static/images/Events And Joins.png diff --git a/contents/_static/images/HotelMicroservices.png b/source/shared/_static/images/HotelMicroservices.png similarity index 100% rename from contents/_static/images/HotelMicroservices.png rename to source/shared/_static/images/HotelMicroservices.png diff --git a/contents/_static/images/LogTailing.png b/source/shared/_static/images/LogTailing.png similarity index 100% rename from contents/_static/images/LogTailing.png rename to source/shared/_static/images/LogTailing.png diff --git a/contents/_static/images/Microservices.png b/source/shared/_static/images/Microservices.png similarity index 100% rename from contents/_static/images/Microservices.png rename to source/shared/_static/images/Microservices.png diff --git a/contents/_static/images/Orchestration.png b/source/shared/_static/images/Orchestration.png similarity index 100% rename from contents/_static/images/Orchestration.png rename to source/shared/_static/images/Orchestration.png diff --git a/contents/_static/images/OutboxPattern.png b/source/shared/_static/images/OutboxPattern.png similarity index 100% rename from contents/_static/images/OutboxPattern.png rename to source/shared/_static/images/OutboxPattern.png diff --git a/contents/_static/images/PipesAndFilters.png b/source/shared/_static/images/PipesAndFilters.png similarity index 100% rename from contents/_static/images/PipesAndFilters.png rename to source/shared/_static/images/PipesAndFilters.png diff --git a/contents/_static/images/ReferenceData.png b/source/shared/_static/images/ReferenceData.png similarity index 100% rename from contents/_static/images/ReferenceData.png rename to source/shared/_static/images/ReferenceData.png diff --git a/contents/_static/images/RequestDrivenArchitecture.png b/source/shared/_static/images/RequestDrivenArchitecture.png similarity index 100% rename from contents/_static/images/RequestDrivenArchitecture.png rename to source/shared/_static/images/RequestDrivenArchitecture.png diff --git a/contents/_static/images/RussianDoll.png b/source/shared/_static/images/RussianDoll.png similarity index 100% rename from contents/_static/images/RussianDoll.png rename to source/shared/_static/images/RussianDoll.png diff --git a/contents/_static/images/TaskQueues.png b/source/shared/_static/images/TaskQueues.png similarity index 100% rename from contents/_static/images/TaskQueues.png rename to source/shared/_static/images/TaskQueues.png