Merge pull request #44 from BrighterCommand/cobra_enabled

Support documentation for multiple versions at the same time

contents/9/AWSSQSConfiguration.md

@@ -0,0 +1,168 @@
+# AWS SQS Configuration
+
+## General
+
+SNS and SQS are proprietary message-oriented-middleware available on the AWS platform. Both are well documented: see [SNS](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) and [SQS](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html). Brighter handles the details of sending to SNS using an SQS queue for the consumer. You might find the [documentation for the AWS .NET SDK](https://docs.aws.amazon.com/sdk-for-net/) helpful when debugging, but you should not have to interact with it directly to use Brighter.
+
+It is useful to understand the relationship between these two components:
+
+- **SNS**: A routing table, [SNS](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) provides routing for messages to subscribers. Subscribers include, but are not limited to, SQS [see SNS Subscribe Protocol](https://docs.aws.amazon.com/sns/latest/api/API_Subscribe.html). An entry in the table is a **Topic**.
+- **SQS**: A store-and-forward queue over which a consumer receives messages. A message is locked whilst a consumer has read it, until they ack it, upon which it is deleted from the queue, or nack it, upon which it is unlocked. A policy controls movement of messages that cannot be delivered to a DLQ. [SQS](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html) may be used for point-to-point integration, and does not require SNS.
+
+Brighter only supports the scenario where SNS is used as a routing table, and an SQS subscribes to a **topic**. It does not support stand-alone SQS queues. Point-to-point scenarios can be modelled as an SNS **topic** with one subscribing queue.
+
+## Connection
+
+The Connection to AWS is provided by an **AWSMessagingGatewayConnection**. This is a wrapper around AWS credentials and region, that allows us to create the .NET clients that abstract various AWS HTTP APIs. We require the following parameters:
+
+- **Credentials**: An instance of *AWSCredentials*. Storing and retrieving the credentials is a detail for your application and may vary by environment. There is AWS discussion of credentials resolution [here](https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/creds-assign.html)
+- **Region**: The *RegionEndpoint* to use. SNS is a regional service, so we need to know which region to provision infrastructure in, or find it from.
+
+``` csharp
+public void ConfigureServices(IServiceCollection services)
+{
+    if (!new CredentialProfileStoreChain().TryGetAWSCredentials("default", out var credentials)
+	    throw InvalidOperationException("Missing AWS Credentials);
+
+    services.AddBrighter(...)
+        .UseExternalBus(new SnsProducerRegistryFactory
+                    new AwsMessagingGatewayConnection(credentials, Environment.GetEnvironmentVariable("AWS_REGION"))
+                    ,
+            ... //publication, see below
+        ).Create()
+}
+```
+## Publication
+
+For more on a *Publication* see the material on an *External Bus* in [Basic Configuration](/contents/BrighterBasicConfiguration.md#using-an-external-bus).
+
+Brighter's **Routing Key** represents the [SNS Topic Name](https://docs.aws.amazon.com/sns/latest/api/API_CreateTopic.html).
+
+### Finding and Creating Topics
+Depending on the option you choose for how we handle required messaging infrastructure (Create, Validate, Assume), we will need to determine if a **Topic** already exists, when we want to create it if missing, or validate it. 
+
+Naively using the AWS SDK's **FindTopic** method is an expensive operation. This enumerates all the **Topics** in that region, looking for those that have a matching name. Under-the-hood the client SDK pages through your topics. If you have a significant number of topics, this is expensive and subject to rate limiting. 
+
+As creating a **Topic** is an *idempotent* operation in SNS, if asked to Create we do so without first searching to see if it already exists because of the cost of validation.
+
+If you create your infrastructure out-of-band, and ask us validate it exists, to mitigate the cost of searching for topics, we provide several options under **FindTopicBy**.
+
+- **FindTopicBy**: How do we find the topic: 
+    - **TopicFindBy.Arn** -> On a *Publication*, the routing key is the **Topic** name, but you explicitly supply the ARN in another field: **TopicArn**. On a *Subscription* the routing key is the **Topic** ARN.
+    - **TopicFindBy.Convention** -> The routing key is the **Topic** name, and we use convention to construct the ARN from it
+    - **TopicFindBy.Name** -> The routing key is the **Topic** name & we use ListTopics to find it (rate limited 30/s)
+
+#### TopicFindBy.Arn 
+We use **GetTopicAttributesAsync** SDK method to request attributes of a Topic with the ARN supplied in **TopicArn**. If this call fails with a NotFoundException, we know that the Topic does not exist. This is a *hack*, but is much more efficient than enumeration as a way of determining if the ARN exists.
+
+#### TopicFindBy.Convention 
+If you supply only the **Topic** name via the routing key, we construct the ARN by convention as follows:
+
+``` csharp
+var arn = new Arn
+    {
+        Partition = //derived from the partition of the region you supplied to us,
+        Service = "sns",
+        Region = //derived from the system name of the region you supplied to us,
+        AccountId = //your account id - derived from the credentials you supplied,
+        Resource = topicName
+    }
+```
+
+These assumptions work, if the topic is created by the account your credentials belong to. If not, you can't use by convention.
+
+Once we obtain an ARN by convention, we can then use the optimized approach described under [TopicFindBy.Arn](#topicfindbyarn) to confirm that your topic exists.
+
+#### TopicFindBy.Name
+If you supply a name, but we can't construct the ARN via the above conventions, we have to fall back to the **SDKs** **FindTopic** approach. 
+
+Because creation is idempotent, and **FindTopic** is expensive, you are almost always better off choosing to create over validating a topic by name. 
+
+If you are creating the topics out-of-band, by CloudFormation for example, and so do not want Brighter the risk that Brighter will create them, then you will have an ARN. In that case you should use [TopicFindBy.Arn](#topicfindbyarn) or assume that any required infrastructure exists. 
+
+### Other Attributes
+- **SNSAttributes**: This property lets you pass through an instance of **SNSAttributes** which has properties representing the attributes used when creating a **Topic**. These are only used if you are creating a **Topic**.
+    - **DeliveryPolicy**: The policy that defines how Amazon SNS retries failed deliveries to HTTP/S endpoints.
+    - **Policy**: The policy that defines who can access your topic. By default, only the topic owner can publish or subscribe to the topic.
+    - **Tags**: A list of resource tags to use.
+
+``` csharp
+public void ConfigureServices(IServiceCollection services)
+{
+    if (!new CredentialProfileStoreChain().TryGetAWSCredentials("default", out var credentials)
+	    throw InvalidOperationException("Missing AWS Credentials);
+
+    services.AddBrighter(...)
+        .UseExternalBus(
+            ...,//connection, see above
+            new SnsPublication[]
+            {
+                new SnsPublication()
+                {
+                    Topic = new RoutingKey("GreetingEvent"),
+                    FindTopicBy = TopicFindBy.Convention
+                }
+            }
+        ).Create()
+}
+```
+
+## Subscription
+
+As normal with Brighter, we allow **Topic** creation from the *Subscription*. Because this works in the same way as the *Publication* see the notes under [Publication](#publication) for further detail on the options that you can configure around creation or validation.
+
+In SNS you need to [subscribe](https://docs.aws.amazon.com/sns/latest/api/API_Subscribe.html) to a **Topic** to receive messages from that **Topic**. Brighter subscribes using an SQS queue (there are other options for SNS, but Brighter does not use those). Much of the *Subscription* configuration allows you to control the parameters of that *Subscription*.
+
+We support the following properties on an *SQS Subscription* most of which relate to the creation of the [SQS Queue](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_CreateQueue.html) with which we subscribe:
+
+- **LockTimeout**: How long, in seconds, a 'lock' is held on a message for one consumer before it times out(*VisibilityTimeout*). Default is 10s.
+- **DelaySeconds**:  The length of time, in seconds, for which the delivery of all messages in the queue is delayed. Default is 0.
+- **MessageRetentionPeriod**: The length of time, in seconds, for which Amazon SQS retains a message on a queue before deleting it. Default is 4 days.
+- **IAMPolicy**: The queue's policy. A valid [AWS policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html).
+- **RawMessageDelivery**: Indicate that the Raw Message Delivery setting is enabled or disabled. Defaults to true.
+- **RedrivePolicy**: The parameters for the dead-letter queue functionality of the source queue. An instance of the **RedrivePolicy** class, which has the following parameters:
+    - **MaxReceiveCount**: The maximum number of requeues for a message before we push it to the DLQ instead
+    - **DeadlLetterQueueName**: The name of the dead letter queue we want to associate with any redrive policy.
+
+
+``` csharp
+private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services)
+{
+    (AWSCredentials credentials, RegionEndpoint region) = CredentialsChain.GetAwsCredentials();
+    var awsConnection = new AWSMessagingGatewayConnection(credentials, region);
+
+     var subscriptions = new Subscription[]
+    {
+        new SqsSubscription<GreetingEvent>(
+            name: new SubscriptionName("Subscription-Name),
+            channelName: new ChannelName("Channel-Name"),
+            routingKey: new RoutingKey("arn:aws:sns:us-east-2:444455556666:MyTopic"),
+            findTopicBy: TopicFindBy.Arn,
+            makeChannels: OnMissingChannel.Validate
+        );
+    }
+
+     var sqsMessageConsumerFactory = new SqsMessageConsumerFactory(awsConnection);
+
+    services.AddServiceActivator(options =>
+        {
+            options.Subscriptions = subscriptions;
+            options.ChannelFactory = new ChannelFactory(sqsMessageConsumerFactory);
+            ... //see Basic Configuration
+        })
+```
+
+### Ack and Nack
+
+As elsewhere, Brighter only Acks after your handler has run to process the message. We will Ack unless you throw a **DeferMessageAction**. See [Handler Failure](/contents/HandlerFailure.md) for more.
+
+An Ack will delete the message from the SQS queue using the SDK's **DeleteMessageAsync**.
+
+In response to a DeferMessageAction we will requeue, using the SDK's **ChangeMessageVisibilityAsync** to make the message available again to other consumers.
+
+On a Nack, we will move the message to a DLQ, if there is one. We Nack when we exceed the requeue count for a message, or we raise a ConfigurationException.
+
+
+
+
+

contents/9/AsyncDispatchARequest.md

@@ -0,0 +1,98 @@
+# Dispatching Requests Asynchronously
+
+Once you have [implemented your Request Handler](ImplementingAHandler.html), you will want to dispatch **Commands** or **Events** to that Handler.
+
+## Usage
+
+In the following example code we register a handler, create a command processor, and then use that command processor to send a command to the handler asynchronously.
+
+
+``` csharp
+    public class Program
+    {
+        private static async Task Main(string[] args)
+        {
+            var host = Host.CreateDefaultBuilder()
+                .ConfigureServices((hostContext, services) =>
+
+                    {
+                        services.AddBrighter()
+                            .AutoFromAssemblies();
+                    }
+                )
+                .UseConsoleLifetime()
+                .Build();
+
+            var commandProcessor = host.Services.GetService<IAmACommandProcessor>();
+
+            await commandProcessor.SendAsync(new GreetingCommand("Ian"));
+
+            await host.RunAsync();
+        }
+```
+
+## Registering a Handler
+
+In order for a **Command Dispatcher** to find a Handler for your **Command** or **Event** you need to register the association between that **Command** or **Event** and your Handler.
+
+Brighter's **HostBuilder** support provides **AutoFromAssemblies** to register any *Request Handlers* in the project. See [Basic Configuration](/contents/BrighterBasicConfiguration.md) for more.
+
+### Pipelines Must be Homogeneous
+
+Brighter only supports pipelines that are solely **IHandleRequestsAsync** or **IHandleRequests**. In particular, note that middleware (attributes on your handler) must be of the same type as the rest of your pipeline. A common mistake is to **UsePolicy** when you mean **UsePolicyAsync**.
+
+## Dispatching Requests
+
+Once you have registered your Handlers, you can dispatch requests to them. To do that you simply use the **commandProcessor.SendAsync()** (or **commandProcessor.PublishAsync()**) method passing in an instance of your command. *Send* expects one handler, *Publish* expects zero or more. (You can use **commandProcessor.DepositPostAsync** and **commandProcessor.ClearOutboxAsync** with an External Bus).
+
+``` csharp
+await commandProcessor.SendAsync(new GreetingCommand("Ian"));
+```
+
+### Returning results to the caller.
+
+A Command does not have return value and **CommandDispatcher.Send()** does not return anything. Please see a discussion on how to handle this in [Returning Results from a Handler](/contents/ReturningResultsFromAHandler.md).
+
+### Cancellation
+
+Brighter supports the cancellation of asynchronous operations.
+
+The asynchronous methods: **SendAsync** and **PublishAsync** accept a **CancellationToken** and pass this token down the pipeline. The parameter defaults to default(CancellationToken) where the call does not intend to cancel.
+
+The responsibility for checking for a cancellation request lies with the individual handlers, which must determine what action to take if cancellation had been signalled.
+
+### Async Callback Context
+
+When an awaited method completes, what thread runs any completion code? There are two options:
+
+- The original thread that was running when the await began
+- A new thread allocated from the thread pool
+
+Why does this matter? Because if you needed to access anything that is thread local, being called back on a new thread means you will not have access to those variables.
+
+As a result, when awaiting it is possible to configure how the continuation runs. 
+
+- To run on the original thread, requires the CLR to capture information on the thread you were using. This is the SynchronizationContext; because the CLR must record this information, we refer to it as a captured context. Your execution will be queued back on to the original context, which has a performance cost.
+- To run on a new thread, using the Task Scheduler to allocate from the thread pool.
+
+You can use ConfigureAwait to control this. This article explains why you might wish to use [ConfigureAwait](https://devblogs.microsoft.com/dotnet/configureawait-faq/), in more depth.
+
+As a library, we need to allow you to make this choice for your handler chain. For this reason, our *Async methods support the parameter **continueOnCapturedContext**. 
+
+Library writers are encouraged to default to false i.e. use the Task Scheduler instead of the SychronizationContext. Brighter adopts this default, but it might not be what you want if your handler needs to run in the context of the original thread. As a result we let you use this parameter on the **\*Async** calls to change the behaviour throughout your pipeline.
+
+``` csharp
+await commandProcessor.SendAsync(new GreetingCommand("Ian"), continueOnCapturedContext: true);
+```
+
+A handler exposes the parameter you supply via  the property **ContinueOnCapturedContext**. 
+
+You should pass this value via **ConfigureAwait** if you need to be able to support making this choice at the call site. For example, when you call the base handler in your return statement, to ensure that the decision as to whether to use the scheduler or the context flows down the pipeline.
+
+``` csharp
+return await base.HandleAsync(command, ct).ConfigureAwait(ContinueOnCapturedContext);
+```
+
+You can ignore this, if you want to default to using the Task Scheduler.
+
+

contents/9/AzureBlobArchiveProvider.md

@@ -0,0 +1,40 @@
+# Azure Blob Archive Provider
+
+## Usage
+The Azure Blob Archive Provider is a provider for [Outbox Archiver](/contents/BrighterOutboxSupport.md#outbox-archiver).
+
+For this we will need the *Archive* packages for the Azure *Archive Provider*.
+
+* **Paramore.Brighter.Archive.Azure**
+
+``` csharp
+private static IHostBuilder CreateHostBuilder(string[] args) =>
+    Host.CreateDefaultBuilder(args)
+        .ConfigureServices(hostContext, services) =>
+        {
+            ConfigureBrighter(hostContext, services);
+        }
+
+private static void ConfigureBrighter(HostBuilderContext hostContext, IServiceCollection services)
+{
+    services.AddBrighter(options =>
+        { ...  })
+        .UseOutboxArchiver(
+            new AzureBlobArchiveProvider(new AzureBlobArchiveProviderOptions()
+                {
+                    BlobContainerUri = "https://brighterarchivertest.blob.core.windows.net/messagearchive",
+                    TokenCredential = New AzCliCredential();
+                }
+            ),
+            options => {
+                TimerInterval = 5; // Every 5 seconds
+                BatchSize  = 500; // 500 messages at a time
+                MinimumAge = 744; // 1 month
+            }
+        );
+}
+
+...
+
+```
+

contents/9/AzureBlobConfiguration.md

@@ -0,0 +1,24 @@
+# Azure Archive Provider Configuration
+
+## General
+Azure Service Bus (ASB) is a fully managed enterprise message broker and is [well documented](https://docs.microsoft.com/en-us/azure/service-bus-messaging/) Brighter handles the details of sending to or receiving from ASB.  You may find it useful to understand the [concepts](https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-queues-topics-subscriptions) of the ASB.
+
+## Connection
+At this time Azure Blob Archive Provider only supports Token Credential for authentication
+
+## Permissions
+For the archiver to work the calling credential will require the role **Storage Blob Data Owner** however if **TagBlobs** is set to False then **Storage Blob Data Contributor** will be adequate.  If you feel that Data Owner is too high you can create a custom role encompasing Contributor and 'Microsoft.Storage/storageAccounts/blobServices/containers/blobs/tags/write'
+
+## Options
+
+* **BlobContainerUri** : The URI of the Blob container to store messages in (i.e. "https://BlobTest.blob.core.windows.net/messagearchive)
+* **TokenCredential** : The Credential to use when writing the Blob
+* **AccessTier** : The Access Tier to write to the blob
+* **TagBlobs** : if this is set to True the defined in **TagsFunc** will be written to the blobs
+* **TagsFunc**  : The function to arrange the tags to add when storing, please note that **TagBlobs** must be True for these to be used, default Tags :
+    - topic
+    - correlationId
+    - message_type
+    - timestamp
+    - content_type
+* **StorageLocationFunc** : The function to provide the location to store the message inside of the Blob container, default location : The Id of the message at the root of the **BlobContainerUri**