[Docs] Akka.Delivery Documentation (#6757)

* scaffolding Akka.Delivery documentation

* added images

* starting on point-to-point delivery documentation

* fleshing out article

* fix spelling errors

* added updated code references and samples

* working on code sample

* don't make `Docs.Tests` a friend assembly

* Revert "don't make `Docs.Tests` a friend assembly"

This reverts commit 0e5c506.

* added API approvals

* updating docs

* added doc spec

* spelling out `Producer` behavior

* stubbing out rest of article

* completed docs on message chunking

* added durable queue spec

* completed all notes about the durable producer queue

* fixed typos and markdown linting errors

* fixed DocFx issues

* fleshing out Akka.Cluster.Sharding.Delivery documentation

* completed first draft of Akka.Cluster.Sharding.Delivery documentation

* fixed markdown errors

* fixed DocFx errors and warnings

* added reference to reliable delivery in sharding documentation

* added reference to Akka.Delivery documentation in older article about deliverability

* fixed typo

* more typos

* fix typo

docs/articles/actors/toc.yml

@@ -26,4 +26,6 @@
   href: testing-actor-systems.md
 - name: Coordinated Shutdown
   href: coordinated-shutdown.md
+- name: Reliable Message Delivery
+  href: reliable-delivery.md
 

docs/articles/clustering/cluster-overview.md

@@ -5,6 +5,9 @@ title: Akka.Cluster Overview
 
 # Akka.Cluster Overview
 
+> [!IMPORTANT]
+> See "[Reliable Akka.NET Message Delivery with Akka.Delivery](xref:reliable-delivery)" for tips on how to achieve reliable messaging over Akka.Remote and Akka.Cluster.
+
 ## What Is a "Cluster"?
 
 A cluster represents a fault-tolerant, elastic, decentralized peer-to-peer network of Akka.NET applications with no single point of failure or bottleneck. Akka.Cluster is the module that gives you the ability to create these applications.
@@ -83,7 +86,7 @@ The first step towards using Akka.Cluster is to install the [Akka.Cluster NuGet
 PM> Install-Package Akka.Cluster
 ```
 
-Once you've installed Akka.Cluster, we need to update our HOCON configuration to turn on the [`ClusterActorRefProvider`](http://api.getakka.net/docs/stable/html/CC0676F0.htm "Akka.NET API Docs - ClusterActorRefProvider class"), configure an Akka.Remote transport, and enable at least 1 seed node.
+Once you've installed Akka.Cluster, we need to update our HOCON configuration to turn on the `akka.actor.provider = cluster`, configure an Akka.Remote transport, and enable at least 1 seed node.
 
 > [!NOTE]
 > Akka.Cluster depends on Akka.Remote.

docs/articles/clustering/cluster-sharded-daemon-process.md

@@ -1,3 +1,8 @@
+---
+uid: sharded-daemon-process
+title: Akka.Cluster.Sharding Daemon Processes - Distributing Workers
+---
+
 # Sharded Daemon Process
 
 > [!WARNING]

docs/articles/clustering/cluster-sharding-delivery.md

@@ -0,0 +1,127 @@
+---
+uid: cluster-sharding-delivery
+title: Reliable Delivery over Akka.Cluster.Sharding
+---
+
+# Reliable Delivery over Akka.Cluster.Sharding
+
+> [!TIP]
+> Please see "[Reliable Message Delivery with Akka.Delivery](xref:reliable-delivery)" before reading this documentation. Akka.Cluster.Sharding.Delivery builds upon all of the concepts and tools implemented in the base Akka.Delivery APIs.
+
+If you're using [Akka.Cluster.Sharding](xref:cluster-sharding) to distribute state via one or more `ShardRegion`s across your Akka.Cluster, Akka.Cluster.Sharding.Delivery can help you guarantee delivery of messages from the rest of your `ActorSystem`s to each of your entity actors.
+
+## Point to Point Delivery
+
+Akka.Cluster.Sharding.Delivery only uses [point-to-point delivery mode from Akka.Delivery](xref:reliable-delivery) and **message chunking is not supported** in this mode.
+
+### Typed Messaging Protocol
+
+Akka.Cluster.ShardingDelivery uses a .NET generic-typed protocol and the `ShardingProducerController` and `ShardingConsumerController` are also both strongly typed. This means that end-users need to organize their messages into "protocol groups" in order to be effective, like so:
+
+[!code-csharp[Message Protocol](../../../src/core/Akka.Docs.Tests/Delivery/DeliveryDocSpecs.cs?name=MessageProtocol)]
+
+The common interface that all of the messages in this protocol implement is typically the type you'll want to use for your generic argument `T` in the Akka.Delivery or [Akka.Cluster.Sharding.Delivery](xref:cluster-sharding-delivery) method calls and types, as shown below:
+
+[!code-csharp[Starting Typed Actors](../../../src/core/Akka.Docs.Tests/Delivery/DeliveryDocSpecs.cs?name=ProducerRegistration)]
+
+### Built-in Actors and Messages
+
+![Overview of built-in Akka.Cluster.Sharding.Delivery actors](/images/cluster/delivery/1-sharding-delivery-registration.png)
+
+The Akka.Cluster.Sharding.Delivery relationship consists of the following actors:
+
+* **`Producer`** - this is a user-defined actor that is responsible for the production of messages. It receives [`ShardingProducerController.RequestNext<T>`](xref:Akka.Cluster.Sharding.Delivery.ShardingProducerController.RequestNext`1) messages from the `ShardingProducerController` when capacity is available to deliver additional messages.
+* **`ShardingProducerController`** - this actor is built into Akka.Cluster.ShardingDelivery and does most of the work. **You typically only need a single `ShardingProducerController` per-`ActorSystem` / per-`ShardRegion`** (or you can use [Sharded Daemon Processes](xref:sharded-daemon-process) to host a fixed number of producers per-cluster.) The `ShardingProducerController` is responsible for spawning a `ProducerController` per-entity and delivering those messages to the `ShardRegion` `IActorRef`.
+* **`ShardingConsumerController`** - this actor is built into Akka.Cluster.Sharding.Delivery and typically resides on the opposite site of the network from the `ShardingProducerController`. This actor wraps around your normal Akka.Cluster.Sharding entity actors and is created directly by the `ShardRegion` each time an entity is messaged. The `ShardingConsumerController` will spawn your entity actor directly and will additionally spawn a `ConsumerController` for each unique `ProducerId` detected in the message stream. Each of the `ConsumerController`s spawned by the `ShardingConsumerController` will deliver messages via their usual [`ConsumerController.Delivery<T>`](xref:Akka.Delivery.ConsumerController.Delivery`1) to the `Consumer`.
+* **`Consumer`** - this is your entity actor hosted via Akka.Cluster.Sharding. The `Consumer` processes messages of type `T` and must send `ConsumerController.Confirmation` messages back to the `ConsumerController` once it has successfully processed each `ConsumerController.Delivery<T>`. The `Consumer` actor is spawned by the `ShardingConsumerController`.
+
+#### Integration with ShardRegions
+
+In order to make use of Akka.Cluster.Sharding.Delivery, we have to change the way we spawn our `ShardRegion`'s entity actors:
+
+[!code-csharp[Launching ShardRegion with ShardingConsumerController configured](../../../src/examples/Cluster/ClusterSharding/ShoppingCart/Program.cs?name=LaunchShardRegion)]
+
+1. The `ShardingConsumerController` needs to be the actor initially created by the `ShardRegion` each time an entity is spawned;
+2. The `ShardingConsumerController.Create` method takes an argument of type `Func<IActorRef, Props>` - this allows you to pass in the `IActorRef` of the `ShardingConsumerController` itself down to your entity actor, the `Props` of which should be returned by this function.
+3. The `Consumer` actor must send a `ConsumerController.Start<T>` message, typically during `PreStart`, to the `ShardingConsumerController` in order to trigger message delivery.
+
+[!code-csharp[Consumer signalling to ShardingConsumerController that it's ready to receive messages](../../../src/examples/Cluster/ClusterSharding/ShoppingCart/Customers.cs?name=ShardingConsumerRegistration)]
+
+Do this for each entity type / `ShardRegion` you wish to guarantee delivery for via the `ShardingProducerController`.
+
+Next, we have to create our `Producer` and `ShardingProducerController` instances:
+
+[!code-csharp[Launching ShardRegion with ShardingConsumerController configured](../../../src/examples/Cluster/ClusterSharding/ShoppingCart/Program.cs?name=StartSendingMessage)]
+
+1. We have to launch our `ShardingProducerController` and our `Producer` actors - each `ShardingProducerController` must have a *globally unique* `ProducerId` value (similar to a `PersistentId`).
+2. `ShardingProducerController`s can be optionally made persistent via the same [`EventSourcedProducerQueue`](xref:Akka.Persistence.Delivery.EventSourcedProducerQueue) that can be used by an individual `ProducerController`.
+3. The `ShardingProducerController` must have a reference to the `IActorRef` of the `ShardRegion` to which it will be delivering messages.
+4. The `ShardingProducerController` must receive a [`ShardingProducerController.Start<T>`](xref:Akka.Cluster.Sharding.Delivery.ShardingProducerController.Start`1) message that contains the `Producer`'s `IActorRef` in order to begin message production.
+
+> [!TIP]
+> Unlike Akka.Delivery, there is no need to have the `ProducerController` and `ConsumerController` explicitly register with the other - this will be handled automatically by the Akka.Cluster.Sharding messaging system.
+
+### Message Production
+
+Once the `Producer` has been successfully registered with its `ProducerController`, it will begin to receive `ShardingProducerController.RequestNext<T>` messages - each time it receives one of these messages the `Producer` can send a burst of messages to the `ShardingProducerController`.
+
+[!code-csharp[Launching ShardRegion with ShardingConsumerController configured](../../../src/examples/Cluster/ClusterSharding/ShoppingCart/Producer.cs?name=MessageProduction)]
+
+> [!IMPORTANT]
+> It is crucial that the `Prodcuer` send its messages of type `T` wrapped inside a [`ShardingEnvelope`](xref:Akka.Cluster.Sharding.ShardingEnvelope) - otherwise the `ShardingProducerController` won't know which messages should be routed to which unique `entityId`. Additionally - your `HashCodeMessageExtractor` that you use with your `ShardRegion` must also be able to handle the `ShardingEnvelope` to ensure that this message is processed correctly on the receiving side. There is a proposal in-place to automate some of this work in a future release of Akka.NET: [#6717](https://github.com/akkadotnet/akka.net/issues/6717).
+
+One important distinction between [`ShardingProducerController.RequestNext<T>`](xref:Akka.Cluster.Sharding.Delivery.ShardingProducerController.RequestNext`1) and [`ProducerController.RequestNext<T>`](xref:Akka.Delivery.ProducerController.RequestNext`1) - because the `ShardingProducerController` has to deliver to multiple `Consumer`s, it retains a much larger outbound delivery buffer. You can check the status of which entities are currently buffered or which ones have active demand by inspecting the `ShardingProducerController.RequestNext<T>.BufferedForEntitiesWithoutDemand` or `ShardingProducerController.RequestNext<T>.EntitiesWithDemand` properties respectively.
+
+![Akka.Cluster.Sharding.Delivery message production cycle.](/images/cluster/delivery/2-sharding-message-production.png)
+
+Once the `ShardingProducerController` begins receiving messages of type `T` (wrapped in a `ShardingEnvelope`) from the `Producer`, it will spawn `ProducerController`s for each unique entity and begin routing those messages to the `ShardRegion`.
+
+### Message Consumption
+
+The `ProducerController`s will all send `ConsumerController.SequencedMessage<T>` over the wire, wrapped inside `ShardingEnvelope`s - the `ShardRegion` must be programmed to handle these types:
+
+![Akka.Cluster.Sharding.Delivery message consumption process.](/images/cluster/delivery/3-sharding-message-consumption.png)
+
+[!code-csharp[ShardRegion message extractor](../../../src/examples/Cluster/ClusterSharding/ShoppingCart/MessageExtractor.cs?name=ExtractorClass)]
+
+As the `ConsumerController.SequencedMessage<T>`s are delivered, the `ShardRegion` will spawn the `ShardingConsumerController` for each entity, which will in turn spawn the entity actor itself (the `Consumer`) as well as one `ConsumerController` per unique `ProducerId`. These actors are all cheap and maintain a finite amount of buffer space.
+
+1. The `Consumer` receives the `ConsumerController.Delivery<T>` message from the `ShardingConsumerController`;
+2. The `Consumer` replies to the `IActorRef` stored inside `ConsumerController.Delivery<T>.DeliverTo` with a `ConsumerController.Confirmed` message - this marks the message as "processed;"
+3. The `ConsumerController` marks the message as delivered, removes it from the buffer, and requests additional messages from the `ProducerController`; and
+4. This in turn causes the `ShardingProducerController` to update its aggregate state and send additional `ShardingProducerController.RequestNext<T>` demand to the `Producer`.
+
+### Guarantees, Constraints, and Caveats
+
+See ["Reliable Message Delivery with Akka.Delivery - Guarantees, Constraints, and Caveats"](xref:reliable-delivery#guarantees-constraints-and-caveats) - these are the same in Akka.Cluster.Sharding.Delivery.
+
+With one notable exception: **Akka.Cluster.Sharding.Delivery does not support message chunking** and there are no plans to add it at this time.
+
+## Durable Reliable Delivery over Akka.Cluster.Sharding
+
+By default the `ShardingProducerController` will run using without any persistent storage - however, if you reference the [Akka.Persistence library](xref:persistence-architecture) in your Akka.NET application then you can make use of the [`EventSourcedProducerQueue`](xref:Akka.Persistence.Delivery.EventSourcedProducerQueue) to ensure that your `ShardingProducerController` saves and un-acknowledged messages to your Akka.Persistence Journal and SnapshotStore.
+
+[!code-csharp[Durable ShardingProducerController configuration](../../../src/contrib/cluster/Akka.Cluster.Sharding.Tests/Delivery/DurableShardingSpec.cs?name=SpawnDurableProducer)]
+
+> [!TIP]
+> The `EventSourcedProducerQueue` can be customized via the [`EventSourcedProducerQueue.Settings` class](xref:Akka.Persistence.Delivery.EventSourcedProducerQueue.Settings) - for instance, you can customize it to use a separate Akka.Persistence Journal and SnapshotStore.
+
+Each time a message is sent to the `ShardingProducerController` it will persist a copy of the message to the Akka.Persistence journal.
+
+![ShardingProducerController persistence with EventSourcedProducerQueue.](/images/cluster/delivery/4-sharding-message-persistence.png)
+
+> [!TIP]
+> All messages for all entities are stored in the same `EventSourcedProducerQueue` instance.
+
+### Confirmation of Outbound Messages Persisted
+
+If the `Producer` needs to confirm that all of its outbound messages have been successfully persisted, this can be accomplished via the `ShardingProducerController.RequestNext<T>.AskNextTo` method:
+
+[!code-csharp[Starting ProducerController with EventSourcedProducerQueue enabled](../../../src/core/Akka.Docs.Tests/Delivery/DeliveryDocSpecs.cs?name=ConfirmableMessages)]
+
+The `AskNextTo` method will return a `Task<long>` that will be completed once the message has been confirmed as stored inside the `EventSourcedProducerQueue` - the `long` in this case is the sequence number that has been assigned to this message via the `ShardingProducerController`'s outbound queue.
+
+In addition to outbound deliveries, confirmation messages from the `ShardingConsumerController` will also be persisted - and these will cause the `EventSourcedProducerQueue` to gradually compress its footprint in the Akka.Persistence journal by taking snapshots.
+
+> [!TIP]
+> By default the `EventSourcedProducerQueue` will take a new snapshot every 1000 events but this can be configured via the [`EventSourcedProducerQueue.Settings` class](xref:Akka.Persistence.Delivery.EventSourcedProducerQueue.Settings).

docs/articles/clustering/cluster-sharding.md

@@ -1,8 +1,8 @@
 ---
 uid: cluster-sharding
-title: Akka.Cluster.Sharding module
+title: Akka.Cluster.Sharding - Reliable, Automatic State Distribution with Akka.Cluster
 ---
-# Akka.Cluster.Sharding Module
+# Akka.Cluster.Sharding
 
 Cluster sharding is useful in cases when you want to contact with cluster actors using their logical id's, but don't want to care about their physical location inside the cluster or manage their creation. Moreover it's able to re-balance them, as nodes join/leave the cluster. It's often used to represent i.e. Aggregate Roots in Domain Driven Design terminology.
 
@@ -83,6 +83,10 @@ As you may have seen in the examples above shard resolution algorithm is one of
 
 By default re-balancing process always happens from nodes with the highest number of shards, to the ones with the smallest one. This can be configured into by specifying custom implementation of the `IShardAllocationStrategy` interface in `ClusterSharding.Start` parameters.
 
+## Reliable Delivery of Messages to Sharded Entity Actors
+
+If you are interested in ensuring that all messages are guaranteed to be delivered to your entity actors even across restarts, re-balancing operations, or crashes then please see "[Reliable Delivery over Akka.Cluster.Sharding](xref:cluster-sharding-delivery)."
+
 ## Passivation
 
 To reduce memory consumption, you may decide to stop entities after some period of inactivity using `Context.SetReceiveTimeout(timeout)`. In order to make cluster sharding aware of stopping entities, **DON'T use `Context.Stop(Self)` on the entities**, as this may result in losing messages. Instead send a `ShardRegion.Passivate` message to current entity `Context.Parent` (which is shard itself in this case). This will inform shard to stop forwarding messages to target entity, and buffer them instead until it's terminated. Once that happens, if there are still some messages buffered, entity will be reincarnated and messages flushed to it automatically.

docs/articles/clustering/toc.yml

@@ -16,6 +16,8 @@
   href: cluster-client.md
 - name: Cluster Sharding
   href: cluster-sharding.md
+- name: Reliable Delivery over Cluster Sharding
+  href: cluster-sharding-delivery.md
 - name: Sharded Daemon Process
   href: cluster-sharded-daemon-process.md
 - name: Cluster Metrics

docs/articles/concepts/message-delivery-reliability.md

@@ -73,6 +73,9 @@ most expensive—and has consequently worst performance—because in addition to
 the second it requires state to be kept at the receiving end in order to filter
 out duplicate deliveries.
 
+> [!TIP]
+> Please see "[Reliable Message Delivery with Akka.Delivery](xref:reliable-delivery)" if you need a delivery guarantee that is more robust than "at most once" delivery of messages.
+
 ### Discussion: Why No Guaranteed Delivery?
 
 At the core of the problem lies the question what exactly this guarantee shall

docs/articles/persistence/at-least-once-delivery.md

@@ -2,7 +2,10 @@
 uid: at-least-once-delivery
 title: At-Least-Once Delivery
 ---
-# At-Least-Once Delivery
+# At-Least-Once Delivery (Obsolete)
+
+> [!WARNING]
+> `AtLeastOnceDelivery` actors in Akka.Persistence are being deprecated in favor of [Akka.Delivery](xref:reliable-delivery) and [Akka.Cluster.Sharding.Delivery](xref:cluster-sharding-delivery). Please look at those articles for further details.
 
 To send messages with at-least-once delivery semantics to destinations you can mix-in `AtLeastOnceDelivery` class to your `PersistentActor` on the sending side. It takes care of re-sending messages when they have not been confirmed within a configurable timeout.
 

docs/articles/remoting/messaging.md

@@ -7,6 +7,9 @@ title: Remote Messaging
 
 Once you [form an association between two `ActorSystem`s](xref:remote-overview#how-to-form-associations-between-remote-systems), you can now send messages transparently between actors regardless of where they are on the network.
 
+> [!IMPORTANT]
+> See "[Reliable Akka.NET Message Delivery with Akka.Delivery](xref:reliable-delivery)" for tips on how to achieve reliable messaging over Akka.Remote and Akka.Cluster.
+
 ## Serialization
 
 [Serialization of messages in Akka.NET is transparent](xref:serialization), but in order to achieve that transparency there are some practices you need to observe in how you design your project.

src/Directory.Build.props

@@ -2,7 +2,7 @@
   <PropertyGroup>
     <Copyright>Copyright © 2013-2023 Akka.NET Team</Copyright>
     <Authors>Akka.NET Team</Authors>
-    <VersionPrefix>1.5.4</VersionPrefix>
+    <VersionPrefix>1.5.7</VersionPrefix>
     <PackageIcon>akkalogo.png</PackageIcon>
     <PackageProjectUrl>https://github.com/akkadotnet/akka.net</PackageProjectUrl>
     <PackageLicenseUrl>https://github.com/akkadotnet/akka.net/blob/master/LICENSE</PackageLicenseUrl>
@@ -40,7 +40,7 @@
     <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
   </PropertyGroup>
   <PropertyGroup>
-    <PackageReleaseNotes>placeholder for nightlies*</PackageReleaseNotes>
+    <PackageReleaseNotes>Placeholder for nightlies*</PackageReleaseNotes>
   </PropertyGroup>
   <!-- SourceLink support for all Akka.NET projects -->
   <ItemGroup>