CFP AVAD: Adds Container IsFeedRangePartOfAsync support for FeedRange

[philipthomas-MSFT]

Pull Request Template

Description

Details #4483

The customer wants to validate if one FeedRange is a subset of another FeedRange. This means checking if all the data in the y FeedRange is entirely within the boundaries of the x FeedRange, ensuring that no data is processed twice or skipped. It is understood that FeedRange could also be represented by a PartitionKey or FeedRangeEpk.

Feature: PartitionKey and FeedRange Validation

  Scenario: Validate if the y (FeedRange) is part of the x (PartitionKey)
    Given x represents the PartitionKey
    And y represents the FeedRange
    And both x and y have inclusive minimum and maximum boundaries (x.IsMinInclusive=true, x.IsMaxInclusive=true, y.IsMinInclusive=true, y.IsMaxInclusive=true)
    Then the scenario is valid, and both x and y include their start and end values

  Scenario: Validate NotSupportedException for incompatible boundaries between x (PartitionKey) and y (FeedRange)
    Given x represents the PartitionKey
    And y represents the FeedRange
    And x has an inclusive maximum boundary (x.IsMaxInclusive=true)
    And y has an exclusive maximum boundary (y.IsMaxInclusive=false)
    When the boundaries are compared
    Then a NotSupportedException is thrown due to incompatible boundary configurations

  Scenario: Validate minimum boundary inclusiveness for y (FeedRange) when compared to x (PartitionKey)
    Given x represents the PartitionKey
    And y represents the FeedRange
    And y has an exclusive minimum boundary (y.IsMinInclusive=false)
    And x has an inclusive minimum boundary (x.IsMinInclusive=true)
    When the boundaries are compared
    Then an error is thrown because y's minimum value must be inclusive (IsMinInclusive must be true)

  Scenario: Validate that both y's (FeedRange) minimum and maximum are exclusive when compared to x (PartitionKey)
    Given x represents the PartitionKey
    And y represents the FeedRange
    And both y's minimum and maximum values are exclusive (y.IsMinInclusive=false, y.IsMaxInclusive=false)
    When the boundaries are compared
    Then an error is thrown because y's minimum value must be inclusive (IsMinInclusive must be true)

Scenario Description	x.IsMinInclusive	x.IsMaxInclusive	y.IsMinInclusive	y.IsMaxInclusive	Expected Outcome
Validate if the y partition key is part of the x feed range	true	true	true	true	Scenario is valid, both x and y include their start and end values
Validate NotSupportedException for incompatible boundaries	true	true	true	false	NotSupportedException is thrown due to incompatible boundary configurations
Validate minimum boundary inclusiveness for y	true	true	false	true	Error is thrown because y's minimum value must be inclusive
Validate y's exclusiveness results in error	true	true	false	false	Error is thrown because y's minimum value must be inclusive

---

Feature: FeedRange and PartitionKey Validation

  Scenario: Validate that both x (FeedRange) and y (PartitionKey) are fully inclusive
    Given x represents the FeedRange
    And y represents the PartitionKey
    And x has inclusive minimum and maximum boundaries (x.IsMinInclusive=true, x.IsMaxInclusive=true)
    And y has inclusive minimum and maximum boundaries (y.IsMinInclusive=true, y.IsMaxInclusive=true)
    Then the scenario is valid because both x and y include their start and end values

  Scenario: Validate that x (FeedRange) has inclusive minimum and exclusive maximum while y (PartitionKey) is fully inclusive
    Given x represents the FeedRange
    And y represents the PartitionKey
    And x has inclusive minimum and exclusive maximum boundaries (x.IsMinInclusive=true, x.IsMaxInclusive=false)
    And y has inclusive minimum and maximum boundaries (y.IsMinInclusive=true, y.IsMaxInclusive=true)
    Then the scenario is valid because x and y are compatible in terms of their boundaries

  Scenario: Validate that x's (FeedRange) minimum is exclusive while y (PartitionKey) is fully inclusive
    Given x represents the FeedRange
    And y represents the PartitionKey
    And x has an exclusive minimum and inclusive maximum boundary (x.IsMinInclusive=false, x.IsMaxInclusive=true)
    And y has inclusive minimum and maximum boundaries (y.IsMinInclusive=true, y.IsMaxInclusive=true)
    Then the scenario is invalid because x's minimum value must be inclusive (IsMinInclusive must be true)

  Scenario: Validate that both x's (FeedRange) minimum and maximum are exclusive while y (PartitionKey) is fully inclusive
    Given x represents the FeedRange
    And y represents the PartitionKey
    And x has exclusive minimum and maximum boundaries (x.IsMinInclusive=false, x.IsMaxInclusive=false)
    And y has inclusive minimum and maximum boundaries (y.IsMinInclusive=true, y.IsMaxInclusive=true)
    Then the scenario is invalid because x's minimum value must be inclusive (IsMinInclusive must be true)

Scenario Description	x.IsMinInclusive	x.IsMaxInclusive	y.IsMinInclusive	y.IsMaxInclusive	Expected Outcome
Validate that both x and y are fully inclusive	true	true	true	true	Scenario is valid because both x and y include their start and end values
Validate that x has inclusive minimum and exclusive maximum while y is fully inclusive	true	false	true	true	Scenario is valid because x and y are compatible in terms of their boundaries
Validate that x's minimum is exclusive while y is fully inclusive	false	true	true	true	Scenario is invalid because x's minimum value must be inclusive (IsMinInclusive must be true)
Validate that both x's minimum and maximum are exclusive while y is fully inclusive	false	false	true	true	Scenario is invalid because x's minimum value must be inclusive (IsMinInclusive must be true)

---

Feature: FeedRange and FeedRange Validation

  Scenario: Validate when both x (FeedRange) and y (FeedRange) have inclusive minimum and maximum boundaries
    Given x represents the FeedRange
    And y represents the FeedRange
    And both x and y have inclusive minimum and maximum boundaries (x.IsMinInclusive=true, x.IsMaxInclusive=true, y.IsMinInclusive=true, y.IsMaxInclusive=true)
    Then the scenario is valid because both x and y include their start and end values

  Scenario: Validate when x (FeedRange) has inclusive boundaries and y (FeedRange) has an exclusive maximum boundary
    Given x represents the FeedRange
    And y represents the FeedRange
    And x has inclusive minimum and maximum boundaries (x.IsMinInclusive=true, x.IsMaxInclusive=true)
    And y has an exclusive maximum boundary (y.IsMinInclusive=true, y.IsMaxInclusive=false)
    Then a NotSupportedException is thrown because y's maximum boundary is exclusive while x's is inclusive

  Scenario: Validate when x (FeedRange) has inclusive boundaries and y (FeedRange) has an exclusive minimum boundary
    Given x represents the FeedRange
    And y represents the FeedRange
    And x has inclusive minimum and maximum boundaries (x.IsMinInclusive=true, x.IsMaxInclusive=true)
    And y has an exclusive minimum boundary (y.IsMinInclusive=false, y.IsMaxInclusive=true)
    Then the scenario is invalid because y's minimum boundary must be inclusive

  Scenario: Validate when x (FeedRange) has inclusive boundaries and y (FeedRange) has both exclusive minimum and maximum boundaries
    Given x represents the FeedRange
    And y represents the FeedRange
    And x has inclusive minimum and maximum boundaries (x.IsMinInclusive=true, x.IsMaxInclusive=true)
    And y has both exclusive minimum and maximum boundaries (y.IsMinInclusive=false, y.IsMaxInclusive=false)
    Then the scenario is invalid because y's minimum boundary must be inclusive

  Scenario: Validate when x (FeedRange) has an inclusive minimum boundary and an exclusive maximum boundary, and y (FeedRange) has fully inclusive boundaries
    Given x represents the FeedRange
    And y represents the FeedRange
    And x has an inclusive minimum and exclusive maximum boundary (x.IsMinInclusive=true, x.IsMaxInclusive=false)
    And y has fully inclusive boundaries (y.IsMinInclusive=true, y.IsMaxInclusive=true)
    Then the scenario is valid because x's and y's boundaries are compatible

  Scenario: Validate when both x (FeedRange) and y (FeedRange) have inclusive minimum boundaries and exclusive maximum boundaries
    Given x represents the FeedRange
    And y represents the FeedRange
    And both x and y have inclusive minimum boundaries and exclusive maximum boundaries (x.IsMinInclusive=true, x.IsMaxInclusive=false, y.IsMinInclusive=true, y.IsMaxInclusive=false)
    Then the scenario is valid because both ranges are compatible

  Scenario: Validate when x (FeedRange) has an inclusive minimum boundary and an exclusive maximum boundary, and y (FeedRange) has an exclusive minimum boundary
    Given x represents the FeedRange
    And y represents the FeedRange
    And x has an inclusive minimum boundary and an exclusive maximum boundary (x.IsMinInclusive=true, x.IsMaxInclusive=false)
    And y has an exclusive minimum boundary but an inclusive maximum boundary (y.IsMinInclusive=false, y.IsMaxInclusive=true)
    Then the scenario is invalid because y's minimum boundary must be inclusive

  Scenario: Validate when x (FeedRange) has an inclusive minimum boundary and an exclusive maximum boundary, and y (FeedRange) has both exclusive minimum and maximum boundaries
    Given x represents the FeedRange
    And y represents the FeedRange
    And x has an inclusive minimum boundary and an exclusive maximum boundary (x.IsMinInclusive=true, x.IsMaxInclusive=false)
    And y has both exclusive minimum and maximum boundaries (y.IsMinInclusive=false, y.IsMaxInclusive=false)
    Then the scenario is invalid because y's minimum boundary must be inclusive

  Scenario: Validate when x (FeedRange) has an exclusive minimum boundary and an inclusive maximum boundary, and y (FeedRange) has fully inclusive boundaries
    Given x represents the FeedRange
    And y represents the FeedRange
    And x has an exclusive minimum boundary and an inclusive maximum boundary (x.IsMinInclusive=false, x.IsMaxInclusive=true)
    And y has fully inclusive boundaries (y.IsMinInclusive=true, y.IsMaxInclusive=true)
    Then the scenario is invalid because x's minimum boundary must be inclusive

  Scenario: Validate when x (FeedRange) has an exclusive minimum boundary and an inclusive maximum boundary, and y (FeedRange) has an exclusive maximum boundary
    Given x represents the FeedRange
    And y represents the FeedRange
    And x has an exclusive minimum boundary and an inclusive maximum boundary (x.IsMinInclusive=false, x.IsMaxInclusive=true)
    And y has an inclusive minimum but an exclusive maximum boundary (y.IsMinInclusive=true, y.IsMaxInclusive=false)
    Then the scenario is invalid because x's minimum boundary must be inclusive

  Scenario: Validate when both x (FeedRange) and y (FeedRange) have exclusive minimum boundaries and inclusive maximum boundaries
    Given x represents the FeedRange
    And y represents the FeedRange
    And both x and y have exclusive minimum boundaries and inclusive maximum boundaries (x.IsMinInclusive=false, x.IsMaxInclusive=true, y.IsMinInclusive=false, y.IsMaxInclusive=true)
    Then the scenario is invalid because both x's and y's minimum boundaries must be inclusive

  Scenario: Validate when both x (FeedRange) and y (FeedRange) have exclusive minimum and maximum boundaries
    Given x represents the FeedRange
    And y represents the FeedRange
    And both x and y have exclusive minimum and maximum boundaries (x.IsMinInclusive=false, x.IsMaxInclusive=false, y.IsMinInclusive=false, y.IsMaxInclusive=false)
    Then the scenario is invalid because both x's and y's minimum boundaries must be inclusive

  Scenario: Validate when x (FeedRange) has exclusive minimum and maximum boundaries, and y (FeedRange) has fully inclusive boundaries
    Given x represents the FeedRange
    And y represents the FeedRange
    And x has exclusive minimum and maximum boundaries (x.IsMinInclusive=false, x.IsMaxInclusive=false)
    And y has fully inclusive boundaries (y.IsMinInclusive=true, y.IsMaxInclusive=true)
    Then the scenario is invalid because x's minimum boundary must be inclusive

Scenario Description	x.IsMinInclusive	x.IsMaxInclusive	y.IsMinInclusive	y.IsMaxInclusive	Expected Outcome
Validate when both x and y have inclusive minimum and maximum boundaries	true	true	true	true	Scenario is valid because both x and y include their boundaries
Validate when x has inclusive boundaries and y has an exclusive maximum boundary	true	true	true	false	NotSupportedException because y's maximum is exclusive
Validate when x has inclusive boundaries and y has an exclusive minimum boundary	true	true	false	true	Scenario is invalid because y's minimum must be inclusive
Validate when x has inclusive boundaries and y has both exclusive minimum and maximum boundaries	true	true	false	false	Scenario is invalid because y's minimum must be inclusive
Validate when x has an inclusive minimum and exclusive maximum, and y has fully inclusive boundaries	true	false	true	true	Scenario is valid because x's and y's boundaries are compatible
Validate when both x and y have inclusive minimum and exclusive maximum boundaries	true	false	true	false	Scenario is valid because both ranges are compatible
Validate when x has an inclusive minimum and exclusive maximum, and y has an exclusive minimum boundary	true	false	false	true	Scenario is invalid because y's minimum must be inclusive
Validate when x has an inclusive minimum and exclusive maximum, and y has both exclusive boundaries	true	false	false	false	Scenario is invalid because y's minimum must be inclusive
Validate when x has an exclusive minimum and inclusive maximum, and y has fully inclusive boundaries	false	true	true	true	Scenario is invalid because x's minimum must be inclusive
Validate when x has an exclusive minimum and inclusive maximum, and y has an exclusive maximum boundary	false	true	true	false	Scenario is invalid because x's minimum must be inclusive
Validate when both x and y have exclusive minimum and inclusive maximum boundaries	false	true	false	true	Scenario is invalid because both x's and y's minimum must be inclusive
Validate when both x and y have exclusive minimum and maximum boundaries	false	false	false	false	Scenario is invalid because both x's and y's minimum must be inclusive
Validate when x has exclusive minimum and maximum boundaries, and y has fully inclusive boundaries	false	false	true	true	Scenario is invalid because x's minimum must be inclusive

---

API

Task<bool> IsFeedRangePartOfAsync(
            Cosmos.FeedRange x,
            Cosmos.FeedRange y,
            CancellationToken cancellationToken = default)

Levelsetting

Container

Diagram 1: Final state of PR

classDiagram
    class Container {
        <<abstract>>
        +IsFeedRangePartOfAsync() : Task<bool>
    }
    
    class ContainerInternal {
        +IsFeedRangePartOfAsync() : Task<bool>
    }

    class ContainerCore {
        -container : Container
        +IsFeedRangePartOfAsync() : Task<bool>
    }

    class ContainerInlineCore {
        -container : Container
        +IsFeedRangePartOfAsync() : Task<bool>
    }

    class EncryptionContainer {
        -container : Container
        +IsFeedRangePartOfAsync() : Task<bool>
    }

    class CustomEncryptionContainer {
        -container : Container
        +IsFeedRangePartOfAsync() : Task<bool>
    }

    ContainerInternal --|> Container : "is a"
    ContainerCore --|> ContainerInternal : "is a"
    ContainerInlineCore --|> ContainerCore : "is a"
    EncryptionContainer --|> Container : "is a"
    CustomEncryptionContainer --|> Container : "is a"
    ContainerCore --> Container : "has a"
    ContainerInlineCore --> Container : "has a"
    EncryptionContainer --> Container : "has a"
    CustomEncryptionContainer --> Container : "has a"

Loading

FeedRange

classDiagram
    class FeedRange {
    <<abstract>>
    }

    class FeedRangeInternal {
        -range : Range
    }

    class FeedRangeEpk {
        -range : Range
    }

    class FeedRangePartitionKey {
        -range : Range
    }

    class FeedRangePartitionKeyRange {
        -range : Range
    }

    class Range {
        +Min : string
        +Max : string
        +IsMinInclusive : bool
        +IsMaxInclusive : bool
    }

    FeedRangeInternal --|> FeedRange : "is a"
    FeedRangeEpk --|> FeedRangeInternal : "is a"
    FeedRangePartitionKey --|> FeedRangeInternal : "is a"
    FeedRangePartitionKeyRange --|> FeedRangeInternal : "is a"
    FeedRangeInternal --> Range : "has a"
    FeedRangePartitionKeyRange --> Range : "has a"
    FeedRangePartitionKey --> Range : "has a"
    FeedRangeEpk --> Range : "has a"

Loading

https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.cosmos.feedrange?view=azure-dotnet

A FeedRange in Azure Cosmos DB represents a segment of data, often based on partition keys, that can be processed or monitored. FeedRanges are typically used in the context of change feed processing to allow multiple workers to consume data in parallel by splitting the container’s data into smaller, manageable pieces. Each FeedRange corresponds to a specific partition key range.

Min: Lower boundary of a range

Max Upper boundary of a range

IsMinInclusive: Is the lower boundary inclusive or exclusive?

• if true, the partition key value at Min is included in the range.
• if false, the range starts just after the Min value.

IsMaxInclusive: Is the upper boundary inclusive or exclusive?

• if true, the partition key at Max is included in the range.
• if false, the range stops just before the Max value.

"MaxExclusive": The upper boundary of a range is exclusive; the range does not include the maximum value itself.

• IsMaxInclusive = false means that the upper boundary is exclusive, so IsMaxInclusive is that same as the unofficial term, "MaxExclusive".

Examples based on unit tests

FeedRange

A FeedRange represents a segment of data that can be processed or read in parallel. FeedRanges can be used with or without partition keys, depending on the context. They are especially useful in change feed processing, where multiple workers can process different ranges in parallel.

Container container = default;

try
{
    ContainerResponse containerResponse = await this.cosmosDatabase.CreateContainerIfNotExistsAsync(
        id: Guid.NewGuid().ToString(),
        partitionKeyPath: "/pk");

    container = containerResponse.Container;

    bool actualIsSubset = await container.IsFeedRangePartOfAsync(
        new FeedRangeEpk(new Documents.Routing.Range<string>(xMin, xMax, true, false)),
        new FeedRangeEpk(new Documents.Routing.Range<string>(yMin, yMax, true, false)),
        cancellationToken: CancellationToken.None);

    Assert.AreEqual(expected: expectedIsSubset, actual: actualIsSubset);
}
finally
{
    if (container != null)
    {
        await container.DeleteContainerAsync();
    }
}

Hierarchical PartitionKey

In hierarchical partitioning, the partition key is composed of multiple levels (or paths), which can be combined in a hierarchical manner. This means that data is partitioned based on a combination of keys, such as {"region": "US", "userId": "123"}, which makes the partitioning more granular and allows for more specific partition key ranges.

Container container = default;

try
{
    ContainerProperties containerProperties = new ContainerProperties()
    {
        Id = Guid.NewGuid().ToString(),
        PartitionKeyPaths = new Collection<string> { "/pk", "/id" }
    };

    ContainerResponse containerResponse = await this.cosmosDatabase.CreateContainerIfNotExistsAsync(containerProperties);

    container = containerResponse.Container;

    PartitionKey partitionKey = new PartitionKeyBuilder()
        .Add("WA")
        .Add(Guid.NewGuid().ToString())
        .Build();

    FeedRange feedRange = FeedRange.FromPartitionKey(partitionKey);

    bool actualIsSubset = await container.IsFeedRangePartOfAsync(
        new FeedRangeEpk(new Documents.Routing.Range<string>(xMin, xMax, true, false)),
        feedRange,
        cancellationToken: CancellationToken.None);

    Assert.AreEqual(expected: expectedIsSubset, actual: actualIsSubset);
}
finally
{
    if (container != null)
    {
        await container.DeleteContainerAsync();
    }
}

Partition Key

In non-hierarchical partitioning, the partition key is a single value (or path). For instance, a partition key might be {"userId": "123"}. Data is distributed based solely on this one-dimensional key.

Container container = default;

try
{
    ContainerResponse containerResponse = await this.cosmosDatabase.CreateContainerIfNotExistsAsync(
        id: Guid.NewGuid().ToString(),
        partitionKeyPath: "/pk");

    container = containerResponse.Container;

    PartitionKey partitionKey = new("WA");
    FeedRange feedRange = FeedRange.FromPartitionKey(partitionKey);

    bool actualIsSubset = await container.IsFeedRangePartOfAsync(
        new FeedRangeEpk(new Documents.Routing.Range<string>(xMin, xMax, true, false)),
        feedRange,
        cancellationToken: CancellationToken.None);

    Assert.AreEqual(expected: expectedIsSubset, actual: actualIsSubset);
}
finally
{
    if (container != null)
    {
        await container.DeleteContainerAsync();
    }
}

Invalid FeedRangeEpk with IsMinInclusive: false throws ArgumentOutOfRangeException

FeedRangeEpk is a more specific form of FeedRange that focuses on effective partition keys (EPK), which represent the internal structure of how data is distributed across partitions in Azure Cosmos DB.

FeedRangeEpk targets specific physical partitions, and these boundaries are managed internally by Cosmos DB. These boundaries are fixed based on the hash values of the partition keys and are not modifiable by the user. As a result, there is no need for the notion of inclusivity, because the partition boundary is rigid and defined by the physical partition structure, not by a logical partition key.

Container container = default;

try
{
    ContainerResponse containerResponse = await this.cosmosDatabase.CreateContainerIfNotExistsAsync(
        id: Guid.NewGuid().ToString(),
        partitionKeyPath: "/pk");

    container = containerResponse.Container;

    bool actualIsSubset = await container
            .IsFeedRangePartOfAsync(
                new FeedRangeEpk(new Documents.Routing.Range<string>("", "3FFFFFFFFFFFFFFF", false, true)),
                new FeedRangeEpk(new Documents.Routing.Range<string>("", "FFFFFFFFFFFFFFFF", true, false)),
                cancellationToken: CancellationToken.None));
}
catch (ArgumentOutOfRangeException exception)
{
     Console.WriteLine(exception.Message); // "IsMinInclusive must be true."
}
finally
{
    if (container != null)
    {
        await container.DeleteContainerAsync();
    }
}

EnsureConsistentInclusitivy

Validates if all ranges in the list have consistent inclusivity for both IsMinInclusive and IsMaxInclusive boundaries.
Throws an InvalidOperationException if there are inconsistencies.

List<Documents.Routing.Range<string>> ranges = new List<Documents.Routing.Range<string>>
{
    new Documents.Routing.Range<string> { IsMinInclusive = true, IsMaxInclusive = false },
    new Documents.Routing.Range<string> { IsMinInclusive = true, IsMaxInclusive = true },
    new Documents.Routing.Range<string> { IsMinInclusive = true, IsMaxInclusive = false },
    new Documents.Routing.Range<string> { IsMinInclusive = false, IsMaxInclusive = false }
};

EnsureConsistentInclusivity(ranges);

IsSubset

Determines whether the y range is a subset of the x range.

Documents.Routing.Range<string> x= new Documents.Routing.Range<string>("A", "Z", true, true);
Documents.Routing.Range<string> y= new Documents.Routing.Range<string>("B", "Y", true, true);

bool isSubset = ContainerCore.IsSubset(x, y);

Type of change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

Closing issues

To automatically close an issue: closes #4483

[github-actions]

All good!

As per ask from Matias

[NaluTripician]

LGTM

[FabianMeiswinkel]

LGTM except for the two NITs