Cosmos: allow for concurrency token on '_etag' property

[jviau]

#17299

Adds the following:

1. UseEtagConcurrency extension method for EntityTypeBuilder and EntityTypeBuilder<TEntity>, which adds a shadow property for the _etag field and sets it as the concurrency token.
2. Validates only a json property _etag is the concurrency token for a cosmos entity.
3. Passes the current _etag value in the IfMatchEtag portion of the request options during ReplaceItemStreamAsync.

Things to consider:

• Should this be added to DeleteItemAsync as well?
• Should an extension method be added for PropertyBuilder to configure it as an etag?
• Should this also validate that the _etag property is of string type?

TODO:

• Add functional tests.

If there was another design in mind and this one isn't acceptable, no worries! Feel free to reject or ask for other changes. I am just sharing back the changes I made locally when playing around with cosmos efcore.

[AndriySvyryd]

Should this be added to DeleteItemAsync as well?

Yes. #10443 tracks making this configurable

Should an extension method be added for PropertyBuilder to configure it as an etag?

Yes, it would also set the JSON name to "_etag"

Should this also validate that the _etag property is of string type?

Yes. The provider type should be string, i.e. property.GetTypeMapping().Converter?.ProviderClrType ?? property.ClrType;

Add functional tests.

Ideally they would derive from OptimisticConcurrencyTestBase, but that model might be too hard to properly map in Cosmos. So you can just use a part of it and only make the relevant tests run.

[jviau]

Should this be added to DeleteItemAsync as well?

Yes. #10443 tracks making this configurable

Reading that I am unsure what is the correct behavior here. Should I add the IfMatchEtag option to all delete requests when etag concurrency is present? This would mean if an item is modified externally before a delete, the delete will fail. Is that the desired behavior? It sounds like that issue OP doesn't want that.

[AndriySvyryd]

Reading that I am unsure what is the correct behavior here. Should I add the IfMatchEtag option to all delete requests when etag concurrency is present? This would mean if an item is modified externally before a delete, the delete will fail. Is that the desired behavior? It sounds like that issue OP doesn't want that.

Yes, the IfMatchEtag should be added. This will make the Cosmos provider behave in the same way as other providers do, even though it might not be ideal.

[jviau]

Update on this: I am working on implementing OptimisticConcurrencyTestBase for these changes. However, it is proving fairly difficult as I am encountering several various features this test base relies on that Cosmos provider does not support yet. These tests use IDbContextTransactionManager for examble, where the cosmos implementation just throws NotSupportedException.

[AndriySvyryd]

@jviau You don't need to implement OptimisticConcurrencyTestBase, just follow its general structure. Don't use ExecutionStrategy or transactions, just clean the database before each test or if you just make one test work that should be enough for now.

[jviau]

@AndriySvyryd. Okay I have made a separate test class and have the it working for the most part. But I have hit an interesting issue, wondering if this is a problem with my code, a known issue, or something new. Properties with ValueGeneratedOnAddOrUpdate() are not being refreshed when added, saved, then retrieved with the same DbContext. So:

public class MyEntity
{
  public string Id { get; set; }

  public string Generated { get; set; ] // this is configured with .ValueGeneratedOnAddOrUpdate()
}

using var ctx = new MyContext();
ctx.MyEntities.Add(new MyEntity { Id = "1" });
ctx.SaveChanges()
var myEntity = ctx.MyEntities.Single(e => e.Id == "1");
myEntity.Generated != null; // false! this is still null. but...

using var ctx2 = new MyContext();
var myEntity2 = ctx2.MyEntities.Single(e => e.Id == "1");
myEntity2.Generated != null; // true! When fetched from a different context, this is finally retrieved.

[AndriySvyryd]

Properties with ValueGeneratedOnAddOrUpdate() are not being refreshed

That's expected, since the etags are the only thing that can be store generated by Cosmos, so there's currently no general support to get the updated values. You would need to modify CosmosClientWrapper.CreateItemOnceAsync and CosmosClientWrapper.ReplaceItemOnceAsync to get the ETag header value from the response and update the etag property if one is present.

[jviau]

Ah I see. That seems pretty important for _etag to work properly. But that can be done in a later PR I suppose.

If there are any further changes you want in this PR, or more cases tested, let me know.

[AndriySvyryd]

@jviau If you could add the store generated support in this PR, it would be better, should be like 3-4 lines in each method.

[jviau]

Do you mean the work to read back the new _etag value and update the store? Okay, reading back the document response is simple enough. Where do I take that returned json and update the store though? I am unfamiliar with how I would do that.

[AndriySvyryd]

I was thinking of adding an IUpdateEntry entry parameter to CosmosClientWrapper.CreateItemOnceAsync and CosmosClientWrapper.ReplaceItemOnceAsync and then adding this before the return:

var etagProperty = entry.EntityType.GetETagProperty();
if (etagProperty != null)
{
    entry.SetStoreGeneratedValue(etagProperty, response.Headers.ETag);
}

With this you could actually remove the concurrencyToken and partitionKey parameters and just calculate the values in place using entry

[jviau]

Ah okay. I was not aware of SetStoreGeneratedValue. That makes it much simpler!

[jviau]

@AndriySvyryd the store update is done.

Some things we can address if you want:

1. The concurrency test does not clean the DB between each test case. Just on each run. I use different IDs to avoid tests conflicting with one another.
2. The parameter list for CosmosClientWrapper is getting a bit unwieldy. Could consider extracting it into some transaction object? It could wrap the document and update entry, supplying the necessary things to the client wrapper on demand. e.g.:

public class CosmosUpsertRequest // name TBD
{
    public JToken Document { get; }
    public string PartitionKey { get; }
    public CosmosConcurrencyToken ConcurrencyToken { get; }
    public void ProcessResponse(ResponseMessage response) { } // can update the store values here.
}

[AndriySvyryd]

The parameter list for CosmosClientWrapper is getting a bit unwieldy.

As I said:

With this you could actually remove the concurrencyToken and partitionKey parameters and just calculate the values in place using entry

[jviau]

The parameter list for CosmosClientWrapper is getting a bit unwieldy.

As I said:

With this you could actually remove the concurrencyToken and partitionKey parameters and just calculate the values in place using entry

Good point, missed that in the first message. Took care of it.

[AndriySvyryd]

Thanks for your contribution! This is a very useful feature.

[richardrobberse]

@AndriySvyryd @jviau I guess this will be available in 3.1.2?

[AndriySvyryd]

@richardrobberse No, this will ship in 5.0.0, we don't add features in patch releases. You can also use the daily builds

[richardrobberse]

@AndriySvyryd Ouch, I see.
Few more questions:

• For when is the 5.0.0 release planned?
• Is there a way to get back the etag when a record is added?

[AndriySvyryd]

For when is the 5.0.0 release planned?

@richardrobberse November

Is there a way to get back the etag when a record is added?

You mean without this feature? You can add a property mapped to "_etag", however the value won't be used for concurrency checks.