[Azure Function Isolated] Cosmos change feed listener tries to start before CreateContainerIfNotExistsAsync

[andersson09]

For isolated azure functions, I believe that the cosmos change feed tries to listen before containers have been created. How do you execute code before the runtime loads the function?

Thanks

Setup:

Program.cs

var host = new HostBuilder()
...
.Build()

var client = host.Services.GetRequiredService<CosmosClient>();
var databaseResponse = await client.CreateDatabaseIfNotExistsAsync(Configuration["DbName"]);
var database = databaseResponse.Database;

await database.CreateContainerIfNotExistsAsync("container", "/id");

host.Run();

Function.cs

[Function("DoSomething")]
public async Task Run(
    [CosmosDBTrigger("%DbName%", "container", 
    ConnectionStringSetting = "CosmosDbConnectionString", 
    LeaseCollectionName = "leases", 
    CreateLeaseCollectionIfNotExists = true)] IReadOnlyList<Entity> input)
{
} 

Error message on local machine:

The listener for function 'DoSomething' was unable to start. Microsoft.Azure.WebJobs.Extensions.CosmosDB: Either the source collection 'container' (in database 'db') or the lease collection 'leases' (in database 'db') does not exist. Both collections must exist before the listener starts. To automatically create the lease collection, set 'CreateLeaseCollectionIfNotExists' to 'true'

Versions

Microsoft.Azure.Functions.Worker.Sdk 1.3.0
Microsoft.Azure.Functions.Worker.Extensions.Http 3.0.13
Microsoft.Azure.Cosmos 3.26.1
Microsoft.Azure.Functions.Worker.Extensions.CosmosDB 3.0.9

[ealsur]

Reference: Azure/azure-cosmos-dotnet-v3#3150 (comment)

Extensions do not control the events that happen outside of the Extension scope. This repo just contains the Extensions (in this case the CosmosDBTrigger) and the Trigger will start when the Host tells it to start, but it has no idea when that is and what you are trying to do is outside of the Extensions.

You are trying to run some code on the Host initialization.

One idea would be to attempt to execute things before the Start is called, maybe like: https://github.com/Azure/azure-webjobs-sdk-extensions/blob/dev/test/WebJobs.Extensions.CosmosDB.Tests/CosmosDBEndToEndTests.cs#L93-L120, but someone from the Host repo might be better advisors.

[andersson09]

@ealsur Do you mind transferring the issue since I don't think I have permissions.

[kshyju]

@andersson09 Thanks for the sample code. I can repro the issue. As you mentioned in the issue, the root cause for the problem is the extension listener starting before the language worker (dotnet isolated project) process starts. So, the listener gets the error the first time since the container does not exist.

As a workaround, you could consider moving the database/container creation step as part of your release/deployment process where the database/container gets created before the app is deployed.

We will tag this use case for discussion as part of other reliability improvements we are working on and will share an update as we make progress. Thanks!

[kshyju]

cc @pragnagopa @liliankasem @fabiocav

[andersson09]

As a workaround, you could consider moving the database/container creation step as part of your release/deployment process where the database/container gets created before the app is deployed.

I feel release processes are already complicated enough and would rather have this in code if its possible. Also makes creating databases much easier on local machine.

We will tag this use case for discussion as part of other reliability improvements we are working on and will share an update as we make progress. Thanks!

Thanks this would be great. Is there a rough ETA?

[kshyju]

@andersson09 Update on this. Discussed this with the team. We do not have any current plans to support this from the dotnet-worker or host. This could be an enhancement request which the cosmos extension team could consider adding to the extension (similar to the creation of the lease collection container if not exist). cc @ealsur

As mentioned in my last comment, you should consider moving this container creation as a step in your deployment pipeline.

[brettsam]

Does the trigger initialization retry? I thought it should retry and eventually start working...

As for adding a CreateIfNotExists, I believe we explicitly excluded hat option from the Trigger b/c it wasn't a good design choice. I don't recall exactly why, but you can see that the output binding does allow this.

Ultimately, you're trying to start listening to something that doesn't exist, and the Extension and Worker are racing to start at the same time, resulting in what you're seeing. The source of the trigger really needs to exist before the host starts and today there's no way to delay that or run custom code before listeners begin.