CRON Function stops randomly in consumption plan

[crgarcia12]

Check for a solution in the Azure portal

I have multiple functions apps in different regions. I deploy the same function to all of them using func azure functionapp publish. Functions run for a while but then they stop firing.
All the Apps share the same storage account

Investigative information

- Timestamp: 2020-07-07 05:08
- Function App version: 2.0.13900.0
- Function App name: crgar-apim-synthetictransaction-australiaeast-app
- Function name(s) (as appropriate): SyntheticTransaction
- Invocation ID: Last OperationId: 377133debf77ee4991a741fff109e639
**- Region: ** AustraliaEast, Brazil, West Europe, Central US, etc

Repro steps

My function App has this CRON expression:
public static async Task Run([TimerTrigger("0 */2 * * * *")]TimerInfo myTimer, ILogger log)

This is the function.json that I see in the portal

{
  "generatedBy": "Microsoft.NET.Sdk.Functions-3.0.3",
  "configurationSource": "attributes",
  "bindings": [
    {
      "type": "timerTrigger",
      "schedule": "0 */2 * * * *",
      "useMonitor": true,
      "runOnStartup": false,
      "name": "myTimer"
    }
  ],
  "disabled": false,
  "scriptFile": "../bin/SyntheticTransactions.dll",
  "entryPoint": "SyntheticTransactions.SyntheticTransaction.Run"
}

Diagnostics shows this report:

Is runOnStartup enabled? | False
TimeZone | (UTC) Coordinated Universal Time
Cron Expression | 0 */2 * * * *
Latest 10 executions | 
2020-07-06T06:34:00.0036339+00:00  
2020-07-06T06:32:00.0036853+00:00  
2020-07-06T06:30:00.0008163+00:00  
2020-07-06T06:28:00.0061657+00:00  
2020-07-06T06:25:59.9951292+00:00  
2020-07-06T06:24:00.0208449+00:00  
2020-07-06T06:21:59.9885030+00:00  
2020-07-06T06:19:59.9966712+00:00  
2020-07-06T06:18:00.0105661+00:00  
2020-07-06T06:15:59.9981813+00:00

Last execution was one day ago

Expected behavior

Function app to run every two minutes.

Actual behavior

Function App runs several times, and then it does not get triggered anymore

Known workarounds

I can manually invoke the function (using the Test) and that seems to unblock it and start firing for a while.

Related information

*** Programming language used:** C#
*** Links to source:** https://github.com/crgarcia12/azure-apim-demo/tree/master/backend/synthetic-transactions/SyntheticTransactions

Source

[FunctionName(nameof(SyntheticTransaction))]
        public static async Task Run([TimerTrigger("0 */2 * * * *")]TimerInfo myTimer, ILogger log)
        {
            log.LogInformation($"{nameof(SyntheticTransaction)} trigger function v5 executed at: '{DateTime.Now}'");

            string endpoint = Environment.GetEnvironmentVariable("ApplicationEndpoint");

            using (HttpClient client = new HttpClient())
            {
                for (int i = 0; i < 100; i++)
                {
                    string response = await client.GetStringAsync(endpoint);
                    log.LogInformation($"{nameof(SyntheticTransaction)} responded '{endpoint}' trigger '{i}': '{response}'");
                }
            }
        }

[ankitkumarr]

Hi @crgarcia12, did you happen to delete your function app? I couldn't find it.

Looking at old logs, I see that your app ran successfully until the time you mentioned, and then it stopped running the function, as you said. I see errors in the logs with functions host trying to get the lease to monitor and run your function. Timer trigger functions acquire Azure storage lease in order to identify the primary instance that should run your function when the timer goes. You can find some more doc here.
However, because the instance failed to grab the lease, I think it stopped monitoring the timer, and didn't execute. Then, I see that you function app starting executing properly the next day and continued to execute as you'd expect.

My question to you is, do you know if something happened in your storage account, that was configured with the function app in a way that would interfere with lease management?

[ankitkumarr]

Actually, I just realized that your function app name is fairly long (over 32 chars). Do you happen to be sharing your storage account between mutliple function apps? Or are you using slots?

We have a known issue where if the function app name is longer than 32 chars, it can cause collisions in the storage account (if multiple function apps have same 32 char prefix). The collision would mean that only one of the collided function apps would have the lease I mentioned earlier. Here's the issue with more info.

I am almost certain that's what is likely causing the issue. But I will wait for your response to see if you indeed have multiple apps with same prefix using the same storage, or if you are using slots. If so, you can follow the workarounds mentioned in the other issue.

Thank you for reporting this!

@fabiocav / @mathewc, this may be another case of Host ID collision. Just fyi.

[crgarcia12]

Yes! I am sharing the storage account! so that might be it. The function started again the next day because I manually triggered it with the "Test your function" in the portal - Thanks!

[ankitkumarr]

Thanks for the confirmation! I will close this issue in favor of the other one. Please let us know if you follow-up questions and we can re-open the issue.