Improve Timer trigger behavior with host lock

[michaeldaw]

Hi,

I've found that if I'm testing a Function locally that uses a Timer trigger, it doesn't always get run by the CLI. Typically, when running a timer trigger function, the CLI will output the clock time for the next five times that the function is scheduled to run. However, if you end a session of func.exe and then quickly start it again, the timer triggered functions will not be added to the job.

[lindydonna]

@michaeldaw Can you provide more specific repro steps? Are you saying that restarting func.exe with a short interval will not register the timer correctly?

[michaeldaw]

Hi Donna,

Sorry for the awkward wording in my original post, but your response is essentially correct. Here is a screenshot of a command prompt where I start a local host with a single Timer function that I haven't modified from the default (created using the Visual Studio tooling).

I start the host using func host start. Everything seems to work normally. The host starts, the timer function is run once, and a schedule is written out to the console indicating the next 5 scheduled occurrences. At the appropriate time, the function is again executed.

I then cancelled the host and immediately again ran func host start. The host "finds" the timer function, but neither immediately executes it, nor displays the next 5 schedules occurrences. If I wait until the time at which the next execution should occur, nothing seems to happen.

However, if I cancel the host and wait for a period of time (a minute or so) before again executing func host start, the function and schedule will be properly detected.

I've been seeing this behavior occur consistently for as long as I've been using the Functions CLI. I hope this explanation is of a little more help. Let me know if I can provide further details.

[lindydonna]

@michaeldaw That is rather odd behavior! I'll try to repro and see what's going on. I suspect it might have to do with the underlying behavior of the runtime, but I'll find out.

[markmonster]

I've experienced this behavior as well. Seems to happen more often when you stop the Azure Function host pretty soon after start, and immediately restarting.

[lindydonna]

@mathewc Could you weigh in on the behavior here? I'm guessing there's something specific to the script host going on.

[AdallomRoy]

I'm having the exact same behavior.

[lindydonna]

It turns out the CLI should do some special work to make the timer trigger behave as expected.

From @mathewc:

This is likely due to the fact that on startup, we acquire a listener level blob lease lock – this is how we ensure that across scaled out instances, only one instance is running the timer. If you stop then restart quickly, there can be a delay until the previous lock expires. See https://github.com/Azure/azure-webjobs-sdk-extensions/wiki/TimerTrigger#local-development for details.

Also, this lock is based on host ID, so if you’re using the same host ID local as you are in production, and production is running, the local won’t acquire the lock and won’t run. All by design.

[michaeldaw]

Thank you for the explanation, Donna. It sounds like we could configure the host.json file in local development to change the ListenerLockPeriod setting according to this schema: https://github.com/Azure/azure-webjobs-sdk-script/wiki/host.json
Being able to set that to the minimum 15 seconds as described in the link from mathewc will definitely be of help for the time being.
Thanks for looking in to it.

[michaeldaw]

Just to follow up, I've been using the 15 second "ListenerLockPeriod" setting described above and it significantly improves the development/debugging experience. In case it's helps anyone, here's an example host.json file that will put this setting in to effect:

{
  "singleton": {
    "listenerLockPeriod": "00:00:15" 
  }
}

[Inspyro]

Can you clarify wehter this setting will have negative affects on azure (online) as it is in the host.json file? Is there a reason for the default being higher than 15s?

[mathewc]

In steady state production, you want to accept the default of 60s because it will reduce the number of lease renewal attempts the host has to do. The logic is to start renewal attempts once half the interval has expired, speeding up renewal attempts as the expiry gets closer.

You shouldn't have to set the default to 15s yourself - the CLI should be doing that automatically, putting the host into "debug mode" optimized for local development. @ahmelsayed You're right that you only want that lower value for local debugging, not production.
88
You can set a local environment variable AzureWebJobsEnv to value "Development" (v1 host reads/applies this here). This will configure the lock period to a low debug value as you're doing, and will also set other values for optimal local development. Details here.

[Inspyro]

@mathewc Thanks for the clarification

[ahmelsayed]

Closing extenal issue.