Hangfire.Storage.DistributedLockTimeoutException: Timeout expired

[trihanhcie]

Hi, I'm using Hangfire with Mssql.
I keep having timeout issues on recurring jobs.

Hangfire.Storage.DistributedLockTimeoutException
Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'HangFire:HangfireBootstrapper.BackgroundTaskRunner.RunAll' resource.

Hangfire.Storage.DistributedLockTimeoutException: Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'HangFire:HangfireBootstrapper.BackgroundTaskRunner.RunAll' resource.
at Hangfire.SqlServer.SqlServerDistributedLock.Acquire(IDbConnection connection, String resource, TimeSpan timeout)
at Hangfire.SqlServer.SqlServerConnection.AcquireLock(String resource, TimeSpan timeout)
at Hangfire.SqlServer.SqlServerConnection.AcquireDistributedLock(String resource, TimeSpan timeout)
at Hangfire.DisableConcurrentExecutionAttribute.OnPerforming(PerformingContext filterContext)
at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation)

Any idea how to fix that?

Thanks

[VesselinVassilev]

I am having the same issue. I have this attribute on my method:

[DisableConcurrentExecution(timeoutInSeconds: 10 * 60)]

In my case it is a recurring task that executes every hour.

[Echarnus]

Having the same problem over here. Is there some configuration missing? Am using RavenDB for storage.

Hangfire.Server.DelayedJobScheduler:Debug: An exception was thrown during acquiring distributed lock on the locks:schedulepoller resource within 60 seconds. The scheduled jobs have not been handled this time.
It will be retried in 15 seconds

Hangfire.Storage.DistributedLockTimeoutException: Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'HangFire/locks:schedulepoller' resource.
at Hangfire.Raven.DistributedLocks.RavenDistributedLock.Acquire(TimeSpan timeout)
at Hangfire.Raven.DistributedLocks.RavenDistributedLock..ctor(RavenStorage storage, String resource, TimeSpan timeout, RavenStorageOptions options)
at Hangfire.Raven.RavenConnection.AcquireDistributedLock(String resource, TimeSpan timeout)
at Hangfire.Server.DelayedJobScheduler.UseConnectionDistributedLock[T](JobStorage storage, Func`2 action)
Hangfire.Server.RecurringJobScheduler:Debug: An exception was thrown during acquiring distributed lock the recurring-jobs:lock resource within 60 seconds. The recurring jobs have not been handled this time.

Hangfire.Storage.DistributedLockTimeoutException: Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'HangFire/recurring-jobs:lock' resource.
at Hangfire.Raven.DistributedLocks.RavenDistributedLock.Acquire(TimeSpan timeout)
at Hangfire.Raven.DistributedLocks.RavenDistributedLock..ctor(RavenStorage storage, String resource, TimeSpan timeout, RavenStorageOptions options)
at Hangfire.Raven.RavenConnection.AcquireDistributedLock(String resource, TimeSpan timeout)
at Hangfire.Server.RecurringJobScheduler.UseConnectionDistributedLock(JobStorage storage, Func`2 action)

[FrenchMasta]

Hi has anyone found a solution to this yet?

[kashyapus]

Happening for me too. I wish to find a solution to this. Again, it's only for long running jobs that run into each other.

[davidrevoledo]

Having the same issue w a Long running job

[kashyapus]

Set QueuePollInterval to 30 and it resolved the issue for me.

[urbankrzysztof]

@kashyapus u mean QueuePollInterval = TimeSpan.FromSeconds(30) (30 seconds or other value u meant?
Thanks in advance for answer :)

[kashyapus]

@kashyapus u mean QueuePollInterval = TimeSpan.FromSeconds(30) (30 seconds or other value u meant?
Thanks in advance for answer :)

Yes you are correct. TimeSpan.FromSeconds(30)

[MTrachsel]

What does QueuePollInterval when set to a higher value actually do? Does it decrease the amount of polls per timespan?

[weirdyang]

Did anyone manage to resolve this?

I tried increasing QueuePollInterval to 35seconds and still getting this issue.

[JAspeling]

Any feedback on this? We are also getting this error on a recurring job.

[weirdyang]

For my case, it was because I called app.UseHangfireServer(), when I've already added services.AddHangfireServer(); in the configure method above. Removing that line solved this issue for me.

[MustafaModern]

Everything works perfectly in development mode; this only happens to me when I deploy my service to the service. I have the following configuration:
[AutomaticRetry(Attempts = 2)]
[DisableConcurrentExecution(0)]

[hippieZhou]

the same to me , I also got this issue in the Production environment:

Stopped program because of exception||Hangfire.Storage.DistributedLockTimeoutException: Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'HangFire:lock:recurring-job:PeriodicCleanupFileStorageJob' resource.
   at Hangfire.SqlServer.SqlServerDistributedLock.Acquire(IDbConnection connection, String resource, TimeSpan timeout)
   at Hangfire.SqlServer.SqlServerConnection.AcquireLock(String resource, TimeSpan timeout)
   at Hangfire.SqlServer.SqlServerConnection.AcquireDistributedLock(String resource, TimeSpan timeout)
   at Hangfire.RecurringJobExtensions.AcquireDistributedRecurringJobLock(IStorageConnection connection, String recurringJobId, TimeSpan timeout)
   at Hangfire.RecurringJobManager.AddOrUpdate(String recurringJobId, Job job, String cronExpression, RecurringJobOptions options)
   at Hangfire.RecurringJobManagerExtensions.AddOrUpdate(IRecurringJobManager manager, String recurringJobId, Job job, String cronExpression, TimeZoneInfo timeZone, String queue)

[mortenmoulder]

I'm getting the same errors as the people above.

@odinserj Can you please look into it? It's quite annoying and it's basically breaking our entire environment.

[odinserj]

Please try to use Microsoft.Data.SqlClient package of the latest version instead as shown here – #1735 (comment). There are strange problems with connections in System.Data.SqlClient that's solved by using MD.SqlClient, and connection can hold locks.

[odinserj]

All the recent problems with SQL Server were fixed by using that new package, some reported to support, and one of them is #2065. So there's a high chance this problem can also be resolved by a simple upgrade.

[Excalib88]

Did you resolve it? I am trying call service from service and receive error too.

[nicpon]

Everything works perfectly in development mode; this only happens to me when I deploy my service to the service. I have the following configuration:
[AutomaticRetry(Attempts = 2)]
[DisableConcurrentExecution(0)]
@MustafaModern
Have you found solution to your problem ?

[ChetanChandan]

@odinserj i am facing the same issue "Hangfire.Storage.DistributedLockTimeoutException
Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the resource."

can you please let me know if this issue is fixed or not. Using attributes like
[AutomaticRetry(Attempts = 0)]
[DisableConcurrentExecution(timeoutInSeconds: 30)]

[nycgavin]

I was having the same issue, there's nothing wrong with the application, it's just that this is a recurring job that runs once per 15 minutes. The programmer added the attribute to ensure that only 1 instance is running in case if the job runs for like an hour. if the next job runs before the first job finish, you will see that error in the hangfire admin site

[hheexx]

Happened to me today. Recurring jobs all stopped. Lock got stuck.
I see non-mssql database backends have locks table that you have to empty.
On MSSQL there is no locks table, they probably use some builtin locking system.
In the end restarting the database engine helped....