Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unknown error 258 : The timeout period elapsed prior to completion #1435

Closed
Head-Strong opened this issue Apr 9, 2020 · 10 comments
Closed

Comments

@Head-Strong
Copy link

Hi Team,

We are facing timeout issue when try to insert data in a table. The issue is intermittent. The table has only 4 columns and primary key as well.

Details of the code and Infra: -

  1. Connection timeout is : 90
  2. Command Timeout is default (30).
  3. Application is in dotnet core 2.2
  4. Dapper version is 1.50.5
  5. Sql Version is 12.0.6372.1
  6. Application deployed in Linux container in PCF (Ubuntu 18.04 (Bionic Beaver))

Error details: -
ExceptionMessage: 'Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.';
Inner exception message: 'Unknown error 258'
Message: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

Thanks in advance.

@NickCraver
Copy link
Member

Without code, we won't be able to help here. All I can tell you based on that info is: it's timing out. This isn't likely due to Dapper, but more likely due to a table lock on the SQL side preventing the insert for an extended period of time. To determine this you'll need to look on the server when it's happening, e.g. via sp_who or another route.

Note: you don't need the error to see it most of the time. Generally speaking, a situation like this is going to occur far more often than you're getting a timeout...it's just not stalling long enough to timeout in the other cases. I'd advise hopping on the server and observing for a bit to see who's blocking who.

@frankyuan
Copy link

frankyuan commented Jul 9, 2020

I met a similar issue, the log is like below, only a little part of queries has this issue. After google, found https://stackoverflow.com/questions/57270245/sql-server-dbcommand-timeout-with-net-core-container-under-load, the author said it is a threading issue (I'm not sure if it is the reason here). Dapper's version is 2.0.35. The application runs in Linux docker.

Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.\n
---> System.ComponentModel.Win32Exception (258): Unknown error 258\n
at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)\n
at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)\n
at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)\n
at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()\n
at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()\n
at Microsoft.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()\n
at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadByteArray(Span`1 buff, Int32 len, Int32& totalRead)\n
at Microsoft.Data.SqlClient.TdsParser.TrySkipValue(SqlMetaDataPriv md, Int32 columnOrdinal, TdsParserStateObject stateObj)\n
at Microsoft.Data.SqlClient.TdsParser.TrySkipRow(_SqlMetaDataSet columns, Int32 startCol, TdsParserStateObject stateObj)\n
at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)\n
at Microsoft.Data.SqlClient.SqlDataReader.TryReadInternal(Boolean setTimeout, Boolean& more)\n
at Microsoft.Data.SqlClient.SqlDataReader.ReadAsync(CancellationToken cancellationToken)\n
--- End of stack trace from previous location where exception was thrown ---\n
at Dapper.SqlMapper.QueryAsync[T](IDbConnection cnn, Type effectiveType, CommandDefinition command) in /_/Dapper/SqlMapper.Async.cs:line 437\n

@mgravell
Copy link
Member

mgravell commented Jul 9, 2020

Well, ReadSniSyncOverAsync doesn't sound promising - that does indeed sound like it could contribute to a "spiral of death" when under load, but: that is not a Dapper thing - that's the underlying ADO.NET provider, that we don't have any influence over.

@frankyuan
Copy link

Got it, thanks, maybe I need to ask in SqlClient repo.

@NickCraver
Copy link
Member

Given this is a SQL issue not specific to Dapper, closing out here and watching dotnet/SqlClient#647 :)

@ephra-samuel
Copy link

have the some problem here,
with inner and stack trace here :
Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.System.ComponentModel.Win32Exception (258): Unknown error 258
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)\n at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)\n at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)\n at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)\n at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)\n at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource1 completion, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite, String methodName)\n
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()\n
at Dapper.SqlMapper.ExecuteCommand(IDbConnection cnn, CommandDefinition& command, Action2 paramReader) in C:\\projects\\dapper\\Dapper\\SqlMapper.cs:line 2806\n at Dapper.SqlMapper.ExecuteImpl(IDbConnection cnn, CommandDefinition& command) in C:\\projects\\dapper\\Dapper\\SqlMapper.cs:line 568\n at Dapper.SqlMapper.Execute(IDbConnection cnn, String sql, Object param, IDbTransaction transaction, Nullable1 commandTimeout, Nullable`1 commandType) in C:\projects\dapper\Dapper\SqlMapper.cs:line 441\n

@luarvic
Copy link

luarvic commented Apr 8, 2021

Have the same problem:

Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
 ---> System.ComponentModel.Win32Exception (258): Unknown error 258
   at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__164_0(Task`1 result)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location where exception was thrown ---

@cuongvu0203
Copy link

Have same problem here:
Anyway to fix this :(

18/03/2022 07:22:34 - Error - Qi.TnNew.Core.Interfaces.IGlobalExceptionHandlerError: On url: http://study.hanoi.edu.vn/huong-dan-thi/on-tap-thcs-mon-tieng-anh-9-e-so-18-302 | 1173060 System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (258): Unknown error 258 at System.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__126_0(Task1 result) at System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at Dapper.SqlMapper.QueryRowAsync[T](IDbConnection cnn, Row row, Type effectiveType, CommandDefinition command) in /_/Dapper/SqlMapper.Async.cs:line 473 at Qi.TnNew.Infrastructure.Data.TestDetailsLogData.GetRawTestDetailsByKey(String key) in /src/src/Qi.TnNew.Infrastructure/Data/TestDetailsLogData.cs:line 34 at Qi.TnNew.Core.Services.ExamService.CacheGetRawTestDetailsAsync(String detailsLogKey) in /src/src/Qi.TnNew.Core/Services/ExamService.cs:line 504 at Qi.TnNew.Core.Services.ExamService.CreateTestDetailsAsync(Int64 examId, Int64 userId, String detailsLogKey) in /src/src/Qi.TnNew.Core/Services/ExamService.cs:line 520 at Qi.TnNew.Core.Services.ExamService.CreateTestDetailForUserAsync(Int64 examId, Int64 userId, String detailsLogKey) in /src/src/Qi.TnNew.Core/Services/ExamService.cs:line 176 at Qi.TnNew.Web.Controllers.HomeController.<TestGuide>g__TakeAnExamAsync|24_1(Int64 userId, Int64 examId, String examTypeCode) in /src/src/Qi.TnNew.Web/Controllers/HomeController.cs:line 514 at Qi.TnNew.Web.Controllers.HomeController.TestGuide(Int64 id) in /src/src/Qi.TnNew.Web/Controllers/HomeController.cs:line 515 at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask1 actionResultValueTask) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Logged|17_1(ResourceInvoker invoker) at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger) at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult) at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context) at Qi.TnNew.Web.Startup.<>c.<b__8_1>d.MoveNext() in /src/src/Qi.TnNew.Web/Startup.cs:line 177 --- End of stack trace from previous location --- at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task) ClientConnectionId:017c41d4-d10b-445b-b356-9197e1814fd0 Error Number:-2,State:0,Class:11`

@dazinator
Copy link

dazinator commented Oct 9, 2023

There are two different errors / stack traces here.

The ReadSniSyncOverAsync error is one problem and is tracked here: dotnet/SqlClient#647

However @cuongvu0203 @luarvic issueand my issue is a different stack trace nothing to do with ReadSniSyncOverAsync.

In my case I am using EF Core with Azure SQL. The thing "timing out" is a select query. However I can't see anything wrong with the query and when it works it runs quite quickly. My suspicion is this:

  • By getting the DbConnection from EF Core DbContext and passing it to dapper to query with directly - I assume this will bypass EF Core's notion of an "execution strategy" and thus bypass its "retry policies"? https://learn.microsoft.com/en-us/ef/core/miscellaneous/connection-resiliency
  • Azure SQL instance I am using can go down after a period of inactivity, then on first query it starts up again. The EF Core provider for SQL has a retry on failure execution policy configured, which handles specific transient error codes and implements retries. I am guessing that this logic may handle timeouts for azure sql instances caused by when its starting up.. but as dapper is bypassing this, it won't be handled?
 builder = builder.UseSqlServer(connectionString, (a) =>
                    {
                        a.EnableRetryOnFailure();

/// <summary>
        ///     <para>
        ///         Configures the context to use the default retrying <see cref="IExecutionStrategy" />.
        ///     </para>
        ///     <para>
        ///         This strategy is specifically tailored to SQL Server (including SQL Azure). It is pre-configured with
        ///         error numbers for transient errors that can be retried.
        ///     </para>
        ///     <para>
        ///         Default values of 6 for the maximum retry count and 30 seconds for the maximum default delay are used.
        ///     </para>
        /// </summary>
        /// <remarks>
        ///     See <see href="https://aka.ms/efcore-docs-connection-resiliency">Connection resiliency and database retries</see>
        ///     for more information.
        /// </remarks>
        public virtual SqlServerDbContextOptionsBuilder EnableRetryOnFailure()
            => ExecutionStrategy(c => new SqlServerRetryingExecutionStrategy(c));

I am going to try wrapping my dapper query in its own polly retry to see if this "solves" the issue, and then see if I can tie up the logs with the azure sql "pausing" on inactivity.

@CesarD
Copy link

CesarD commented Jan 19, 2024

There are two different errors / stack traces here.

The ReadSniSyncOverAsync error is one problem and is tracked here: dotnet/SqlClient#647

However @cuongvu0203 @luarvic issueand my issue is a different stack trace nothing to do with ReadSniSyncOverAsync.

In my case I am using EF Core with Azure SQL. The thing "timing out" is a select query. However I can't see anything wrong with the query and when it works it runs quite quickly. My suspicion is this:

  • By getting the DbConnection from EF Core DbContext and passing it to dapper to query with directly - I assume this will bypass EF Core's notion of an "execution strategy" and thus bypass its "retry policies"? https://learn.microsoft.com/en-us/ef/core/miscellaneous/connection-resiliency
  • Azure SQL instance I am using can go down after a period of inactivity, then on first query it starts up again. The EF Core provider for SQL has a retry on failure execution policy configured, which handles specific transient error codes and implements retries. I am guessing that this logic may handle timeouts for azure sql instances caused by when its starting up.. but as dapper is bypassing this, it won't be handled?
 builder = builder.UseSqlServer(connectionString, (a) =>
                    {
                        a.EnableRetryOnFailure();
/// <summary>
        ///     <para>
        ///         Configures the context to use the default retrying <see cref="IExecutionStrategy" />.
        ///     </para>
        ///     <para>
        ///         This strategy is specifically tailored to SQL Server (including SQL Azure). It is pre-configured with
        ///         error numbers for transient errors that can be retried.
        ///     </para>
        ///     <para>
        ///         Default values of 6 for the maximum retry count and 30 seconds for the maximum default delay are used.
        ///     </para>
        /// </summary>
        /// <remarks>
        ///     See <see href="https://aka.ms/efcore-docs-connection-resiliency">Connection resiliency and database retries</see>
        ///     for more information.
        /// </remarks>
        public virtual SqlServerDbContextOptionsBuilder EnableRetryOnFailure()
            => ExecutionStrategy(c => new SqlServerRetryingExecutionStrategy(c));

I am going to try wrapping my dapper query in its own polly retry to see if this "solves" the issue, and then see if I can tie up the logs with the azure sql "pausing" on inactivity.

@dazinator Have you got this working? I got a similar scenario myself... Trying to understand if it actually is as you explained. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants