Unobserved task exception in PrometheusHttpListener

[veleek]

Package

OpenTelemetry.Exporter.Prometheus.HttpListener

Package Version

Package Name	Version
OpenTelemetry.Exporter.Prometheus.HttpListener	1.6.0-rc1
OpenTelemetry.Instrumentation.Http	1.6.0-rc1
OpenTelemetry.Exporter.OpenTelemetryProtocol	1.6.0

Runtime Version

net6.0

Description

When shutting down a web service which uses a PromethusHttpListener, it is possible to see an unobserved task exception.

HttpListener.GetContextAsync does not take a CancellationToken, so the PrometheusHttpListener.WorkerSproc uses a .Wait with a cancellation token to handle cancellation. When the token is cancelled, an OperationCancelledException is thrown from .Wait which is handled by the loop, but the underlying get context task is not cancelled. Later on in the finally for the WorkerProc when the HttpListener is stopped this will cause that task to throw an exception which is then unobserved.

I have confirmed that this code has not changed in the most recent branches either.

Steps to Reproduce

1. Register a PrometheusHttpListener

   var meterProvider = Sdk.CreateMeterProviderBuilder()
       .AddPrometheusHttpListener(...)
       .Build();
   

2. Dispose the meter provider.

   meterProvider.Dispose();
   

3. Exit the application, and observe an unobserved exception!

Expected Result

PrometheusHttpListener should cleanly shutdown.

Actual Result

Unobserved task exception is thrown.

Additional Context

Stack trace from exception:

Unobserved task exception! Sender: System.Threading.Tasks.Task`1[System.Net.HttpListenerContext]
System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (The I/O operation has been aborted because of either a thread exit or an application request.)
 ---> System.Net.HttpListenerException (995): The I/O operation has been aborted because of either a thread exit or an application request.
   at System.Net.HttpListener.EndGetContext(IAsyncResult asyncResult)
   at System.Net.HttpListener.<>c.<GetContextAsync>b__44_1(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
   --- End of inner exception stack trace ---

The inner exception (The I/O operation has been aborted because of either a thread exit or an application request.) is expected when the listener is being shutdown.

You can replicate the behavior in LinqPad using something like this. Note that the worker loop exits cleanly, but the ctxTask is faulted and if you don't handle the unobserved exception (which LinqPad does implicitly) you'll see a crash.

CancellationTokenSource tokenSource = new CancellationTokenSource();
HttpListener httpListener = new System.Net.HttpListener();

async Task Main()
{
	Task<HttpListenerContext> ctxTask = null;
	this.httpListener.Start();
	Task workerProc = Task.Run(() =>
	{
		try
		{
			while (!this.tokenSource.IsCancellationRequested)
			{
				ctxTask = this.httpListener.GetContextAsync();
				"Waiting on getContext".Dump();
				ctxTask.Wait(this.tokenSource.Token);
				var ctx = ctxTask.Result;

			}
		}
		catch (OperationCanceledException)
		{
			"Worker Loop is exiting...".Dump();
		}
		catch (Exception e)
		{
			e.Dump("Unhandled exception");
		}
		finally
		{
			"Stopping listener".Dump();
			this.httpListener.Stop();
		}

		"Exiting Cleanly".Dump();
	});

	await Task.Delay(500);
	this.tokenSource.Cancel();
	await Task.Delay(500);
	ctxTask.Status.Dump("After");
	ctxTask.Exception.Dump();
	
	"Done!".Dump();
}

[veleek]

The correct solution here is probably to stop the HttpListener at the same time the token is cancelled (or maybe even just stop using the cancellation token in the worker loop) and handle the HttpListenerException.

try
{
  // ...
}
catch (HttpListenerException e) when (e.ErrorCode == 995)
{
  // Expected exception when HttpListener is shutting down.
}

[reyang]

Package

OpenTelemetry

@veleek would you share why did you choose "OpenTelemetry" package instead of "OpenTelemetry.Exporter.Prometheus.HttpListener"? I'm trying to see if the issue template needs to be fixed/improved.

[veleek]

@reyang Nope, no reason. I just totally missed that when filling out the form. Updated now. Thank you!

[reyang]

@reyang Nope, no reason. I just totally missed that when filling out the form. Updated now. Thank you!

Thanks @veleek! I guess the dropdown is probably not very noticeable when it is sitting beside textareas. Let me see if I can improve it.