Getting a endless HttpMessageHandler cleanup cycle

[Eilon]

From @masterjs on Tuesday, 28 August 2018 14:21:50

Hello,
I'm using the HTTP Client factory for ASPNETCORE 2.1. But now, after my second http request, I'm starting to get some logs that are getting me worried.

HttpMessageHandler expired after 120000ms for client 'default'
Starting HttpMessageHandler cleanup cycle with 1 items
BDC.ClientPortal.ApiManager> [10:11:49 DBG] Ending HttpMessageHandler cleanup cycle after 0.0342ms - processed: 0 items - remaining: 1 items
BDC.ClientPortal.ApiManager> [10:11:59 DBG] Starting HttpMessageHandler cleanup cycle with 1 items
BDC.ClientPortal.ApiManager> [10:11:59 DBG] Ending HttpMessageHandler cleanup cycle after 0.0149ms - processed: 0 items - remaining: 1 items
BDC.ClientPortal.ApiManager> [10:12:09 DBG] Starting HttpMessageHandler cleanup cycle with 1 items
BDC.ClientPortal.ApiManager> [10:12:09 DBG] Ending HttpMessageHandler cleanup cycle after 0.0025ms - processed: 0 items - remaining: 1 items
BDC.ClientPortal.ApiManager> [10:12:19 DBG] Starting HttpMessageHandler cleanup cycle with 1 items
BDC.ClientPortal.ApiManager> [10:12:19 DBG] Ending HttpMessageHandler cleanup cycle after 0.0017ms - processed: 0 items - remaining: 1 items ...

These logs are endless. Is that normal? (doesn't look normal)

thank you,
JS

***** dotnet --info *****
.NET Core SDK (reflecting any global.json):
Version: 2.1.400
Commit: 8642e60a0f

Runtime Environment:
OS Name: Windows
OS Version: 10.0.14393
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\2.1.400\

Host (useful for support):
Version: 2.1.2
Commit: 811c3ce6c0

.NET Core SDKs installed:
1.1.0 [C:\Program Files\dotnet\sdk]
1.1.4 [C:\Program Files\dotnet\sdk]
...
2.1.202 [C:\Program Files\dotnet\sdk]
2.1.400 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 1.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 1.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
....
Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Copied from original issue: dotnet/aspnetcore#3470

[Eilon]

From @rynowak on Tuesday, 28 August 2018 18:00:02

This means that there's a client/message handler that's been expired, but someone is still holding a reference to it. You can analyze what's keeping it alive by capturing a memory dump.

[mixandmatch025]

I'm experiencing the same issue, in the memory dump I don't see any HttpClient kept alive. Any other pointers on where to look?

[mixandmatch025]

After further investigation, the following object are incremented on each call to my web service:

• HttpClientHandler
• SocketsHttpHandler,
• HttpConnectionSettings
• DiagnosticsHandler
• LoggingScopeHttpMessageHandler
• ExpiredHandlerTrackingEntry

I am using a custom asp.net core middleware that on each call uses IServiceProvider to create an instance of a request handler, which uses IHttpClientFactory to create an HttpClient.

[rynowak]

If you're looking at a memory dump you should be able to find the "path to root" of one of these objects - that will show you what's keeping it around in memory. What's the "path to root" of LoggingScopeHttpMessageHandler?

[mixandmatch025]

The path to root is:
LoggingScopeHttpMessageHandler -> ExpiredHandlerTrackingEntry -> DefaultHttpClientFactory -> Dictionary<object, object> -> ServiceProviderEngineScope -> KestrelServerOptions -> Http1Connection -> AsyncTaskMethodBuilder... -> Dictionary<Int32, Task>

[rynowak]

Is there an instance of LifetimeTrackingHttpMessageHandler alive anywhere?

[rynowak]

It would be great to see the path to root of HttpClientHandler

[mixandmatch025]

Is there an instance of LifetimeTrackingHttpMessageHandler alive anywhere?

Doesn't seem to be.

It would be great to see the path to root of HttpClientHandler

HttpClientHandler -> LoggingHttpMessageHandler -> LoggingScopeHttpMessageHandler -> ExpiredHandlerTrackingEntry -> DefaultHttpClientFactory -> Dictionary<object, object> -> ServiceProviderEngineScope -> KestrelServerOptions -> Http1Connection -> AsyncTaskMethodBuilder... -> Dictionary<Int32, Task>

[rynowak]

When you're collecting these logs is your application mostly idle? Everything you've shown me so far seems to indicate that things are working correctly, but the weakreference hasn't been killed yet.

ExpiredHandlerTrackingEntry should have a strong reference to LoggingScopeHttpMessageHandler and a weak reference to LifetimeTrackingHttpMessageHandler

https://github.com/aspnet/HttpClientFactory/blob/master/src/Microsoft.Extensions.Http/ExpiredHandlerTrackingEntry.cs

Once LoggingScopeHttpMessageHandler goes out of scope, the weak reference will get expired and the 'real' handler should be eligible for disposal.

[mixandmatch025]

My app is completely idle, it has finished handling the requests. Do you have any ideas why is LoggingScopeHttpMessageHandler kept in scope, or how I can find out?

[mixandmatch025]

It seems this is the expected behavior, as it happens with an empty app that just uses IHttpClientFactory to create an HttpClient. After a lot of requests the items are cleaned. Is there a certain number of items needed before cleanup starts processing items?

[mixandmatch025]

Or maybe this has to do with the underlying connection handling by kestrel?

[davidfowl]

Kestrel handles sever resources not client resources (incoming connections not outgoing connections)