Grpc Error using .Net 7 on a Windows Server 2022 Standard

[hiou1210]

Describe the bug:

Error when testing a Google call on the CI / CD server. I cannot go further if a simple test do not pass. It was perfectly working using useGRPCcore = true. Once i removed the useGrpcCore=true, then i am supposed to use http2, based on the error, it looks like it use Http3. When i do the same test on my local machine using Windows 11, i dont have any issue.

Steps to Reproduce:
var bidOverrideRestoreEntities = criterions
.Where(crit => crit.CriterionBidType == BiddingBehaviorType.BidOverride)
.Select(crit => crit as AdGroupBidOverrideCriterion)
.Select(bidOverrides =>
new AdGroupCriterion
{
ResourceName = ResourceNames.AdGroupCriterion(googleAccountId, bidOverrides.ProviderEntityId, bidOverrides.ProviderCriterionId),
CpcBidMicros = (long)(bidOverrides.Value * (decimal)MicroAmountFactor)
});
var bidOverrideRestoreOperations = bidOverrideRestoreEntities.Select(x => new AdGroupCriterionOperation
{
Update = x,
UpdateMask = FieldMasks.AllSetFieldsOf(x)
});

foreach (var operationBatch in bidOverrideRestoreOperations.InSetsOf(2000))
{
var operationArray = operationBatch.ToArray();
var bidOverrideRequest = new MutateAdGroupCriteriaRequest
{
CustomerId = googleAccountId.ToString()
};
bidOverrideRequest.Operations.AddRange(operationArray);
service.MutateAdGroupCriteria(bidOverrideRequest);
}

Expected behavior:

No error and the bid modifier applied to the adgroup

Client library version and API version:
Client library version: 18
Google Ads API version: 15
.NET version: .Net 7
Operating system (Linux, Windows, ...) and version (if the bug is platform-specific): Windows Server 2022 Standard 20348.2227

Request/Response Logs:
GoogleAds.DetailedRequestLogs Information: 1 : [2024-02-21 21:59:40Z] -
---------------BEGIN API CALL---------------

Request

Method Name: /google.ads.googleads.v15.services.AdGroupCriterionService/MutateAdGroupCriteria
Host:
Headers: {
"x-goog-api-client": "gl-dotnet/2.0.0 gapic/18.0.0 gax/4.2.0+a8085e4f36ad24e2747b5e550f11079d4a891e78 grpc/2.46.3 gccl/3.1.0 pb/3.21.5+638779f353731a0a04496bde20d14164684c3d93",
"developer-token": "REDACTED",
"login-customer-id": "XXXXXXX",
"x-goog-request-params": "customer_id=XXXXXXX"
}

{ "customerId": "XXXXXXX", "operations": [ { "update": { "resourceName": "customers/XXXXXXXXX/adGroupCriteria/120214107244~10", "bidModifier": 1.1 }, "updateMask": "resourceName,bidModifier" } ] }

Response

Headers: {}

Fault: Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error starting gRPC call. HttpRequestException: The connection timed out from inactivity. (googleads.googleapis.com:443) QuicException: The connection timed out from inactivity.", DebugException="System.Net.Http.HttpRequestException: The connection timed out from inactivity. (googleads.googleapis.com:443)
---> System.Net.Quic.QuicException: The connection timed out from inactivity.
at System.Net.Quic.QuicConnection.HandleEventShutdownInitiatedByTransport(SHUTDOWN_INITIATED_BY_TRANSPORT_e__Struct& data)
at System.Net.Quic.QuicConnection.HandleConnectionEvent(QUIC_CONNECTION_EVENT& connectionEvent)
at System.Net.Quic.QuicConnection.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_CONNECTION_EVENT* connectionEvent)
--- End of stack trace from previous location ---
at System.Net.Quic.ValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Quic.QuicConnection.FinishConnectAsync(QuicClientConnectionOptions options, CancellationToken cancellationToken)
at System.Net.Quic.QuicConnection.ConnectAsync(QuicClientConnectionOptions options, CancellationToken cancellationToken)
at System.Net.Quic.QuicConnection.ConnectAsync(QuicClientConnectionOptions options, CancellationToken cancellationToken)
at System.Net.Http.ConnectHelper.ConnectQuicAsync(HttpRequestMessage request, DnsEndPoint endPoint, TimeSpan idleTimeout, SslClientAuthenticationOptions clientAuthenticationOptions, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.ConnectQuicAsync(HttpRequestMessage request, DnsEndPoint endPoint, TimeSpan idleTimeout, SslClientAuthenticationOptions clientAuthenticationOptions, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp3ConnectionAsync(HttpRequestMessage request, HttpAuthority authority, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in //src/Grpc.Net.Client/Balancer/Internal/BalancerHttpHandler.cs:line 114
at Grpc.Net.Client.Internal.GrpcCall2.RunCall(HttpRequestMessage request, Nullable1 timeout) in /_/src/Grpc.Net.Client/Internal/GrpcCall.cs:line 481")
at Grpc.Net.Client.Internal.GrpcCall2.GetResponseHeadersCoreAsync() in /_/src/Grpc.Net.Client/Internal/GrpcCall.cs:line 300 at Google.Ads.GoogleAds.Logging.LoggingHandler.HandleAsyncUnaryLogging[TRequest,TResponse](TRequest request, ClientInterceptorContext2 context, Task1 oldTask, AsyncUnaryCall1 call)
----------------END API CALL----------------

[RyanThomas73]

We've been receiving these QuicException: The connection timed out from inactivity. a lot recently at my day job. We've got a lot of different components but the most common ones this had been occurring for are targetting .NET 8 and using the 19.0.1 version of the google-ads-dotnet nuget package.

It appears to happen consistently at about 1 minute into the request and I believe some changes are needed so we can configure a longer connection idle timeout.

@AnashOommen - Tagging you since you appear to be the previous committer for the code in question that I believe should be updated:

Specifically - The CreateGrpNetClientChannel logic here needs to be switched to using a SocketsHttpHandler and needs to allow consumers a way to pass in a TimeSpan value for the socketsHttpHandler.PooledConnectionIdleTimeout property.

This is necessary because:

1. The HttpClientHandler class does not provide any mechanisms to specify the PooledConnectionIdleTimeout on the SocketsHttpHandler _underlyingHandler that it wraps. This causes the wrapped SocketsHttpHandler to always use the default PooledConnectionIdleTimeout which is 1 minutes on .NET 6 and greater (2 minutes on .NET 5 and lower .NET Core versions).

2. The Grpc.Net.Client.GrpcChannel object has it's own ConnectionIdleTimeout property which it will populate off of the SocketsHttpHandler.PooledConnectionIdleTimeout when using that handler type. When using the HttpClientHandler however the GrpcChannel.ConnectionIdleTimeout property is left null which will also result in falling back to a default of 1 minute

[Raibaz]

@RyanThomas73, 2 days ago we had an internal, widespread issue that caused increased latency on the whole API, not just the .NET client library, so that was likely the reason why you were getting API timeouts.

This being said, your suggestion to allow consumers to specify their own timeout make perfect sense: I'll see if I can put together a PR about that.

[RyanThomas73]

@RyanThomas73, 2 days ago we had an internal, widespread issue that caused increased latency on the whole API, not just the .NET client library, so that was likely the reason why you were getting API timeouts.

This being said, your suggestion to allow consumers to specify their own timeout make perfect sense: I'll see if I can put together a PR about that.

Thanks Raibaz. We did encounter an uptick during the latency issue but we've been encountering them regularly for several weeks now on some components. Ultimately realized it was introduced/increased when we upgraded from older .NET versions to .NET 8 due to the decrease in the default value.

[AnashOommen]

Closing this issue.

[RyanThomas73]

@AnashOommen @Raibaz Can you clarify what closing this issue means with respect to the request to allow consumers to configure the connection idle timeout value beyond the default?

Has support been added? Or has the decision been made to not support configuring the timeout?

[AnashOommen]

Oh, I misunderstood your previous comment to say this was an issue at your end. We do have a timeout parameter, but I don't think we are setting it to the PooledConnectionIdleTimeout property.

Could you promote your previous comment into its own feature request bug? I think it's easier to follow up that way.