Test outcome different when running normal vs. running with debugger attached

[Skyppid]

Well, I have the following (integration) test which has multiple arguments specified for different cases:

[Test]
[Arguments(SecurityUserRoles.Admin, false)]
[Arguments(SecurityUserRoles.Customer, false)]
[Arguments(SecurityUserRoles.Employee, false)]
[Arguments(SecurityUserRoles.BI, true)]
public async Task Fail_403_NonBIUsers(string role, bool isAuthenticated)
{
    // Arrange
    await using var scope = Factory.Services.CreateAsyncScope();
    scope.ServiceProvider.GetRequiredService<ITestUserProvider>().SetCurrent(MockUser.Generate([role]));
    var client = Factory.CreateClient();

    // Test
    var response = await client.GetAsync($"{CONTROLLER_ROUTE}/completion-infos");

    // Assert
    await Assert.That(response.StatusCode).IsEqualTo(isAuthenticated ? HttpStatusCode.OK : HttpStatusCode.Unauthorized);
}

When I run it normally it fails with this exception:

Cannot access a disposed object.
Object name: 'IServiceProvider'.
at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(ServiceIdentifier serviceIdentifier, ServiceProviderEngineScope serviceProviderEngineScope)
at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType)
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.CreateAsyncScope(IServiceProvider provider)
at CHS.Tests.IT.IntegrationTests.BIController.CompletionInfoEndpointTests.Fail_403_NonBIUsers(String role, Boolean isAuthenticated) in ***\CompletionInfoEndpointTests.cs:line 34
at TUnit.Core.AsyncConvert.Convert(Func1 action) at TUnit.Core.DiscoveredTest1.ExecuteTest(CancellationToken cancellationToken)
at TUnit.Engine.Helpers.Timings.Record(String name, TestContext context, Func1 action) at TUnit.Engine.Services.TestInvoker.Invoke(DiscoveredTest discoveredTest, CancellationToken cancellationToken, List1 cleanupExceptions)
at TUnit.Engine.Services.TestInvoker.Invoke(DiscoveredTest discoveredTest, CancellationToken cancellationToken, List1 cleanupExceptions) at TUnit.Engine.Services.SingleTestExecutor.ExecuteTestMethodWithTimeout(DiscoveredTest discoveredTest, CancellationToken cancellationToken, List1 cleanupExceptions)
at TUnit.Engine.Services.SingleTestExecutor.ExecuteWithCancellationTokens(DiscoveredTest discoveredTest, List1 cleanupExceptions) at TUnit.Engine.Services.SingleTestExecutor.ExecuteWithRetries(DiscoveredTest discoveredTest, List1 cleanupExceptions)
at TUnit.Engine.Services.SingleTestExecutor.ExecuteWithRetries(DiscoveredTest discoveredTest, List`1 cleanupExceptions)
at TUnit.Engine.Services.SingleTestExecutor.ExecuteTestInternalAsync(DiscoveredTest test, ITestExecutionFilter filter, ExecuteRequestContext context, Boolean isStartedAsDependencyForAnotherTest)
at TUnit.Engine.Services.SingleTestExecutor.ExecuteTestInternalAsync(DiscoveredTest test, ITestExecutionFilter filter, ExecuteRequestContext context, Boolean isStartedAsDependencyForAnotherTest)
at TUnit.Engine.Services.SingleTestExecutor.ExecuteTestInternalAsync(DiscoveredTest test, ITestExecutionFilter filter, ExecuteRequestContext context, Boolean isStartedAsDependencyForAnotherTest)
at TUnit.Engine.Services.SingleTestExecutor.ExecuteTestInternalAsync(DiscoveredTest test, ITestExecutionFilter filter, ExecuteRequestContext context, Boolean isStartedAsDependencyForAnotherTest)
at TUnit.Engine.Services.TestsExecutor.ProcessTest(DiscoveredTest test, ITestExecutionFilter filter, ExecuteRequestContext context, CancellationToken cancellationToken)

When I debug this test case it does not fire this exception but instead run through to the assertions and fail due to an expected issue (test setup not yet done correctly).

The weird thing is: When I run any other test case on this test I get success back. Only the one that should actually succeed fails with this exception. It fails the assertion if debugged through.

What could be the issue? I don't understand why one scenario would cause a disposed exception while others work fine. And why does it not fail when I debug through?

Seems like there is a bug in TUnit. Could be some kind of race condition?

[thomhurst]

Would you be able to create a reproduction repo? Without knowing what the factory and services you're using are I can't really replicate it

[Skyppid]

Not sure how I could do this without basically sharing our whole project. It's an integration test so each and every service is loaded into it. Trying to strip the service but keep the setup of services is quite tedious and in my experience often so invasive that potential bugs appear or disappear by these changes.

I hoped the odd scenario and the stacktrace might already help giving a clue as to what could be wrong. Is there anything I can do to get more info? I mean it's odd that something should be disposed mid-test. The logs are not that easy to read but there are some oddities to be found. To me it seems like there's something wrong with the injection maybe?

The test class inherits:

public abstract class TestBase
{
    [ClassDataSource<ServiceFactory>(Shared = SharedType.PerTestSession)]
    public required ServiceFactory Factory { get; set; }
}

and the factory itself looks like this:

public sealed class ServiceFactory : WebApplicationFactory<Program>, IAsyncInitializer
{
    public Task InitializeAsync()
    {
        // You can also override certain services here to mock things out

        // Grab a reference to the server
        // This forces it to initialize.
        // By doing it within this method, it's thread safe.
        // And avoids multiple initialisations from different tests if parallelisation is switched on
        _ = Server;

        return Task.CompletedTask;
    }

    protected override void ConfigureWebHost(IWebHostBuilder builder)
    {
        builder.ConfigureTestServices(services =>
        {
            services.AddTransient<ITestUserProvider, TestUserProvider>();
            services.AddAuthentication("TestScheme").AddScheme<AuthenticationSchemeOptions, TestingAuthenticationHandler>("TestScheme", options => { });
        });
        base.ConfigureWebHost(builder);
    }
}

[Dalet]

I have a similar setup with a base class except my property-injected WebApplicationFactory is SharedType.None.

I started having the same exception after upgrading TUnit from 0.10.26. In my case the issue seems to be at least from version 0.12.14. It's difficult to pinpoint exactly when it started because it doesn't seem to happen on every run.

[Skyppid]

Well I can at least tell that switching to XUnit fixed it. I can try to reproduce it with a small repository the coming days when I find time. I guess it shouldn't be that hard to reproduce. Seems like this can occur even with a basic setup.

[thomhurst]

I'll try and reproduce it - And if anyone beats me to it, please share as it'll be very useful 😄

[thomhurst]

Could you try v0.14.13 please and let me know if it still occurs

[Dalet]

Still happening with v0.14.13

[thomhurst]

@Dalet if you can get me a reproduction at any point that'll help as I can't recreate it