Error testing from CLI

[mdima]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

Using Visual Studio, when I run the tests from Test Explorer they succeed, but when I run them from the CLI after the test execution I get the following error:

An exception occurred while invoking executor 'executor://xunit/VsTestRunner3/netcore/': The type initializer for 'Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Constants' threw an exception.
Stack trace:
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable`1 executorUriExtensionMap, Int64 totalTests) in /_/src/Microsoft.TestPlatform.CrossPlatEngine/Execution/BaseRunTests.cs:line 493

Inner exception: Object reference not set to an instance of an object.

I tried both Xunit than MSTests, the behavior is the same.

Expected Behavior

When I run:

dotnet test --no-build --verbosity detailed --logger "console;verbosity=detailed" --collect "XPlat Code Coverage"

the test should succeed, like they do using the Visual Studio Test Explorer.

Steps To Reproduce

You can reproduce the issue in this project (I hope it is not considered a complex project):
https://github.com/mdima/traceroute

Just open the Developer PowerShell, cd UnitTests, then run:
dotnet test --no-build --verbosity detailed --logger "console;verbosity=detailed" --collect "XPlat Code Coverage"

Exceptions (if any)

No response

.NET Version

9.0.304

Anything else?

PS C:\Users\Michele DM\source\repos\Michele-traceroute\UnitTests> dotnet test --no-build --verbosity detailed --logger "console;verbosity=detailed" --collect "XPlat Code Coverage"
A total of 1 test files matched the specified pattern.
C:\Users\Michele DM\source\repos\Michele-traceroute\UnitTests\bin\Debug\net9.0\UnitTests.dll
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v3.1.3+b1b99bdeb3 (64-bit .NET 9.0.8)
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v3.1.3+b1b99bdeb3 (64-bit .NET 9.0.8)
[xUnit.net 00:00:00.07]   Discovering: UnitTests
[xUnit.net 00:00:00.07]   Discovering: UnitTests
[xUnit.net 00:00:00.10]   Discovered:  UnitTests
[xUnit.net 00:00:00.10]   Discovered:  UnitTests
[xUnit.net 00:00:00.12]   Starting:    UnitTests
[xUnit.net 00:00:00.12]   Starting:    UnitTests
  Passed UnitTests.Services.TraceRouteServiceTests.TestTraceRoute [107 ms]
  Passed UnitTests.Services.TraceRouteServiceTests.TestTraceRouteFull [2 s]
  Passed UnitTests.Components.Pages.HomeTests.TestHome [260 ms]
  Passed UnitTests.Services.ReverseLookupServiceTests.GetHostNameNull [10 ms]
  Passed UnitTests.Services.ReverseLookupServiceTests.GetHostName [1 s]
  Passed UnitTests.Controllers.APIControllerTests.GetServerInfo [48 ms]
  Passed UnitTests.Controllers.APIControllerTests.ReceivePresence [117 ms]
  Passed UnitTests.Controllers.APIControllerTests.TraceRouteOK [3 s]
info: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[63]
info: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[63]
      User profile is available. Using 'C:\Users\Michele DM\AppData\Local\ASP.NET\DataProtection-Keys' as key repository and Windows DPAPI to encrypt keys at rest.
      User profile is available. Using 'C:\Users\Michele DM\AppData\Local\ASP.NET\DataProtection-Keys' as key repository and Windows DPAPI to encrypt keys at rest.
info: TraceRoute.Services.ServerListService[0]
info: TraceRoute.Services.ServerListService[0]
      Starting the ServerListService
      Starting the ServerListService
info: TraceRoute.Services.ServerListService[0]
info: TraceRoute.Services.ServerListService[0]
      The HttpContext last URL is not available
      The HttpContext last URL is not available
crit: Microsoft.AspNetCore.Hosting.Diagnostics[11]
crit: Microsoft.AspNetCore.Hosting.Diagnostics[11]
      Hosting startup assembly exception
      Hosting startup assembly exception
      System.InvalidOperationException: Startup assembly  failed to execute. See the inner exception for more details.
      System.InvalidOperationException: Startup assembly  failed to execute. See the inner exception for more details.
       ---> System.ArgumentException: The value cannot be an empty string. (Parameter 'assemblyName')
       ---> System.ArgumentException: The value cannot be an empty string. (Parameter 'assemblyName')
         at System.ArgumentException.ThrowNullOrEmptyException(String argument, String paramName)
         at System.ArgumentException.ThrowNullOrEmptyException(String argument, String paramName)
         at System.Reflection.AssemblyName..ctor(String assemblyName)
         at System.Reflection.AssemblyName..ctor(String assemblyName)
         at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.ExecuteHostingStartups()
         at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.ExecuteHostingStartups()
         --- End of inner exception stack trace ---
         --- End of inner exception stack trace ---
  Passed UnitTests.Controllers.APIControllerTests.GetServerList [2 ms]
  Passed UnitTests.Controllers.APIControllerTests.SecurityChecks [1 ms]
  Passed UnitTests.Services.TraceRouteApiClientTests.TestGetServerInfo [46 ms]
  Passed UnitTests.Services.TraceRouteApiClientTests.TestGetServerList [33 ms]
  Passed UnitTests.Services.TraceRouteApiClientTests.TestSendPresence [68 ms]
  Passed UnitTests.Services.TraceRouteApiClientTests.TestRemoteTrace [57 ms]
  Passed UnitTests.Components.Molecules.AboutTests.TestAbout [4 ms]
  Passed UnitTests.Components.Molecules.AboutTests.TestCheckVersion [27 ms]
  Passed UnitTests.ProgramTests.Program [250 ms]
  Passed UnitTests.Components.Layout.MainLayoutTests.TestShowServerDetails [22 ms]
  Passed UnitTests.Components.Layout.MainLayoutTests.TestRefreshServerList [4 ms]
  Passed UnitTests.Components.Layout.MainLayoutTests.TestShowServerEntry [2 ms]
  Passed UnitTests.Components.Layout.MainLayoutTests.TestOnShowIpDetails [24 ms]
  Passed UnitTests.Components.Layout.MainLayoutTests.TestInitialHost [6 ms]
  Passed UnitTests.Components.Layout.MainLayoutTests.TestOnShowHopDetails [3 ms]
  Passed UnitTests.Components.Layout.MainLayoutTests.TestBeginTraceRoute [11 s]
  Passed UnitTests.Components.Layout.MainLayoutTests.TestLayout [8 ms]
  Passed UnitTests.Helpers.BashHelperTests.TraceKnown [1 ms]
  Passed UnitTests.Services.BogonIPServiceTests.IsBogonIP [< 1 ms]
  Passed UnitTests.Services.ServerListServiceTests.GetRemoteServerInfo [21 ms]
  Passed Can start and stop the service [16 s]
  Passed UnitTests.Services.ServerListServiceTests.AddServer [1 ms]
  Passed Can clean the server list [73 ms]
  Passed UnitTests.Services.ServerListServiceTests.SendPresenceToMainHost [127 ms]
  Passed UnitTests.Services.StoreServerURLFilterTests.TestFilter [7 ms]
  Passed UnitTests.Services.IpApiClientTests.GetUnKnownIP [47 ms]
  Passed UnitTests.Services.IpApiClientTests.GetKnownIP [51 ms]
  Passed UnitTests.Services.IpApiClientTests.GetErrorIP [< 1 ms]
  Passed UnitTests.Components.Molecules.IpDetailsComponentTests.TestIpDetails [4 ms]
  Passed UnitTests.Components.Molecules.SettingTests.TestSettings [54 ms]
[xUnit.net 00:00:36.77]   Finished:    UnitTests
[xUnit.net 00:00:36.77]   Finished:    UnitTests
An exception occurred while invoking executor 'executor://xunit/VsTestRunner3/netcore/': The type initializer for 'Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Constants' threw an exception.
Stack trace:
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable`1 executorUriExtensionMap, Int64 totalTests) in /_/src/Microsoft.TestPlatform.CrossPlatEngine/Execution/BaseRunTests.cs:line 493

Inner exception: Object reference not set to an instance of an object.

Stack trace:
   at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.ProcessHelper.GetCurrentProcessLocation() in /_/src/Microsoft.TestPlatform.PlatformAbstractions/netcore/System/ProcessHelper.cs:line 21
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Constants..cctor() in /_/src/Microsoft.TestPlatform.CrossPlatEngine/Constants.cs:line 26

An exception occurred while invoking executor 'executor://xunit/VsTestRunner3/netcore/': The type initializer for 'Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Constants' threw an exception.
Stack trace:
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable`1 executorUriExtensionMap, Int64 totalTests) in /_/src/Microsoft.TestPlatform.CrossPlatEngine/Execution/BaseRunTests.cs:line 493

Inner exception: Object reference not set to an instance of an object.

Stack trace:
   at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.ProcessHelper.GetCurrentProcessLocation() in /_/src/Microsoft.TestPlatform.PlatformAbstractions/netcore/System/ProcessHelper.cs:line 21
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Constants..cctor() in /_/src/Microsoft.TestPlatform.CrossPlatEngine/Constants.cs:line 26

  Passed UnitTests.Helpers.ConfigurationHelperTests.GetNumericValue_NoDefault [< 1 ms]
  Passed UnitTests.Helpers.ConfigurationHelperTests.GetSetHostRemoteTraces [< 1 ms]
  Passed UnitTests.Helpers.ConfigurationHelperTests.GetSetEnableRemoteTraces [< 1 ms]
  Passed UnitTests.Helpers.ConfigurationHelperTests.GetAppSetting [< 1 ms]
  Passed UnitTests.Helpers.ConfigurationHelperTests.GetCurrentSettings [53 ms]
  Passed UnitTests.Helpers.ConfigurationHelperTests.GetNumericValue [1 ms]
  Passed UnitTests.Helpers.ConfigurationHelperTests.GetCacheMinutes [< 1 ms]
  Passed UnitTests.Helpers.ConfigurationHelperTests.IsRootNode [1 ms]
  Passed UnitTests.Helpers.ConfigurationHelperTests.GetRootNode [< 1 ms]

Test Run Failed.
Total tests: 47
     Passed: 47
 Total time: 37,9807 Seconds
  UnitTests test failed with 1 error(s) (38,2s)
    C:\Program Files\dotnet\sdk\9.0.304\Microsoft.TestPlatform.targets(48,5): error MSB6006: "dotnet.exe" exited with code 1.

Test summary: total: 47; failed: 0; succeeded: 47; skipped: 0; duration: 38,2s
Build failed with 1 error(s) in 38,4s

Attachments:
  C:\Users\Michele DM\source\repos\Michele-traceroute\UnitTests\TestResults\0825f757-b5ae-4be0-9369-673819de55ae\coverage.cobertura.xml

[mdima]

While the same test run with VS succeed:

[martincostello]

From seeing this part of the stack trace:

      System.InvalidOperationException: Startup assembly  failed to execute. See the inner exception for more details.
       ---> System.ArgumentException: The value cannot be an empty string. (Parameter 'assemblyName')
       ---> System.ArgumentException: The value cannot be an empty string. (Parameter 'assemblyName')
         at System.ArgumentException.ThrowNullOrEmptyException(String argument, String paramName)
         at System.ArgumentException.ThrowNullOrEmptyException(String argument, String paramName)
         at System.Reflection.AssemblyName..ctor(String assemblyName)
         at System.Reflection.AssemblyName..ctor(String assemblyName)
         at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.ExecuteHostingStartups()
         at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.ExecuteHostingStartups()

This looks like an issue with the test setup for APIControllerTests. My guess would be it's do to with whatever the current directory is in the context where the tests are being run where they fail and/or the name of the entry assembly (which might be the test host, and then cause the startup to fail because it hasn't been explicitly set to the right name (the name of the assembly of the project you're testing) using ApplicationName like this #58403 (comment))

I would recommend refactoring the test to use WebApplicationFactory<T> so you don't need to setup so many manual services for your system under test in the constructor.

[mdima]

@martincostello: I tried (quickly because unfortunately in this moment I cannot dedicate so much time on it), but the problem persists.

I set up a WebApplicationFactory then I refactored the tests (partially, disabling the rest), but the problem persists.
I also tried to give the application a custom name, just to try if the error was something different, but was the same.

Thanks,
Michele

[mdima]

I pushed here the WebApplicationFactory refactory, just in case:
https://github.com/mdima/traceroute/tree/Test-refactory

Thanks!