FileNotFoundException using own TestRunner

[Sputnik24]

Hi

referencing to my comment on NUnit 4.0.0 ( nunit/nunit#4563 (comment) ) I want to report the following issue.

Runtime: .NET 8
NUnit: 4.2.1
NUnit.Engine: 3.18.1

I developed a custom NUnit agent which instantiates a local test runnter the following way:

TestEngine.WorkDirectory = Path.GetDirectoryName(testFile);
TestEngine.InternalTraceLevel = InternalTraceLevel.Off;
TestPackage = new TestPackage(testFile);
TestRunner = TestEngine.GetRunner(TestPackage);
TestNodes = TestRunner.Explore(TestFilter.Empty);

With NUnit 3.14.0 / Engine 3.16.3 this was working.

With the latest versions, I get

27.08.2024 09:07:04.275 NUnit.Engine.NUnitEngineException: An exception occurred in the driver while loading tests.
 ---> System.IO.FileNotFoundException: Could not load file or assembly 'UnitTests, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. Das System kann die angegebene Datei nicht finden.
File name: 'UnitTests, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'
   at System.Reflection.RuntimeAssembly.InternalLoad(AssemblyName assemblyName, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext, RuntimeAssembly requestingAssembly, Boolean throwOnFileNotFound)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
   at NUnit.Engine.Drivers.NUnitNetStandardDriver.Load(String testAssembly, IDictionary`2 settings)
   at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)
   --- End of inner exception stack trace ---
   at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)
   at NUnit.Engine.Runners.DirectTestRunner.LoadPackage()
   at NUnit.Engine.Runners.DirectTestRunner.EnsurePackageIsLoaded()
   at NUnit.Engine.Runners.DirectTestRunner.Explore(TestFilter filter)
   at NUnit.Engine.Runners.MasterTestRunner.Explore(TestFilter filter)
   at NUnitAgent.Program.ProcessMessageAsync(String rawMsg, CancellationToken ctx) in NUnitAgent\Program.cs:line 306

I guess the comment from @stevenaw ( nunit/nunit#4563 (reply in thread) ) gives a hint about the root cause.

I hope that this gets fixed. If you need further information or support, please contact me.

Thanks a lot
Daniel

[OsirisTerje]

I'll move this to the console/engine repo.

[CharliePoole]

@Sputnik24 As a first shot at this, I'll take a look at the PRs listed to see if they were all ported to the new version3 branch.

[CharliePoole]

@Sputnik24 I'd like to find out if I need to do anymore on this before I release 3.18.2. It looks like all the issues cited by @stevenaw have been ported. Can you try the latest dev build from our MyGet feed and let me know?

[Sputnik24]

@CharliePoole I tested with NUnit.Engine 3.18.2-dev00040 and NUnit 4.3.0-alpha.0.6 and still get the same exception

NUnit.Engine.NUnitEngineException: An exception occurred in the driver while loading tests.
 ---> System.IO.FileNotFoundException: Could not load file or assembly 'UnitTests, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. Das System kann die angegebene Datei nicht finden.
File name: 'UnitTests, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'
   at System.Reflection.RuntimeAssembly.InternalLoad(AssemblyName assemblyName, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext, RuntimeAssembly requestingAssembly, Boolean throwOnFileNotFound)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
   at NUnit.Engine.Drivers.NUnitNetStandardDriver.Load(String testAssembly, IDictionary`2 settings)
   at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)
   --- End of inner exception stack trace ---
   at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)
   at NUnit.Engine.Runners.DirectTestRunner.LoadPackage()
   at NUnit.Engine.Runners.DirectTestRunner.EnsurePackageIsLoaded()
   at NUnit.Engine.Runners.DirectTestRunner.Explore(TestFilter filter)
   at NUnit.Engine.Runners.MasterTestRunner.Explore(TestFilter filter)

Edit: I wanted to roll back and saw that 3.16.3 is not available on nuget.org anymore :( - luckily it was still in my cache. This is the only version working for me.

[CharliePoole]

@Sputnik24 3.16.3 should be available but unlisted. Nuget doesn't allow deletions. I believe you can get it via the command-line by specifying the version.

Version 3.16.3 made things worst for most people. In the end all the 3.16 builds were unlisted. I think I'll look at all commits between 3.16.2 and 3.16.3 and see what was and wasn't ported to the latest master. That may give me a hint.

Do you have a repro handy that you are able to share?

[CharliePoole]

I can find nothing there that seems to cause this. However, I notice in the stack that you are using the NUnitNETStandardDriver rather than the NUnitNetCore31Driver. If your test target .NET 8.0, I'd expect to see the latter. Am I missing something about what you're doing here?

[Sputnik24]

@CharliePoole Good question.

I created a minimal example which reproduces the issue: https://github.com/Sputnik24/NUnitOwnAgent
Build the Tests project first and than run the console application. It will throw the error. Maybe I'm doing something wrong.

Thanks a lot for your great support.

[Sputnik24]

Hi @CharliePoole
could you reproduce the issue using my example? Do you need further support?

[CharliePoole]

Yes, I could, but it works after I replace line 6 pf Program.cs with

    TestPackage package = new TestPackage("../../../../../../../../Tests/bin/Debug/net8.0/Tests.dll");

In situations like this, I open File Explorer and count the number of times I need to go back from bin\debug out loud to myself. :-)

[Sputnik24]

Hi @CharliePoole
thanks for re-testing. In my real world NUnitAgent the test dll file is selected by a file dialog, so the path can't be wrong, as it anyway gives the absolute path.

I added a File.Exists test to my code (I comitted it):

• Using my string, File.Exists says true but I got the above FileNotFoundException
• Using your string, File.Exists says false but the ITestRunner is working.
• Using the absolute path, File.Exists to true but I got your exception

Can you please re-open this issue and maybe test with the absolute path?

At least, I found a workaround: When I set the ITestengine.WorkDirectory to the path of the dll file and provide only the dll file name without path, it is working.

Second point is your statement that you would expect NUnitNetCore31Driver in the stack trace instead of NUnitNetStandardDriver

[CharliePoole]

I'll retest in the morning using your updated repro example.

[CharliePoole]

I have the same results as you and I'm re-opening the issue for more examination. To be honest, I am still suspicious that the problem may be in your code, but I need to be sure.

A few points...

It is only a little surprising a relative path will succeed with File.Exists but fail to be found when passed in a test package. The two use cases can employ different bases for relative paths. However, I didn't think they did, so it is a small surprise and I'll look into why that's the case.

OTOH it's a BIG surprise to me that setting WorkDirectory has this effect. The NUnit WorkDirectory is a directory used for output of results, logs and other reports. It has no connection with the Current working directory, except as a default value. If you change WorkDirectory, your reports should go elsewhere but nothing else should change unless you are using WorkDirectory some place in your own code in a way it wasn't intended. OR unless there's a bug in the engine.

I'll continue to look at this but please tell me if you have misused WorkDirectory somewhere. :-) I recognize that it's badly named.

[Sputnik24]

Thanks @CharliePoole
What about passing an absolute path to the TestPackage, this does not work, neither, which might be the biggest deal.

[CharliePoole]

That's odd as well. I'm looking.

FYI... Your example using WorkDirectory doesn't work either. It simply doesn't throw an exception. The engine is not supposed to throw an exception if a file is not found, it merely reports it. Here's the output I get using that approach(indented for clarity):

<test-run id="1" name="Tests.dll" fullname="C:\Users\charlie\dev\NUnit\NUnitOwnAgent\OwnAgent\bin\Debug\net8.0\Tests.dll" runstate="Runnable" testcasecount="0">
  <test-suite type="Assembly" id="1-1" name="Tests.dll" fullname="C:\Users\charlie\dev\NUnit\NUnitOwnAgent\OwnAgent\bin\Debug\net8.0\Tests.dll" testcasecount="0" runstate="NotRunnable">
    <properties>
      <property name="_SKIPREASON" value="File not found: C:\Users\charlie\dev\NUnit\NUnitOwnAgent\OwnAgent\bin\Debug\net8.0\Tests.dll" />
    </properties>
  </test-suite>
</test-run>

In other words, the engine is looking in the current directory. It does this whether I set the WorkDirectory or not, so that resolves my first surprise.

[CharliePoole]

@Sputnik24 Your example is working for me using the my current build, which is still unreleased. It fails using 3.18.2-dev00042 and also using the 3.18.2 release itself.

I'll keep this open until a new dev build (3.18.3-devxxxxx) is released.

[Sputnik24]

@CharliePoole Thanks a lot.
At least good news that you can reproduce and confirm the issue. I didn't check the content of the xml output, as well, as I was too happy that it works :D.
Ping me here, when 3.18.3-dev is released, I will re-test and confirm.

[CharliePoole]

@Sputnik24 Since your main program is .NET 8.0, I assume you are using the .NET standard build of the engine because it's the only one you should be able to reference. If that's true, then the new build I just released will probably not fix the problem, but please try it out anyway. It's version 3.18.3-dev00002.

[Sputnik24]

Edit3 (but most important, therefore, at the beginning):
I cloned 3.18.2, did a single change, compiled it locally and it works. As assumed in Edi2, I only added netcoreapp3.1 to the TargetFrameworks in nunit.engine.csproj and now the correct driver is given back by the DriverFactory.

@CharliePoole as you assumed, issue still occurs with 3.18.3-dev00002

Additionally, I found the code in your source where the DriverFactory decides which driver to load, NUnitNetCore31Driver or NUnitNetStandardDriver; and I am also surprised why NetStandard and not NetCorre31.

Edit: @CharliePoole
I guess, the additional point is the root cause. I rolled my example back to 3.16.3 and went through the Explore code step by step with the debugger until I reach the NUnit3DriverFactory class.
With 3.16.3 the debugger shows that it returns the NUnitNetCore31Driver and with new version the debugger shows returning the NUnitNetStandardDriver.

Can you confirm on your side? Did something change in building the package?

Edit2: I did some further investigations what have changed in the DriverFactory.
In 3.1.6.2, back to 3.14.0 where is is also working, in NUnit.Engine .NET Core 3.1 was referenced in csproj. Beginning with 3.15.-beta1 .NET Core 3.1 is removed and only .NET Standard and .NET Framework are referenced.
Maybe this gives a hint about the root cause.

[CharliePoole]

@Sputnik24

Yes, that's really the underlying problem. I've been trying to handle this issue with a small fix, but we do actually need a new engine. Even if we can get past this particular issue in your case, you will hit other roadblocks later because the lack of a modern engine build makes it difficult to create a third-party runner using .NET 8.0. That said, there will be a .NET 8.0 runner in 3.19, so it would be nice if we could get you past this File Not Found problem so you can continue to work.

In order to debug the code on my own system, I added the engine build itself as part of the solution. But, that changed the environment enough that there was nothing to debug! The code works. I suspect that this is the result of the .deps.json file created by compiling the engine along with your runner. Can you post a link to that file in your own case?

One temporary workaround may be for you to use an older version of the engine until 3.19 is released. Is that possible?

[Sputnik24]

@CharliePoole
Thanks a lot. .deps.json of which project do you need? NUnitOwnAgent?

Regarding your question: Sure, it is possible to stay on 3.16.3 what I do for month, now. But I have a couple of question, also to understand better the usage of the engine:

• Why was netcore3.1 removed from targetframeworks in nunig.engine?
• Why do you think that re-adding it will not fix it (as it works in 3.16.3)? I cloned 3.18.2, re-added netcore3.1 to nunit.engine project, made a local build and use it in my own agent. It is working.
• Looking forward to runner in 3.19. I understand it will still be possible to greate own ITestRunner with .net8.0 (and hopefully .net9.0). Can you say something about timeline?

[CharliePoole]

@Sputnik24
The netcore3.1 build of the engine was removed as an experiment. At the same time we added a .NET 6.0 agent and a .NET 6.0 build of nunit.engine.core, which contains the code to actually locate dependencies. All our tests continued to pass, so we left it out in the 3.15 release. I think that's because most of our focus was on the standard runner, which uses agents to run the tests.

Anyway, there is now a .NET 8.0 build of the engine. It's experimental and only available on the MyGet feed, version 3.18.3-dev00006. Give it a try.

[Sputnik24]

@CharliePoole I need some support how to use the net8.0 build of the engine. I updated NuGet package NUnit.Engine to 3.18.3-dev00006 but it did not solve the problem, same issue occurs (I cleaned solution and Nuget cache before).
I wanted to install NUnit.ConsoleRunner.Net80 but this does not work.

I'm pretty sure, that I do something wrong. What is the right way to reference the net8.0 engine?

Edit: I cloned the main branch, build it w/o any changes and copied Microsoft.Extensions.DependencyModel.dll, nunit.engine.api.dll, nunit.engine.core.dll, nunit.engine.dll and testcentric.engine.metadata.dll from ./bin/net8.0 folder to my OwnAgent project and it is working. Maybe something wrong with the nuget package?

But, having a look into the assembly info in Rider (from the locally build dll), I see still .NETStandard 2.0:

Same, when referencing the nuget package:

[CharliePoole]

I'm not sure why you're copying assemblies. Of course, that's how it was done in the past, but today you should be using the package management system. Your main runner program should include a PackageReference for the correct version of the engine. NuGet will take care of everything else making all the necessary dependencies available to you, most likely 100s of them.

In the case of the net 8.0 runner, you can install the package wherever you want from the command-line using nuget.exe. Once again, you should execute the runner from that directory rather than copying it elsewhere. Or if you feel you must copy it, at least copy everything.

[CharliePoole]

BTW, I assume you have put our myget feed into your nuget.config. Otherwise, you may only get what's on nuget.org, which is likely to be 3.18.2 as the closest match to 3.18.3-dev00006.

[Sputnik24]

Hi Charlie,
of course, I added you myget feed to the nuget config and reference the latest 3.18.3 version, as you see from the second screenshot in my previous post.
I build and copied locally to crosscheck if there is something wrong. And at least, I see a difference between locally build 3.18.3 (main branch) and 3.18.3. My locally build nuning.engine.core version shows .NETCoreApp v8.0 while the one from MyGet shows .NETSTandard v.2.0.

My Nuget package manager:

[CharliePoole]

OK, I think I misunderstood you about copying assemblies. I see now that you did that only to try to debug the problem. So, if I understand correctly, when you copy the correct .NET 8.0 builds it works but they are not being copied or selected automatically by referencing the package normally. Have I got that right? And your runner is built targeting .NET 8.0?

My understanding is that the net8.0 implementation should be used in this situation. Later today, I'll set up a separate new project that references the engine and see if I get the same results as you.

[CharliePoole]

Looking at your example, I don't see how you create the engine. Can you post that?

[CharliePoole]

I tested in a separate project and I have the same results as you. The package I am now seeing on MyGet only has net462 and netstandard2.0. The same is true for the net8.0 runner package, which uses the netstandard2.0 engine. OTOH my local copies of the package, as built, are all correct.

Thanks for pointing this out. The problem isn't in the engine but in my release scripts, which are not creating and pushing the package correctly. I'll work on a new package release today.

[Sputnik24]

Glad to read that you find the issue. Let me know when a new version is ready for testing.

To answer your question how I create the eingine: It is line 11 in my Program.cs:
ITestEngine testEngine = TestEngineActivator.CreateInstance();
Is this the correct way?

Thanks a lot for your great and fast support and your kindness.

[CharliePoole]

That's correct historically. But now I wonder if I also need a .NET 8.0 build of the api assembly for this to work. I'll test that before i release a new package.

I'll post here when there's a new package.

[CharliePoole]

New package is 3.18.3-dev00009

[CharliePoole]

@Sputnik24
Closing as original example is now working for me using 3.18.3-dev00009. You can continue to comment here of course.

[Sputnik24]

Hi @CharliePoole
I confirm that 3.18.3-dev00009 is working. Thanks a lot for this quick solution. Looking forward to 3.18.3.