From 93ab816f8c852ee29f3f7eb3982b41b0fa985237 Mon Sep 17 00:00:00 2001 From: Toni Solarin-Sodara Date: Tue, 15 Dec 2020 21:56:50 +0100 Subject: [PATCH] Update documentation --- CONTRIBUTING.md | 2 +- Documentation/Changelog.md | 1 - Documentation/ConsumeNightlyBuild.md | 31 ++-- Documentation/DeterministicBuild.md | 28 ++-- Documentation/DriversFeatures.md | 27 ++-- Documentation/Examples.md | 8 +- Documentation/GlobalTool.md | 9 +- Documentation/KnownIssues.md | 205 ++++++++++++++------------- Documentation/MSBuildIntegration.md | 27 ++-- Documentation/ReleasePlan.md | 101 ++++++------- Documentation/Roadmap.md | 16 +-- Documentation/Troubleshooting.md | 46 ++++-- Documentation/VSTestIntegration.md | 65 +++++---- README.md | 8 +- 14 files changed, 314 insertions(+), 260 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 36fd9d2b5..5d5c71993 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -11,7 +11,7 @@ Contributions are highly welcome, however, except for very small changes, kindly Clone this repo: - git clone https://github.com/tonerdo/coverlet.git + git clone https://github.com/coverlet-coverage/coverlet.git cd coverlet Building, testing, and packing use all the standard dotnet commands: diff --git a/Documentation/Changelog.md b/Documentation/Changelog.md index abc061d9a..ccefd0110 100644 --- a/Documentation/Changelog.md +++ b/Documentation/Changelog.md @@ -113,4 +113,3 @@ coverlet.collector 1.1.0 -Fix property attribute detection [#477](https://github.com/tonerdo/coverlet/pull/477) by https://github.com/amweiss -Fix instrumentation serialization bug [#458](https://github.com/tonerdo/coverlet/pull/458) -Fix culture for cobertura xml report [#464](https://github.com/tonerdo/coverlet/pull/464) - diff --git a/Documentation/ConsumeNightlyBuild.md b/Documentation/ConsumeNightlyBuild.md index c048e5889..e65da90e4 100644 --- a/Documentation/ConsumeNightlyBuild.md +++ b/Documentation/ConsumeNightlyBuild.md @@ -1,12 +1,13 @@ # Consume nightly build -You can check metadata of nightly build packages here: +You can check the metadata of nightly build packages here: -Msbuild https://www.myget.org/feed/coverlet-dev/package/nuget/coverlet.msbuild +MSBuild https://www.myget.org/feed/coverlet-dev/package/nuget/coverlet.msbuild VSTest collector https://www.myget.org/feed/coverlet-dev/package/nuget/coverlet.collector -.Net tools https://www.myget.org/feed/coverlet-dev/package/nuget/coverlet.console +.NET tools https://www.myget.org/feed/coverlet-dev/package/nuget/coverlet.console + +To consume nightly builds, create a `NuGet.Config` in your root solution directory and add the following content: -To consume nightly build create a `NuGet.Config` on your root solution directory and add following content ```xml @@ -23,34 +24,38 @@ To consume nightly build create a `NuGet.Config` on your root solution directory ### Install packages -You can install nightly package using visual studio +Visual Studio: ![File](images/nightly.PNG) -Nuget(PM console) -``` +NuGet (Package Manager console): + +```powershell PM> Install-Package coverlet.msbuild -Version 2.6.25-g6209239d69 -Source https://www.myget.org/F/coverlet-dev/api/v3/index.json ``` -.NET CLI -``` +.NET CLI: + +```bash dotnet add package coverlet.msbuild --version 2.6.25-g6209239d69 --source https://www.myget.org/F/coverlet-dev/api/v3/index.json ``` -.csproj +MSBuild project file: -``` +```xml ``` ### How to verify version -You can understand which version you're using comparing nightly build release date and repo commits. +You can understand which version you're using by comparing nightly build release date with repo commits. + For instance if we want to consume last msbuild nightly build: + * Go to https://www.myget.org/feed/coverlet-dev/package/nuget/coverlet.msbuild * Scroll down the page and check release date ![File](images/nightly_1.PNG) * Go to repo commits and compare date and first part of commit hash ![File](images/nightly_2.PNG) -As you can see we build at 00.00 UTC and build takes some seconds, so it's possible that release date won't be the same of commit repo. \ No newline at end of file +As you can see we build at 00.00 UTC and build takes some seconds, so it's possible that release date won't be the same as repo commits. diff --git a/Documentation/DeterministicBuild.md b/Documentation/DeterministicBuild.md index 315a94eb0..ae39da596 100644 --- a/Documentation/DeterministicBuild.md +++ b/Documentation/DeterministicBuild.md @@ -1,25 +1,23 @@ # Deterministic build -Deterministic build **is supported only** by `msbuild`(`/p:CollectCoverage=true`) and `collectors`(`--collect:"XPlat Code Coverage"`) drivers. +Support for deterministic builds is available **only** in the `msbuild` (`/p:CollectCoverage=true`) and `collectors` (`--collect:"XPlat Code Coverage"`) drivers. Deterministic builds are important because they enable verification that the resulting binary was built from the specified sources. For more information on how to enable deterministic builds, I recommend you to take a look at [Claire Novotny](https://github.com/clairernovotny)'s [guide](https://github.com/clairernovotny/DeterministicBuilds). -Deterministic builds are important as they enable verification that the resulting binary was built from the specified source and provides traceability. -For more information on how to enable deterministic build I recommend you to take a look at [@clairernovotny](https://github.com/clairernovotny)'s guide https://github.com/clairernovotny/DeterministicBuilds +From a coverage perspective, deterministic builds create some challenges because coverage tools usually need access to complete source file metadata (ie. local path) during instrumentation and report generation. These files are reported inside the `.pdb` files, where debugging information is stored. -From coverage perspective deterministic build put some challenge because usually coverage tools need access to source file metadata (ie. local path) during instrumentation and report generation. -These files are reported inside `pdb` files, where are stored debugging information needed by debuggers and also by tools that needs to work with "build" metadata information, for example generated `dll` modules and `source files` used. -In local (non-CI) builds metadata emitted to pdbs are not "deterministic", that means that for instance source files path are reported with full path. -If for instance we build same project on different machine we'll have different paths emitted inside pdbs, so builds are "non deterministic" because same project build won't generates same artifacts. +In local (non-CI) builds, metadata emitted to pdbs are not "deterministic", which means that source files are reported with their full paths. For example, when we build the same project on different machines we'll have different paths emitted inside pdbs, hence, builds are "non deterministic". -As explained above, to improve the level of security of generated artifacts (suppose for instance DLLs inside the nuget package), we need to apply some signature (signing with certificate) and validate before usage to avoid possible security issues like tampering. -Finally thanks to deterministic CI builds (with the `ContinuousIntegrationBuild` property set to `true`) plus signature we can validate artifacts and be sure that binary was build from a specific sources (because there is no hard-coded variables metadata like paths from different build machines). +As explained above, to improve the level of security of generated artifacts (for instance, DLLs inside the NuGet package), we need to apply some signature (signing with certificate) and validate before usage to avoid possible security issues like tampering. + +Finally, thanks to deterministic CI builds (with the `ContinuousIntegrationBuild` property set to `true`) plus signature we can validate artifacts and be sure that the binary was built from specific sources (because there is no hard-coded variable metadata, like paths from different build machines). **Deterministic build is supported without any workaround since version 3.1.100 of .NET Core SDK** ## Workaround only for .NET Core SDK < 3.1.100 -At the moment deterministic build works thanks to Roslyn compiler that emits deterministic metadata if `DeterministicSourcePaths` is enabled. Take a look here for more information https://github.com/dotnet/sourcelink/tree/master/docs#deterministicsourcepaths. -To allow coverlet to correctly do his work we need to provide information to translate deterministic path to real local path for every project referenced by tests project. -The current workaround is to add on top of your repo a `Directory.Build.targets` with inside a simple snippet with custom `target` that supports coverlet resolution algorithm. +At the moment, deterministic build works thanks to the Roslyn compiler emitting deterministic metadata if `DeterministicSourcePaths` is enabled. Take a look [here](https://github.com/dotnet/sourcelink/tree/master/docs#deterministicsourcepaths) for more information. + +To allow Coverlet to correctly do its work, we need to provide information to translate deterministic paths to real local paths for every project referenced by the test project. The current workaround is to add at the root of your repo a `Directory.Build.targets` with a custom `target` that supports Coverlet resolution algorithm. + ```xml ``` -If you already have a `Directory.Build.targets` file on your repo root you can simply copy `DeterministicBuild.targets` you can find on coverlet repo root next to yours and import it in your targets file. -This target will be used by coverlet to generate on build phase a file on output folder with mapping translation informations, the file is named `CoverletSourceRootsMapping`. + +If you already have a `Directory.Build.targets` file on your repo root you can simply copy `DeterministicBuild.targets` (which can be found at the root of this repo) next to yours and import it in your targets file. This target will be used by Coverlet to generate, at build time, a file that contains mapping translation information, the file is named `CoverletSourceRootsMapping` and will be in the output folder of your project. You can follow our [step-by-step sample](Examples.md) -Feel free to file an issue in case of troubles! +Feel free to file an issue in case you run into any trouble! diff --git a/Documentation/DriversFeatures.md b/Documentation/DriversFeatures.md index 08914ffe7..ee9b2a37e 100644 --- a/Documentation/DriversFeatures.md +++ b/Documentation/DriversFeatures.md @@ -1,19 +1,18 @@ # Driver feature differences -Since the beginning all coverlet drivers shared the same coverage engine. -Since version 3.0.0 we decided to consolidate versioning across drivers so for every new release all drivers will have the same version number. -We think that keeping the versions in sync express better the set of features every release will have. -This does not mean that all drivers will support every functionality/feature or have the same behaviours, since they are limited by the context they're running in. -In the table below we keep track of main differences: +All Coverlet drivers share the same coverage engine. Since version 3.0.0, we've consolidated versioning across drivers, so for every new release, all drivers will have the same version number. + +We think that keeping the versions in sync better expresses the set of features every release will have. This does not mean that all drivers will support every functionality/feature or have the same behaviours, since they are limited by the context they're running in. -| Feature | MsBuild | .NET Tool | DataCollectors | -|----------|:-------------:|-------------:|----------------:| -| .NET Core support(>= 2.0) | Yes | Yes |Yes | -| .NET Framework support(>= 4.6.1) | Yes | Yes |Yes(since 3.0.0) | -| Show result on console | Yes | Yes | No | -| Deterministic reports output folder | Yes | Yes |No | -| Merge reports | Yes | Yes |No | -| Coverage threshold validation | Yes | Yes |No | +In the table below we keep track of main differences: +| Feature | MSBuild | .NET Tool | DataCollectors | +|:-----------------------------------|:--------------|--------------|------------------| +| .NET Core support(>= 2.0) | Yes | Yes | Yes | +| .NET Framework support(>= 4.6.1) | Yes | Yes | Yes(since 3.0.0) | +| Show result on console | Yes | Yes | No | +| Configurable reports output folder | Yes | Yes | No | +| Merge reports | Yes | Yes | No | +| Coverage threshold validation | Yes | Yes | No | -If possible we advice you to use the collectors integration (vstest engine integration), since it is fully integrated inside the test pipeline and does not suffer of the [known issues](KnownIssues.md) of the other drivers. \ No newline at end of file +When possible, we advice you to use the collectors integration (VSTest engine integration), since it is fully integrated inside the test pipeline and does not suffer from the [known issues](KnownIssues.md) of the other drivers. diff --git a/Documentation/Examples.md b/Documentation/Examples.md index c0fb58ad9..55f8169f0 100644 --- a/Documentation/Examples.md +++ b/Documentation/Examples.md @@ -1,10 +1,10 @@ # Examples - ## MSBuild Integration -* Use `/p:MergeWith` feature `Documentation/Examples/MSBuild/MergeWith/MergeWith.sln` + +* Using `/p:MergeWith` feature `Documentation/Examples/MSBuild/MergeWith/MergeWith.sln` * Deterministic build feature `Documentation/Examples/MSBuild/DeterministicBuild/DeterministicBuild.sln` ## VSTest Integration -* 'HelloWorld' sample `Documentation/Examples/VSTest/HelloWorld/HelloWorld.sln` -* Deterministic build feature `Documentation/Examples/VSTest/DeterministicBuild/DeterministicBuild.sln` +* HelloWorld sample `Documentation/Examples/VSTest/HelloWorld/HelloWorld.sln` +* Deterministic build feature `Documentation/Examples/VSTest/DeterministicBuild/DeterministicBuild.sln` diff --git a/Documentation/GlobalTool.md b/Documentation/GlobalTool.md index 1ca5f3852..fe5328cfc 100644 --- a/Documentation/GlobalTool.md +++ b/Documentation/GlobalTool.md @@ -119,7 +119,7 @@ coverlet --target --targetargs --output teamcit The currently supported [TeamCity statistics](https://confluence.jetbrains.com/display/TCD18/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-ServiceMessages) are: | TeamCity Statistic Key | Description | -| :--- | :--- | +|:------------------------|:-------------------------------| | CodeCoverageL | Line-level code coverage | | CodeCoverageB | Branch-level code coverage | | CodeCoverageM | Method-level code coverage | @@ -226,9 +226,14 @@ Both `--exclude` and `--include` options can be used together but `--exclude` ta You can also include coverage of the test assembly itself by specifying the `--include-test-assembly` flag. +## SourceLink + +Coverlet supports [SourceLink](https://github.com/dotnet/sourcelink) custom debug information contained in PDBs. When you specify the `--use-source-link` flag, Coverlet will generate results that contain the URL to the source files in your source control instead of local file paths. + ## Exit Codes -Coverlet outputs specific exit codes to better support build automation systems for determining failures and take action on it. +Coverlet outputs specific exit codes to better support build automation systems for determining the kind of failure so the appropriate action can be taken. + ```bash 0 - Success. 1 - If any test fails. diff --git a/Documentation/KnownIssues.md b/Documentation/KnownIssues.md index 181309225..6ba92cfe5 100644 --- a/Documentation/KnownIssues.md +++ b/Documentation/KnownIssues.md @@ -1,65 +1,67 @@ # Known Issues -## 1) VSTest stops process execution early(`dotnet test`) +## VSTest stops process execution early -*Affected drivers*: msbuild(`dotnet test`) , dotnet tool(if you're using ` --targetargs "test ... --no-build"`) +*Affected drivers*: msbuild (`dotnet test`) , dotnet tool(if you're using ` --targetargs "test ... --no-build"`) - *Symptoms:* - * warning or error like +*Symptoms:* - `Unable to read beyond end of stream` +* warning or error like: - `warning : [coverlet] Hits file:'C:\Users\REDACTED\AppData\Local\Temp\testApp_ac32258b-fd4a-4bb4-824c-a79061e97c31' not found for module: 'testApp'` + `Unable to read beyond end of stream` - * zero coverage result (often only on CI but not on local) -``` -Calculating coverage result... -C:\Users\REDACTED\.nuget\packages\coverlet.msbuild\2.6.0\build\netstandard2.0\coverlet.msbuild.targets(21,5): warning : [coverlet] Hits file:'C:\Users\REDACTED\AppData\Local\Temp\testApp_ac32258b-fd4a-4bb4-824c-a79061e97c31' not found for module: 'testApp' [C:\Users\REDACTED\Documents\repo\testapp\testapp.Tests\testapp.Tests.csproj] -C:\Users\REDACTED\.nuget\packages\coverlet.msbuild\2.6.0\build\netstandard2.0\coverlet.msbuild.targets(21,5): warning : [coverlet] Hits file:'C:\Users\REDACTED\AppData\Local\Temp\testApp.Tests_ac32258b-fd4a-4bb4-824c-a79061e97c31' not found for module: 'testApp.Tests' [C:\Users\REDACTED\Documents\repo\testapp\testapp.Tests\testapp.Tests.csproj] - Generating report 'C:\Users\REDACTED\Documents\repo\testapp\lcov.info' - -+---------------+------+--------+--------+ -| Module | Line | Branch | Method | -+---------------+------+--------+--------+ -| testApp | 0% | 0% | 0% | -+---------------+------+--------+--------+ -| testApp.Tests | 0% | 100% | 0% | -+---------------+------+--------+--------+ + `warning : [coverlet] Hits file:'C:\Users\REDACTED\AppData\Local\Temp\testApp_ac32258b-fd4a-4bb4-824c-a79061e97c31' not found for module: 'testApp'` -+---------+------+--------+--------+ -| | Line | Branch | Method | -+---------+------+--------+--------+ -| Total | 0% | 0% | 0% | -+---------+------+--------+--------+ -| Average | 0% | 0% | 0% | -+---------+------+--------+--------+ -``` +* zero coverage result (often only on CI but not on local) -The issue is related to vstest platform https://github.com/microsoft/vstest/issues/1900#issuecomment-457488472 -``` -However if testhost doesn't shut down within 100ms(as the execution is completed, we expect it to shutdown fast). vstest.console forcefully kills the process. -``` + ```bash + Calculating coverage result... + C:\Users\REDACTED\.nuget\packages\coverlet.msbuild\2.6.0\build\netstandard2.0\coverlet.msbuild.targets(21,5): warning : [coverlet] Hits file:'C:\Users\REDACTED\AppData\Local\Temp\testApp_ac32258b-fd4a-4bb4-824c-a79061e97c31' not found for module: 'testApp' [C:\Users\REDACTED\Documents\repo\testapp\testapp.Tests\testapp.Tests.csproj] + C:\Users\REDACTED\.nuget\packages\coverlet.msbuild\2.6.0\build\netstandard2.0\coverlet.msbuild.targets(21,5): warning : [coverlet] Hits file:'C:\Users\REDACTED\AppData\Local\Temp\testApp.Tests_ac32258b-fd4a-4bb4-824c-a79061e97c31' not found for module: 'testApp.Tests' [C:\Users\REDACTED\Documents\repo\testapp\testapp.Tests\testapp.Tests.csproj] + Generating report 'C:\Users\REDACTED\Documents\repo\testapp\lcov.info' + + +---------------+------+--------+--------+ + | Module | Line | Branch | Method | + +---------------+------+--------+--------+ + | testApp | 0% | 0% | 0% | + +---------------+------+--------+--------+ + | testApp.Tests | 0% | 100% | 0% | + +---------------+------+--------+--------+ + + +---------+------+--------+--------+ + | | Line | Branch | Method | + +---------+------+--------+--------+ + | Total | 0% | 0% | 0% | + +---------+------+--------+--------+ + | Average | 0% | 0% | 0% | + +---------+------+--------+--------+ + ``` + +The issue is related to VSTest platform https://github.com/microsoft/vstest/issues/1900#issuecomment-457488472 -Coverlet collect and write hits data on process exist, if for some reason process is too slow to close will be killed and we cannot collect coverage result. -This happen also if there are other "piece of code" during testing that slow down process exit. -We found problem for instance with test that uses RabbitMQ. +> However if testhost doesn't shut down within 100ms (as the execution is completed, we expect it to shutdown fast). vstest.console forcefully kills the process. + +Coverlet collects and writes hits data on process exit, if for some reason the process is too slow to close, it will be killed and we cannot collect coverage result. +This happen also if there are other "piece of code" during testing that slow down process exit. We found problem for instance with tests that use RabbitMQ. *Solution:* -The only way to solve this issue is to use collectors integration https://github.com/coverlet-coverage/coverlet#vstest-integration-preferred-due-to-known-issue-supports-only-net-core-application. -With collector we're injected in test host through a in-proc collector that talk with vstest platform so we can signal when we end our work. -## 2) Upgrade `coverlet.collector` to version > 1.0.0 +The only way to get around this issue is to use collectors integration https://github.com/coverlet-coverage/coverlet#vstest-integration-preferred-due-to-known-issue-supports-only-net-core-application. With the collector, we're injected in test host through a in-proc collector that communicates with the VSTest platform so we can signal when we end our work. + +## Upgrade `coverlet.collector` to version > 1.0.0 + +*Affected drivers*: VSTest integration `dotnet test --collect:"XPlat Code Coverage"` -*Affected drivers*: vstest integration `dotnet test --collect:"XPlat Code Coverage"` +*Symptoms:* The same as "known issue 1". - *Symptoms:* The same as known issue 1. +There is a bug inside the VSTest platform: https://github.com/microsoft/vstest/issues/2205. -There is a bug inside vstest platform https://github.com/microsoft/vstest/issues/2205. -If you upgrade collector package with version greater than 1.0.0 in-proc collector won't be loaded so you could incur known issue number 1 and get zero coverage result +If you upgrade the collector package with a version greater than 1.0.0, in-proc collector won't be loaded so you could incur "known issue number 1" and get zero coverage result *Solutions:* + 1) Reference `Microsoft.NET.Test.Sdk` with version *greater than* 16.4.0 -For instance + ```xml ... @@ -67,7 +69,7 @@ For instance ... ``` -2) You can pass custom *runsetting* file like this +2) You can pass custom *coverage.runsettings* file like this ```xml @@ -90,80 +92,93 @@ For instance ``` + And pass it to command line + ```bash -dotnet test --settings runsetting +dotnet test --settings coverage.runsettings ``` -## 3) Nerdbank.GitVersioning and `/p:UseSourceLink=true` option + +## Nerdbank.GitVersioning and `/p:UseSourceLink=true` option *Affected drivers*: all drivers that support `/p:UseSourceLink=true` - *Symptoms:* some tool like SonarSource doesn't work well see https://github.com/tonerdo/coverlet/issues/482 +*Symptoms:* some tool like SonarSource doesn't work well see https://github.com/coverlet-coverage/coverlet/issues/482 + +`Nerdbank.GitVersioning` generates a version file on the fly but this file is not part of user solution and it's not commited to repo so the generated remote source file reference does not exit, i.e. + +``` +... + +... +``` + +*Solution:* we can exclude `Nerdbank.GitVersioning` autogenerated file from instrumentation using filters - `Nerdbank.GitVersioning` generates a version file on the fly but this file is not part of user solution and it's not commited to repo so the generated remote source file reference does not exit, i.e. - ``` - ... - - ... - ``` +```bash +/p:ExcludeByFile=\"**/*Json.Version.cs\" +``` - *Solution:* we can exclude `Nerdbank.GitVersioning` autogenerated file from instrumentation using filter - ```bash - /p:ExcludeByFile=\"**/*Json.Version.cs\" - ``` -## 4) Failed to resolve assembly during instrumentation +## Failed to resolve assembly during instrumentation *Affected drivers*: all drivers - *Symptoms:* during build/instrumentation you get exception like - ``` - [coverlet] Unable to instrument module: ..\UnitTests\bin\Debug\netcoreapp2.1\Core.Messaging.dll because : Failed to resolve assembly: 'Microsoft.Azure.ServiceBus, Version=3.4.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c' [..\UnitTests.csproj] - ``` - - In the instrumentation phase coverlet needs to load all reference used by your instrumented module. Sometime the build phase(out of coverlet control) does not copy those dll to output target because references are resolved for instance at runtime or in publish phase from nuget packages folders. - - *Solution:* we need to tell to msbild to copy nuget dll reference to output using msbuild switch `CopyLocalLockFileAssemblies` - ```bash - dotnet test /p:CollectCoverage=true /p:CopyLocalLockFileAssemblies=true - ``` - or adding the attribute `` to project - file - ```xml - - ... - true - ... +*Symptoms:* during build/instrumentation you may get an exception like: + +``` +[coverlet] Unable to instrument module: ..\UnitTests\bin\Debug\netcoreapp2.1\Core.Messaging.dll because : Failed to resolve assembly: 'Microsoft.Azure.ServiceBus, Version=3.4.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c' [..\UnitTests.csproj] +``` + +In the instrumentation phase, Coverlet needs to load all references used by your instrumented module. Sometimes the build phase (out of Coverlet's control) does not copy those dlls to the output folder because they are not resolved till runtime or at publish phase from the NuGet packages folders. + +*Solution:* we need to tell to MSBuild to copy all NuGet dll reference to output using MSBuild switch `CopyLocalLockFileAssemblies` + +```bash +dotnet test /p:CollectCoverage=true /p:CopyLocalLockFileAssemblies=true +``` + +or by adding the property `` to the project file + +```xml + +... + true +... - ``` - NB. This **DOESN'T ALWAYS WORK**, for instance in case of shared framework https://github.com/dotnet/cli/issues/12705#issuecomment-536686785 - We can do nothing at the moment this is a build behaviour out of our control. - This issue should not happen for .net runtime version >= 3.0 because the new default behavior is copy all assets to the build output https://github.com/dotnet/cli/issues/12705#issuecomment-535150372 +``` + +NB. This **DOESN'T ALWAYS WORK**, for example in case of the shared framework https://github.com/dotnet/cli/issues/12705#issuecomment-536686785 + +We can do nothing at the moment as this is a build behaviour out of our control. This issue should not happen for .NET runtime version >= 3.0 because the new default behavior is to copy all assets to the build output https://github.com/dotnet/cli/issues/12705#issuecomment-535150372 - In this case the only workaround for the moment is to *manually copy* missing dll to output folder https://github.com/tonerdo/coverlet/issues/560#issue-496440052 "The only reliable way to work around this problem is to drop the DLL in the unit tests project's bin\Release\netcoreapp2.2 directory." +In this case the only workaround at the moment is to *manually copy* missing dlls to the output folder: https://github.com/coverlet-coverage/coverlet/issues/560#issue-496440052 - ## 5) Tests fail if assembly is strong named +> The only reliable way to work around this problem is to drop the DLL in the unit tests project's bin\Release\netcoreapp2.2 directory. - *Affected drivers*: all drivers +## Tests fail if assembly is strong named - *Symptoms:* Running coverage on .NET Framework runtime(i.e. .NET 4.6.1) and get error like: - ``` - Failed Tests.MinMax.Min_AsyncSelector_Int32_4 +*Affected drivers*: all drivers + +*Symptoms:* Running coverage on .NET Framework runtime(i.e. .NET 4.6.1) and get error like: + +``` +Failed Tests.MinMax.Min_AsyncSelector_Int32_4 Error Message: System.TypeInitializationException : The type initializer for 'Tests.AsyncEnumerableTests' threw an exception. ---- System.IO.FileLoadException : Could not load file or assembly 'System.Linq.Async, Version=4.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263' or one of its dependencies. Strong name signature could not be verified. The assembly may have been tampered with, or it was delay signed but not fully signed with the correct private key. (Exception from HRESULT: 0x80131045) Stack Trace: - at Tests.AsyncEnumerableTests..ctor() - at Tests.MinMax..ctor() + at Tests.AsyncEnumerableTests..ctor() + at Tests.MinMax..ctor() ----- Inner Stack Trace ----- - at Tests.AsyncEnumerableTests..cctor() - ``` + at Tests.AsyncEnumerableTests..cctor() +``` - *Solution:* Looks like this is caused by xUnit's app domains. For `dotnet test`, it can be disabled with the following argument: `-- RunConfiguration.DisableAppDomain=true` +*Solution:* Looks like this is caused by xUnit's app domains. For `dotnet test`, it can be disabled with the following argument: `-- RunConfiguration.DisableAppDomain=true` - NB. Workaround doesn't work if test method itself explicitly creates an appdomain and uses shadow copying in order to test that the assembly behaves properly in those conditions. +NB. Workaround doesn't work if test method itself explicitly creates an appdomain and uses shadow copying in order to test that the assembly behaves properly in those conditions. -## 6) Code coverage returns NaN% +## Code coverage returns NaN% *Symptoms:* You are getting following result when running Coverlet within CI/CD pipeline: @@ -181,6 +196,6 @@ Stack Trace: +---------+------+--------+--------+ ``` -SUT(System Under Test) assembly is also not listed in MSBuild logs - "Instrumented module" is missing for your dll. +SUT (System Under Test) assembly is also not listed in MSBuild logs - "Instrumented module" is missing for your dll. -*Solution*: Check whether deterministic build is turned on for your solution, if so, follow instructions how to handle [Deterministic build](DeterministicBuild.md). +*Solution*: Check whether deterministic build is turned on for your solution, if so, follow the [instructions](DeterministicBuild.md) on how to handle deterministic builds. diff --git a/Documentation/MSBuildIntegration.md b/Documentation/MSBuildIntegration.md index 615b57d42..5d6675523 100644 --- a/Documentation/MSBuildIntegration.md +++ b/Documentation/MSBuildIntegration.md @@ -1,4 +1,4 @@ -# Coverlet Integration with MSBuild +# Coverlet integration with MSBuild In this mode, Coverlet doesn't require any additional setup other than including the NuGet package in the unit test project. It integrates with the `dotnet test` infrastructure built into the .NET Core CLI and when enabled, will automatically generate coverage results after tests are run. @@ -44,7 +44,7 @@ To specify a directory where all results will be written to (especially if using dotnet test /p:CollectCoverage=true /p:CoverletOutput='./results/' ``` -The coverlet MSBuild task sets the `CoverletReport` MSBuild item so that you can easily use the produced coverlet reports. For example, using [ReportGenerator](https://github.com/danielpalme/ReportGenerator#usage--command-line-parameters) to generate an html coverage report. +The Coverlet MSBuild task sets the `CoverletReport` MSBuild item so that you can easily use the produced coverage reports. For example, using [ReportGenerator](https://github.com/danielpalme/ReportGenerator#usage--command-line-parameters) to generate an html coverage report. ```xml @@ -63,7 +63,7 @@ dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=teamcity The currently supported [TeamCity statistics](https://confluence.jetbrains.com/display/TCD18/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-ServiceMessages) are: | TeamCity Statistic Key | Description | -| :--- | :--- | +|:------------------------|:-------------------------------| | CodeCoverageL | Line-level code coverage | | CodeCoverageB | Branch-level code coverage | | CodeCoverageM | Method-level code coverage | @@ -76,14 +76,13 @@ The currently supported [TeamCity statistics](https://confluence.jetbrains.com/d ## Merging Results -With Coverlet you can combine the output of multiple coverage runs into a single result. +With Coverlet, you can combine the output of multiple coverage runs into a single result. ```bash dotnet test /p:CollectCoverage=true /p:MergeWith='/path/to/result.json' ``` -The value given to `/p:MergeWith` **must** be a path to Coverlet's own json result format. The results in `result.json` will be read, and added to the new results written to by Coverlet. -[Check the sample](Examples.md). +The value given to `/p:MergeWith` **must** be a path to Coverlet's own json result format. The results in `result.json` will be read, and added to the new results written to by Coverlet. [Check the sample](Examples.md). ## Threshold @@ -117,7 +116,7 @@ dotnet test /p:CollectCoverage=true /p:Threshold=80 /p:ThresholdType=line /p:Thr ### Attributes -You can ignore a method an entire class or assembly from code coverage by creating and applying the `ExcludeFromCodeCoverage` attribute present in the `System.Diagnostics.CodeAnalysis` namespace. +You can ignore a method, an entire class or assembly from code coverage by creating and applying the `ExcludeFromCodeCoverage` attribute present in the `System.Diagnostics.CodeAnalysis` namespace. You can also ignore additional attributes by using the `ExcludeByAttribute` property (short name, i.e. the type name without the namespace, supported only): @@ -126,6 +125,7 @@ dotnet test /p:CollectCoverage=true /p:ExcludeByAttribute="Obsolete,GeneratedCod ``` ### Source Files + You can also ignore specific source files from code coverage using the `ExcludeByFile` property - Use single or multiple paths (separate by comma) - Use file path or directory path with globbing (e.g `dir1/*.cs`) @@ -135,6 +135,7 @@ dotnet test /p:CollectCoverage=true /p:ExcludeByFile=\"**/dir1/class1.cs,**/dir2 ``` ### Filters + Coverlet gives the ability to have fine grained control over what gets excluded using "filter expressions". Syntax: `/p:Exclude=[Assembly-Filter]Type-Filter` @@ -177,17 +178,19 @@ Methods that do not return can be marked with attributes to cause statements aft Attributes can be specified with the following syntax. Syntax: `/p:DoesNotReturnAttribute="DoesNotReturnAttribute,OtherAttribute"` -### Note for Powershell / VSTS users -To exclude or include multiple assemblies when using Powershell scripts or creating a .yaml file for a VSTS build ```%2c``` should be used as a separator. Msbuild will translate this symbol to ```,```. +### Note for Powershell / Azure DevOps users + +To exclude or include multiple assemblies when using Powershell scripts or creating a .yaml file for an Azure DevOps build ```%2c``` should be used as a separator. Msbuild will translate this symbol to ```,```. ```/p:Exclude="[*]*Examples?%2c[*]*Startup"``` -VSTS builds do not require double quotes to be unescaped: +Azure DevOps builds do not require double quotes to be unescaped: + ``` dotnet test --configuration $(buildConfiguration) --no-build /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=$(Build.SourcesDirectory)/TestResults/Coverage/ /p:Exclude="[MyAppName.DebugHost]*%2c[MyAppNamet.WebHost]*%2c[MyAppName.App]*" ``` -## Note for Linux users +### Note for Linux users [There is an issue with MSBuild on Linux](https://github.com/microsoft/msbuild/issues/3468) that affects the ability to escape quotes while specifying multiple comma-separated values. Linux MSBuild automatically translates `\` to `/` in properties, tasks, etc. before using them, which means if you specified `/p:CoverletOutputFormat=\"json,opencover\"` in an MSBuild script, it will be converted to `/p:CoverletOutputFormat=/"json,opencover/"` before execution. This yields an error similar to the following: @@ -202,4 +205,4 @@ The workaround is to use the .NET Core `dotnet msbuild` command instead of using ## SourceLink -Coverlet supports [SourceLink](https://github.com/dotnet/sourcelink) custom debug information contained in PDBs. When you specify the `--use-source-link` flag in the global tool or `/p:UseSourceLink=true` property in the MSBuild command, Coverlet will generate results that contain the URL to the source files in your source control instead of absolute file paths. +Coverlet supports [SourceLink](https://github.com/dotnet/sourcelink) custom debug information contained in PDBs. When you specify the `/p:UseSourceLink=true` property, Coverlet will generate results that contain the URL to the source files in your source control instead of local file paths. diff --git a/Documentation/ReleasePlan.md b/Documentation/ReleasePlan.md index d76ecc1fb..68c6b8910 100644 --- a/Documentation/ReleasePlan.md +++ b/Documentation/ReleasePlan.md @@ -15,32 +15,33 @@ Additional labels for pre-release and build metadata are available as extensions ## Release Calendar -We release 3 components as nuget packages: +We release 3 components as NuGet packages: -**coverlet.msbuild.nupkg** -**coverlet.console.nupkg** -**coverlet.collector.nupkg** +**coverlet.msbuild.nupkg** +**coverlet.console.nupkg** +**coverlet.collector.nupkg** We plan 1 release [once per quarter](https://en.wikipedia.org/wiki/Calendar_year) if there is *at least* 1 new commit of source code on master. This release may be a major, minor, or patch version upgrade from the previous release depending on impact to consumers. **We release intermediate packages in case of severe bug or to unblock users.** ### Current versions -| Package | **coverlet.msbuild** | -| :-------------: |:-------------:| -|**coverlet.msbuild** | 2.9.0 | -|**coverlet.console** | 1.7.2 | -|**coverlet.collector** | 1.3.0 | +| Package | Version | +|:----------------------|:--------| +|**coverlet.msbuild** | 2.9.0 | +|**coverlet.console** | 1.7.2 | +|**coverlet.collector** | 1.3.0 | ### Proposed next versions -We bump version based on Semantic Versioning 2.0.0 spec. -If we add features to **coverlet.core.dll** we bump MINOR version of all packages. -If we do breaking changes on **coverlet.core.dll** we bump MAJOR version of all packages. -We MANUALLY bump versions on production release, so we have different release plan between prod and nigntly packages. +We bump version based on Semantic Versioning 2.0.0 spec: -| Release Date | **coverlet.msbuild** | **coverlet.console** | **coverlet.collector** | **commit hash**| **notes** | -| :-------------: |:-------------:|:-------------:|:-------------:|:-------------:|:-------------:| +1. If we add features to **coverlet.core.dll** we bump MINOR version of all packages. +1. If we do breaking changes on **coverlet.core.dll** we bump MAJOR version of all packages. +1. We MANUALLY bump versions on production release, so we have different release plan between prod and nigntly packages. + +| Release Date | coverlet.msbuild | coverlet.console | coverlet.collector | commit hash| notes | +| :-------------- |:--------------|:--------------|:--------------|:--------------|:--------------| | <1 Jul 2020> | 3.0.0 | 3.0.0 | 3.0.0 | | Align versions | 30 May 2020 | 2.9.0 | 1.7.2 | 1.3.0 | 83a38d45b3f9c231d705bfed849efbf41b3aaa86 | deterministic build support | 04 April 2020 | 2.8.1 | 1.7.1 | 1.2.1 | 3f81828821d07d756e02a4105b2533cedf0b543c @@ -53,29 +54,28 @@ We MANUALLY bump versions on production release, so we have different release pl To get the list of commits between two version use git command ```bash - git log --oneline hashbefore currenthash +git log --oneline hashbefore currenthash ``` -# How to manually release packages to Nuget.org +# How to manually release packages to nuget.org -This is the steps to do to release new packages to Nuget.org +This is the steps to release new packages to nuget.org -1) Update projects version in file `version.json` in root of repo (remove `-preview.{height}` and adjust version) +1. Update projects version in file `version.json` in root of repo (remove `-preview.{height}` and adjust version) -Update core lib project file version https://github.com/tonerdo/coverlet/blob/master/src/coverlet.core/coverlet.core.csproj. -The version of core lib project file is the version we'll report on github repo releases https://github.com/tonerdo/coverlet/releases +Update core lib project file version https://github.com/coverlet-coverage/coverlet/blob/master/src/coverlet.core/coverlet.core.csproj. +The version of core lib project file is the version we'll report on github repo releases https://github.com/coverlet-coverage/coverlet/releases Do a PR and merge to master. -2) Clone repo, **remember to build packages from master and not from your fork or metadata links will point to your forked repo.** -Run `git log -5` from repo root to verify last commit. +2. Clone repo, **remember to build packages from master and not from your fork or metadata links will point to your forked repo.** . Run `git log -5` from repo root to verify last commit. +3. From new cloned, aligned and versions updated repo root run pack command -3) From new cloned, aligned and versions updated repo root run pack command -``` -dotnet pack -c release /p:TF_BUILD=true /p:PublicRelease=true -... - coverlet.console -> D:\git\coverlet\src\coverlet.console\bin\Release\netcoreapp2.2\coverlet.console.dll + ``` + dotnet pack -c release /p:TF_BUILD=true /p:PublicRelease=true + ... + coverlet.console -> D:\git\coverlet\src\coverlet.console\bin\Release\netcoreapp2.2\coverlet.console.dll coverlet.console -> D:\git\coverlet\src\coverlet.console\bin\Release\netcoreapp2.2\publish\ Successfully created package 'D:\git\coverlet\bin\Release\Packages\coverlet.msbuild.2.8.1.nupkg'. Successfully created package 'D:\git\coverlet\bin\Release\Packages\coverlet.msbuild.2.8.1.snupkg'. @@ -83,25 +83,26 @@ dotnet pack -c release /p:TF_BUILD=true /p:PublicRelease=true Successfully created package 'D:\git\coverlet\bin\Release\Packages\coverlet.console.1.7.1.snupkg'. Successfully created package 'D:\git\coverlet\bin\Release\Packages\coverlet.collector.1.2.1.nupkg'. Successfully created package 'D:\git\coverlet\bin\Release\Packages\coverlet.collector.1.2.1.snupkg'. -``` -4) Sign the packages using SignClient tool https://www.nuget.org/packages/SignClient - -```powershell -❯ SignClient "Sign" ` ->> --baseDirectory "REPO ROOT DIRECTORY\bin" ` ->> --input "**/*.nupkg" ` ->> --config "ROOT REPO DIRECTORY\eng\signclient.json" ` ->> --user "USER" ` ->> --secret "SECRET" ` ->> --name "Coverlet" ` ->> --description "Coverlet" ` ->> --descriptionUrl "https://github.com/coverlet-coverage/coverlet" -``` - -5) Upload *.nupkg files to Nuget.org site. **Check all metadata(url links, deterministic build etc...) before "Submit"** - -6) **On your fork**: -* Align to master -* Bump version by one(fix part) and re-add `-preview.{height}` -* Create release on repo https://github.com/tonerdo/coverlet/releases using https://github.com/tonerdo/coverlet/blob/master/src/coverlet.core/coverlet.core.csproj assembly version -* Update the [Release Plan](https://github.com/tonerdo/coverlet/blob/master/Documentation/ReleasePlan.md)(this document) and [ChangeLog](https://github.com/tonerdo/coverlet/blob/master/Documentation/Changelog.md) + ``` + +4. Sign the packages using SignClient tool https://www.nuget.org/packages/SignClient + + ```powershell + ❯ SignClient "Sign" ` + >> --baseDirectory "REPO ROOT DIRECTORY\bin" ` + >> --input "**/*.nupkg" ` + >> --config "ROOT REPO DIRECTORY\eng\signclient.json" ` + >> --user "USER" ` + >> --secret "SECRET" ` + >> --name "Coverlet" ` + >> --description "Coverlet" ` + >> --descriptionUrl "https://github.com/coverlet-coverage/coverlet" + ``` + +5. Upload *.nupkg files to Nuget.org site. **Check all metadata(url links, deterministic build etc...) before "Submit"** + +6. **On your fork**: + * Align to master + * Bump version by one (fix part) and re-add `-preview.{height}` + * Create release on repo https://github.com/coverlet-coverage/coverlet/releases using https://github.com/coverlet-coverage/coverlet/blob/master/src/coverlet.core/coverlet.core.csproj assembly version + * Update the [Release Plan](https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/ReleasePlan.md)(this document) and [ChangeLog](https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/Changelog.md) diff --git a/Documentation/Roadmap.md b/Documentation/Roadmap.md index efb8dbc74..1c08d4ae6 100644 --- a/Documentation/Roadmap.md +++ b/Documentation/Roadmap.md @@ -12,19 +12,19 @@ Default priority order "should" be: 1) Bugs: we should fix bugs as soon as possible and for first bugs related to coverage because this is the goal of coverlet. -Coverage bugs: https://github.com/tonerdo/coverlet/issues?q=is%3Aissue+is%3Aopen+label%3Abug+label%3Atenet-coverage -Other bugs: https://github.com/tonerdo/coverlet/issues?q=is%3Aissue+is%3Aopen+label%3Abug +Coverage bugs: https://github.com/coverlet-coverage/coverlet/issues?q=is%3Aissue+is%3Aopen+label%3Abug+label%3Atenet-coverage +Other bugs: https://github.com/coverlet-coverage/coverlet/issues?q=is%3Aissue+is%3Aopen+label%3Abug 2) New features: analyze and add new features, we have three drivers so features could be related to one of these. -Feature requests: https://github.com/tonerdo/coverlet/issues?q=is%3Aissue+is%3Aopen+label%3Afeature-request +Feature requests: https://github.com/coverlet-coverage/coverlet/issues?q=is%3Aissue+is%3Aopen+label%3Afeature-request 3) Performance: we never worked on performance aspect of coverlet, it makes sense for a "new project with some hope", but today coverlet is the facto the dotnet coverage tool, so we HAVE TO approach this aspect. -Performance issues: https://github.com/tonerdo/coverlet/issues?q=is%3Aissue+is%3Aopen+label%3Atenet-performance +Performance issues: https://github.com/coverlet-coverage/coverlet/issues?q=is%3Aissue+is%3Aopen+label%3Atenet-performance Some new features have got a `Discussion` label if we don't have and agreement yet on semantics. -Discussions: https://github.com/tonerdo/coverlet/issues?q=is%3Aissue+is%3Aopen+label%3Adiscussion +Discussions: https://github.com/coverlet-coverage/coverlet/issues?q=is%3Aissue+is%3Aopen+label%3Adiscussion ## New features roadmap @@ -32,14 +32,14 @@ This is the list of features we should develop soon as possible: ### High priority -- Allow merge reports solution wide on all flavours https://github.com/tonerdo/coverlet/issues/662 https://github.com/tonerdo/coverlet/issues/357 +- Allow merge reports solution wide on all flavours https://github.com/coverlet-coverage/coverlet/issues/662 https://github.com/coverlet-coverage/coverlet/issues/357 - Some perf improvements https://github.com/coverlet-coverage/coverlet/issues/836 ### Low priority -- Rethink hits reports strategy https://github.com/tonerdo/coverlet/issues/808 -- Support coverage on netfx for collectors integration https://github.com/tonerdo/coverlet/issues/705 +- Rethink hits reports strategy https://github.com/coverlet-coverage/coverlet/issues/808 +- Support coverage on netfx for collectors integration https://github.com/coverlet-coverage/coverlet/issues/705 ## Maintainers discussion channel diff --git a/Documentation/Troubleshooting.md b/Documentation/Troubleshooting.md index 3d8d00d09..6c57b7494 100644 --- a/Documentation/Troubleshooting.md +++ b/Documentation/Troubleshooting.md @@ -1,8 +1,9 @@ # Troubleshooting -## Msbuild integration +## MSBuild integration 1) Generate verbose log + ``` dotnet test test\coverlet.core.tests\coverlet.core.tests.csproj -c debug /p:CollectCoverage=true /p:CoverletOutputFormat=opencover /p:Include=[coverlet.*]* -verbosity:diagnostic -bl:msbuild.binlog -noconsolelogger ``` @@ -17,19 +18,24 @@ dotnet test test\coverlet.core.tests\coverlet.core.tests.csproj -c debug /p:Coll ``` dotnet test --collect:"XPlat Code Coverage" --settings runsettings --diag:log.txt ``` + You'll get logs file in same folder similar to + ``` log.datacollector.19-09-12_14-55-17_64755_5.txt log.host.19-09-12_14-55-18_82700_6.txt log.txt ``` Search inside with filter '[coverlet]' + ## Coverlet Global Tool ``` coverlet "C:\git\coverlet\test\coverlet.core.tests\bin\Debug\netcoreapp2.0\coverlet.core.tests.dll" --target "dotnet" --targetargs "test C:\git\coverlet\test\coverlet.core.tests --no-build" --verbosity detailed ``` + Sample output + ``` ... Instrumented module: 'C:\git\coverlet\test\coverlet.core.tests\bin\Debug\netcoreapp2.0\coverlet.core.dll' @@ -72,7 +78,7 @@ Hits file:'C:\Users\Marco\AppData\Local\Temp\coverlet.core_703263e9-21f0-4d1c-9c +---------+--------+--------+--------+ ``` -## Use local build(no collectors) +## Use local build (no collectors) Sometimes is useful test local updated source to fix issue. You can "load" your local build using simple switch: @@ -110,11 +116,10 @@ Build succeeded. Time Elapsed 00:00:07.42 D:\git\coverlet (fixjsonserializerbug -> origin) -λ - ``` * Go to repro project and run + ``` D:\git\Cake.Codecov\Source\Cake.Codecov.Tests (develop -> origin) λ dotnet test /p:CollectCoverage=true /p:Exclude="[xunit.*]*" /p:CoverletToolsPath=D:\git\coverlet\src\coverlet.msbuild.tasks\bin\Debug\netstandard2.0\ @@ -129,6 +134,7 @@ In this way you can add `Debug.Launch()` inside coverlet source and debug. To use/debug local collectors build we need to tell to our project to restore and use our local build nuget package. 1) Build local package + ``` C:\git\coverlet\src\coverlet.collector (master -> origin) λ dotnet pack @@ -177,9 +183,10 @@ Copyright (C) Microsoft Corporation. All rights reserved. - ``` + 4) Run test command + ``` dotnet test XUnitTestProject1\ --collect:"XPlat Code Coverage" ``` @@ -187,7 +194,8 @@ Copyright (C) Microsoft Corporation. All rights reserved. You can also attach/debug your code adding some line of code on collectors i.e. Attach using vs "Attach to Process" -```cs + +```csharp while(!System.Diagnostics.Debugger.IsAttached) { System.Threading.Thread.Sleep(1000); @@ -195,15 +203,19 @@ while(!System.Diagnostics.Debugger.IsAttached) ``` Fire attach -```cs + +```csharp System.Diagnostics.Debugger.Launch(); ``` -If you want debug in-process collector you need to set VSTEST_HOST_DEBUG(https://github.com/microsoft/vstest/issues/2158) environment variable +If you want debug in-process collector, you need to set VSTEST_HOST_DEBUG(https://github.com/microsoft/vstest/issues/2158) environment variable + ``` set VSTEST_HOST_DEBUG=1 ``` + Test host will wait for debugger + ``` Starting test execution, please wait... Logging Vstest Diagnostics in file: C:\git\coverletissue\collectorlog\XUnitTestProject1\log.txt @@ -211,26 +223,30 @@ Host debugging is enabled. Please attach debugger to testhost process to continu Process Id: 800, Name: dotnet ``` -**Every time you update code and rebuild new package remember to remove local nuget cache(`RMDIR "C:\Users\[winUser]\.nuget\packages\coverlet.collector" /S /Q`) otherwise you'll load old collector code because the package version wasn't changed** +**Every time you update code and rebuild new package, remember to remove local nuget cache (`RMDIR "C:\Users\[winUser]\.nuget\packages\coverlet.collector" /S /Q`) otherwise you'll load old collector code because the package version wasn't changed** ## Enable injected tracker log -Coverlet works thanks to ModuleTracker that is injected during instrumentation for every covered module. -This piece of code is run as a part of tests and doesn't have any connection with coverlet. +Coverlet works thanks to ModuleTracker that is injected during instrumentation for every covered module. This piece of code is run as a part of tests and doesn't have any connection with Coverlet. + We can collect logs from trackers through an enviroment variable + ``` set COVERLET_ENABLETRACKERLOG=1 ``` -When enabled, tracking event will be collected in log file near to module location. -File name will be something like `moduleName.dll_tracker.txt` and files with detailed hits will be in a folder named `TrackersHitsLog`. + +When enabled, tracking event will be collected in a log file near to module location. File name will be something like `moduleName.dll_tracker.txt` and files with detailed hits will be in a folder named `TrackersHitsLog`. ## Enable msbuild task instrumentation debugging You can live attach and debug msbuild tasks with `COVERLET_MSBUILD_INSTRUMENTATIONTASK_DEBUG` env variable + ``` set COVERLET_MSBUILD_INSTRUMENTATIONTASK_DEBUG=1 ``` + You'll get this message during test run + ``` dotnet test -p:Include="[test_coverage.]" -p:Exclude="[*.Test.*]*" -p:CollectCoverage=true -p:CoverletOutputFormat=cobertura -p:CoverletOutput=coverage.cobertura.xml Coverlet msbuild instrumentation task debugging is enabled. Please attach debugger to process to continue @@ -240,13 +256,15 @@ Process Id: 29228 Name: dotnet ## Enable collector instrumentation debugging You can live attach and debug collectors with `COVERLET_DATACOLLECTOR_OUTOFPROC_DEBUG` and `COVERLET_DATACOLLECTOR_INPROC_DEBUG` env variable + ``` set COVERLET_DATACOLLECTOR_OUTOFPROC_DEBUG=1 set COVERLET_DATACOLLECTOR_INPROC_DEBUG=1 ``` + You will be asked to attach a debugger through UI popup. To enable exceptions log for in-process data collectors + ``` set COVERLET_DATACOLLECTOR_INPROC_EXCEPTIONLOG_ENABLED=1 ``` - diff --git a/Documentation/VSTestIntegration.md b/Documentation/VSTestIntegration.md index 5471a00db..143904e08 100644 --- a/Documentation/VSTestIntegration.md +++ b/Documentation/VSTestIntegration.md @@ -1,6 +1,7 @@ # Coverlet integration with VSTest (a.k.a. Visual Studio Test Platform) -**Supported runtime versions:** +**Supported runtime versions**: + Before version `3.0.0` - .NET Core >= 2.0 - .NET Framework not fully supported(only out of process collector, could suffer of [known issue](https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/KnownIssues.md#1-vstest-stops-process-execution-earlydotnet-test)) @@ -9,8 +10,8 @@ Since version `3.0.0` - .NET Core >= 2.0 - .NET Framework >= 4.6.1 +As explained in quick start section, to use collectors you need to run *SDK v2.2.401* or newer and your project file must reference `coverlet.collector.dll` and a minimum version of `Microsoft.NET.Test.Sdk`. -As explained in quick start section to use collectors you need to run *SDK v2.2.401* or newer and your project file must reference `coverlet.collector.dll` and a minimum version of `Microsoft.NET.Test.Sdk`. A sample project file looks like: ```xml @@ -39,7 +40,9 @@ With correct reference in place you can run coverage through default dotnet test ``` dotnet test --collect:"XPlat Code Coverage" ``` + or + ``` dotnet publish ... @@ -48,9 +51,11 @@ dotnet publish ... dotnet vstest C:\project\bin\Debug\netcoreapp3.0\publish\testdll.dll --collect:"XPlat Code Coverage" ``` + As you can see in case of `vstest` verb you **must** publish project before. -At the end of tests you'll find the coverage file data under default vstest plat directory `TestResults` +At the end of tests you'll find the coverage file data under default VSTest platform directory `TestResults` + ``` Attachments: C:\git\coverlet\Documentation\Examples\VSTest\HelloWorld\XUnitTestProject1\TestResults\bc5e983b-d7a8-4f17-8c0a-8a8831a4a891\coverage.cobertura.xml @@ -59,8 +64,10 @@ Total tests: 1 Passed: 1 Total time: 2,5451 Seconds ``` -You can change the position of files using standard `dotnet test` switch `[-r|--results-directory]` -*NB: By design vstest platform will create your file under a random named folder(guid string) so if you need stable path to load file to some gui report system(i.e. coveralls, codecov, reportgenerator etc..) that doesn't support glob patterns or hierarchical search, you'll need to manually move resulting file to a predictable folder* + +You can change the position of files using standard `dotnet test` switch `[-r|--results-directory]` + +*NB: By design VSTest platform will create your file under a random named folder(guid string) so if you need stable path to load file to some gui report system(i.e. coveralls, codecov, reportgenerator etc..) that doesn't support glob patterns or hierarchical search, you'll need to manually move resulting file to a predictable folder* ## Coverlet options supported by VSTest integration @@ -68,29 +75,31 @@ At the moment VSTest integration doesn't support all features of msbuild and .NE We're working to fill the gaps. *PS: if you don't have any other way to merge reports(for instance your report generator doesn't support multi coverage file) you can for the moment exploit a trick reported by one of our contributor Daniel Paz(@p4p3) https://github.com/tonerdo/coverlet/pull/225#issuecomment-573896446* +### Default option (if you don't specify a runsettings file) + +| Option | Summary | +|:-------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +|Format | Results format in which coverage output is generated. Default format is cobertura. Supported format lcov, opencover, cobertura, teamcity, json (default coverlet proprietary format) | -#### Default option (if you don't specify a runsettings file) -| Option | Summary | -|-------------|------------------------------------| -|Format | Results format in which coverage output is generated. Default format is cobertura. Supported format lcov, opencover, cobertura, teamcity, json (default coverlet proprietary format)| +### Advanced Options (Supported via runsettings) -#### Advanced Options (Supported via runsettings) These are a list of options that are supported by coverlet. These can be specified as datacollector configurations in the runsettings. -| Option | Summary | -|------------- |------------------------------------------------------------------------------------------| -|Format | Coverage output format. These are either cobertura, json, lcov, opencover or teamcity as well as combinations of these formats. | -|Exclude | Exclude from code coverage analysing using filter expressions. | -|ExcludeByFile | Ignore specific source files from code coverage. | -|Include | Explicitly set what to include in code coverage analysis using filter expressions. | -|IncludeDirectory| Explicitly set which directories to include in code coverage analysis. | -|SingleHit | Specifies whether to limit code coverage hit reporting to a single hit for each location.| -|UseSourceLink | Specifies whether to use SourceLink URIs in place of file system paths. | -|IncludeTestAssembly | Include coverage of the test assembly. | -|SkipAutoProps | Neither track nor record auto-implemented properties. | -|DoesNotReturnAttribute | Methods marked with these attributes are known not to return, statements following them will be excluded from coverage | +| Option | Summary | +|:-------------------------|:----------------------------------------------------------------------------------------------------------------------------------| +| Format | Coverage output format. These are either cobertura, json, lcov, opencover or teamcity as well as combinations of these formats. | +| Exclude | Exclude from code coverage analysing using filter expressions. | +| ExcludeByFile | Ignore specific source files from code coverage. | +| Include | Explicitly set what to include in code coverage analysis using filter expressions. | +| IncludeDirectory | Explicitly set which directories to include in code coverage analysis. | +| SingleHit | Specifies whether to limit code coverage hit reporting to a single hit for each location. | +| UseSourceLink | Specifies whether to use SourceLink URIs in place of file system paths. | +| IncludeTestAssembly | Include coverage of the test assembly. | +| SkipAutoProps | Neither track nor record auto-implemented properties. | +| DoesNotReturnAttribute | Methods marked with these attributes are known not to return, statements following them will be excluded from coverage | How to specify these options via runsettings? + ``` @@ -124,10 +133,12 @@ This runsettings file can easily be provided using command line option as given Take a look at our [`HelloWorld`](Examples/VSTest/HelloWorld/HowTo.md) sample. -#### Passing runsettings arguments through commandline +### Passing runsettings arguments through commandline + +You can avoid passing a `runsettings` file to `dotnet test` driver by using the xml flat syntax in the command line. -You can avoid passing a `runsettings` file to `dotnet test` driver by using the xml flat syntax in the command line. For instance if you want to set the `Format` element as a runsettings option you can use this syntax: + ``` dotnet test --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=json,cobertura,lcov,teamcity,opencover ``` @@ -137,11 +148,11 @@ Take a look here for further information: https://github.com/microsoft/vstest-do ## How it works Coverlet integration is implemented with the help of [datacollectors](https://github.com/Microsoft/vstest-docs/blob/master/docs/extensions/datacollector.md). -When we specify `--collect:"XPlat Code Coverage"` vstest platform tries to load coverlet collectors inside `coverlet.collector.dll` +When we specify `--collect:"XPlat Code Coverage"` VSTest platform tries to load coverlet collectors inside `coverlet.collector.dll` -1. Outproc Datacollector : The outproc collector run in a separate process(datacollector.exe/datacollector.dll) than the process in which tests are being executed(testhost*.exe/testhost.dll). This datacollector is responsible for calling into coverlet APIs for instrumenting dlls, collecting coverage results and sending the coverage output file back to test platform. +1. Out-of-proc Datacollector: The outproc collector run in a separate process(datacollector.exe/datacollector.dll) than the process in which tests are being executed(testhost*.exe/testhost.dll). This datacollector is responsible for calling into Coverlet APIs for instrumenting dlls, collecting coverage results and sending the coverage output file back to test platform. -2. Inproc Datacollector : The inproc collector is loaded in the testhost process executing the tests. This collector will be needed to remove the dependency on the process exit handler to flush the hit files and avoid to hit this [serious known issue](https://github.com/tonerdo/coverlet/blob/master/Documentation/KnownIssues.md#1-vstest-stops-process-execution-earlydotnet-test) +2. In-proc Datacollector: The in-proc collector is loaded in the testhost process executing the tests. This collector will be needed to remove the dependency on the process exit handler to flush the hit files and avoid to hit this [serious known issue](https://github.com/tonerdo/coverlet/blob/master/Documentation/KnownIssues.md#1-vstest-stops-process-execution-earlydotnet-test) ## Known Issues diff --git a/README.md b/README.md index f07d36bbf..76519e9b8 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Coverlet -[![Build Status](https://dev.azure.com/tonerdo/coverlet/_apis/build/status/coverlet-coverage.coverlet?branchName=master)](https://dev.azure.com/tonerdo/coverlet/_build/latest?definitionId=5&branchName=master) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/tonerdo/coverlet/blob/master/LICENSE) +[![Build Status](https://dev.azure.com/tonerdo/coverlet/_apis/build/status/coverlet-coverage.coverlet?branchName=master)](https://dev.azure.com/tonerdo/coverlet/_build/latest?definitionId=5&branchName=master) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/coverlet-coverage/coverlet/blob/master/LICENSE) | Driver | Current version | Downloads | |---|---|---| @@ -34,7 +34,7 @@ Coverlet can be used through three different *drivers* Coverlet supports only SDK-style projects https://docs.microsoft.com/en-us/visualstudio/msbuild/how-to-use-project-sdk?view=vs-2019 -### VSTest Integration (preferred due to [known issue](https://github.com/tonerdo/coverlet/blob/master/Documentation/KnownIssues.md#1-vstest-stops-process-execution-earlydotnet-test) supports only .NET Core application) +### VSTest Integration (preferred due to [known issue](https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/KnownIssues.md#1-vstest-stops-process-execution-earlydotnet-test) supports only .NET Core application) At the moment collectors integration **does not support** .NET Framework application. @@ -62,7 +62,7 @@ See [documentation](Documentation/VSTestIntegration.md) for advanced usage. ``` -### MSBuild Integration (suffers of possible [known issue](https://github.com/tonerdo/coverlet/blob/master/Documentation/KnownIssues.md#1-vstest-stops-process-execution-earlydotnet-test)) +### MSBuild Integration (suffers of possible [known issue](https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/KnownIssues.md#1-vstest-stops-process-execution-earlydotnet-test)) ### Installation ```bash @@ -85,7 +85,7 @@ See [documentation](Documentation/MSBuildIntegration.md) for advanced usage. #### Requirements Requires a runtime that support _.NET Standard 2.0 and above_ -### .NET Global Tool ([guide](https://docs.microsoft.com/en-us/dotnet/core/tools/global-tools), suffers of possible [known issue](https://github.com/tonerdo/coverlet/blob/master/Documentation/KnownIssues.md#1-vstest-stops-process-execution-earlydotnet-test)) +### .NET Global Tool ([guide](https://docs.microsoft.com/en-us/dotnet/core/tools/global-tools), suffers from possible [known issue](https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/KnownIssues.md#1-vstest-stops-process-execution-earlydotnet-test)) ### Installation