Skip to content

Commit

Permalink
Use 1ES-hosted pools in .NET Core checkin pipeline (Azure#6655)
Browse files Browse the repository at this point in the history 
Ubuntu 18.04 MS-hosted agents in Azure Pipelines will soon be deprecated. The original intent of this change was to convert our .NET checkin pipeline from MS-hosted agents to 1ES-agents, which it does.

However, rather than converting the Windows MS-hosted agent used for code coverage to a _Windows_ 1ES-hosted agent, I decided to update our code coverage job to work on a Linux 1ES-hosted agent, since our latest versions of IoT Edge don't directly target Windows. This ended up being a larger change, and includes:
- Updating our code coverage .runsettings file to use configuration settings supported by Coverlet ([doc](https://github.com/coverlet-coverage/coverlet/blob/1f1a10143517da4e7b48b50caf16b407abe5a233/Documentation/VSTestIntegration.mdhttps://github.com/coverlet-coverage/coverlet/blob/1f1a10143517da4e7b48b50caf16b407abe5a233/Documentation/VSTestIntegration.md))
- Replacing verbose Bash@3 task syntax to the shortcut "script" syntax for readability
- Changing from the Windows-only "code coverage" collector to the "xplat code coverage" collector (which uses Coverlet)
- Switching from line coverage to branch coverage because it's supposed to be a more accurate metric per some documentation I read
- Reducing the coverage goal to 50%. As near as I can tell, the new code coverage run is measuring the same assemblies as the old run, but due to differences in how it measures and the switch to branch coverage, the resulting number is about 15% lower. We can address gaps in coverage with other PRs.
- Updating to a newer version of the Microsoft.NET.Test.Sdk, Moq, and xunit* dependencies in our tests, which fixed a problem I was seeing with Coverlet
- Adding Coverlet as a dependency to all unit test projects
- Fixing some xunit compile errors resulting from the updated xunit version
- Building Microsoft.Azure.Devices.Edge.Util.Test.Common as a library instead of an app, so that xunit doesn't look for unit tests in that assembly
- Installing another package (needed by rocksdb unit tests) in installPrereqs.sh

One benefit of the switch to Coverlet/Cobertura is that we get a nice, formatted report in the Code Coverage tag in the pipeline UI!

_Note_: due to some reformatting in dotnet-checkin.yaml, it will be easier to review the changes there if you hide whitespace.

## Azure IoT Edge PR checklist:
  • Loading branch information
@damonbarry
damonbarry authored Nov 2, 2022
1 parent d796a36 commit aa82555
Show file tree
Hide file tree
Showing 46 changed files with 265 additions and 352 deletions.
155 changes: 12 additions & 143 deletions CodeCoverage.runsettings
View file
Original file line number Diff line number Diff line change
@@ -1,151 +1,20 @@
<?xml version="1.0" encoding="utf-8"?>
<!-- File name extension must be .runsettings -->
<RunSettings>
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
<DataCollector friendlyName="XPlat Code Coverage">
<Configuration>
<CodeCoverage>
<!--
Additional paths to search for .pdb (symbol) files. Symbols must be found for modules to be instrumented.
If .pdb files are in the same folder as the .dll or .exe files, they are automatically found. Otherwise, specify them here.
Note that searching for symbols increases code coverage runtime. So keep this small and local.
-->
<!--
<SymbolSearchPaths>
<Path>C:\Users\User\Documents\Visual Studio 2012\Projects\ProjectX\bin\Debug</Path>
<Path>\\mybuildshare\builds\ProjectX</Path>
</SymbolSearchPaths>
-->

<!--
About include/exclude lists:
Empty "Include" clauses imply all; empty "Exclude" clauses imply none.
Each element in the list is a regular expression (ECMAScript syntax). See http://msdn.microsoft.com/library/2k3te2cs.aspx.
An item must first match at least one entry in the include list to be included.
Included items must then not match any entries in the exclude list to remain included.
-->

<!-- Match assembly file paths: -->
<ModulePaths>
<Include>
<ModulePath>.*\.dll$</ModulePath>
<ModulePath>.*\.exe$</ModulePath>
</Include>
<Exclude>
<!-- test/prod dependencies -->
<ModulePath>.*windowsazure\..*</ModulePath>
<ModulePath>.*moq\..*</ModulePath>
<ModulePath>.*app.metrics.*\..*</ModulePath>
<ModulePath>.*autofac.*</ModulePath>
<ModulePath>.*fluentassertions.*</ModulePath>
<ModulePath>.*microsoft.codeanalysis\..*</ModulePath>
<ModulePath>.*microsoft.identitymodel\..*</ModulePath>
<ModulePath>.*newtonsoft\..*</ModulePath>
<ModulePath>.*nito\..*</ModulePath>
<ModulePath>.*nunit.framework\..*</ModulePath>
<ModulePath>.*nunit3.testadapter\..*</ModulePath>
<ModulePath>.*prometheus-net.*</ModulePath>
<ModulePath>.*protobuf-net.*</ModulePath>
<ModulePath>.*stackexchange\..*</ModulePath>
<ModulePath>.*system.identitymodel\..*</ModulePath>
<ModulePath>.*system.reactive.*</ModulePath>
<ModulePath>.*system.linq.async.*</ModulePath>
<ModulePath>.*xunit\..*</ModulePath>
<ModulePath>.*edgex509authdownstreamdevice.*\..*</ModulePath>
<ModulePath>.*edgedownstreamdevice.*\..*</ModulePath>
<!-- test binaries -->
<ModulePath>.*test\..*</ModulePath>
<!-- test/sample edge modules -->
<ModulePath>.*cloudtodevicemessagetester\..*</ModulePath>
<ModulePath>.*deploymenttester\..*</ModulePath>
<ModulePath>.*directmethodreceiver\..*</ModulePath>
<ModulePath>.*directmethodsender\..*</ModulePath>
<ModulePath>.*edgehubrestarttester\..*</ModulePath>
<ModulePath>.*iotedgequickstart\..*</ModulePath>
<ModulePath>.*leafdevice\..*</ModulePath>
<ModulePath>.*load-gen\..*</ModulePath>
<ModulePath>.*metricscollector\..*</ModulePath>
<ModulePath>.*metricsvalidator\..*</ModulePath>
<ModulePath>.*numberlogger\..*</ModulePath>
<ModulePath>.*modulerestarter\..*</ModulePath>
<ModulePath>.*networkcontroller\..*</ModulePath>
<ModulePath>.*relayer\..*</ModulePath>
<ModulePath>.*simulatedtemperaturesensor\..*</ModulePath>
<ModulePath>.*temperaturefilter\..*</ModulePath>
<ModulePath>.*testanalyzer\..*</ModulePath>
<ModulePath>.*testresultcoordinator\..*</ModulePath>
<ModulePath>.*twintester\..*</ModulePath>
<ModulePath>.*microsoft.azure.webjobs.*</ModulePath>
</Exclude>
</ModulePaths>

<!-- Match fully qualified names of functions: -->
<!-- (Use "\." to delimit namespaces in C# or Visual Basic, "::" in C++.) -->
<Functions>
<Exclude>
<Function>^Fabrikam\.UnitTest\..*</Function>
<Function>^std::.*</Function>
<Function>^ATL::.*</Function>
<Function>.*::__GetTestMethodInfo.*</Function>
<Function>^Microsoft::VisualStudio::CppCodeCoverageFramework::.*</Function>
<Function>^Microsoft::VisualStudio::CppUnitTestFramework::.*</Function>
</Exclude>
</Functions>

<!-- Match attributes on any code element: -->
<Attributes>
<Exclude>
<!-- Don't forget "Attribute" at the end of the name -->
<Attribute>^System\.Diagnostics\.DebuggerHiddenAttribute$</Attribute>
<Attribute>^System\.Diagnostics\.DebuggerNonUserCodeAttribute$</Attribute>
<Attribute>^System\.Runtime\.CompilerServices.CompilerGeneratedAttribute$</Attribute>
<Attribute>^System\.CodeDom\.Compiler.GeneratedCodeAttribute$</Attribute>
<Attribute>^System\.Diagnostics\.CodeAnalysis.ExcludeFromCodeCoverageAttribute$</Attribute>
</Exclude>
</Attributes>

<!-- Match the path of the source files in which each method is defined: -->
<Sources>
<Exclude>
<Source>.*\\atlmfc\\.*</Source>
<Source>.*\\vctools\\.*</Source>
<Source>.*\\public\\sdk\\.*</Source>
<Source>.*\\microsoft sdks\\.*</Source>
<Source>.*\\vc\\include\\.*</Source>
<Source>.*\\edge-util\\src\\es6numberserializer\\.*</Source>
<Source>.*\\edge-util\\src\\jsoncanonicalizer\\.*</Source>
</Exclude>
</Sources>

<!-- Match the company name property in the assembly: -->
<CompanyNames>
<Exclude>
<CompanyName>.*microsoft.*</CompanyName>
</Exclude>
</CompanyNames>

<!-- Match the public key token of a signed assembly: -->
<PublicKeyTokens>
<!-- Exclude Visual Studio extensions: -->
<Exclude>
<PublicKeyToken>^B77A5C561934E089$</PublicKeyToken>
<PublicKeyToken>^B03F5F7F11D50A3A$</PublicKeyToken>
<PublicKeyToken>^31BF3856AD364E35$</PublicKeyToken>
<PublicKeyToken>^89845DCD8080CC91$</PublicKeyToken>
<PublicKeyToken>^71E9BCE111E9429C$</PublicKeyToken>
<PublicKeyToken>^8F50407C4E9E73B6$</PublicKeyToken>
<PublicKeyToken>^E361AF139669C375$</PublicKeyToken>
</Exclude>
</PublicKeyTokens>

<!-- We recommend you do not change the following values: -->
<UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
<AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
<CollectFromChildProcesses>True</CollectFromChildProcesses>
<CollectAspDotNet>False</CollectAspDotNet>

</CodeCoverage>
<IncludeTestAssembly>false</IncludeTestAssembly>
<Include>
[Microsoft.Azure.Devices.Edge.*]*,
[Microsoft.Azure.Devices.Routing.Core]*,
[Microsoft.Azure.WebJobs.Extensions.EdgeHub]*,
</Include>
<Exclude>
[Microsoft.Azure.Devices.Edge.*Test]*,
[Microsoft.Azure.Devices.Edge.*Test.Common]*,
[Microsoft.Azure.Devices.Edge.ModuleUtil]*,
</Exclude>
</Configuration>
</DataCollector>
</DataCollectors>
Expand Down
191 changes: 93 additions & 98 deletions builds/checkin/dotnet.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,111 +2,106 @@ trigger: none
pr:
branches:
include:
- main
- release/*
- main
- release/*

jobs:
################################################################################
- job: check_run_pipeline
- job: check_run_pipeline
################################################################################
displayName: Check pipeline preconditions (changes ARE NOT in either edgelet, docs, or mqtt folder)
pool:
vmImage: "ubuntu-20.04"
steps:
- bash: |
git log -m -1 --name-only --first-parent --pretty="" | egrep -i -v '^(edgelet|doc|mqtt)'
if [[ $? == 0 ]]; then
echo "Detected changes outside of edgelet, docs and mqtt folders"
echo "##vso[task.setvariable variable=RUN_PIPELINE;isOutput=true]TRUE"
fi
displayName: Check changes in sources
name: check_files
displayName: Check pipeline preconditions (changes ARE NOT in either edgelet, docs, or mqtt folder)
pool:
name: $(pool.linux.name)
demands:
- ImageOverride -equals agent-aziotedge-ubuntu-20.04-msmoby
steps:
- bash: |
git log -m -1 --name-only --first-parent --pretty="" | egrep -i -v '^(edgelet|doc|mqtt)'
if [[ $? == 0 ]]; then
echo "Detected changes outside of edgelet, docs and mqtt folders"
echo "##vso[task.setvariable variable=RUN_PIPELINE;isOutput=true]TRUE"
fi
displayName: Check changes in sources
name: check_files
################################################################################
- job: linux_amd64
- job: linux_amd64
################################################################################
displayName: Linux amd64
dependsOn: check_run_pipeline
condition: eq(dependencies.check_run_pipeline.outputs['check_files.RUN_PIPELINE'], 'true')
pool:
vmImage: "ubuntu-20.04"
steps:
- task: Bash@3
displayName: Install Prerequisites
inputs:
filePath: scripts/linux/installPrereqs.sh
- task: Bash@3
displayName: Build
inputs:
filePath: scripts/linux/buildBranch.sh
arguments: -c "$(configuration)"
- task: Bash@3
displayName: Test
inputs:
filePath: scripts/linux/runTests.sh
arguments: '"--filter Category=Unit&Category!=GetLogsTests"'
- task: Bash@3
displayName: Run GetLogsTests Tests
inputs:
filePath: scripts/linux/runTests.sh
arguments: '"--filter Category=GetLogsTests"'
- task: PublishTestResults@2
displayName: Publish Test Results
inputs:
testRunner: VSTest
testResultsFiles: "**/TestResults/*.trx"
condition: succeededOrFailed()
displayName: Linux amd64
dependsOn: check_run_pipeline
condition: eq(dependencies.check_run_pipeline.outputs['check_files.RUN_PIPELINE'], 'true')
pool:
name: $(pool.linux.name)
demands:
- ImageOverride -equals agent-aziotedge-ubuntu-20.04-msmoby
steps:
- script: scripts/linux/installPrereqs.sh
displayName: Install Prerequisites
- script: scripts/linux/buildBranch.sh -c '$(configuration)'
displayName: Build
- script: scripts/linux/runTests.sh '--filter Category=Unit&Category!=GetLogsTests'
displayName: Test
- script: scripts/linux/runTests.sh '--filter Category=GetLogsTests'
displayName: Run GetLogsTests Tests
- task: PublishTestResults@2
displayName: Publish Test Results
inputs:
testRunner: VSTest
testResultsFiles: '**/TestResults/*.trx'
condition: succeededOrFailed()

################################################################################
- job: code_coverage
- job: code_coverage
################################################################################
displayName: Code coverage
dependsOn: check_run_pipeline
condition: eq(dependencies.check_run_pipeline.outputs['check_files.RUN_PIPELINE'], 'true')
variables:
coverage.goal: 60
pool:
vmImage: windows-2022
steps:
- task: DotNetCoreCLI@2
displayName: Build
inputs:
command: build
arguments: '-o target'
- task: VSTest@2
displayName: Run unit tests with code coverage
inputs:
testSelector: testAssemblies
testAssemblyVer2: |
target\*Test.dll
testFiltercriteria: Category=Unit&Category!=GetLogsTests
runInParallel: true
runTestsInIsolation: true
codeCoverageEnabled: true
runSettingsFile: CodeCoverage.runsettings
publishRunAttachments: true
- task: VSTest@2
displayName: Run GetLogsTests unit tests with code coverage
inputs:
testSelector: testAssemblies
testAssemblyVer2: |
target\*Test.dll
testFiltercriteria: Category=GetLogsTests
runInParallel: true
runTestsInIsolation: true
codeCoverageEnabled: true
runSettingsFile: CodeCoverage.runsettings
publishRunAttachments: true
- task: PublishTestResults@2
displayName: Publish code coverage
inputs:
testRunner: VSTest
testResultsFiles: '$(Agent.TempDirectory)/TestResults/*.trx'
condition: succeededOrFailed()
- task: mspremier.BuildQualityChecks.QualityChecks-task.BuildQualityChecks@5
displayName: 'Check build quality'
inputs:
checkCoverage: true
coverageFailOption: fixed
coverageType: lines
coverageThreshold: $(coverage.goal)
displayName: Code coverage
dependsOn: check_run_pipeline
condition: eq(dependencies.check_run_pipeline.outputs['check_files.RUN_PIPELINE'], 'true')
variables:
coverage.goal: 50
pool:
name: $(pool.linux.name)
demands:
- ImageOverride -equals agent-aziotedge-ubuntu-20.04-msmoby
steps:
- script: scripts/linux/installPrereqs.sh
displayName: Install Prerequisites
- script: |
dotnet test \
--logger trx \
--results-directory '$(Agent.TempDirectory)' \
--filter 'Category=Unit&Category!=GetLogsTests' \
--collect 'xplat code coverage' \
--settings CodeCoverage.runsettings
displayName: Run unit tests with code coverage
- script: |
dotnet test 'edge-agent/test/Microsoft.Azure.Devices.Edge.Agent.Core.Test' \
--logger trx \
--results-directory '$(Agent.TempDirectory)' \
--filter 'Category=GetLogsTests' \
--collect 'xplat code coverage' \
--settings CodeCoverage.runsettings
displayName: Run GetLogsTests unit tests with code coverage
condition: succeededOrFailed()
- script: |
dotnet new tool-manifest
dotnet tool install dotnet-reportgenerator-globaltool
dotnet tool run reportgenerator \
-reports:$(Agent.TempDirectory)/**/coverage.cobertura.xml \
-targetdir:$(Build.SourcesDirectory) \
-reporttypes:"Cobertura"
displayName: Generate coverage report
condition: succeededOrFailed()
- task: PublishCodeCoverageResults@1
displayName: Publish coverage report
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: '$(Build.SourcesDirectory)/Cobertura.xml'
condition: succeededOrFailed()
- task: BuildQualityChecks@8
displayName: 'Check build quality'
inputs:
checkCoverage: true
coverageFailOption: fixed
coverageType: branch
coverageThreshold: $(coverage.goal)
condition: succeededOrFailed()
12 changes: 8 additions & 4 deletions ...ft.Azure.Devices.Edge.Agent.Core.Test/Microsoft.Azure.Devices.Edge.Agent.Core.Test.csproj
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,14 @@
</PropertyGroup>

<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.4.0" />
<PackageReference Include="Moq" Version="4.10.1" />
<PackageReference Include="xunit" Version="2.4.1" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.4.1">
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.3.1" />
<PackageReference Include="Moq" Version="4.18.2" />
<PackageReference Include="xunit" Version="2.4.2" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.4.5">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers</IncludeAssets>
</PackageReference>
<PackageReference Include="coverlet.collector" Version="3.1.2">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers</IncludeAssets>
</PackageReference>
Expand Down
Loading

0 comments on commit aa82555

Please sign in to comment.