[ILASM] Add support for deterministic builds and PDB checksums

[amanasifkhalid]

Follow-up to #85344. Fixes #8293. Fixes #62484.

This PR simply pulls in the SHA-256 wrappers added in #109559 into the prototype in #85344.

[amanasifkhalid]

The determinism tests were failing because I was using different assembly names as inputs into ilasm, which would change the CodeView debug directory entry. This should be fixed

[amanasifkhalid]

/azp run runtime-coreclr ilasm

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[amanasifkhalid]

/azp run runtime-coreclr ilasm

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[amanasifkhalid]

/azp run runtime-coreclr ilasm

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jkotas]

Was it problematic in some way to set the file name here?

[amanasifkhalid]

Looks like a no-op change -- reverted.

[amanasifkhalid]

Moving the assemblies produced by ILAsm around to compare them in the determinism tests was causing sporadic failures on Windows. @jkotas are you ok with me reverting to comparing the assemblies' hashes?

[amanasifkhalid]

/azp run runtime-coreclr ilasm

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jkotas]

Moving the assemblies produced by ILAsm around to compare them in the determinism tests was causing sporadic failures on Windows. @jkotas are you ok with me reverting to comparing the assemblies' hashes?

What is an example of the sporadic failure that you are trying to work around?

[amanasifkhalid]

What is an example of the sporadic failure that you are trying to work around?

From the last run (example log), after running the initial ILDasm/ILAsm roundtrip, the os.replace() call to rename the resultant binary to <binary name>.base fails for seemingly random tests. If the prior ilasm run failed to produce the binary, then I'd expect the roundtrip tests to have terminated before getting to the determinism leg. And unlike os.rename(), os.replace() is supposed to be able to handle platform-specific semantics for overwriting files (i.e. no error on Windows if the new filename already exists). I haven't been able to reproduce this locally, though I wonder if some other process is still holding onto the binary when we try to rename it; I suspect the hashing approach isn't failing because it only needs to read the binary.

[jkotas]

Antivirus scanners tend to lock the binary after it was created for some time on Windows. It is best to avoid renaming or deleting the binary shortly after it was created. If you do that, you typically have to have a retry loop around it to make it reliable.

Is it possible to change the test such that it does not rename or delete the binaries shortly after they are created?

[amanasifkhalid]

Is it possible to change the test such that it does not rename or delete the binaries shortly after they are created?

The problem is the output name we give to ILAsm has to be consistent to not break determinism, but we also don't want the second run of ILAsm to overwrite the previous assembly since we need to compare their contents. We can avoid renaming the files by reading each binary completely into memory after each ILAsm invocation, but that seems impractical for large assemblies. With the hashing strategy, we don't have to keep each file completely in memory to do the comparison.

[jkotas]

The problem is the output name we give to ILAsm has to be consistent to not break determinism,

Does the full path need to be the same or just the file name?

[amanasifkhalid]

I believe the whole path has to be the same, as the path of the corresponding PDB is included in the CodeView entry. Ex:

005AFF2B cv 42 00002160 360 Format: RSDS, {E4632B33-1CEC-8C2E-513D-F9C4DF5B2A1E}, 1, C:\wk\runtime2\IL-RT\trythrowexcept_d.pdb

[jkotas]

I suspect that you may see intermittent failures with the hashing approach as well since you are still overwriting a recently created binary. The hashing may have reduced the probability of the intermittent failures by adding a delay that is sufficient for antivirus to finish the scan and let the binary go most of the time.

I do not have strong opinion about this. If you want to go with the hashing, it is fine with me.

[amanasifkhalid]

Sounds good, I'll keep this in mind. Thank you for the reviews!

ILAsm CI failure on macOS looks related to infra.