Base layer manipulation (microsoft#1637)

* Simple baseLayerReader to export parentless layers

This is the inverse of the baseLayerWriter: It walks Files/ and
UtilityVM/Files/ (if present) and ignores the rest of the layer data,
as it will be recreated when the layer is imported.

Signed-off-by: Paul "TBBle" Hampson <[email protected]>

* Introduce hcsshim.ConvertToBaseLayer

This API allows turning any collection of files into a WCOW base layer.

It will create the necessary files in Files/ for
hcsshim.ProcessBaseLayer to function, validate the necessary files for
hcsshim.ProcessUtilityVMImage if UtilityVM/ exists, and then call those
two APIs to complete the process.

Calling this on a directory containing an untarred base layer OCI
tarball, gives a very similar outcome to passing the tar stream through
ociwclayer.ImportLayer.

The new API is used in `TestSCSIAddRemoveWCOW` to create nearly-empty
base layers for the scratch layers attached and removed from the utility
VM.

A wclayer command is also introduced: `makebaselayer` for testing and
validation purposes.

Signed-off-by: Paul "TBBle" Hampson <[email protected]>

* Include hard-linked files as hard-links in the tarstream

Signed-off-by: Paul "TBBle" Hampson <[email protected]>

* Use offline registry library to generate min hive

This change adds functions to generate valid, empty hives.

Signed-off-by: Gabriel Adrian Samfira <[email protected]>

* Rename ofreg.go and close key

Signed-off-by: Gabriel Adrian Samfira <[email protected]>

* Fix temp dir creation

Signed-off-by: Gabriel Adrian Samfira <[email protected]>

* Cleanup tests

Signed-off-by: Gabriel Adrian Samfira <[email protected]>

* Fix ORCloseHive definition

Signed-off-by: Gabriel Adrian Samfira <[email protected]>

* Remove unused ctx from baseLayerReader

Signed-off-by: Gabriel Adrian Samfira <[email protected]>

* Use string in sys definition and check for err

  * We can use string instead of *uint16 in the //sys definition and allow
mksyscall to generate the proper boilerplate.
  * do not shadow err if it's not nil

Signed-off-by: Gabriel Adrian Samfira <[email protected]>

* Close the r.proceed channel

Signed-off-by: Gabriel Adrian Samfira <[email protected]>

* Return if backup reader is nil

Signed-off-by: Gabriel Adrian Samfira <[email protected]>

---------

Signed-off-by: Paul "TBBle" Hampson <[email protected]>
Signed-off-by: Gabriel Adrian Samfira <[email protected]>
Co-authored-by: Paul "TBBle" Hampson <[email protected]>

wcow: support graceful termination of servercore containers (microsof…

…t#1416) (microsoft#1640)

* This commit includes the changes to enable graceful termination of WCOW containers

(cherry picked from commit 5cfbc2a)

Signed-off-by: Kirtana Ashok <[email protected]>
Co-authored-by: Kirtana Ashok <[email protected]>

fix: temp file leak during hash computation (microsoft#1641)

Fix a temp file leak when computing dmverity root hash. This
mainly affects `dmverity-vhd` tool and users may see their temp
storage filling up.

Signed-off-by: Maksim An <[email protected]>

test: Add CRI benchmarks for container operations (microsoft#1569)

* test: Add CRI benchmarks for container operations

Add CRI API benchmarks to `cri-containerd.test.exe`, and update helper
functions in `cri-containerd` to accept `tb testing.TB` instead of
`t *testing.T`.

Switched `cri-containerd` to `test\internal\flag` so that
`test\internal\require` can be used to check if features are present
since that implementation already mirrors what `cri-containerd` does,
but accepts a `testing.TB`, and deleted old
`test/internal/stringsetflag.go` file.

Renamed `scripts/Test-Functionl.ps1` to `scripts/Run-Tests.ps1`, and
updated it to run both functional and CRI tests and benchmarks.

Signed-off-by: Hamza El-Saawy <[email protected]>

* PR: clarifying comment

Signed-off-by: Hamza El-Saawy <[email protected]>

---------

Signed-off-by: Hamza El-Saawy <[email protected]>

adding tarball support for generating root layer hashes (microsoft#1600)

This will be used in a "clean-room" scenario for use to security policy generation. Clean-room in this instance is for generating a security policy on computers without internet access or the docker daemon (or similar) running.

The &tag passed in defaults to "latest" if only the image name is passed in. If the value of the tag is nil, the tarball must only have one image in it. Otherwise, many images can be stored in the tarball and be searched by their image name and tag.

Signed-off-by: Seth Hollandsworth <[email protected]>

[release/0.8] Remove blocking wait on container exit for every exec c…

…reated (microsoft#1605)

* Remove blocking wait on container exit for every exec created

Commit fixes the memory leak seen in the shim.
It removes creation of channel that waits on container exit
for every new exec. Instead, the container wait channel is exposed
through WaitChannel() function which callers can use to decide
if container has exited or not.

It also fixes CI to use the same version of golang and updates golangci-lint version to v1.48

Signed-off-by: Kirtana Ashok <[email protected]>
(cherry picked from commit 5fc00c5)
Signed-off-by: Kirtana Ashok <[email protected]>

* update golangci-lint version

Signed-off-by: Kirtana Ashok <[email protected]>

Signed-off-by: Kirtana Ashok <[email protected]>
Co-authored-by: Kirtana Ashok <[email protected]>

Merge pull request microsoft#1601 from kiashok/memLeakShim

Remove blocking on container exit for every new exec created

Remove blocking wait on container exit for every exec created (micros…

…oft#1604)

Commit fixes the memory leak seen in the shim.
It removes creation of channel that waits on container exit
for every new exec. Instead, the container wait channel is exposed
through WaitChannel() function which callers can use to decide
if container has exited or not.

Signed-off-by: Kirtana Ashok <[email protected]>
(cherry picked from commit 5fc00c5)
Signed-off-by: Kirtana Ashok <[email protected]>

Signed-off-by: Kirtana Ashok <[email protected]>
Co-authored-by: Kirtana Ashok <[email protected]>

plumb AMD certs to workload containers (microsoft#1549)

confidential containers: Add AMD cert plumbing

Add logic to plumb AMD certificates to workload containers. The
assumption is that the certificates will be "fresh enough" for
necessary attestation and key release by the workflow and third
party services.

Additionally add error logging when UVM reference info file
is not found

Signed-off-by: Maksim An <[email protected]>

Merge pull request microsoft#1554 from kiashok/terminateOnTimeout

[release/0.9] Call container.Terminate() on shutdown timeouts