Update kuttl e2e to work with OCP on AWS

[JohnStrunk]

Describe what this PR does
Adds updates to the kuttl-based e2e suite so that it works in OpenShift on AWS

Overview of updates:

• EBS volumes can't be cloned, so CopyMethod must be Snapshot
• EBS volumes are locked to one AZ. The "verify" task must mount both the source and dest to do the compare, so they must both be provisioned from the same AZ. This is done via Pod affinity rules.
  • Adding pod affinity rules required an update to the Kind script to assign a topology label to the (single) kind node so that pods could still be scheduled when running there.
• Due to multi-node environment, the RWO volumes can only be accessed by a single Pod at a time, so the source bust be shut down prior to the verify job
• The source job needs to call sync before the snapshot happens or there's no guarantee the latest data will make it into the EBS snapshot.
• Mover containers all needed to call sync before exiting to ensure snapshots have all the data

Is there anything that requires special attention?

Related issues:

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JohnStrunk

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [JohnStrunk]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[JohnStrunk]

cc: @sallyom

[codecov]

Codecov Report

Merging #214 (16a9017) into master (9eb97fc) will decrease coverage by 0.1%.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           master    #214     +/-   ##
========================================
- Coverage    76.3%   76.1%   -0.2%     
========================================
  Files           9       9             
  Lines        1956    1956             
========================================
- Hits         1493    1490      -3     
- Misses        359     360      +1     
- Partials      104     106      +2     

Impacted Files	Coverage Δ
controllers/replicationsource_controller.go	77.8% <0.0%> (-0.6%)	⬇️
controllers/volumehandler.go	68.6% <0.0%> (ø)

[JohnStrunk]

For testing the e2e on OCP/AWS, I have:

• Used latest nightly CI. This has a default VSC called csi-aws-vsc that works ootb with the gp2-csi driver
• Switched the gp2-csi driver to be default SC
• Installed Minio:
  helm install --create-namespace -n minio --set accessKey.password=access --set secretKey.password=password --set defaultBuckets=mybucket --set containerSecurityContext.enabled=false --set podSecurityContext.enabled=false minio bitnami/minio
• Started Scribe:
  helm upgrade --install --create-namespace -n scribe-system --set image.tag=latest --set rclone.tag=latest --set rclone.repository=quay.io/johnstrunk/scribe-mover-rclone --set restic.tag=latest --set rsync.tag=latest --set rsync.repository=quay.io/johnstrunk/scribe-mover-rsync --set metrics.disableAuth=true scribe ./helm/scribe
  Note that the above is using the containers from my personal repo. That's just to pick up the changes to the movers in this PR. It won't be needed once this merges
• Ran the tests: make test-e2e

--- PASS: kuttl (349.75s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/simple-rsync (118.46s)
        --- PASS: kuttl/harness/restic-with-manual-trigger (197.72s)
        --- PASS: kuttl/harness/simple-rclone (144.57s)
        --- PASS: kuttl/harness/restic-without-trigger (143.22s)
PASS

[JohnStrunk]

/assign sallyom

[cooktheryan]

/lgtm