2024 07 s3 docs (capillariesio#71)

Added s3 setup instructions

doc/s3.md

@@ -0,0 +1,64 @@
+# S3 data access
+
+[Integration tests suite](./testing.md#integration-tests) contains tests that use data and config files stored in AWS S3. [k8s POC](../test/k8s/README.md) also uses S3 files. You have to perform some AWS setup steps to make S3 integration tests and k8s POC work.
+
+For demo purposes, Capillaries Docker-based and k8s POC test environments use user-based permissions (see below), which may not be the preferred way of controlling access in production AWS environment. Using role assumption approach is the recommended way.
+
+## Bucket capillaries-testbucket
+
+Create capillaries-testbucket (this name may be taken already, so you may be forced to use some other name). In `Amazon S3->Buckets->capillaries-testbucket->Permissions` tab, turn `Block all public access` on.
+
+Before running tests, make sure you have this environment variable is set:
+```
+export CAPILLARIES_AWS_TESTBUCKET=capillaries-testbucket
+```
+
+## IAM user UserAccessCapillariesTestbucket
+
+daemon/webapi/toolbelt running in our Docker/k8s tests will this IAM user credentials and access `capillaries-testbucket`.
+
+Create IAM user `UserAccessCapillariesTestbucket`.
+
+Create access key for it, save it to ~/UserAccessCapillariesTestbucket.rc:
+```
+export AWS_ACCESS_KEY_ID=AK...
+export AWS_SECRET_ACCESS_KEY=...
+export AWS_DEFAULT_REGION=us-east-1
+```
+
+Now, everytime you have to run Capillaries test Docker setup or k8s POC, make sure you have those variables set:
+
+```
+source ~/UserAccessCapillariesTestbucket.rc
+```
+
+## Bucket permissions for UserAccessCapillariesTestbucket
+
+In `Amazon S3->Buckets->capillaries-testbucket->Permissions->Bucket policy` specify:
+
+{
+	"Version": "2012-10-17",
+	"Statement": [
+		{
+			"Effect": "Allow",
+			"Principal": {
+				"AWS": "arn:aws:iam::<your_aws_acount>:user/UserAccessCapillariesTestbucket"
+			},
+			"Action": "s3:ListBucket",
+			"Resource": "arn:aws:s3:::capillaries-testbucket"
+		},
+		{
+			"Effect": "Allow",
+			"Principal": {
+				"AWS": "arn:aws:iam::<your_aws_acount>:user/UserAccessCapillariesTestbucket"
+			},
+			"Action": [
+				"s3:DeleteObject",
+				"s3:GetObject",
+				"s3:PutObject"
+			],
+			"Resource": "arn:aws:s3:::capillaries-testbucket/*"
+		}
+	]
+}
+

doc/testing.md

@@ -18,6 +18,8 @@ and open /var/tmp/capillaries.html in a web browser.
 
 ## Integration tests
 
+Some integration tests use data and config files stored in S3. Make sure you have the test bucket and IAM user credentials set up as described in [s3 data access](./s3.md).
+
 To run all integration tests, make sure you have RabbitMQ, Casandra and Capillaries Daemon running (either in Docker containers or as regular applications), test data is present (see ./copy_demo_data.sh) and run:
 
 ```

test/code/common/util.sh

@@ -29,48 +29,21 @@ check_s3()
     if [ "$CAPILLARIES_AWS_TESTBUCKET" = "" ]; then
         echo Error, missing: export CAPILLARIES_AWS_TESTBUCKET=capillaries-testbucket
         echo This is the name of the bucket the user creates to test S3 Capillaries scenarios.
-        echo 'Expected permissions:'
-        echo '{
-    "Version": "2012-10-17",
-    "Statement": [
-        {
-            "Effect": "Allow",
-            "Principal": {
-                "AWS": "arn:aws:iam::<aws_account_id>:user/capillaries-testuser"
-            },
-            "Action": "s3:ListBucket",
-            "Resource": "arn:aws:s3:::capillaries-testbucket"
-        },
-        {
-            "Effect": "Allow",
-            "Principal": {
-                "AWS": "arn:aws:iam::<aws_account_id>:user/capillaries-testuser"
-            },
-            "Action": [
-                "s3:DeleteObject",
-                "s3:GetObject",
-                "s3:PutObject"
-            ],
-            "Resource": "arn:aws:s3:::capillaries-testbucket/*"
-        }
-    ]
-}'
+        echo See s3.md for details on how to set up bucket permissions.
         exit 1
     fi
 
-    if [ ! -e ~/.aws/credentials ]; then
-        echo '~/.aws/credentials not found, expected:'
-        echo '[default]'
-        echo 'aws_access_key_id=AK...'
-        echo 'aws_secret_access_key=...'
+    if [ "$AWS_ACCESS_KEY_ID" == "" ]; then
+        echo Error, please specify export AWS_ACCESS_KEY_ID=...
+        exit 1
+    fi
+    if [ "$AWS_SECRET_ACCESS_KEY" == "" ]; then
+        echo Error, please specify export AWS_SECRET_ACCESS_KEY=...
         exit 1
     fi
 
-    if [ ! -e ~/.aws/config ]; then
-        echo '~/.aws/config not found, expected:'
-        echo '[default]'
-        echo 'region=us-east-1'
-        echo 'output=json'
+    if [ "$AWS_DEFAULT_REGION" == "" ]; then
+        echo Error, please specify export AWS_DEFAULT_REGION=...
         exit 1
     fi
 }

test/k8s/README.md

@@ -6,4 +6,6 @@ This is a POC demonstrating Capillaries running in a Kubernetes cluster.
  - Capillaries docker images were built and uploaded using binaries_build.sh, images_build.sh, images_upload.sh commands
  - Minikube Kuberetes cluster is running
 
-Just run scripts in order.
+Just run scripts in order.
+
+Tests scripts use data and config files stored in S3. Make sure you have the test bucket and IAM user credentials set up as described in [s3 data access](../../doc/s3.md).