diff --git a/docs/admin/iamRoleChaining.md b/docs/admin/iamRoleChaining.md new file mode 100644 index 0000000000..776f8a55ff --- /dev/null +++ b/docs/admin/iamRoleChaining.md @@ -0,0 +1,77 @@ +# AWS IAM Role Chaining +Using an EC2 instance profile enables your Hopsworks cluster to access AWS resources. +This forces all Hopsworks users to share the instance profile role and the resource access policies attached to +that role. To allow for per project access policies you could have your users use AWS credentials directly in +their programs which is not recommended so you should instead use [Role chaining](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_terms-and-concepts.html#iam-term-role-chaining). +To use Role chaining, you need to first setup IAM roles in AWS: + + **Step 1**. Create an instance profile role with policies that will allow it to assume all resource roles that we can + assume +from the Hopsworks cluster. + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "AssumeDataRoles", + "Effect": "Allow", + "Action": "sts:AssumeRole", + "Resource": [ + "arn:aws:iam::123456789011:role/test-role", + "arn:aws:iam::xxxxxxxxxxxx:role/s3-role", + "arn:aws:iam::xxxxxxxxxxxx:role/dev-s3-role", + "arn:aws:iam::xxxxxxxxxxxx:role/redshift" + ] + } + ] +} +``` +
Example policy for assuming four roles.
+ + **Step 2**. Create the resource roles and edit trust relationship and add policy document that will allow the instance + profile +to assume this role. + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "AWS": "arn:aws:iam::xxxxxxxxxxxx:role/instance-profile" + }, + "Action": "sts:AssumeRole" + } + ] +} +``` +
Example policy document.
+ +Role chaining allows the instance profile to assume any role in the policy attached in step 1. To limit access to +iam roles we can create a per-project mapping from the admin page in Hopsworks. + +
+ + Role Chaining + +
Role Chaining
+
+ +Click on your name in the top right corner of the navigation bar and choose _Cluster Settings_ from the dropdown menu. +In the Cluster Settings' _IAM Role Chaining_ tab you can configure the mappings between projects and IAM roles. +You can add mappings by entering the project name, which roles in that project can access the cloud role and the +role ARN. +Optionally you can set a role mapping as default by marking the default checkbox. The default roles can be changed from +the project setting by a Data owner in that project. + +
+ + Create Role Chaining + +
Create Role Chaining
+
+ +Any member of a project can then go to the _Project Settings_ -> +[Assuming IAM Roles](../compute/project/iamRoleChaining.md) page to see which roles they can assume. \ No newline at end of file diff --git a/docs/admin/services.md b/docs/admin/services.md index fa06e38bc6..0d145b7e6f 100644 --- a/docs/admin/services.md +++ b/docs/admin/services.md @@ -5,8 +5,8 @@ You can find the Services page by clicking on your name, in the top right corner _Cluster Settings_ from the dropdown menu and going to the _Services_ tab.
- - services page + + services page
Services page
@@ -16,7 +16,8 @@ It provides information about their status as reported by agents that monitor th Systemd units. Columns in the services table represent machines in your cluster. Each service running on a machine will have a status -_running_ (green), _stopped_ (gray), or _bad health_ (red). +_running_ (green) or _stopped_ (red). If a service is not installed on a machine it will have a status _not installed_ +(gray). Services are divided into groups, and you can search for a service by its name or group. You can also search for machines by their host name. @@ -29,8 +30,8 @@ machines by their host name. After you find the correct service you will be able to **start**, **stop** or **restart** it, by clicking on its status.
- - start services + + start services
Start, Stop and Restart a service
diff --git a/docs/assets/images/admin/iam-role/cluster-settings.png b/docs/assets/images/admin/iam-role/cluster-settings.png new file mode 100644 index 0000000000..74f52fa2a4 Binary files /dev/null and b/docs/assets/images/admin/iam-role/cluster-settings.png differ diff --git a/docs/assets/images/admin/iam-role/new-role-chaining.png b/docs/assets/images/admin/iam-role/new-role-chaining.png new file mode 100644 index 0000000000..50bf568586 Binary files /dev/null and b/docs/assets/images/admin/iam-role/new-role-chaining.png differ diff --git a/docs/assets/images/admin/services/full.png b/docs/assets/images/admin/services/full.png deleted file mode 100644 index 56af083da9..0000000000 Binary files a/docs/assets/images/admin/services/full.png and /dev/null differ diff --git a/docs/assets/images/admin/services/services-page.png b/docs/assets/images/admin/services/services-page.png new file mode 100644 index 0000000000..a1c5426063 Binary files /dev/null and b/docs/assets/images/admin/services/services-page.png differ diff --git a/docs/assets/images/admin/services/services-start.png b/docs/assets/images/admin/services/services-start.png new file mode 100644 index 0000000000..31c8b48c7e Binary files /dev/null and b/docs/assets/images/admin/services/services-start.png differ diff --git a/docs/assets/images/admin/services/services.png b/docs/assets/images/admin/services/services.png index 098658e291..9cc360c91b 100644 Binary files a/docs/assets/images/admin/services/services.png and b/docs/assets/images/admin/services/services.png differ diff --git a/docs/assets/images/admin/services/start.png b/docs/assets/images/admin/services/start.png deleted file mode 100644 index fcb7a0ad1e..0000000000 Binary files a/docs/assets/images/admin/services/start.png and /dev/null differ diff --git a/docs/assets/images/iam-role/project-settings.png b/docs/assets/images/iam-role/project-settings.png new file mode 100644 index 0000000000..f90f04879e Binary files /dev/null and b/docs/assets/images/iam-role/project-settings.png differ diff --git a/docs/compute/project/iamRoleChaining.md b/docs/compute/project/iamRoleChaining.md new file mode 100644 index 0000000000..0bc694f999 --- /dev/null +++ b/docs/compute/project/iamRoleChaining.md @@ -0,0 +1,44 @@ +# Assuming AWS IAM Roles +When deploying Hopsworks on EC2 instances you might need to assume different roles to access resources on AWS. +These roles are configured in AWS and mapped to a project in Hopsworks, for a guide on how to configure this go to +[AWS IAM Role Chaining](../../admin/iamRoleChaining.md). + +After an administrator configured role mappings in Hopsworks you can see the roles you can assume in the Project +Settings IAM Role Chaining tab. +
+ + Role Chaining + +
Role Chaining
+
+ +You can then use the [Hops python library](https://hops-py.logicalclocks.com/) and +[Hops java/scala library](https://github.com/logicalclocks/hops-util) to assume the roles listed in your project’s settings page. + +When calling _assume\_role_ you can pass the role ARN string or use the get role method that takes the role id +as an argument. If you assign a default role for your project you can call _assume\_role_ without arguments. + +You can assign (if you are a Data owner in that project) a default role to you project by clicking on the _default_ +checkbox of the role you want to make default. You can set one default per project role. If a default is set for +a project role (Data scientist or Data owner) and all members (ALL) the default set for the project role will take +precedence over the default set for all members. + +###### python +```python +from hops.credentials_provider import get_role, assume_role +credentials = assume_role(role_arn=get_role(1)) +spark.read.csv("s3a://resource/test.csv").show() +``` + +###### scala +```scala +import io.hops.util.CredentialsProvider +val creds = CredentialsProvider.assumeRole(CredentialsProvider.getRole(1)) +spark.read.csv("s3a://resource/test.csv").show() +``` + +The _assume\_role_ method sets spark hadoop configurations that will allow spark to read s3 buckets. The code examples +above show how to read s3 buckets using Python and Scala. + +The method also sets environment variables **AWS_ACCESS_KEY_ID**, **AWS_SECRET_ACCESS_KEY** and +**AWS_SESSION_TOKEN** so that programs running in the container can use the credentials for the newly assumed role. \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index f5f224d6fd..5eafe2e40f 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -27,6 +27,7 @@ nav: - Project-based Multi-tenancy: compute/project/multiTenancy.md - Delete a Project: compute/project/deleteProject.md - Project Name Reserved Words: compute/project/reservedNames.md + - Assuming IAM Roles: compute/project/iamRoleChaining.md - Python: compute/python.md - Jupyter: compute/jupyter.md - Jobs: compute/jobs.md @@ -51,6 +52,7 @@ nav: - User Management: admin/user.md - Configure Alerts: admin/alert.md - Manage Services: admin/services.md + - IAM Role Chaining: admin/iamRoleChaining.md - Hopsworks.ai: https://docs.hopsworks.ai/hopsworks-cloud/latest/ - Examples: https://examples.hopsworks.ai/ - Community: https://community.hopsworks.ai/