Skip to content

Commit f54d54a

Browse files
HarshCasperHarshCasper
authored
add cloud pods debugging tutorial (#66)
* add cloud pods debugging tutorial * fix image paths * fix link
1 parent f44975e commit f54d54a

File tree

2 files changed

+348
-0
lines changed

2 files changed

+348
-0
lines changed

‎src/assets/images/aws/tutorials/collab-debugging-cloud-pod.png

281 KB
Loading

‎src/content/docs/aws/tutorials/cloud-pods-collaborative-debugging.mdx

Lines changed: 348 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,348 @@
1+
---
2+
title: "How To: Collaborative AWS local development with LocalStack’s Cloud Pods"
3+
description: Replicating development environments ensures that all developers, regardless of their local machine configurations or operating systems, work within an environment that closely mirrors production. This consistency helps identify and solve environment-specific issues early in the development cycle, reducing the "it works on my machine" problem where code behaves differently on different developers' machines.
4+
services:
5+
- iam
6+
- ddb
7+
- agw
8+
- lmb
9+
platform:
10+
- Java
11+
deployment:
12+
- terraform
13+
pro: true
14+
leadimage: "collab-debugging-cloud-pod.png"
15+
---
16+
17+
# **Introduction**
18+
19+
By replicating environments, teams can share the exact conditions under which a bug occurs.
20+
21+
For developing AWS applications locally, the tool of choice is LocalStack, which can sustain a full-blown comprehensive stack.
22+
However, when issues appear, and engineers need a second opinion from a colleague, recreating the environment from scratch can leave
23+
details slipping through the cracks.
24+
This is where Cloud Pods come in, to encapsulate the state of the LocalStack instance and allow for seamless
25+
collaboration.
26+
While databases have snapshots, similarly, LocalStack uses Cloud Pods for reproducing state and data.
27+
28+
In this tutorial, we will explore a common situation where a basic IAM misconfiguration causes unnecessary delays in finding the right solution.
29+
We will also discuss the best practices to prevent this and review some options for configuring Cloud Pod storage.
30+
The full sample application can be found [on GitHub](https://github.com/localstack-samples/cloud-pods-collaboration-demo) to clone, for following along more easily.
31+
32+
### **Prerequisites**
33+
34+
- [LocalStack CLI](/aws/getting-started/installation#localstack-cli) (preferably using `pip`)
35+
- [Docker](https://docs.docker.com/engine/install/)
36+
- [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) or [OpenTofu](https://opentofu.org/docs/intro/install/) and [terraform-local](/aws/integrations/infrastructure-as-code/terraform#install-the-tflocal-wrapper-script)
37+
- Optional for Lambda build & editing: [Maven 3.9.4](https://maven.apache.org/install.html) & [Java 21](https://www.java.com/en/download/help/download_options.html)
38+
39+
- Basic knowledge of AWS services (API Gateway, Lambda, DynamoDB, IAM)
40+
- Basic understanding of Terraform for provisioning AWS resources
41+
42+
In this demo scenario, a new colleague, Bob, joins the company, clones the application repository, and starts working on the Lambda code.
43+
He will add the necessary
44+
resources in the Terraform configuration file and some IAM policies that the functions need in order to access the database.
45+
He is following good practice rules, where the resource has only the necessary permissions.
46+
However, Bob encounters an error despite this.
47+
48+
### Architecture Overview
49+
50+
The stack consists of an API Gateway that exposes endpoints and integrates with two Lambda functions responsible for adding and fetching
51+
products from a DynamoDB database.
52+
IAM policies are enforced to ensure compliance with the
53+
**[principle of least privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege)**, and the logs will be sent to the CloudWatch service.
54+
55+
### Note
56+
57+
This demo application is suitable for AWS and behaves the same as on LocalStack.
58+
You can try this out by running the Terraform configuration file against the AWS platform.
59+
60+
![Application Diagram](/images/aws/cloud-pod-collab.png)
61+
62+
### Starting LocalStack
63+
64+
In the root directory, there is a `docker-compose.yml` file that will spin up version 3.3.0 of LocalStack, with an
65+
important configuration flag, `ENFORCE_IAM=1`, which will facilitate IAM policy evaluation and enforcement.
66+
For this
67+
example, a `LOCALSTACK_AUTH_TOKEN` is needed, which you can find in the LocalStack web app on the
68+
[Getting Started](https://app.localstack.cloud/getting-started) page.
69+
70+
```bash
71+
export LOCALSTACK_AUTH_TOKEN=<your-auth-token>
72+
docker compose up
73+
```
74+
75+
### The Terraform Configuration File
76+
77+
The entire Terraform configuration file for setting up the application stack is available in the same repository at
78+
https://github.com/localstack-samples/cloud-pods-collaboration-demo/blob/main/terraform/main.tf.
79+
To deploy all the resources on LocalStack,
80+
navigate to the project's root folder and use the following commands:
81+
82+
```bash
83+
cd terraform
84+
tflocal init
85+
tflocal plan
86+
tflocal apply --auto-approve
87+
```
88+
89+
`tflocal` is a small wrapper script to run Terraform against LocalStack.
90+
The endpoints for all services are configured to point to the
91+
LocalStack API, which allows you to deploy your unmodified Terraform scripts against LocalStack.
92+
93+
- **`init`**: This command initializes the Terraform working directory, installs any necessary plugins, and sets up the backend.
94+
- **`plan`**: Creates an execution plan, which allows you to review the actions Terraform will take to change your infrastructure.
95+
- **`apply`**: Finally, the **`apply`** command applies the changes required to reach the desired state of the configuration.
96+
If **`-auto-approve`** is used, it bypasses the interactive approval step normally required.
97+
98+
As mentioned previously, there is something missing from this configuration, and that is the **`GetItem`** operation permission
99+
for one of the Lambda functions:
100+
101+
```hcl
102+
resource "aws_iam_policy" "lambda_dynamodb_policy" {
103+
name = "LambdaDynamoDBAccess"
104+
description = "IAM policy for accessing DynamoDB from Lambda"
105+
106+
policy = jsonencode({
107+
Version = "2012-10-17"
108+
Statement = [
109+
{
110+
Action = [
111+
"dynamodb:Scan",
112+
"dynamodb:Query",
113+
"dynamodb:UpdateItem",
114+
"dynamodb:PutItem",
115+
]
116+
Effect = "Allow"
117+
Resource = "*"
118+
},
119+
]
120+
})
121+
}
122+
```
123+
124+
Bob has mistakenly used `dynamodb:Scan` and `dynamodb:Query`, but missed adding the `dynamodb:GetItem` action to the policy document above.
125+
126+
### Reproducing the issue locally
127+
128+
Let’s test out the current state of the application.
129+
The Terraform configuration file outputs the REST API ID of the API Gateway.
130+
We can capture that value and use it further to invoke the **`add-product`** Lambda:
131+
132+
```bash
133+
export rest_api_id=$(cd terraform; tflocal output --raw rest_api_id)
134+
```
135+
136+
The endpoint for the API Gateway is constructed similarly to the one on AWS:
137+
138+
**`
139+
https://<apiId>.execute-api.localhost.localstack.cloud:4566/<stageName>/<path>
140+
`**
141+
142+
So adding two products to the database is straightforward using `curl`:
143+
144+
```bash
145+
curl --location "http://$rest_api_id.execute-api.localhost.localstack.cloud:4566/dev/productApi" \
146+
--header 'Content-Type: application/json' \
147+
--data '{
148+
"id": "34534",
149+
"name": "EcoFriendly Water Bottle",
150+
"description": "A durable, eco-friendly water bottle designed to keep your drinks cold for up to 24 hours and hot for up to 12 hou
151+
s. Made from high-quality, food-grade stainless steel, it'\''s perfect for your daily hydration needs.",
152+
"price": "29.99"
153+
}'
154+
155+
curl --location "http://$rest_api_id.execute-api.localhost.localstack.cloud:4566/dev/productApi?id=82736" \
156+
--header 'Content-Type: application/json' \
157+
--data '{
158+
"id": "82736",
159+
"name": "Sustainable Hydration Flask",
160+
"description": "This sustainable hydration flask is engineered to maintain your beverages at the ideal temperature—cold for 24 hours and hot for 12 hours.
161+
Constructed with premium, food-grade stainless steel, it offers an environmentally friendly solution to stay hydrated throughout the day.",
162+
"price": "31.50"
163+
}'
164+
```
165+
166+
The response is the one that we expect: `Product added/updated successfully.`
167+
168+
However, retrieving one of the products does not return the desired result:
169+
170+
```bash
171+
$ curl --location "http://$rest_api_id.execute-api.localhost.localstack.cloud:4566/dev/productApi?id=34534"
172+
```
173+
174+
```bash title="Output"
175+
Internal server error⏎
176+
```
177+
178+
An `Internal server error⏎` does not give out too much information.
179+
Bob does not know for sure what could be
180+
causing this.
181+
The Lambda code and the configurations look fine to him.
182+
183+
## Using Cloud Pods for collaborative debugging
184+
185+
### Creating a Cloud Pod
186+
187+
To share this exact environment and issue with Alice, a more experienced colleague, Bob only needs to run a simple `localstack pod` command:
188+
189+
```bash
190+
$ localstack pod save cloud-pod-product-app
191+
```
192+
193+
```bash title="Output"
194+
Cloud Pod `cloud-pod-product-app` successfully created ✅
195+
Version: 1
196+
Remote: platform
197+
Services: sts,iam,apigateway,dynamodb,lambda,s3,cloudwatch,logs
198+
```
199+
200+
LocalStack provides a remote storage backend that can be used to store the state of your application and share it with your team members.
201+
202+
The Cloud Pods CLI is included in the LocalStack CLI installation, so there’s no need for additional plugins to begin using it.
203+
The `LOCALSTACK_AUTH_TOKEN` needs to be set as an environment variable.
204+
205+
Additionally, there are other commands for managing Cloud Pods included in the CLI:
206+
207+
```bash
208+
localstack pod --help
209+
```
210+
211+
```bash title="Output"
212+
Usage: localstack pod [OPTIONS] COMMAND [ARGS]...
213+
214+
Manage the state of your instance via Cloud Pods.
215+
216+
Options:
217+
-h, --help Show this message and exit.
218+
219+
Commands:
220+
delete Delete a Cloud Pod
221+
list List all available Cloud Pods
222+
load Load the state of a Cloud Pod into the application runtime
223+
remote Manage Cloud Pod remotes
224+
save Create a new Cloud Pod
225+
versions List all available versions for a Cloud Pod
226+
```
227+
228+
### Pulling and Loading the Cloud Pod
229+
230+
The workflow between Alice and Bob is incredibly easy:
231+
232+
![Bob and Alice Collab](/images/aws/bob-alice-cloud-pod-collab.png)
233+
234+
Now, in a fresh LocalStack instance, Alice can immediately load the Cloud Pod, because she's part of the
235+
same organization:
236+
237+
```bash
238+
localstack pod load cloud-pod-product-app
239+
```
240+
241+
```bash title="Output"
242+
Cloud Pod cloud-pod-product-app successfully loaded
243+
```
244+
245+
### Debugging and Resolving the Issue
246+
247+
Not only can Alice easily reproduce the bug now, but she also has access to the state and data of the services
248+
involved, meaning that the Lambda logs are still in the CloudWatch log groups.
249+
250+
![CloudWatch Logs](/images/aws/cloudwatch-logs.png)
251+
252+
By spotting the error message, there’s an instant starting point for checking the source of the problem.
253+
The error message displayed in the logs is very specific:
254+
255+
`"Error: User: arn:aws:sts::000000000000:assumed-role/productRole/get-product is not authorized to perform: dynamodb:GetItem on resource: arn:aws:dynamodb:us-east-1:000000000000:table/Products because no identity-based policy allows the dynamodb:GetItem action (Service: DynamoDb, Status Code: 400, Request ID: d50e9dad-a01a-4860-8c21-e844a930ba7d)"`
256+
257+
### Identifying the Misconfiguration
258+
259+
The error points to a permissions issue related to accessing DynamoDB.
260+
The action **`dynamodb:GetItem`** is
261+
not authorized for the role, preventing the retrieval of a product by its ID.
262+
This kind of error was not foreseen as one
263+
of the exceptions to be handled in the application.
264+
IAM policies are not always easy and straightforward, so it's a well known fact that
265+
these configurations are prone to mistakes.
266+
267+
To confirm the finding, Alice now has the exact same environment to reproduces the error in.
268+
There are no machine specific configurations and
269+
no other manual changes.
270+
This leads to the next step in troubleshooting: **inspecting the Terraform configuration file** responsible
271+
for defining the permissions attached to the Lambda role for interacting with DynamoDB.
272+
273+
### Fixing the Terraform Configuration
274+
275+
Upon review, Alice discovers that the Terraform configuration does not include the necessary permission **`dynamodb:GetItem`** in the
276+
policy attached to the Lambda role.
277+
This oversight explains the error message.
278+
The Terraform configuration file acts as a
279+
blueprint for AWS resource permissions, and any missing action can lead to errors related to authorization.
280+
281+
This scenario underscores the importance of thorough review and testing of IAM roles and policies when working with AWS resources.
282+
It's easy to overlook a single action in a policy, but as we've seen, such an omission can significantly impact application
283+
functionality.
284+
By carefully checking the Terraform configuration files and ensuring that all necessary permissions are included,
285+
developers can avoid similar issues and ensure a smoother, error-free interaction with AWS services.
286+
287+
The action list should now look like this:
288+
289+
```bash
290+
resource "aws_iam_policy" "lambda_dynamodb_policy" {
291+
name = "LambdaDynamoDBAccess"
292+
description = "IAM policy for accessing DynamoDB from Lambda"
293+
294+
policy = jsonencode({
295+
Version = "2012-10-17"
296+
Statement = [
297+
{
298+
Action = [
299+
"dynamodb:GetItem",
300+
"dynamodb:UpdateItem",
301+
"dynamodb:PutItem",
302+
]
303+
Effect = "Allow"
304+
Resource = "*"
305+
},
306+
]
307+
})
308+
}
309+
```
310+
311+
To double-check, Alice creates the stack on AWS, and observes that the issue is the same, related to policy
312+
misconfiguration:
313+
314+
![AWS CloudWatch Logs](/images/aws/aws-cloudwatch-logs.png)
315+
316+
### Impact on the team
317+
318+
Alice has updated the infrastructure and deployed a new version of the Cloud Pod with the necessary fixes.
319+
Bob will
320+
access the updated infrastructure and proceed with his tasks.
321+
Meanwhile, Carol is developing integration tests for the
322+
CI pipeline.
323+
She will use the stable version of the infrastructure to ensure that the workflows function effectively from
324+
start to finish.
325+
326+
![Carol writes tests](/images/aws/carol-bob-alice-cloud-pod-collab.png)
327+
328+
### Other Remote Options
329+
330+
For organizations with specific data regulations, LocalStack offers multiple remote storage options for Cloud Pods,
331+
allowing full control with on-premises storage if needed.
332+
That way, Bob, Alice and Carol could collaborate using either an S3 bucket remote storage or an ORAS (OCI Registry as Storage) remote storage.
333+
The Cloud Pods command-line interface enables users to manage these remotes with ease, by following the instructions in the
334+
[documentation](/aws/capabilities/state-management/cloud-pods#remotes).
335+
336+
## Conclusion
337+
338+
Cloud Pods play a crucial role in team collaboration, significantly speeding up development processes.
339+
The multiple and
340+
versatile options for remote storage can support different business requirements for companies that prefer using the
341+
environments they control.
342+
Cloud Pods are not just for teamwork; they also excel in other areas, such as creating
343+
resources in Continuous Integration (CI) for ultra-fast testing pipelines.
344+
345+
## Additional resources
346+
347+
- [Cloud Pods documentation](/aws/capabilities/state-management/cloud-pods)
348+
- [Terraform for AWS](https://developer.hashicorp.com/terraform/tutorials/aws-get-started)

0 commit comments

Comments
 (0)