Deep dive on how to scale instances based on AWS SQS
Serverless framework project for building applications on AWS Lambda
Lambda function that is triggered every minute, calculates backlog per instance and publish a custom metric
backlog per instance = SQS.ApproximateNumberOfMessages / runningInstances
Lambda function that is triggered by selected events of aws.autoscaling (debugging lambda)
- 'EC2 Instance Launch Successful'
- 'EC2 Instance Launch Unsuccessful'
- 'EC2 Instance Terminate Successful'
- 'EC2 Instance Terminate Unsuccessful'
A naive implementation of a dockerized nodejs sqs-worker
Each message handling takes 1 min to complete
Whenever a new EC2 instance is launched, it pulls and run this container sqs-worker container
Accepts the following environment variables:
AWS_PROFILE=<profile>
or
AWS_ACCESS_KEY_ID=<access-key>
AWS_SECRET_ACCESS_KEY=<secret-key>
AWS_REGION=<region>
QUEUE_URL=<sqs-queue-url>
LOCALSTACK_ENABLED=<boolean> - you should have localstack installed
Terraform - Infrastructure as code
Terraform and serverless framework must be installed!
Running order is important!
scaling-based-on-aws-sqs/infra> ssh-keygen -t rsa
Enter file in which to save the key: <full-path>/scaling-based-on-aws-sqs/infra/mykey
scaling-based-on-aws-sqs/infra> terraform init
scaling-based-on-aws-sqs/infra> terraform apply
scaling-based-on-aws-sqs/backlog-per-instance-calculator> npm i -g serverless
scaling-based-on-aws-sqs/backlog-per-instance-calculator> npm i
scaling-based-on-aws-sqs/backlog-per-instance-calculator> serverless deploy
Copy the created queue-url and region (from terraform output or from AWS console) and paste it in publishMessages.js
The script publishes batchSize messages to the queue, in each round, and sleeps sleepInMs between each iteration
scaling-based-on-aws-sqs/backlog-per-instance-calculator> npm run publish
Lets you watch the logs of a specific function.
scaling-based-on-aws-sqs/backlog-per-instance-calculator> serverless logs -f logAutoScalingGroupEvents -t
scaling-based-on-aws-sqs/backlog-per-instance-calculator> serverless logs -f calculateBacklogPerInstance -t
Cleanup order is important!
scaling-based-on-aws-sqs/backlog-per-instance-calculator> serverless remove
scaling-based-on-aws-sqs/infra> terraform destroy
This project is inspired by Scaling based on Amazon SQS
Lambda functions are deployed using Serverless Framework
Infrastructure as code Terraform
Node.js middleware engine for AWS Lambda middy