About

The purpose of this project is to make it easy to run various load tests on systems with FoundationDB in the cloud.

We can achieve that by:

1. "Prebaking" cloud-specific VM images for the FoundationDB cluster nodes along with the tester nodes and machines for capturing the telemetry.
2. Using these images to quickly create FoundationDB clusters with the specific configuration.

The first step is handled by the Packer, the second - by the Terraform.

We want to start with a fixed topology:

• foundationDB nodes in the same network, forming a cluster;
• tester machines connected to the FDB.

Number and VM type for the node and load tester machines could be changed within the configuration.

Creating AMIs

First you need to create packer images for FDB and tester machines.

Note, that in this setup we actually create a temporary EC2 instance in the cloud, install all the dependencies there, make a snapshot (AMI) and kill the original machine. With these snapshots we could then quickly launch a dozen of EC2 instances, connecting them into a cluster.

You can create a FoundationDB AMI like this (make sure to fill in your AWS credentials):

$ export AWS_ACCESS_KEY=""
$ export AWS_SECRET_KEY=""
$ cd packer-fdb
$ packer build --only=aws packer.json

Tester AMI can be created exactly like this but you enter packer-tester folder instead.

Creating Docker

Should you want to create docker images instead of AMIs, you can do so via:

$ cd packer-fdb
$ packer build --only=docker packer.json

Deploying Clusters

Setup Terraform

Before deploying pre-baked images into AWS you need to configure your working copy first. Install Terraform, then go to the terraform folder of this repository and execute:

$ terraform init

Then, you would need to create a file .secret.aws.tfvars, filling it with your AWS credentials:

aws_access_key = ""
aws_secret_key = ""
aws_account_id = ""

Afterwards you would also need to create a new ssh key called terraform and place it into your ~/.ssh/ folder. Terraform will install it into all the new machines, making it possible for you to connect to them via ssh.

You can do that with:

$ ssh-keygen -t rsa -b 4096 -C "terraform" -f "$HOME/.ssh/terraform"

Deploy a cluster

In order to deploy a cluster you would need to execute the following in the terraform folder:

# prepare terraform plan, using a separate file with the credentials
$ terraform plan -var-file=.secret.aws.tfvars -out my.plan
# carry out the plan
$ terraform apply "my.plan"

The process should take 3-4 minutes and print out in the end something like this:

aws_instance.fdb[0]: Creation complete after 3m32s (ID: i-0b90a62a90636c1ad)

Apply complete! Resources: 12 added, 0 changed, 0 destroyed.

Outputs:

fdb_address = [
    fdb-01.amazonaws.com,
    fdb-02.amazonaws.com,
    fdb-03.amazonaws.com
]
tester_address = [
    tester-01.amazonaws.com
]

Note that the machine names would be different each time (and much longer). This is just a sample output.

Congratulations, you now have a FoundationDB cluster running in AWS. You can test it by connecting to the test machine with your terraform key:

$ ssh -i ~/.ssh/terraform [email protected]

On your first connection, the ssh might ask you about accepting the new fingerprint. This happens because we have a brand new server running. Just type in 'yes'.

Once connected to the test machine, you could verify that the client tools are installed and the cluster is responding:

$ fdbcli
Using cluster file `/etc/foundationdb/fdb.cluster'.

The database is available.

Welcome to the fdbcli. For help, type `help'.
fdb> status details

Using cluster file `/etc/foundationdb/fdb.cluster'.

Configuration:
  Redundancy mode        - double
  Storage engine         - ssd-2
  Coordinators           - 3
  
...

Congratulations, FDB cluster is up and running!

Destroy the cluster

Keeping AWS instances running costs money. So generally it is advised to destroy all the resources after the experiment.

Terraform makes it easy:

$ terraform destroy -var-file=.secret.aws.tfvars

....

Plan: 0 to add, 0 to change, 12 to destroy.

Do you really want to destroy?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes
  
....

Destroy complete! Resources: 12 destroyed.

Modify the cluster

You can tune the cluster configuration by editing variables.tf file to your liking. Ideally, you would do that before creating a new cluster.

The most important options there are:

variable "aws_fdb_size" {
  default = "t2.medium"
  description = "machine type to run FoundationDB servers"

}
# using only 1 machine will conflict with the default cluster config
# 'configure new memory double'
variable "aws_fdb_count" {
  default = 3
  description = "how many machines do we want in our cluster. Minimum 2"
}

variable "aws_tester_size" {
  default = "m4.xlarge"
  description = "instance type for launching tester machines"
}

Plans

I plan to improve this repository a bit later by:

1. Introducing a load tester tool (pre-installed to the tester image) with common benchmarks.
2. Adding a few scripts to visualize the results.
3. Adding more Terraform configurations tuned for better performance.

Contact

If you have any questions, please don't hesitate to get in touch by sending an email to rinat at abdullin.com.