Overview

The resources here include an advanced, ready-to-use solution (for platform operators and developers) that uses Anthos on Bare Metal (ABM) and Anthos Config Management (ACM) to deploy Kubernetes clusters on the edge at scale. We assume that you're familiar with the challenges of edge deployments, the gcloud and kubectl command-line interfaces, Ansible playbooks and working with projects in Google Cloud. For a more detailed description of this solution, see Rolling out clusters on the edge at scale with Anthos for Bare Metal.

In the quick start guide that follows we emulate nodes in an edge location using Google Compute Engine (GCE) VMs and walkthrough a use-case for installing Anthos on Bare Metal (ABM) in that edge location. The installation process also installs Anthos Config Management (ACM) in the Anthos on Bare Metal clusters. We then show how a Point-Of-Sales application is automatically deployed to this edge location by syncing with the configurations found in this repository using ACM. Finally, we experiment by updating the configurations in this repository to see the changes being adopted on the edge dynamically.

You can use the scripts to replicate this deployment on your own and then customize it for your own requirements. A detailed description of these steps (including instructions for installing in an Intel NUC) can be found in the detailed readme.

This diagram shows the deployement state at the end of this guide
(click image to enlarge)

Prerequisites

• Make sure you have the following already installed in your workstation

  • Docker
  • Google Cloud SDK (aka: gcloud)
  • envsubst CLI tool (usually already installed in *nix based OSes)

• Optional

  • skaffold [>=1.30.0]
  • maven [>=3.6.3]
  • Java [11.*]

Note: The prerequisites marked as (Optional) are only required if you want to modify the source for the Point-Of-Sales application. It is not required for just trying this Quick start.

---

Quick start

The following quick start guide will take approximately 55-60 minutes to complete if you have the prerequisites already setup.

1. Setup Google Cloud Environment

1.1) Make a copy of this repository into any git based version control system you use (e.g. GitHub, GitLab, Bitbucket etc.)

Note: If you want to continue with GitHub, see forking a repository for creating your own copy of this repository in GitHub.

# once you have forked this repository, clone it to your local machine
git clone https://github.com/<YOUR_GITHUB_USERNAME>/anthos-samples

# move into the root of the infrastructure setup directory
cd anthos-samples/anthos-bm-edge-deployment

1.2) Setup environment variables (example values are set for some variables; you can change them if you want to name them something else)

export PROJECT_ID="<YOUR_GCP_PROJECT_ID>"
export REGION="us-central1"
export ZONE="us-central1-a"

# port on the GCE instance we will use to setup the nginx proxy to allow traffic into the AnthosBareMetal cluster
export PROXY_PORT="8082"

# should be a multiple of 3 since N/3 clusters are created with each having 3 nodes
export GCE_COUNT="3"

# fork of this repository: https://github.com/GoogleCloudPlatform/anthos-samples
export ROOT_REPO_URL="<LINK_TO_YOUR_FORK_OF_THIS_REPO>"

# this is the username used to authenticate to your fork of this repository
export SCM_TOKEN_USER="<YOUR_GIT_VERSION_CONTROL_USERNAME>"

# this is the access token that will be used to authenticate against your fork of this repository
export SCM_TOKEN_TOKEN="<ACCESS_TOKEN_FOR_YOUR_GIT_REPO>"

Note: If you are trying this out with GitHub as your git version control and have forked this repository into your GitHub account then:

• Used the link to your forked GitHub repository for ROOT_REPO_URL
• Use your GitHub username for SCM_TOKEN_USER
• Use this link to create a personal access token and use that for SCM_TOKEN_TOKEN
  • Under the "Select scopes" section for creating a token only select the "public_repo" scope

1.3) Choose and configure the Google Cloud Project, Region and Zone you would like to use

Note: This step can take upto 90 seconds to complete

gcloud config set project "${PROJECT_ID}"
gcloud services enable compute.googleapis.com

gcloud config set compute/region "${REGION}"
gcloud config set compute/zone "${ZONE}"

1.4) Setup up GCP Service Account used by the GCE instances

# when asked "Create a new key for GSA? [y/N]" type "y" and press
./scripts/create-primary-gsa.sh

This script will create the JSON key file for the new Google Service Account at ./build-artifacts/consumer-edge-gsa.json. It also sets up KMS keyring and key for SSH private key encryption.

---

2. Provision the GCE instances

2.1) Create SSH keypair for communication between provisioning machine and target machines

ssh-keygen -N '' -o -a 100 -t ed25519 -f ./build-artifacts/consumer-edge-machine

2.2) Encrypt the SSH private key using Cloud KMS

gcloud kms encrypt \
  --key gdc-ssh-key \
  --keyring gdc-ce-keyring \
  --location global \
  --plaintext-file build-artifacts/consumer-edge-machine \
  --ciphertext-file build-artifacts/consumer-edge-machine.encrypted

2.3) Generate the environment configuration file .envrc and source it

Once created inspect the .envrc file to ensure that the environment variables have been replaced with the correct values.

envsubst < templates/envrc-template.sh > .envrc
source .envrc

2.4) Create the GCE instances where Anthos BareMetal will be installed

The output of this command will be stored in ./build-artifacts/gce-info. It includes information about the public IP addresses of the GCE VMs and how to SSH into them. You can refer to this file later to find the necessary information.

./scripts/cloud/create-cloud-gce-baseline.sh -c "$GCE_COUNT" | tee ./build-artifacts/gce-info

Note: This step can take upto 2 minutes to complete for a setup with $GCE_COUNT=3

---

3. Install Anthos on Bare Metal with ansible

This phase involves the creation of the docker image for installation, generating the ansible inventory configuration settings and running the ansible scripts that installs Anthos on Bare Metal into the GCE VMs.

3.1) Create a docker image that will be used

You can find more information about the steps involved in creating the image in the docker-build directory.

gcloud builds submit --config docker-build/cloudbuild.yaml docker-build/

# -----------------------------------------------------#
#                   Expected Output                    #
# -----------------------------------------------------#
...
...
latest: digest: sha256:99ded20d221a0b2bcd8edf3372c8b1f85d6c1737988b240dd28ea1291f8b151a size: 4498
DONE
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ID                                    CREATE_TIME                DURATION  SOURCE                                                                                         IMAGES                                                  STATUS
2238baa2-1f41-440e-a157-c65900b7666b  2022-08-17T19:28:57+00:00  6M53S     gs://my_project_cloudbuild/source/1660764535.808019-69238d8c870044f0b4b2bde77a16111d.tgz  gcr.io/my_project/consumer-edge-install (+1 more)  SUCCESS

3.2) Generate the Ansible “inventory” file from template

envsubst < templates/inventory-cloud-example.yaml > inventory/gcp.yaml

3.3) Run the script that creates the docker container for installation

./install.sh

# -----------------------------------------------------#
#                   Expected Output                    #
# -----------------------------------------------------#
...
...
Check the values above and if correct, do you want to proceed? (y/N): y
Starting the installation
Pulling docker install image...

==============================
Starting the docker container. You will need to run the following 2 commands (cut-copy-paste)
==============================
1: ./scripts/health-check.sh
2: ansible-playbook all-full-install.yaml -i inventory
3: Type 'exit' to exit the Docker shell after installation
==============================
Thank you for using the quick helper script!
(you are now inside the Docker shell)

At this point you must be inside the docker container that was created based off of the image built earlier. You will trigger the ansible installation from inside this container.

3.4) Verify access to the GCE VMs

./scripts/health-check.sh

# -----------------------------------------------------#
#                   Expected Output                    #
# -----------------------------------------------------#
cnuc-2 | SUCCESS => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"},"changed": false,"ping": "pong"}
cnuc-3 | SUCCESS => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"},"changed": false,"ping": "pong"}
cnuc-1 | SUCCESS => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"},"changed": false,"ping": "pong"}

3.5) Run the Ansible playbook for installing Anthos Bare Metal on the GCE instances

This script will configure the GCE instances with all the necessary tools, install Anthos BareMetal, install Anthos Config Management and configure it to sync with the configs at $ROOT_REPO_URL/anthos-bm-edge-deployment/acm-config-sink. Upon completion you will see the Login Token for the cluster printed on screen.

Note: This step can take upto 35 minutes to complete for a setup with $GCE_COUNT=3

• Pre-install configuration of the GCE instances: ~10 minutes
  
• Installing Anthos BareMetal: ~20 minutes
  
• Post-install configuration of the GCE instances: ~5 minutes

ansible-playbook all-full-install.yaml -i inventory | tee ./build-artifacts/ansible-run.log

# -----------------------------------------------------#
#                   Expected Output                    #
# -----------------------------------------------------#
...
...
TASK [abm-login-token : Display login token] **************************************************************************
ok: [cnuc-1] => {
    "msg": "eyJhbGciOiJSUzI1NiIsImtpZCI6Imk2X3duZ3BzckQyWmszb09sZHFMN0FoWU9mV1kzOWNGZzMyb0x2WlMyalkifQ.ey
mljZS1hY2NvdW50LnVpZCI6IjQwYWQxNDk2LWM2MzEtNDhiNi05YmUxLWY5YzgwODJjYzgzOSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYW
iZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImVkZ2Etc2EtdG9rZW4tc2R4MmQiLCJrdWJlcm5ldGVzLmlvL3Nl
cnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZWRnYS1zYSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2Vyd
4CwanGlof6s-fbu8IUy1_bTgCminylNKb3VudC5uYW1lIjoiZWRnYS1zYSIsImt1YmVybmV0ZXuaP-hDEKURb5O6IxulTXWH6dxYxg66x
Njb3VudDpkZWZhdWx0OmVkZ2Etc2EifQ.IXqXwX5pg9RIyNHJZTM6cBKTEWOMfQ4IQQa398f0qwuYlSe12CA1l6P8TInf0S1aood7NJWx
xe-5ojRvcG8pdOuINq2yHyQ5hM7K7R4h2qRwUznRwuzOp_eXC0z0Yg7VVXCkaqnUR1_NzK7qSu4LJcuLzkCYkFdSnvKIQABHSvfvZMrJP
Jlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3V
MgyLOd9FJyhZgjbf-a-3cbDci5YABEzioJlHVnV8GOX_q-MnIagA9-t1KpHA"
}
skipping: [cnuc-2]
skipping: [cnuc-3]

PLAY RECAP ***********************************************************************************************************
cnuc-1                     : ok=205  changed=156  unreachable=0    failed=0    skipped=48   rescued=0    ignored=12
cnuc-2                     : ok=128  changed=99   unreachable=0    failed=0    skipped=108  rescued=0    ignored=2
cnuc-3                     : ok=128  changed=99   unreachable=0    failed=0    skipped=108  rescued=0    ignored=2

---

4. Log in to the ABM kubernetes cluster in the Google Cloud console

To be able to see logs and other information about the cluster in the Google Cloud Platform console, you have to Login to the cluster using one of the different means available. In this section we show how to login to the cluster using the Login Token mechanism.

The Kubernetes resources required to generate the login token will already be created inside the new Anthos on Bare Metal cluster. This is done when the cluster gets Synced with the configuration inside the acm-config-sink directory through Anthos Config Management. The cloud-console-reader.yaml and console-cluster-reader-sa.yaml files inside the acm-config-sink directory describe what these resources are.

As a last step, the ansible script generates the Secret that holds the token. The Login Token for the cluster you created will be printed on screen once the ansible script from the previous step completes. If you don't have access to the console output anymore, check the log file at ./build-artifacts/ansible-run.log to get your token. You may also run the following ansible playbook to fetch the token again.

ansible-playbook all-get-login-tokens.yaml -i inventory

Once you have copied the token, login to the kubernetes cluster from the Kubernetes clusters page in the Google Cloud console.

(click image to enlarge)

Verify that the cluster has synced with the configurations from this repository using Anthos Config Management

(click image to enlarge)

---

5. Configure the reverse proxy to route external traffic to ABM's bundled Metal load balancer

5.1) Setup the nginx configuration to route traffic to the API Server Load Balancer service

Note: The following commands are run inside the admin GCE instance (cnuc-1). You must already be SSH'ed into it from the previous steps

# SSH into the admin node of the cluster
ssh -F ./build-artifacts/ssh-config abm-admin@cnuc-1

# get the IP address of the LoadBalancer type kubernetes service
ABM_INTERNAL_IP=$(kubectl get services api-server-lb -n pos | awk '{print $4}' | tail -n 1)

# update the template configuration file with the fetched IP address
sudo sh -c "sed 's/<K8_LB_IP>/${ABM_INTERNAL_IP}/g' /etc/nginx/nginx.conf.template > /etc/nginx/nginx.conf"

# restart nginx to ensure the new configuration is picked up
sudo systemctl restart nginx

# check and verify the status of the nginx server to be "active (running)"
sudo systemctl status nginx

# -----------------------------------------------------#
#                   Expected Output                    #
# -----------------------------------------------------#
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2021-09-17 02:41:01 UTC; 2s ago
       Docs: man:nginx(8)
    Process: 92571 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 92572 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
   Main PID: 92573 (nginx)
      Tasks: 17 (limit: 72331)
     Memory: 13.2M
     CGroup: /system.slice/nginx.service
             ├─92573 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
             ├─92574 nginx: worker process
             ├─92575 nginx: worker process
             ├─92576 nginx: worker process
             ├─92577 nginx: worker process
             ├─92578 nginx: worker process
             ├─92579 nginx: worker process
             ├─92580 nginx: worker process
             ├─92581 nginx: worker process

# exit out of the admin instance
exit

# exit out of the docker container used for installation
exit

---

6. Access the Point of Sale application

6.1) Get the external IP address of the admin GCE instance and access the UI of the Point of Sales application

Note: The following commands are run in your local workstations. If you are still inside the admin GCE instance via SSH or the Docker container used for the installation, then type exit to get back to your local shell.

EXTERNAL_IP=$(gcloud compute instances list --project ${PROJECT_ID} --filter="name:cnuc-1" --format="get(networkInterfaces[0].accessConfigs[0].natIP)")
echo "Point the browser to: ${EXTERNAL_IP}:${PROXY_PORT}"

# -----------------------------------------------------#
#                   Expected Output                    #
# -----------------------------------------------------#
Point the browser to: 34.134.194.84:8082

---

7. Update the API-Server version and observe the change

7.1) Update the image tag for the api-server service to v2 from v1 and push the change to the upstream repository

# portion of interest in the api-server.yaml file is shown here
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  selector:
  template:
    spec:
      # [START anthosbaremetal_pos_deployment_api_server]
      - name: api-server
        image: us-docker.pkg.dev/anthos-dpe-abm-edge-pos/abm-edge-pos-images/api-server:v1
      # [END anthosbaremetal_pos_deployment_api_server]

# push the change
git add acm-config-sink/namespaces/pos/api-server.yaml
git commit -m "chore: updated api-server version to v2"
git push

7.2) Reload the application to view the changes

First, check if the latest commit has been synched in the Anthos Config Management page (similar to the ACM image in step 4.1). Then, wait for the Pod to be updated (you can monitor it in the console). Once the status of the Pod changes to Running, point your browser to the same url as earlier. This time you should see that the v2 version of the application has been deployed!

Note: You might have to do a hard refresh for the browser to reload the new content.

---

Clean up

Google Cloud

• If you used a fresh Google Cloud Project, then you can simply delete it
• If you used an existing Google Cloud Project, then you have to delete the following resources:
  • All Kubernetes clusters with a name prefixed by "cnuc-"
  • All Compute Engine VMs with a name prefixed by "cnuc-"
  • The Cloud Storage bucket with a name prefixed by "abm-edge-boot"
  • The Firewall Rules allow-pod-ingress and allow-pod-egress
  • The Secrets Manager secret install-pub-key

Local workstation

• In addition, you might also want to cleanup the following changes made in your local workstation:
  • Remove the GCE VM IP addresses added to the /etc/hosts file
  • Remove the SSH configuration for cnuc-* in the ~/.ssh/config file
  • Remove the GCE VM fingerprints from the ~/.ssh/known_hosts file