k8s-zookeeper: zookeeper cluster of DRAFT on Kubernetes

Steps:

• Build docker image via Quay.io

• Create kubernetes pods & services

    kubectl create -f zookeeper.yaml [--namespace=xxx]
  

• Teardown

    ./teardown.sh [--namespace=xxx]
  

---

Notes:

• Enable zookeeper cluster nodes' intra communication via 3 services (1 service for 1 pod: zookeeper-1, zookeeper-2, and zookeeper-3)

• Use "zookeeper" service as a load balancer for clients. It doesn't matter on which zookeeper pod the clients are connected in a zookeeper cluster.

• Since pod name (== hostname), which is generated by k8s replication controller, won't be fixed, we use "service" name (e.g. zookeeper-1, zookeeper-2, zookeeper-3) rather than hostname in zoo.cfg.

• Originally, we use an additional zookeeper service for load balancing client communications (port 2181). But after experimented with spark master HA, it seems not work correctly. Therefore, we won't simplify/abstract with an additional layer for zookeeper client communications.

• Resilience verified OK:

    (1) follower crash: container (docker kill), pod (kubectl delete pods), vm (reboot) 
    (2) leader crash: container (docker kill), pod (kubectl delete pods), vm (reboot)
  

• Cluster isolation verified OK:

    (1) namespaces under the same user context 
        [current-context: default]
        kubectl create -f zookeeper.yaml --namespace=default
        kubectl create -f zookeeper.yaml --namespace=staging
    (2) namespaces under different user context 
        kubectl config use-context develop
        [current-context: develop]
        kubectl create -f zookeeper.yaml --namespace=develop
    Fully isolated among above 3 namespaces
  

---

TODO:

• Kubernetes 1.0.x doesn't support emptyDir volumes for containers running as non-root (it's commit in master branch, not v1.0.0 branch, refer to kubernetes/kubernetes#9384 & kubernetes/kubernetes#12627). Use root rather than zookeeper user instead at this moment. (Done: It's verified OK in kubernetes 1.1.1 in using zookeeper user instead of root)
• Try to spread zookeeper pods (created by different replication controllers, but using same service for external clients) on different nodes by experimenting kube-scheduler.service with policy configuration. https://github.com/kubernetes/kubernetes/blob/master/docs/admin/kube-scheduler.md https://docs.openshift.org/latest/admin_guide/scheduler.html#use-cases