forked from contiv/vpp
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
36c594c
commit 87f7619
Showing
3 changed files
with
163 additions
and
0 deletions.
There are no files selected for viewing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| <?xml version="1.0" encoding="UTF-8"?> | ||
| <mxfile modified="2019-02-11T15:08:45.351Z" host="www.draw.io" agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.96 Safari/537.36" etag="7kBtTpxjHjSosm1fdJAc" version="10.2.2" type="google"><diagram id="137wJS51sst3RdgluW2T" name="Page-1">7Vxbc6M2FP41nmkfkgEJBDwm3t1uZpuMp9lpuo8YFJsuRi6WE7u/vsKAjS4x2AZkp/HObKyjC6BzvnPTwQM4nK1+S/359J6EOB4AI1wN4KcBACYAiP3JKOucggw7J0zSKCwG7QiP0b+4IBoFdRmFeMENpITENJrzxIAkCQ4oR/PTlLzyw55JzF917k+wRHgM/FimPkUhneZUFzg7+lccTabllU3k5T0zvxxcPMli6ofktUKCnwdwmBJC82+z1RDH2eaV+5LP+/JG7/bGUpzQJhOW9+gf3524aImc79+fVun63rmCxTIvfrwsnngAUMwWvJ2zL5PsS0kYi4TF3E9KWpBEJZndQrVHNfqKrufZ5V7m85pZjCxdl9EqN7fZW7ouGZaSZRLi7JkN1v06jSh+nPtB1vvKRJTRpnQWs5bJvj5HcTwkMUk3c+Gznf3L7pWm5Ceu9KDNJ5tBElqh5x9Gj/0xjkdkEdGIJKwvYFzBbNDtC05pxOTpd2EAJdm9+HE0UQ6/KTrGhFIy2z5o1o1Xb0qAuZUrBkhMZpimazakmAC9AnsFFrei+bqTbAgL2rQi1Vs0+gWaJtu1dwLHvhQyd4D86RU/ts9RQD4ksDcJtIxzE0Ab1vCvwiv2kOn6r4x+Deyy/UNof1oVE/PWutoa4TRit51t84a4oH5KbzIrlTEg9heLKCjJX6K4vO6mXZjGfK1QnsSIlSmsVZnwN6Z0XbT9JSWMRFI6JROSMLEgmRwUF8qErrRzQCWGxuaTXWEV0XwvHAMU7WwvzGvDQEV7txdZY11piDuRMwGHkjEWJIsxiizTAO8zaYWX4KcTTPeMA5ZaVFMc+zR64W+kdbmDkuL7ShaUUR6e2H+P1A9+tqpYsBna2FFx1EMO9FE7+LZ5fFsGlPFtqvDtdAVv6yh4O54j4NsB7xPgOxhDG/Ew9mBnMD4BdcXUEYkSWhE8E16jysfh5RAafK/Fr5/rlGLJqgt92FVsdG3zC+dKSFp4I8rbhz9BuiUl8udo1K4/4gY4CFRqY+zalm20ozYsg3dMlXqjHNOL3iidkAP1hukIesNwzA71RgM90VQz7PQJFBQSbNc8o4bmGeq0zkgC1nATKlw9kBCfBrDO3XQRwc/PQI3gEI2R3Y3hB57s2Jsqw291FljKhj8T38eiucPA5x31tgnEece/BtyH2eWSNwlJcLugcxqCztMJOtO+LJY1VdHFYkDQq+B/yGJH0qvflkzbYdqq09JPrGO6gs6Dss6ze1V5TnOfpaH0NvcetmDzwEDwgNAxHlBv/ojbkz+ijiggRJwQQU8ITTqOIFwlIK9GKVmtLxCT2+TC2WDygPRiZ5anYwR5PSHoNEYgbYwQjT7vEmhghGno5IQnqZxhcQxjDB/uTlM6BwdXsygMN35kbXy1HaklvjIdl1dsigCrzHX2o9gMiY13o5v7D/Yp2QdQfXxs9cm+bH57sVb/6arQX0w3d9KrpdtWoNRpWEenggUH+BzaObcL0qGYOjWOCxy2BxomFM4lTWdw6IGGKEy8qOkQLVeraLWaouFE4R2nvBvzFunkLZR523pyTbu6aYfj9aEOOJGT6myBBSHvRhjCEm8cZLZ23igHURoytEeEXbCZUPHycVzyvlojwx+3mxXrVH7/sXv8N+0SJ8I61JJ16pn9aSUcclqzZ5tzlpKkQxCgXr9WLhb95i4Y4d5fZB5bWSealpWZv/zcdAeEqQ+SLT+P/QT/Kg/cY/jM+qhVT6YT8BGliRQnrqpSyu5CSktiz92ItR8wfSWpXM3Glozmiwysi6mfl+PGZBnW73cLu2faQsGJrYjHVYli2NnuySUHX3AcrWp8sgOreZ4xeiPZ4Xhjo6VqHmFvmRJT7K28taCzrZVPHW/v/vh0gTtrCscb2rfWPi5mAGL5tAHqyqQu2VQr8xQt2m8ImtpvvQEmkO33JR82WoZ7LbxQo/u4EXqymb2ktG6boGhcaKg1o2a7R2lQTn++a925Tee6jscFzYa1N2ru+i2TpuKlN3iGslv5Ht4zEQvGocJ/N0GfBePIlLaxqnmLks5TVa2ilKA5RMH5ZyJg0ypGS+/LW++pjhGJZ9O6nZhy4dazzI7LHzHWmM0zeTOjTXg1LXa0taJLLkrcvNZk3Exw0i7CGL7c0FIhzAVjiNp6OUKoFFZaK+Vrz11ZK0tOpX684LLf4xC1pGvLPOz1DRdLDqcvNWNoISGO9uTN7TWvZcmvZ19oytAWzgm0by3ab93rHOWKOXcNV8gimrBDk/7efe+y/vDMEyaWfMy08Q7EM71LrCXuxx1BtlBLrDBlvdYS2/uzls1VAq8O6n6S5QP3+e4bDXGvN+Yub/Oj4LyR2ffq3dVeC84R2Ivx8y0dOxvIay8D68rus+buVxDzUsXdb0nCz/8B</diagram></mxfile> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,161 @@ | ||
| # Calico-VPP Deployment | ||
| This mode allows deployment of VPP in Calico clusters, where some of the nodes | ||
| can be running plain Calico (without VPP) and some of the nodes can be running | ||
| Calico with VPP (programmed using the components normally shipped with Contiv-VPP). | ||
| The desired CNI on the particular node can be specified using the labels; the nodes can be | ||
| either labeled as `cni-type=calico` or `cni-type=vpp`. | ||
|
|
||
| Both types of CNI use the same IPAM plugin, which avoids IP address conflicts and allows | ||
| PODs running on non-VPP nodes to communicate with PODs running on VPP nodes as normally. | ||
|
|
||
| As of now, only no-overlay mode is supported, with BGP used to announce the POD subnets between | ||
| the nodes and the outside network. | ||
|
|
||
| [](../../docs/img/img/calico-vpp.png) | ||
|
|
||
| ## Vagrant Deployment Steps | ||
|
|
||
| Use the `./vagrant-start` script and choose the `Calico-VPP` deployment: | ||
|
|
||
| ```bash | ||
| $ ./vagrant-start | ||
| Please provide the number of workers for the Kubernetes cluster (0-50) or enter [Q/q] to exit: 2 | ||
|
|
||
| Please choose Kubernetes environment: | ||
| 1) Production | ||
| 2) Development | ||
| 3) Quit | ||
| --> 1 | ||
| You chose Production environment | ||
|
|
||
| Please choose deployment scenario: | ||
| 1) Without StealTheNIC 3) Calico 5) Quit | ||
| 2) With StealTheNIC 4) Calico-VPP | ||
| --> 4 | ||
| You chose deployment with Calico-VPP | ||
| ... | ||
| ``` | ||
|
|
||
| The master node is automatically labeled as `cni-type=calico`. The worker nodes are | ||
| waiting to be labeled as either `cni-type=calico` or `cni-type=vpp`. To label them, use | ||
| the `kubectl` tool on the master node: | ||
|
|
||
| ```bash | ||
| $ vagrant ssh k8s-master | ||
|
|
||
| vagrant@k8s-master:~$ kubectl label node k8s-worker1 cni-type=calico | ||
| node/k8s-worker1 labeled | ||
|
|
||
| vagrant@k8s-master:~$ kubectl label node k8s-worker2 cni-type=vpp | ||
| node/k8s-worker2 labeled | ||
| ``` | ||
|
|
||
| Wait until all PODs are running and ready. This may take some time, since it pulls images | ||
| from Dockerhub & looses connectivity to master when doing STN: | ||
|
|
||
| ```bash | ||
| vagrant@k8s-master:~$ kubectl get pods --all-namespaces -o wide | ||
|
|
||
| NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE | ||
| kube-system calico-node-8nzc7 2/2 Running 0 1m 192.168.16.3 k8s-worker1 <none> | ||
| kube-system calico-node-lklxr 2/2 Running 0 11m 192.168.16.2 k8s-master <none> | ||
| kube-system calico-node-vpp-hljxl 1/2 Running 0 1m 192.168.16.4 k8s-worker2 <none> | ||
| kube-system contiv-crd-jvlq5 1/1 Running 0 11m 192.168.16.2 k8s-master <none> | ||
| kube-system contiv-etcd-0 1/1 Running 0 11m 192.168.16.2 k8s-master <none> | ||
| kube-system contiv-ksr-bklfr 1/1 Running 0 11m 192.168.16.2 k8s-master <none> | ||
| kube-system contiv-vswitch-c8l92 1/1 Running 0 1m 192.168.16.4 k8s-worker2 <none> | ||
| kube-system coredns-78fcdf6894-z52hv 1/1 Running 0 11m 10.10.0.3 k8s-master <none> | ||
| kube-system coredns-78fcdf6894-zsjq6 1/1 Running 0 11m 10.10.0.2 k8s-master <none> | ||
| kube-system etcd-k8s-master 1/1 Running 0 10m 192.168.16.2 k8s-master <none> | ||
| kube-system kube-apiserver-k8s-master 1/1 Running 0 10m 192.168.16.2 k8s-master <none> | ||
| kube-system kube-controller-manager-k8s-master 1/1 Running 0 9m 192.168.16.2 k8s-master <none> | ||
| kube-system kube-proxy-f8bxp 1/1 Running 0 3m 192.168.16.4 k8s-worker2 <none> | ||
| kube-system kube-proxy-hcpkh 1/1 Running 0 11m 192.168.16.2 k8s-master <none> | ||
| kube-system kube-proxy-xjm7s 1/1 Running 0 5m 192.168.16.3 k8s-worker1 <none> | ||
| kube-system kube-scheduler-k8s-master 1/1 Running 0 10m 192.168.16.2 k8s-master <none> | ||
| ``` | ||
|
|
||
| Check Calico BGP status. It should contain an BGP peer item for each worker node and one more for the gateway: | ||
| ```bash | ||
| vagrant@k8s-master:~$ sudo calicoctl node status | ||
|
|
||
| Calico process is running. | ||
|
|
||
| IPv4 BGP status | ||
| +----------------+-------------------+-------+----------+-------------+ | ||
| | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | | ||
| +----------------+-------------------+-------+----------+-------------+ | ||
| | 192.168.16.100 | global | up | 13:03:44 | Established | | ||
| | 192.168.16.3 | node-to-node mesh | up | 13:10:09 | Established | | ||
| | 192.168.16.4 | node-to-node mesh | up | 13:10:59 | Established | | ||
| +----------------+-------------------+-------+----------+-------------+ | ||
| ``` | ||
|
|
||
| Deploy applications as normally, the PODs will run either on non-VPP or on VPP nodes. | ||
| They should be able to communicate with each other as usually. | ||
|
|
||
|
|
||
| ## Tainting the VPP Nodes | ||
| By default, any POD can be scheduled on any node. Taints can be used to force k8s | ||
| to schedule PODs on non-VPP nodes by default, and only deploy special PODs (which allow it | ||
| by a toleration) on VPP nodes. | ||
|
|
||
|
|
||
| To taint `k8s-worker2` node so that the `cni-type=vpp` needs to be explicitly allowed to | ||
| schedule a POD on that node: | ||
| ```bash | ||
| vagrant@k8s-master:~$ kubectl taint nodes k8s-worker2 cni-type=vpp:NoSchedule | ||
| node/k8s-worker2 tainted | ||
| ``` | ||
|
|
||
| From this point, no more "standard" PODs should be scheduled on the `k8s-worker2` node. Let's check that | ||
| by deploying a nginx deployment with multiple replicas: | ||
|
|
||
| ```bash | ||
| vagrant@k8s-master:~$ kubectl run nginx --image=nginx --replicas=4 | ||
| deployment.apps/nginx created | ||
| ``` | ||
|
|
||
| Note that none of the nginx PODs was scheduled on the `k8s-worker2` node: | ||
| ```bash | ||
| vagrant@k8s-master:~$ kubectl get pods -o wide | ||
| NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE | ||
| nginx-64f497f8fd-8kvqk 1/1 Running 0 1m 10.10.1.3 k8s-worker1 <none> | ||
| nginx-64f497f8fd-926xw 1/1 Running 0 1m 10.10.1.2 k8s-worker1 <none> | ||
| nginx-64f497f8fd-q8zch 1/1 Running 0 1m 10.10.0.5 k8s-master <none> | ||
| nginx-64f497f8fd-vdvss 1/1 Running 0 1m 10.10.0.4 k8s-master <none> | ||
| ``` | ||
|
|
||
| To allow scheduling of a POD on a VPP node, specify the toleration in its definition. You can also | ||
| combine it with a node selector, if you want to force the scheduler to place it on a VPP node | ||
| (the toleration itself allows deployment on a VPP node, but does not force it - it still may be | ||
| deployed on a non-VPP node as well): | ||
| ```bash | ||
| apiVersion: v1 | ||
| kind: Pod | ||
| metadata: | ||
| name: vpp-app-nginx | ||
| spec: | ||
| containers: | ||
| - name: nginx | ||
| image: nginx | ||
| nodeSelector: | ||
| cni-type: vpp | ||
| tolerations: | ||
| - key: "cni-type" | ||
| operator: "Equal" | ||
| value: "vpp" | ||
| effect: "NoSchedule" | ||
| ``` | ||
|
|
||
| After deplyment of the above yaml, you can verify that it was indeed deployed on the VPP | ||
| node `k8s-worker2`: | ||
| ```bash | ||
| vagrant@k8s-master:~$ kubectl get pods -o wide | ||
| NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE | ||
| nginx-64f497f8fd-8kvqk 1/1 Running 0 6m 10.10.1.3 k8s-worker1 <none> | ||
| nginx-64f497f8fd-926xw 1/1 Running 0 6m 10.10.1.2 k8s-worker1 <none> | ||
| nginx-64f497f8fd-q8zch 1/1 Running 0 6m 10.10.0.5 k8s-master <none> | ||
| nginx-64f497f8fd-vdvss 1/1 Running 0 6m 10.10.0.4 k8s-master <none> | ||
| vpp-app-nginx 1/1 Running 0 48s 10.10.2.2 k8s-worker2 <none> | ||
| ``` |