Kubernetes - High Availability

Learn how to create high availability Kubernetes cluster

July 15, 2016 ta-ching chen

5 minute read

Table of Contents

Prerequisites

Please checkout previous installation tutorial to make sure that nothing lost

Create Reliable Storage Layer

Kubernetes use etcd as backend storage to keep cluster information. If we want to have a high availability of Kubernetes cluster, we need to set up etcd cluster as our reliable distributed key-value storage. Once we setup the etcd cluster, it will help us to populate data to whole etcd cluster.

Stop Single Node etcd

service etcd stop

Create etcd Cluster by Running Etcd on Multiple Nodes

sudo /opt/bin/etcd --name <node name> -data-dir <path to etcd data dir> \
--initial-advertise-peer-urls http://<node ip>:4000 \
--listen-peer-urls http://<node ip>:4000 \
--listen-client-urls http://127.0.0.1:4001,http://<node ip>:4001 \
--advertise-client-urls http://<node ip>:4001 \
--initial-cluster-token etcd-cluster-1 \
--initial-cluster <node1 name>=http://<node1 ip>:4000,<node2 name>=http://<node2 ip>:4000,<node3 name>=http://<node3 ip>:4000 \
--initial-cluster-state new

Example:
- infra0: 10.211.55.14
- infra1: 10.211.55.15
- infra2: 10.211.55.16

# infra0
sudo /opt/bin/etcd --name infra0 -data-dir /opt/etcd --initial-advertise-peer-urls http://10.211.55.14:4000 --listen-peer-urls http://10.211.55.14:4000 --listen-client-urls http://127.0.0.1:4001,http://10.211.55.14:4001 --advertise-client-urls http://10.211.55.14:4001 --initial-cluster-token etcd-cluster-1 --initial-cluster infra0=http://10.211.55.14:4000,infra1=http://10.211.55.15:4000,infra2=http://10.211.55.16:4000 --initial-cluster-state new
# infra1
sudo /opt/bin/etcd --name infra1 -data-dir /opt/etcd --initial-advertise-peer-urls http://10.211.55.15:4000 --listen-peer-urls http://10.211.55.15:4000 --listen-client-urls http://127.0.0.1:4001,http://10.211.55.15:4001 --advertise-client-urls http://10.211.55.15:4001 --initial-cluster-token etcd-cluster-1 --initial-cluster infra0=http://10.211.55.14:4000,infra1=http://10.211.55.15:4000,infra2=http://10.211.55.16:4000 --initial-cluster-state new
# infra2
sudo /opt/bin/etcd --name infra2 -data-dir /opt/etcd --initial-advertise-peer-urls http://10.211.55.16:4000 --listen-peer-urls http://10.211.55.16:4000 --listen-client-urls http://127.0.0.1:4001,http://10.211.55.16:4001 --advertise-client-urls http://10.211.55.16:4001 --initial-cluster-token etcd-cluster-1 --initial-cluster infra0=http://10.211.55.14:4000,infra1=http://10.211.55.15:4000,infra2=http://10.211.55.16:4000 --initial-cluster-state new

Once we create etcd cluster, we can use etcdctl ls to checkout current config in etcd

$ etcdctl ls
/registry

Rebuild Flannel Network

Flannel is a backend overlay network for Kubernetes and it uses etcd as config storage. After create the etcd cluster, we need to generate the config for flannel.

rebuild flannel config entry

Go to each node and execute following command

# following command can be find at kubernetes/cluster/ubuntu/util.sh
# it will rebuild flannel network config entry
FLANNEL_NET="172.16.0.0/16" KUBE_CONFIG_FILE="config-default.sh" DOCKER_OPTS="" ~/kube/reconfDocker.sh ai

and now /coreos.com/network/subnets should contains all nodes network interface information

$ etcdctl ls /coreos.com/network/subnets
/coreos.com/network/subnets/172.16.92.0-24
/coreos.com/network/subnets/172.16.80.0-24
/coreos.com/network/subnets/172.16.2.0-24

$ etcdctl get /coreos.com/network/subnets/172.16.2.0-24
{"PublicIP":"10.211.55.15","BackendType":"vxlan","BackendData":{"VtepMAC":"22:f2:fc:58:41:72"}}

Enable Redundancy Kubernetes Components

To enable Kubernetes high availability, we need to copy the binaries kube-apiserver, kube-controller-manager, kube-scheduler to each node we want it to be the master and launch.

Default Config for Each Binary

/etc/default/kube-apiserver

KUBE_APISERVER_OPTS=" --apiserver-count=<number of apiserver> --insecure-bind-address=0.0.0.0 --insecure-port=8080 --etcd-servers=http://127.0.0.1:4001 --logtostderr=true --service-cluster-ip-range=192.168.3.0/24 --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,SecurityContextDeny,ResourceQuota --service-node-port-range=30000-32767 --advertise-address=<node ip, e.g 10.211.55.14> --client-ca-file=/srv/kubernetes/ca.crt --tls-cert-file=/srv/kubernetes/server.cert --tls-private-key-file=/srv/kubernetes/server.key"

/etc/default/kube-controller-manager

KUBE_CONTROLLER_MANAGER_OPTS=" --master=127.0.0.1:8080 --root-ca-file=/srv/kubernetes/ca.crt --service-account-private-key-file=/srv/kubernetes/server.key --v=2 --leader-elect=true --logtostderr=true"

/etc/default/kube-scheduler

KUBE_SCHEDULER_OPTS=" --logtostderr=true --master=127.0.0.1:8080 --v=2 --leader-elect=true"

Apply HA to API Endpoint

At previouse step, we launch multiple kube-apiserver, however, we may encounter the single point of failure because there is still only one api server in config. Now, we gonna use Pound to set up an local api proxy that points to multiple api servers so that we can escape from SPOF.

Install Pound

Kubernetes use a new RESTful method PATCH that only supported by Pound 2.7, we need to install 2.7 or higher on all nodes in the cluster.

For Ubuntu 16.04+

$ sudo apt-get install pound

For Ubuntu 14.04

Please go to the Pound official site to download the 2.7.tar.gz and compile it.

$ wget http://www.apsis.ch/pound/Pound-2.7.tgz
$ tar -zxvf Pound-2.7.tgz
$ cd Pound-2.7
$ configure && make

or download from here if you trust me. lol

$ wget https://tachingchen.com/2016/07/15/Kubernetes-High-Availability/pound-bin.tar

Pound Setup

change backend ip and port in /etc/pound/pound.cfg

User        "www-data"
Group       "www-data"
LogLevel    1
## check backend every X secs:
Alive       1
## use hardware-accelleration card supported by openssl(1):
#SSLEngine  "<hw>"
# poundctl control socket
Control "/var/run/pound/poundctl.socket"
ListenHTTP
      Address 0.0.0.0
    Port    8080
    xHTTP       1
    Service
        BackEnd
            Address x.x.x.x
            Port    8081
        End
        BackEnd
            Address y.y.y.y
            Port    8081
        End
        BackEnd
            Address z.z.z.z
            Port    8081
        End
    End
End

upstart script for manual installation /etc/init.d/pound

#! /bin/sh
### BEGIN INIT INFO
# Provides:          pound
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Should-Start:      $named
# Should-Stop:       $named
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: reverse proxy and load balancer
# Description:       reverse proxy, load balancer and
#                    HTTPS front-end for Web servers
### END INIT INFO
#
# pound - reverse proxy, load-balancer and https front-end for web-servers
PATH=/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/sbin/pound
DESC="reverse proxy and load balancer"
NAME=pound
# Exit if the daemon does not exist (anymore)
test -f $DAEMON || exit 0
. /lib/lsb/init-functions
# Check if pound is configured or not
if [ -f "/etc/default/pound" ]
then
  . /etc/default/pound
  if [ "$startup" != "1" ]
  then
    log_warning_msg "$NAME will not start unconfigured."
    log_warning_msg "Please configure; afterwards, set startup=1 in /etc/default/pound."
    exit 0
  fi
else
  log_failure_msg "/etc/default/pound not found"
  exit 1
fi
# The real work of an init script
case "$1" in
  start)
  log_daemon_msg "Starting $DESC" "$NAME"
    if [ ! -d "/var/run/pound" ]
    then
        mkdir -p /var/run/pound
    fi
  start_daemon $DAEMON $POUND_ARGS
  log_end_msg $?
  ;;
  stop)
  log_daemon_msg "Stopping $DESC" "$NAME"
  killproc $DAEMON
  log_end_msg $?
  ;;
  restart|force-reload)
  log_daemon_msg "Restarting $DESC" "$NAME"
  killproc $DAEMON
  start_daemon $DAEMON $POUND_ARGS
  echo "."
  ;;
  status)
        pidofproc $DAEMON >/dev/null
  status=$?
  if [ $status -eq 0 ]; then
            log_success_msg "$NAME is running"
        else
            log_success_msg "$NAME is not running"
        fi
  exit $status
        ;;
  *)
  echo "Usage: $0 {start|stop|restart|force-reload|status}"
  exit 1
  ;;
esac
# Fallthrough if work done.
exit 0

Reconfigure kube-apiserver

Change default port 8080 to 8081 in /etc/default/kube-apiserver

--insecure-port=8081

Restart Kubernetes Components

restart related Kubernetes components

sudo service kube-apiserver restart
sudo service kube-controller-manager restart
sudo service kubelet restart
sudo service kube-proxy restart
sudo service kube-scheduler restart
sudo service pound restart

Troubleshooting

You will encounter such situation when you try to execute kubectl exec

$ kubectl exec -it <pod> bash
error: Timeout occured

It’s because Kubernetes use SPDY, which is not supported by Pound yet, as underlying communication protocol to connect to the node that container runs on, so you need to use following command instead:

ip: api server ip
port: api server port not proxy port

$ kubectl -s <ip>:<port> exec -it <pod> bash
$ kubectl -s 127.0.0.1:8081 exec -it <pod> bash

Kubernetes - High Availability

Prerequisites

Create Reliable Storage Layer

Stop Single Node etcd

Create etcd Cluster by Running Etcd on Multiple Nodes

Rebuild Flannel Network

Enable Redundancy Kubernetes Components

Default Config for Each Binary

Apply HA to API Endpoint

Install Pound

Pound Setup

Reconfigure kube-apiserver

Restart Kubernetes Components

Troubleshooting

Reference

See Also

Recent Posts

Pitfall of Go HTTP Header Operation

Fission - Serverless Framework For Kubernetes

How to Develop a Serverless Application with Fission (Part 3)

How to Develop a Serverless Application with Fission (Part 2)

How to Develop a Serverless Application with Fission (Part 1)

Categories