Kubernetes & Ansible: Automating Kubernetes Cluster Operations with Kubespray

Kubernetes is the container orchestration platform that has gained the most traction for deploying and managing containerized applications; however, managing these clusters is anything but a trivial task.

Kubernetes Ansible Kubespray

To mitigate this many cloud providers – like Google’s GKE, Azure’s AKS, and Amazon’s EKS – provide tooling in order to abstract cluster operations such as provisioning, scaling and upgrading. This is all well and good if on-premise/hybrid solutions are not in play.

But, you may ask yourself: “wouldn’t all this be easier if I could use Ansible to automate some of these tasks?”

A question that I would reply with an emphatic “YES!” Enter in Kubespray – a deployment method that uses Ansible to provide us with the following abilities:

  • Deployable on AWS, GCE, Azure, Openstack, vSphere or Baremetal
  • High availability
  • Composable attributes (i.e. choice of network plugin)
  • Support for the most popular Linux distributions
  • Continuous integration tests

Let’s look at how to deploy a cluster with Kubespray.

Deploying a cluster

Requirements

Clone the Kubespray repository from here.

In order to start deploying your cluster each server will need to meet the following requirements from Kubernetes.io:

  • Ansible v2.4 (or newer) and python-netaddr is installed on the machine that will run Ansible commands
  • Jinja 2.9 (or newer) is required to run the Ansible Playbooks
  • The target servers must have access to the Internet in order to pull docker images.
  • The target servers are configured to allow IPv4 forwarding.
  • Your ssh key must be copied to all the servers part of your inventory.
  • The firewalls are not managed, you’ll need to implement your own rules the way you used to. in order to avoid any issue during deployment you should disable your firewall.
  • If kubespray is ran from non-root user account, correct privilege escalation method should be configured in the target servers. Then the ansible_become flag or command parameters --become or -b should be specified.

Any provisioning method will work as long as the machines meet the minimum system requirements for Kubernetes:

  • one or more machines running a supported Linux distribution
  • minimum 2 GB of RAM
  • minimum 2 CPUS

Fortunately, there is also a requirements.txt file handy so you can run sudo pip install -r requirements on the Ansible host machine.

You can use Kubespray in conjunction with other provisioners such as Terraform layout the base infrastructure. Kubespray includes Terraform scripts within it’s repo for AWS and Openstack deployments.

My sample cluster will be using a single Kubernetes master and 2 nodes provisioned with Centos 7 with the following IP addresses:

  • node1(master): 10.0.0.1
  • node2: 10.0.0.2
  • node3: 10.0.0.3

Inventory

Kubespray uses an inventory file in order to keep track of which machines in the cluster will be used for api server, etcd and nodes. First copy the sample inventory file and then use the generator to develop a new customized inventory:

1. Make a copy of the sample file

cp -rfp inventory/sample/* inventory/mycluster

2. Build new inventory file

declare -a IPS=(10.0.0.1 10.0.0.2 10.0.0.3)
CONFIG_FILE=inventory/mycluster/hosts.ini python3 contrib/inventory_builder/inventory.py ${IPS[@]}

Hosts

This will create the following hosts.ini:

Configure ‘ip’ variable to bind kubernetes services on a different ip than the default iface

node1 ansible_ssh_host=10.0.0.1 ip=10.0.0.1
node2 ansible_ssh_host=10.0.0.1 ip=10.0.0.2
node3 ansible_ssh_host=10.0.0.1 ip=10.0.0.3
[kube-master]
node1
node2
[etcd]
node1
node2
node3
[kube-node]
node2
node3
[k8s-cluster:children]
kube-node
kube-master

Note: the host IP will be different from the internal ip in production

Variables

Within the inventory directory there is also a folder for group vars. Within this directory you can set things like bootstrap os, add ingress and other options. Two main files that you will have to modify for most deployments are the all.yml and the k8s-cluster.yml files. The all.yml file contains options for bootstrap_os, directory for binaries, external loadbalancer configuration, dns, and certificate management. The k8s-cluster.yml files contain additional options for the cluster such as naming, container management and the enablement of features like ingress, persistent volumes, etc.

Deploy

After our configuration files are setup we can run the playbook with the following command:

ansible-playbook -i inventory/mycluster/hosts.ini cluster.yml — private-key=testkey.pem — flush-cache -s

After this grab a coffee and soon enough your Kubernetes cluster will be up and running. A default option is to include kubectl on the master node so you can verify that your cluster is healthy by logging in and running kubectl get nodes and verify that your nodes are in a healthy state.

Other Operations

In addition to deploying a cluster, Kubespray also includes playbooks for scaling (adding and removing nodes) and upgrading your cluster.

Scaling

Adding and removing nodes from a cluster is as easy as updating the hosts.ini and running the scale.yml or remove-node.yml playbooks respectively.

Adding a node with the scale.yml playbook:

ansible-playbook -i inventory/mycluster/hosts.ini scale.yml -b -v \
--private-key=~/.ssh/private_key

The remove-node.yml playbook will handle draining, certificate removal and node removal. The extra-vars option is the list of nodes to remove which can be added again:

ansible-playbook -i inventory/mycluster/hosts.ini remove-node.yml -b -v \
--private-key=~/.ssh/private_key \
--extra-vars "node=nodename,nodename2"

Upgrading

Kubespray provides functionality to upgrade the following Kubernetes cluster components:

  • docker_version
  • kube_version
  • etcd_version
  • calico_version
  • calico_cni_version
  • weave_version
  • flannel_version
  • kubedns_version

The playbook updgrade-cluster.yml will perform. “graceful” upgrade which will cordon, drain and uncordon the nodes being upgraded.

Upgrading kube_version on an existing cluster:

ansible-playbook upgrade-cluster.yml -b -i inventory/sample/hosts.ini -e kube_version=v1.11.0


Need help deploying and managing Kubernetes with Ansible? Our certified engineers can help. Contact us today.


Conclusion

These are examples of different cluster operations that are supported by Kubespray in order to streamline deployment and maintenance. This tool can manage small and large clusters on many different infrastructure types. By using a tool like Kubespray powered by Ansible you can do “Kubernetes the not so hard way”.

For more information on the power of Kubespray then please reference the following links:

Related Posts