Kubernetes The Hard Way - Bare Metal with VM and Containers

Containers on Docks

General

This set of documents aims at providing full hands-on guides to set up Kubernetes (aka K8S) clusters on bare metal servers, which can actually be physical servers or virtual machines (VM), on premises or in private and public clouds.

Those documents are adaptations of the excellent “Kubernetes The Hard Way - Bare Metal” guide (by Tobias Schöneberg), itself derived from the famous “Kubernetes The Har Way” guide, by Kelsey Hightower.

Except for the Proxmox host itself, which is based on a Debian distribution, all the nodes are installed with CentOS distributions. Other distributions may of course be used. One distrubtion has to be chosen, so let it be CentOS for these guides.

For the Kubernetes-related tools and binaries (e.g., Docker, Go, kubectl), two variants are documented:

CentOS 7 packaged versions, e.g.:
Upstream versions:
Kubernetes CentOS packages are fairly outdated (v1.5.2). There are various consequences, for instance:
- The DNS-related services are still limited, like (old versions of) Kube-DNS only
- Most of the (recent) documentation on Kubernetes services does not work as is with those older versions. For instance, the Service part of Nginx does not work with CentOS packages of Kubernetes
Upgrading some versions of Kubernetes utilities on CentOS is not an esay task, as all the Docker, Go and Kubernetes tools have been packaged in a consistent way on the CentOS distribution. Either of those cannot be upgraded to newer versions without breaking the CentOS packaging contraint
Both variants have been documented, because depending on the use case, either may be prefered:
- CentOS packages for a quick setup, fully supported and consistent with the rest of the CentOS distribution
- Upstream binaries for a more flexible and up-to-date setup, requiring more configuration and maintenance work

The master and worker nodes usually feature different components:

Master components provide the cluster’s control plane. Master components make global decisions about the cluster (for example, scheduling), and detecting and responding to cluster events (starting up a new pod when a replication controller’s replicas field is unsatisfied).
- kube-apiserver, component on the master that exposes the Kubernetes API. It is the front-end for the Kubernetes control plane. It is designed to scale horizontally – that is, it scales by deploying more instances.
- etcd, consistent and highly-available key value store used as Kubernetes’ backing store for all cluster data.
- kube-scheduler,
- kube-controller-manager, component on the master that runs controllers. Logically, each controller is a separate process, but to reduce complexity, they are all compiled into a single binary and run in a single process. These controllers include:
  - Node Controller: responsible for noticing and responding when nodes go down
  - Replication Controller: responsible for maintaining the correct number of pods for every replication controller object in the system
  - Endpoints Controller: Populates the Endpoints object (that is, joins Services and Pods)
  - Service Account and Token Controllers: Create default accounts and API access tokens for new namespaces
- cloud-controller-manager, cloud-controller-manager runs controllers that interact with the underlying cloud providers. The cloud-controller-manager binary is an alpha feature introduced in Kubernetes release 1.6
Worker node components run on every node, maintaining running pods and providing the Kubernetes runtime environment.
- kubelet, is an agent that runs on each node in the cluster. It makes sure that containers are running in a pod. The kubelet takes a set of PodSpecs that are provided through various mechanisms and ensures that the containers described in those PodSpecs are running and healthy. The kubelet does not manage containers which were not created by Kubernetes.
- kube-proxy enables the Kubernetes service abstraction by maintaining network rules on the host and performing connection forwarding
- Container Runtime is the software that is responsible for running containers. Kubernetes supports several runtimes: Docker, containerd, cri-o, rktlet and any implementation of the Kubernetes CRI (Container Runtime Interface)
On top of those components, come addons, which are pods and services that implement cluster features. The pods may be managed by Deployments, ReplicationControllers, and so on. Namespaced addon objects are created in the kube-system namespace.
- DNS. While the other addons are not strictly required, all Kubernetes clusters should have cluster DNS, as many examples rely on it. Cluster DNS is a DNS server, in addition to the other DNS server(s) in your environment, which serves DNS records for Kubernetes services. Containers started by Kubernetes automatically include this DNS server in their DNS searches.
- Web UI Dashboard is a general purpose, web-based UI for Kubernetes clusters. It allows users to manage and troubleshoot applications running in the cluster, as well as the cluster itself.
- Container Resource Monitoring records generic time-series metrics about containers in a central database, and provides a UI for browsing that data.
- Cluster-level logging is a mechanism responsible for saving container logs to a central log store with search/browsing interface.

As many other documentations, that one will soon be outdated, and imperfect for sure. Contributions are therefore welcome to complemenent that guide. For instance, through issues or pull requests.

With KVM/QEMU Virtual Machines (VM)

Guide to set up a Kubernetes cluster on Proxmox-based KVM/QEMU virtual machines (VM). Using any other virtualization framework than Proxmox should not make much difference.

While full virtualization (with virtual machines) provides more flexibility (e.g., choice of kernel and type of operating system, reserved CPU and memory resources), it implies more overhead than containers. The following section accommodates cases with more resource-conscious requirements.

With LXC Containers

Guide to set up a Kubernetes cluster on Proxmox-based LXC containers. Using LXD or OpenVZ rather than Proxmox should not make much difference.

While there are quite a few articles and solutions on Kubernetes and similar clusters in Docker containers (e.g., KinD, Footloose), the literature remains scarce about Kubernetes deployed on container technologies such as Linux containers (e.g., LXC, OpenVZ). However, Linux containers allow for far more flexibility than Docker containers. Basically, Linux containers aim at executing full-blown (Linux) operating system (OS), whereas Docker containers aim at micro-services and serverless platforms, mainly servicing single processes.

This guide therefore aims at helping filling the gap of documentation in the Linux container space for Kubernetes.

Proxmox Setup

That section provides some details on how to prepare a Proxmox distribution in order to setup Kubernetes clusters on it.

All the Kubernetes cluster nodes are insulated thanks to a gateway: all the traffic from outside the cluster is channelled through the gateway. The set up of such a gateway is also an addition to this set of guides. It allows one to experiment with Kubernetes clusters, including operating some in production-like settings, while keeping some good level of security.

References

Kubernetes The Hard Way - Bare Metal, by Tobias Schöneberg, February 2018, GitHub
Kubernetes The Hard Way, by Kelsey Hightower, 2017-2018, GitHub
Run kubernetes inside LXC container, by Andrei Kvapil (aka kvaps), August 2018, Medium
Kubernetes reference documentation
Getting started guide on installing a multi-node Kubernetes cluster on Fedora with flannel
Kubernetes - Architecture, by Khtan66 - Own work, CC BY-SA 4.0
Kubernetes - Pod Networking, by Marvin The Paranoid - Own work, CC BY-SA 4.0
Kubernetes components, for master and worker nodes