WYJ's Notes

Deploying Kubernetes on Bare Metal

  • VMs Env Prepare
  • Step 1: Precheck
  • Step 2: Installing runtime
  • Step 3: Installing kubeadm
  • Step 4: Initializing control-plane node
  • Step 5: Installing a Pod network add-on
  • Step 6: Joining your nodes
  • Step 7: Clean up
    • Remove the node
    • Clean up the control plane
  • Troubleshooting

VMs Env Prepare

  • Master Node - 16.186.77.143

Step 1: Precheck

  • Letting iptables see bridged traffic
    • Check that the br_netfilter module is loaded: lsmod | grep br_netfilter
    • To load it explicitly: sudo modprobe br_netfilter
    • ensure net.bridge.bridge-nf-call-iptables is set to 1 in your sysctl config
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

Step 2: Installing runtime

# (Install Docker CE)
## Set up the repository
### Install required packages
sudo yum install -y yum-utils device-mapper-persistent-data lvm2

## Add the Docker repository
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

# Install Docker CE
sudo yum update -y && sudo yum install -y \
  containerd.io-1.2.13 \
  docker-ce-19.03.11 \
  docker-ce-cli-19.03.11

## Create /etc/docker
sudo mkdir /etc/docker

# Set up the Docker daemon
cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF

# Create /etc/systemd/system/docker.service.d
sudo mkdir -p /etc/systemd/system/docker.service.d

# Restart Docker
sudo systemctl daemon-reload
sudo systemctl restart docker

# If you want the docker service to start on boot, run the following command:
sudo systemctl enable docker

Step 3: Installing kubeadm

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

# Set SELinux in permissive mode (effectively disabling it)
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

sudo systemctl enable --now kubelet

Step 4: Initializing control-plane node

  • Run kubeadm config images pull prior to kubeadm init to verify connectivity to the gcr.io container image registry.
  • kubeadm init –apiserver-advertise-address=16.186.77.143 –pod-network-cidr 10.244.0.0/16
  • systemctl enable kubelet.service
# kubeconfig setup
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
  • (Optional)Control plane node isolation: kubectl taint nodes --all node-role.kubernetes.io/master-

Step 5: Installing a Pod network add-on

  • wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
  • Set network CIDR for net-conf.json in kube-flannel.yml
  • kubectl apply -f kube-flannel.yml -n kube-system

Step 6: Joining your nodes

  • SSH to the machine and Become root (e.g. sudo su -)
  • kubeadm join 16.186.77.143:6443 --token p9lnj7.k6mso572xodjkgqn --discovery-token-ca-cert-hash sha256:be13c400e2cd6826abf0f2a292044d3297aadd98e98a93b2f464aee01ca7df1c
  • kubeadm token list/create
  • kubectl create (cluster)rolebinding.
    • kubeadm alpha kubeconfig user --client-name <CN> - generate an unique credential to which you grant privileges.
    • kubectl create (cluster)rolebinding

Step 7: Clean up

Remove the node

  • Talking to the control-plane node with the appropriate credentials: kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
  • kubeadm reset
  • reset or clean up iptables rules or IPVS tables: iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
  • reset the IPVS tables: ipvsadm -C
  • remove the node: kubectl delete node <node name>

Clean up the control plane

  • kubeadm reset
  • systemctl stop kubelet
  • systemctl stop docker
  • rm -rf /var/lib/cni/
  • rm -rf /var/lib/kubelet/*
  • rm -rf /etc/cni/
  • ifconfig cni0 down
  • ifconfig flannel.1 down
  • ifconfig docker0 down

Troubleshooting

  • NetworkPlugin cni failed to set up pod “xxxxx” network: failed to set bridge addr: “cni0” already has an IP address different from10.x.x.x - Error
ip link set cni0 down
# if cannot delete, restart the vm
brctl delbr cni0
  • The route problem can be solved by flush iptables: “dial tcp 10.96.0.1:443: connect: no route to host…”
systemctl stop kubelet
systemctl stop docker
iptables --flush
iptables -tnat --flush
systemctl start kubelet
systemctl start docker

Tags:  Kubernetes