一、安装要求

本文主要在centos7系统上基于containerdv3.24.5版本的calico组件部署v1.29.3版本的堆叠ETCD高可用k8s原生集群

  • 操作系统CentOS7.x, 内核5.4.x
  • 硬件配置:2GB或更多RAM,2个CPU或更多CPU,硬盘30GB或更多
  • 集群中的所有机器之间网络互通
  • 可以访问外网 要拉取镜像
  • 禁止swap分区
IP Hostname cri
10.1.1.1 10-1-1-1.k8s.host docker
10.1.1.2 10-1-1-2.k8s.host docker
10.1.1.3 10-1-1-3.k8s.host containerd
10.1.1.4 10-1-1-4.k8s.host containerd
172.1.0.0/16 podSubnet
172.0.0.0/16 serviceSubnet

二、系统初始化

1 配置hosts

1
2
3
4
5
6
cat >> /etc/hosts <<EOF
10.1.1.1 10-1-1-1.k8s.host k8s-tst-cluster
10.1.1.2 10-1-1-2.k8s.host
10.1.1.3 10-1-1-3.k8s.host
10.1.1.4 10-1-1-4.k8s.host
EOF

2 关闭防火墙

systemctl stop firewalld && systemctl disable firewalld

3 关闭selinux

sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config

4 关闭swap分区

swapoff -a
sed -i '/swap / s/^\(.*\)$/#\1/g' /etc/fstab

5 设置内核参数

配置内核加载br_netfilteriptables放行ipv6ipv4的流量,确保集群内的容器能够正常通信。

1
2
3
4
5
6
7
8
9
10
11
12
13
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
vm.swappiness = 0
net.bridge.bridge-nf-call-iptables = 1
EOF


sudo sysctl --system

6 开启ipvs支持

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# 在使用ipvs模式之前确保安装了ipset和ipvsadm
sudo yum install ipset ipvsadm -y

# 手动加载ipvs相关模块
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack

# 配置开机自动加载ipvs相关模块
cat <<EOF | sudo tee /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
EOF


$ lsmod | grep -e ip_vs -e nf_conntrack
nf_conntrack_ipv6 18935 1
nf_defrag_ipv6 35104 1 nf_conntrack_ipv6
nf_conntrack_netlink 40492 0
nf_conntrack_ipv4 19149 3
nf_defrag_ipv4 12729 1 nf_conntrack_ipv4
ip_vs_sh 12688 0
ip_vs_wrr 12697 0
ip_vs_rr 12600 0
ip_vs 145458 6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack 143360 10 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_nat_masquerade_ipv6,nf_conntrack_netlink,nf_conntrack_ipv4,nf_conntrack_ipv6
libcrc32c 12644 4 xfs,ip_vs,nf_nat,nf_conntrack

$ cut -f1 -d " " /proc/modules | grep -e ip_vs -e nf_conntrack
nf_conntrack_ipv6
nf_conntrack_netlink
nf_conntrack_ipv4
ip_vs_sh
ip_vs_wrr
ip_vs_rr
ip_vs
nf_conntrack

7 时间同步

1
2
yum -y install ntpdate
ntpdate time.windows.com

8 安装cni-plugins

选择系统对应的架构版本

1
tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.3.0.tgz

三、安装集群

1 cri

一般建议一个集群使用同一个cri,但是为了文档包含内容多些,这集群使用两个cri
并且建议用containerd

容器运行时 默认路径
containerd unix:///var/run/containerd/containerd.sock
CRI-O unix:///var/run/crio/crio.sock
Docker Engine(使用 cri-dockerd) unix:///var/run/cri-dockerd.sock

1.1 安装Docker

1.1.1 配置阿里云Docker Yum源

sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

1.1.2 安装Docker最新版本

查看版本

yum list docker-ce --showduplicates | sort -r
sudo yum install -y docker-ce-20.10.24-3.el7

1.1.3 设置docker

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
cat >> /etc/docker/daemon.json  << EOF
{
"registry-mirrors" : [
"https://mirror.ccs.tencentyun.com",
"http://registry.docker-cn.com",
"http://docker.mirrors.ustc.edu.cn",
"http://hub-mirror.c.163.com"
],
"storage-driver": "overlay2",
"log-opts": {
"max-file": "2",
"max-size": "256m"
},
"exec-opts": ["native.cgroupdriver=systemd"],
"live-restore": true,
"log-level": "info",
"metrics-addr" : "127.0.0.1:9100",
"experimental" : true,
"data-root": "/data/docker"
}
EOF

1.1.4 启动Docker服务

systemctl enable docker
systemctl start docker

1.1.5 安装cri-dockerd

Kubernetes自v1.24移除了对docker-shim的支持,而Docker Engine默认又不支持CRI规范,因而二者将无法直接完成整合。
为此,Mirantis和Docker联合创建了cri-dockerd项目,用于为Docker Engine提供一个能够支持到CRI规范的垫片,从而能够让Kubernetes基于CRI控制Docker

可能下载不下来,建议翻墙下载离线安装 选择对应包下载

yum -y install https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.2/cri-dockerd-0.3.2-3.el7.x86_64.rpm

1.1.6 指定 pause 镜像地址

1
2
3
4
5
6
7
cat /usr/lib/systemd/system/cri-docker.service
...
ExecStart=/usr/bin/cri-dockerd --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6 --container-runtime-endpoint fd://
...

systemctl start cri-docker
systemctl enable cri-docker

1.2 containerd

1.2.1 配置内核

1
2
3
4
5
6
7
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

1.2.2 安装

1
2
3
4
5
6
7
8
9
10
11
12
13
# 导入docker官方的yum源
sudo yum install -y yum-utils device-mapper-persistent-data lvm2

sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

# 查看yum源中存在的各个版本的containerd.io
yum list containerd.io --showduplicates | sort -r

# 直接安装最新版本的containerd.io
yum install containerd.io -y

sudo systemctl start containerd
sudo systemctl enable --now containerd

1.2.3 配置cgroup drivers

CentOS7使用的是systemd来初始化系统并管理进程,初始化进程会生成并使用一个 root 控制组 (cgroup), 并充当 cgroup 管理器。 Systemd 与 cgroup 集成紧密,并将为每个 systemd 单元分配一个 cgroup。 我们也可以配置容器运行时和 kubelet 使用 cgroupfs。 连同 systemd 一起使用 cgroupfs 意味着将有两个不同的 cgroup 管理器。而当一个系统中同时存在cgroupfs和systemd两者时,容易变得不稳定,因此最好更改设置,令容器运行时和 kubelet 使用 systemd 作为 cgroup 驱动,以此使系统更为稳定。 对于containerd, 需要设置配置文件/etc/containerd/config.toml中的 SystemdCgroup 参数。

参考k8s官方的说明文档:
https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd-systemd

1
2
3
4
5
6
7
containerd config default > /etc/containerd/config.toml
# 修改SystemdCgroup参数并重启
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml
systemctl restart containerd
# 重启之后我们再检查配置就会发现已经启用了SystemdCgroup
containerd config dump | grep SystemdCgroup
SystemdCgroup = true

2 安装kubernetes 组件

2.1 添加kubernetes阿里云yum源

1
2
3
4
5
6
7
8
9
10
cat >> /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

2.2 kubeadm、kubelet、kubectl

由于版本更新频繁,所以指定版本号部署
yum -y install kubelet-1.26.5 kubeadm-1.26.5 kubectl-1.26.5
systemctl enable kubelet

四、master节点初始化

配置参考

  • kubernetesVersion字段用来指定我们要安装的k8s版本
  • localAPIEndpoint参数需要修改为我们的master节点的IP和端口,初始化之后的k8s集群的apiserver地址就是这个
  • podSubnetserviceSubnetdnsDomain两个参数默认情况下可以不用修改,这里我按照自己的需求进行了变更
  • nodeRegistration里面的name参数修改为对应master节点的hostname
  • controlPlaneEndpoint参数配置的才是我们前面配置的集群高可用apiserver的地址
  • cri-socket: 通过 systemctl cat docker |grep run 查看
  • 新增配置块使用ipvs,具体可以参考官方文档

出现问题解决不了清空执行: kubeadm reset -f ; ipvsadm --clear

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 10.1.1.3
bindPort: 6443
nodeRegistration:
criSocket: unix:///var/run/cri-dockerd.sock
imagePullPolicy: IfNotPresent
name: 10-1-1-3.k8s.host.ccops.com
taints: null
---
apiServer:
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: tst-kubernetes
controlPlaneEndpoint: k8s-tst-cluster:6443
controllerManager: {}
dns: {}
etcd:
local:
dataDir: /data/server/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.26.5
apiServer:
extraArgs:
audit-log-path: /var/log/k8s-audit.log
networking:
dnsDomain: tst-cluster.local
podSubnet: 172.1.0.0/16
serviceSubnet: 172.0.0.0/16
scheduler: {}
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

kubeadm init --config kubeadm-config.yaml --upload-certs
# 保存以下结果,后面加节点需要
You can now join any number of the control-plane node running the following command on each as root:

kubeadm join k8s-tst-cluster:6443 --token 0j24t3.rqaulkmomt6u10ar \
--discovery-token-ca-cert-hash sha256:5e35746495fcedb50c2db881604d43b46e5857d0493c1f9d0ad4acc35eadb8cf \
--control-plane --certificate-key 7c2695c6791ee290ff70acca2e45e2a154f9b3728658dc513ffbdad14e7feead

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join k8s-tst-cluster:6443 --token 0j24t3.rqaulkmomt6u10ar \
--discovery-token-ca-cert-hash sha256:5e35746495fcedb50c2db881604d43b46e5857d0493c1f9d0ad4acc35eadb8cf

1 配置kubectl

1
2
3
4
mkdir -p $HOME/.kube
sudo ln -s /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

1.1 自动补全

1
2
3
4
5
6
7
8
9
10
yum install bash-completion
echo "source /usr/share/bash-completion/bash_completion" >> ~/.bashrc
echo 'source <(kubectl completion bash)' >>~/.bashrc
source ~/.bashrc
# 命令自定义
cat >>/root/.bashrc<<EOF
alias k=kubectl
complete -F __start_kubectl k
EOF
source /root/.bashrc

2 部署calico

CNI的部署我们参考官网的自建K8S部署教程,官网主要给出了两种部署方式,分别是通过Calico operatorCalico manifests来进行部署和管理calico,operator是通过deployment的方式部署一个calico的operator到集群中,再用其来管理calico的安装升级等生命周期操作。manifests则是将相关都使用yaml的配置文件进行管理,这种方式管理起来相对前者比较麻烦,但是对于高度自定义的K8S集群有一定的优势。

需要注意的是,Calico 维护的网络在默认配置下,是一个被称为“Node-to-Node Mesh”的模式。这时候,每台宿主机上的 BGP Client 都需要跟其他所有节点的 BGP Client 进行通信以便交换路由信息。但是,随着节点数量 N 的增加,这些连接的数量就会以 N²的规模快速增长,从而给集群本身的网络带来巨大的压力。
所以,Node-to-Node Mesh 模式一般推荐用在少于 100 个节点的集群里。而在更大规模的集群中,我们需要用到的是一个叫作 Route Reflector 的模式。参考: 如何优雅的使用Calico路由反射模式

这里我们使用operator的方式进行部署。

不要用apply

1
2
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/tigera-operator.yaml
curl https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/custom-resources.yaml -O

2.1 修改配置

主要修改cidr为kubeadm里配置的podSubnet

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

cat custom-resources.yaml
# This section includes base Calico installation configuration.
# For more information, see: https://projectcalico.docs.tigera.io/master/reference/installation/api#operator.tigera.io/v1.Installation
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
name: default
spec:
# Configures Calico networking.
calicoNetwork:
# Note: The ipPools section cannot be modified post-install.
ipPools:
- blockSize: 26
cidr: 172.1.0.0/16
encapsulation: VXLANCrossSubnet
natOutgoing: Enabled
nodeSelector: all()
---

# This section configures the Calico API server.
# For more information, see: https://projectcalico.docs.tigera.io/master/reference/installation/api#operator.tigera.io/v1.APIServer
apiVersion: operator.tigera.io/v1
kind: APIServer
metadata:
name: default
spec: {}

kubectl create -f custom-resources.yaml

五、节点初始化

1 管理节点

接下来我们在剩下的两个master节点上面执行上面输出的命令,注意要执行带有--control-plane --certificate-key这两个参数的命令,其中--control-plane参数是确定该节点为master控制面节点,而--certificate-key参数则是把我们前面初始化集群的时候通过--upload-certs上传到k8s集群中的证书下载下来使用。

多网卡建议指定apiserver-advertise-address

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
kubeadm join k8s-tst-cluster:6443 --token zjoy9h.62mwhybt502sye7f \
--discovery-token-ca-cert-hash sha256:1640dcfc5e996864c5ebcfffd8051e8302f8cf455d09fd3cea6de4a299fea770 \
--control-plane --certificate-key 9d0da8e47ae6c1216ed5d464dabd18f1f3913ee6610a7bb20eabb7d5e5f5fb64 --cri-socket=unix:///var/run/cri-dockerd.sock --apiserver-advertise-address=10.1.1.1
This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.

To start administering your cluster from this node, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

2 node节点

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
kubeadm join k8s-tst-cluster:6443 --token zjoy9h.62mwhybt502sye7f \
--discovery-token-ca-cert-hash sha256:1640dcfc5e996864c5ebcfffd8051e8302f8cf455d09fd3cea6de4a299fea770
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

3 查看集群状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
kubectl get node
NAME STATUS ROLES AGE VERSION
10-1-1-1.k8s.host.ccops.com Ready control-plane 38m v1.26.5
10-1-1-2.k8s.host.ccops.com Ready control-plane 30m v1.26.5
10-1-1-3.k8s.host.ccops.com Ready control-plane 44m v1.26.5
10-1-1-4.k8s.host.ccops.com Ready <none> 48s v1.26.5
[root@10-1-1-3 calico]# kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
calico-system calico-kube-controllers-85f9866f46-k8wjt 1/1 Running 0 14m
calico-system calico-node-5wrpz 1/1 Running 0 54s
calico-system calico-node-bht57 1/1 Running 0 28m
calico-system calico-node-plqj7 1/1 Running 0 28m
calico-system calico-node-ww7bk 1/1 Running 0 28m
calico-system calico-typha-767bbcc4f4-twbd7 1/1 Running 0 28m
calico-system calico-typha-767bbcc4f4-zmfbb 1/1 Running 0 53s
calico-system csi-node-driver-7mjrt 2/2 Running 0 28m
calico-system csi-node-driver-gqgtr 2/2 Running 0 28m
calico-system csi-node-driver-q2gh2 2/2 Running 0 28m
calico-system csi-node-driver-xp9g6 2/2 Running 0 54s
kube-system coredns-5bbd96d687-lw5bk 1/1 Running 0 44m
kube-system coredns-5bbd96d687-rtjrd 1/1 Running 0 44m
kube-system etcd-10-1-1-1.k8s.host.ccops.com 1/1 Running 0 33m
kube-system etcd-10-1-1-2.k8s.host.ccops.com 1/1 Running 0 30m
kube-system etcd-10-1-1-3.k8s.host.ccops.com 1/1 Running 0 44m
kube-system kube-apiserver-10-1-1-1.k8s.host.ccops.com 1/1 Running 0 34m
kube-system kube-apiserver-10-1-1-2.k8s.host.ccops.com 1/1 Running 0 30m
kube-system kube-apiserver-10-1-1-3.k8s.host.ccops.com 1/1 Running 0 44m
kube-system kube-controller-manager-10-1-1-1.k8s.host.ccops.com 1/1 Running 0 33m
kube-system kube-controller-manager-10-1-1-2.k8s.host.ccops.com 1/1 Running 0 29m
kube-system kube-controller-manager-10-1-1-3.k8s.host.ccops.com 1/1 Running 0 44m
kube-system kube-proxy-78wrg 1/1 Running 0 38m
kube-system kube-proxy-jvfnd 1/1 Running 0 54s
kube-system kube-proxy-pbv2m 1/1 Running 0 44m
kube-system kube-proxy-ztx74 1/1 Running 0 30m
kube-system kube-scheduler-10-1-1-1.k8s.host.ccops.com 1/1 Running 0 33m
kube-system kube-scheduler-10-1-1-2.k8s.host.ccops.com 1/1 Running 0 30m
kube-system kube-scheduler-10-1-1-3.k8s.host.ccops.com 1/1 Running 0 44m
tigera-operator tigera-operator-5d6845b496-zvjcm 1/1 Running 0 29m