一、安装要求
本文主要在centos7系统上基于containerd
和v3.24.5
版本的calico组件部署v1.29.3
版本的堆叠ETCD高可用k8s原生集群
操作系统CentOS7.x, 内核5.4.x
硬件配置:2GB或更多RAM,2个CPU或更多CPU,硬盘30GB或更多
集群中的所有机器之间网络互通
可以访问外网 要拉取镜像
禁止swap分区
IP
Hostname
cri
10.1.1.1
10-1-1-1.k8s.host
docker
10.1.1.2
10-1-1-2.k8s.host
docker
10.1.1.3
10-1-1-3.k8s.host
containerd
10.1.1.4
10-1-1-4.k8s.host
containerd
172.1.0.0/16
podSubnet
172.0.0.0/16
serviceSubnet
二、系统初始化
1 配置hosts
1 2 3 4 5 6 cat >> /etc/hosts <<EOF 10.1.1.1 10-1-1-1.k8s.host k8s-tst-cluster 10.1.1.2 10-1-1-2.k8s.host 10.1.1.3 10-1-1-3.k8s.host 10.1.1.4 10-1-1-4.k8s.host EOF
2 关闭防火墙
systemctl stop firewalld && systemctl disable firewalld
3 关闭selinux
sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config
4 关闭swap分区
swapoff -a
sed -i '/swap / s/^\(.*\)$/#\1/g' /etc/fstab
5 设置内核参数
配置内核加载br_netfilter
和iptables
放行ipv6
和ipv4
的流量,确保集群内的容器能够正常通信。
1 2 3 4 5 6 7 8 9 10 11 12 13 cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf br_netfilter EOF cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 vm.swappiness = 0 net.bridge.bridge-nf-call-iptables = 1 EOF sudo sysctl --system
6 开启ipvs支持
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 sudo yum install ipset ipvsadm -ymodprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack cat <<EOF | sudo tee /etc/modules-load.d/ipvs.conf ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack EOF $ lsmod | grep -e ip_vs -e nf_conntrack nf_conntrack_ipv6 18935 1 nf_defrag_ipv6 35104 1 nf_conntrack_ipv6 nf_conntrack_netlink 40492 0 nf_conntrack_ipv4 19149 3 nf_defrag_ipv4 12729 1 nf_conntrack_ipv4 ip_vs_sh 12688 0 ip_vs_wrr 12697 0 ip_vs_rr 12600 0 ip_vs 145458 6 ip_vs_rr,ip_vs_sh,ip_vs_wrr nf_conntrack 143360 10 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_nat_masquerade_ipv6,nf_conntrack_netlink,nf_conntrack_ipv4,nf_conntrack_ipv6 libcrc32c 12644 4 xfs,ip_vs,nf_nat,nf_conntrack $ cut -f1 -d " " /proc/modules | grep -e ip_vs -e nf_conntrack nf_conntrack_ipv6 nf_conntrack_netlink nf_conntrack_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack
7 时间同步
1 2 yum -y install ntpdate ntpdate time.windows.com
8 安装cni-plugins
选择系统对应的架构版本
1 tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.3.0.tgz
三、安装集群
1 cri
一般建议一个集群使用同一个cri,但是为了文档包含内容多些,这集群使用两个cri
并且建议用containerd
容器运行时
默认路径
containerd
unix:///var/run/containerd/containerd.sock
CRI-O
unix:///var/run/crio/crio.sock
Docker Engine(使用 cri-dockerd)
unix:///var/run/cri-dockerd.sock
1.1 安装Docker
1.1.1 配置阿里云Docker Yum源
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
1.1.2 安装Docker最新版本
查看版本
yum list docker-ce --showduplicates | sort -r
sudo yum install -y docker-ce-20.10.24-3.el7
1.1.3 设置docker
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 cat >> /etc/docker/daemon.json << EOF { "registry-mirrors" : [ "https://mirror.ccs.tencentyun.com", "http://registry.docker-cn.com", "http://docker.mirrors.ustc.edu.cn", "http://hub-mirror.c.163.com" ], "storage-driver": "overlay2", "log-opts": { "max-file": "2", "max-size": "256m" }, "exec-opts": ["native.cgroupdriver=systemd"], "live-restore": true, "log-level": "info", "metrics-addr" : "127.0.0.1:9100", "experimental" : true, "data-root": "/data/docker" } EOF
1.1.4 启动Docker服务
systemctl enable docker
systemctl start docker
1.1.5 安装cri-dockerd
Kubernetes自v1.24移除了对docker-shim的支持,而Docker Engine默认又不支持CRI规范,因而二者将无法直接完成整合。
为此,Mirantis和Docker联合创建了cri-dockerd项目,用于为Docker Engine提供一个能够支持到CRI规范的垫片,从而能够让Kubernetes基于CRI控制Docker
可能下载不下来,建议翻墙下载离线安装 选择对应包下载
yum -y install https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.2/cri-dockerd-0.3.2-3.el7.x86_64.rpm
1.1.6 指定 pause 镜像地址
1 2 3 4 5 6 7 cat /usr/lib/systemd/system/cri-docker.service... ExecStart=/usr/bin/cri-dockerd --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6 --container-runtime-endpoint fd:// ... systemctl start cri-docker systemctl enable cri-docker
1.2 containerd
1.2.1 配置内核
1 2 3 4 5 6 7 cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf overlay br_netfilter EOF sudo modprobe overlaysudo modprobe br_netfilter
1.2.2 安装
1 2 3 4 5 6 7 8 9 10 11 12 13 sudo yum install -y yum-utils device-mapper-persistent-data lvm2sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repoyum list containerd.io --showduplicates | sort -r yum install containerd.io -y sudo systemctl start containerdsudo systemctl enable --now containerd
1.2.3 配置cgroup drivers
CentOS7使用的是systemd
来初始化系统并管理进程,初始化进程会生成并使用一个 root 控制组 (cgroup
), 并充当 cgroup
管理器。 Systemd
与 cgroup
集成紧密,并将为每个 systemd
单元分配一个 cgroup
。 我们也可以配置容器运行时
和 kubelet
使用 cgroupfs
。 连同 systemd
一起使用 cgroupfs
意味着将有两个不同的 cgroup 管理器
。而当一个系统中同时存在cgroupfs和systemd两者时,容易变得不稳定,因此最好更改设置,令容器运行时和 kubelet 使用 systemd
作为 cgroup
驱动,以此使系统更为稳定。 对于containerd
, 需要设置配置文件/etc/containerd/config.toml
中的 SystemdCgroup
参数。
参考k8s官方的说明文档:
https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd-systemd
1 2 3 4 5 6 7 containerd config default > /etc/containerd/config.toml sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml systemctl restart containerd containerd config dump | grep SystemdCgroup SystemdCgroup = true
2 安装kubernetes 组件
2.1 添加kubernetes阿里云yum源
1 2 3 4 5 6 7 8 9 10 cat >> /etc/yum.repos.d/kubernetes.repo << EOF [kubernetes] name=kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF
2.2 kubeadm、kubelet、kubectl
由于版本更新频繁,所以指定版本号部署
yum -y install kubelet-1.26.5 kubeadm-1.26.5 kubectl-1.26.5
systemctl enable kubelet
四、master节点初始化
配置参考
kubernetesVersion
字段用来指定我们要安装的k8s版本
localAPIEndpoint
参数需要修改为我们的master节点的IP和端口,初始化之后的k8s集群的apiserver地址就是这个
podSubnet
、serviceSubnet
和dnsDomain
两个参数默认情况下可以不用修改,这里我按照自己的需求进行了变更
nodeRegistration
里面的name
参数修改为对应master节点的hostname
controlPlaneEndpoint
参数配置的才是我们前面配置的集群高可用apiserver的地址
cri-socket
: 通过 systemctl cat docker |grep run
查看
新增配置块使用ipvs,具体可以参考官方文档
出现问题解决不了清空执行: kubeadm reset -f ; ipvsadm --clear
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 cat kubeadm-config.yamlapiVersion: kubeadm.k8s.io/v1beta3 bootstrapTokens: - groups : - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 10.1.1.3 bindPort: 6443 nodeRegistration: criSocket: unix:///var/run/cri-dockerd.sock imagePullPolicy: IfNotPresent name: 10-1-1-3.k8s.host.ccops.com taints: null --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta3 certificatesDir: /etc/kubernetes/pki clusterName: tst-kubernetes controlPlaneEndpoint: k8s-tst-cluster:6443 controllerManager: {} dns: {} etcd: local : dataDir: /data/server/etcd imageRepository: registry.aliyuncs.com/google_containers kind: ClusterConfiguration kubernetesVersion: 1.26.5 apiServer: extraArgs: audit-log-path: /var/log/k8s-audit.log networking: dnsDomain: tst-cluster.local podSubnet: 172.1.0.0/16 serviceSubnet: 172.0.0.0/16 scheduler: {} --- apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration cgroupDriver: systemd --- apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration mode: ipvs kubeadm init --config kubeadm-config.yaml --upload-certs You can now join any number of the control-plane node running the following command on each as root: kubeadm join k8s-tst-cluster:6443 --token 0j24t3.rqaulkmomt6u10ar \ --discovery-token-ca-cert-hash sha256:5e35746495fcedb50c2db881604d43b46e5857d0493c1f9d0ad4acc35eadb8cf \ --control-plane --certificate-key 7c2695c6791ee290ff70acca2e45e2a154f9b3728658dc513ffbdad14e7feead Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use "kubeadm init phase upload-certs --upload-certs" to reload certs afterward.Then you can join any number of worker nodes by running the following on each as root: kubeadm join k8s-tst-cluster:6443 --token 0j24t3.rqaulkmomt6u10ar \ --discovery-token-ca-cert-hash sha256:5e35746495fcedb50c2db881604d43b46e5857d0493c1f9d0ad4acc35eadb8cf
1 配置kubectl
1 2 3 4 mkdir -p $HOME /.kubesudo ln -s /etc/kubernetes/admin.conf $HOME /.kube/configsudo chown $(id -u):$(id -g) $HOME /.kube/config
1.1 自动补全
1 2 3 4 5 6 7 8 9 10 yum install bash-completion echo "source /usr/share/bash-completion/bash_completion" >> ~/.bashrcecho 'source <(kubectl completion bash)' >>~/.bashrcsource ~/.bashrccat >>/root/.bashrc<<EOF alias k=kubectl complete -F __start_kubectl k EOF source /root/.bashrc
2 部署calico
CNI的部署我们参考官网的自建K8S部署教程 ,官网主要给出了两种部署方式 ,分别是通过Calico operator
和Calico manifests
来进行部署和管理calico
,operator是通过deployment的方式部署一个calico的operator到集群中,再用其来管理calico的安装升级等生命周期操作。manifests则是将相关都使用yaml的配置文件进行管理,这种方式管理起来相对前者比较麻烦,但是对于高度自定义的K8S集群有一定的优势。
需要注意的是,Calico 维护的网络在默认配置下,是一个被称为“Node-to-Node Mesh”的模式。这时候,每台宿主机上的 BGP Client 都需要跟其他所有节点的 BGP Client 进行通信以便交换路由信息。但是,随着节点数量 N 的增加,这些连接的数量就会以 N²的规模快速增长,从而给集群本身的网络带来巨大的压力。
所以,Node-to-Node Mesh 模式一般推荐用在少于 100 个节点的集群里。而在更大规模的集群中,我们需要用到的是一个叫作 Route Reflector 的模式。参考: 如何优雅的使用Calico路由反射模式
这里我们使用operator的方式进行部署。
不要用apply
1 2 kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/tigera-operator.yaml curl https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/custom-resources.yaml -O
2.1 修改配置
主要修改cidr
为kubeadm里配置的podSubnet
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 cat custom-resources.yaml apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: calicoNetwork: ipPools: - blockSize: 26 cidr: 172.1 .0 .0 /16 encapsulation: VXLANCrossSubnet natOutgoing: Enabled nodeSelector: all() --- apiVersion: operator.tigera.io/v1 kind: APIServer metadata: name: default spec: {}kubectl create -f custom-resources.yaml
五、节点初始化
1 管理节点
接下来我们在剩下的两个master节点上面执行上面输出的命令,注意要执行带有--control-plane --certificate-key
这两个参数的命令,其中--control-plane
参数是确定该节点为master控制面节点,而--certificate-key
参数则是把我们前面初始化集群的时候通过--upload-certs
上传到k8s集群中的证书下载下来使用。
多网卡建议指定apiserver-advertise-address
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 kubeadm join k8s-tst-cluster:6443 --token zjoy9h.62mwhybt502sye7f \ --discovery-token-ca-cert-hash sha256:1640dcfc5e996864c5ebcfffd8051e8302f8cf455d09fd3cea6de4a299fea770 \ --control-plane --certificate-key 9d0da8e47ae6c1216ed5d464dabd18f1f3913ee6610a7bb20eabb7d5e5f5fb64 --cri-socket=unix:///var/run/cri-dockerd.sock --apiserver-advertise-address=10.1.1.1 This node has joined the cluster and a new control plane instance was created: * Certificate signing request was sent to apiserver and approval was received. * The Kubelet was informed of the new secure connection details. * Control plane label and taint were applied to the new node. * The Kubernetes control plane instances scaled up. * A new etcd member was added to the local /stacked etcd cluster. To start administering your cluster from this node, you need to run the following as a regular user: mkdir -p $HOME /.kube sudo cp -i /etc/kubernetes/admin.conf $HOME /.kube/config sudo chown $(id -u):$(id -g) $HOME /.kube/config Run 'kubectl get nodes' to see this node join the cluster.
2 node节点
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 kubeadm join k8s-tst-cluster:6443 --token zjoy9h.62mwhybt502sye7f \ --discovery-token-ca-cert-hash sha256:1640dcfc5e996864c5ebcfffd8051e8302f8cf455d09fd3cea6de4a299fea770 [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
3 查看集群状态
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 kubectl get node NAME STATUS ROLES AGE VERSION 10-1-1-1.k8s.host.ccops.com Ready control-plane 38m v1.26.5 10-1-1-2.k8s.host.ccops.com Ready control-plane 30m v1.26.5 10-1-1-3.k8s.host.ccops.com Ready control-plane 44m v1.26.5 10-1-1-4.k8s.host.ccops.com Ready <none> 48s v1.26.5 [root@10-1-1-3 calico]# kubectl get po -A NAMESPACE NAME READY STATUS RESTARTS AGE calico-system calico-kube-controllers-85f9866f46-k8wjt 1/1 Running 0 14m calico-system calico-node-5wrpz 1/1 Running 0 54s calico-system calico-node-bht57 1/1 Running 0 28m calico-system calico-node-plqj7 1/1 Running 0 28m calico-system calico-node-ww7bk 1/1 Running 0 28m calico-system calico-typha-767bbcc4f4-twbd7 1/1 Running 0 28m calico-system calico-typha-767bbcc4f4-zmfbb 1/1 Running 0 53s calico-system csi-node-driver-7mjrt 2/2 Running 0 28m calico-system csi-node-driver-gqgtr 2/2 Running 0 28m calico-system csi-node-driver-q2gh2 2/2 Running 0 28m calico-system csi-node-driver-xp9g6 2/2 Running 0 54s kube-system coredns-5bbd96d687-lw5bk 1/1 Running 0 44m kube-system coredns-5bbd96d687-rtjrd 1/1 Running 0 44m kube-system etcd-10-1-1-1.k8s.host.ccops.com 1/1 Running 0 33m kube-system etcd-10-1-1-2.k8s.host.ccops.com 1/1 Running 0 30m kube-system etcd-10-1-1-3.k8s.host.ccops.com 1/1 Running 0 44m kube-system kube-apiserver-10-1-1-1.k8s.host.ccops.com 1/1 Running 0 34m kube-system kube-apiserver-10-1-1-2.k8s.host.ccops.com 1/1 Running 0 30m kube-system kube-apiserver-10-1-1-3.k8s.host.ccops.com 1/1 Running 0 44m kube-system kube-controller-manager-10-1-1-1.k8s.host.ccops.com 1/1 Running 0 33m kube-system kube-controller-manager-10-1-1-2.k8s.host.ccops.com 1/1 Running 0 29m kube-system kube-controller-manager-10-1-1-3.k8s.host.ccops.com 1/1 Running 0 44m kube-system kube-proxy-78wrg 1/1 Running 0 38m kube-system kube-proxy-jvfnd 1/1 Running 0 54s kube-system kube-proxy-pbv2m 1/1 Running 0 44m kube-system kube-proxy-ztx74 1/1 Running 0 30m kube-system kube-scheduler-10-1-1-1.k8s.host.ccops.com 1/1 Running 0 33m kube-system kube-scheduler-10-1-1-2.k8s.host.ccops.com 1/1 Running 0 30m kube-system kube-scheduler-10-1-1-3.k8s.host.ccops.com 1/1 Running 0 44m tigera-operator tigera-operator-5d6845b496-zvjcm 1/1 Running 0 29m