下面命令只需要在master执行即可

3.0 master初始化

kubeadm init \
--apiserver-advertise-address=172.16.31.21 \
--image-repository registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images \
--kubernetes-version v1.21.0 \
--service-cidr=10.96.0.0/16 \
--pod-network-cidr=192.168.0.0/16 --v=5
 
#要求所有网络范围不重叠 --pod-network-cidr  --service-cidr --apiserver-advertise-address 都不能重叠

#初始化失败执行这个，重新初始化
sudo kubeadm reset

3.1 master 初始化日志

I0716 23:16:09.034852    6166 initconfiguration.go:104] detected and using CRI socket: /var/run/dockershim.sock
[init] Using Kubernetes version: v1.21.0
[preflight] Running pre-flight checks
I0716 23:16:09.147485    6166 checks.go:577] validating Kubernetes and kubeadm version
    [WARNING KubernetesVersion]: Kubernetes version is greater than kubeadm version. Please consider to upgrade kubeadm. Kubernetes version: 1.21.0. Kubeadm version: 1.20.x
I0716 23:16:09.147595    6166 checks.go:166] validating if the firewall is enabled and active
I0716 23:16:09.154919    6166 checks.go:201] validating availability of port 6443
I0716 23:16:09.155021    6166 checks.go:201] validating availability of port 10259
I0716 23:16:09.155032    6166 checks.go:201] validating availability of port 10257
I0716 23:16:09.155046    6166 checks.go:286] validating the existence of file /etc/kubernetes/manifests/kube-apiserver.yaml
I0716 23:16:09.155053    6166 checks.go:286] validating the existence of file /etc/kubernetes/manifests/kube-controller-manager.yaml
I0716 23:16:09.155057    6166 checks.go:286] validating the existence of file /etc/kubernetes/manifests/kube-scheduler.yaml
I0716 23:16:09.155060    6166 checks.go:286] validating the existence of file /etc/kubernetes/manifests/etcd.yaml
I0716 23:16:09.155066    6166 checks.go:432] validating if the connectivity type is via proxy or direct
I0716 23:16:09.155106    6166 checks.go:471] validating http connectivity to first IP address in the CIDR
I0716 23:16:09.155119    6166 checks.go:471] validating http connectivity to first IP address in the CIDR
I0716 23:16:09.155125    6166 checks.go:102] validating the container runtime
I0716 23:16:09.227451    6166 checks.go:128] validating if the "docker" service is enabled and active
    [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0716 23:16:09.336219    6166 checks.go:335] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I0716 23:16:09.336306    6166 checks.go:335] validating the contents of file /proc/sys/net/ipv4/ip_forward
I0716 23:16:09.336330    6166 checks.go:649] validating whether swap is enabled or not
I0716 23:16:09.336345    6166 checks.go:376] validating the presence of executable conntrack
I0716 23:16:09.336363    6166 checks.go:376] validating the presence of executable ip
I0716 23:16:09.336369    6166 checks.go:376] validating the presence of executable iptables
I0716 23:16:09.336377    6166 checks.go:376] validating the presence of executable mount
I0716 23:16:09.336384    6166 checks.go:376] validating the presence of executable nsenter
I0716 23:16:09.336391    6166 checks.go:376] validating the presence of executable ebtables
I0716 23:16:09.336396    6166 checks.go:376] validating the presence of executable ethtool
I0716 23:16:09.336401    6166 checks.go:376] validating the presence of executable socat
I0716 23:16:09.336407    6166 checks.go:376] validating the presence of executable tc
I0716 23:16:09.336411    6166 checks.go:376] validating the presence of executable touch
I0716 23:16:09.336419    6166 checks.go:520] running all checks
I0716 23:16:09.417428    6166 checks.go:406] checking whether the given node name is reachable using net.LookupHost
I0716 23:16:09.417790    6166 checks.go:618] validating kubelet version
I0716 23:16:09.501007    6166 checks.go:128] validating if the "kubelet" service is enabled and active
I0716 23:16:09.506921    6166 checks.go:201] validating availability of port 10250
I0716 23:16:09.506963    6166 checks.go:201] validating availability of port 2379
I0716 23:16:09.506972    6166 checks.go:201] validating availability of port 2380
I0716 23:16:09.506987    6166 checks.go:249] validating the existence and emptiness of directory /var/lib/etcd
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0716 23:16:09.540280    6166 checks.go:839] image exists: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-apiserver:v1.21.0
I0716 23:16:09.579551    6166 checks.go:839] image exists: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-controller-manager:v1.21.0
I0716 23:16:09.615551    6166 checks.go:839] image exists: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-scheduler:v1.21.0
I0716 23:16:09.651421    6166 checks.go:839] image exists: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-proxy:v1.21.0
I0716 23:16:09.692049    6166 checks.go:839] image exists: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/pause:3.2
I0716 23:16:09.734836    6166 checks.go:839] image exists: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/etcd:3.4.13-0
I0716 23:16:09.779173    6166 checks.go:839] image exists: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/coredns:1.7.0
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I0716 23:16:09.779231    6166 certs.go:110] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
I0716 23:16:09.936554    6166 certs.go:474] validating certificate period for ca certificate
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master] and IPs [10.96.0.1 172.16.31.21]
[certs] Generating "apiserver-kubelet-client" certificate and key
I0716 23:16:10.049704    6166 certs.go:110] creating a new certificate authority for front-proxy-ca
[certs] Generating "front-proxy-ca" certificate and key
I0716 23:16:10.221141    6166 certs.go:474] validating certificate period for front-proxy-ca certificate
[certs] Generating "front-proxy-client" certificate and key
I0716 23:16:10.262809    6166 certs.go:110] creating a new certificate authority for etcd-ca
[certs] Generating "etcd/ca" certificate and key
I0716 23:16:10.324995    6166 certs.go:474] validating certificate period for etcd/ca certificate
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master] and IPs [172.16.31.21 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master] and IPs [172.16.31.21 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
I0716 23:16:11.016517    6166 certs.go:76] creating new public/private key files for signing service account users
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0716 23:16:11.133874    6166 kubeconfig.go:101] creating kubeconfig file for admin.conf
[kubeconfig] Writing "admin.conf" kubeconfig file
I0716 23:16:11.345718    6166 kubeconfig.go:101] creating kubeconfig file for kubelet.conf
[kubeconfig] Writing "kubelet.conf" kubeconfig file
I0716 23:16:11.504030    6166 kubeconfig.go:101] creating kubeconfig file for controller-manager.conf
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0716 23:16:11.594761    6166 kubeconfig.go:101] creating kubeconfig file for scheduler.conf
[kubeconfig] Writing "scheduler.conf" kubeconfig file
I0716 23:16:11.837787    6166 kubelet.go:63] Stopping the kubelet
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
I0716 23:16:11.899153    6166 manifests.go:96] [control-plane] getting StaticPodSpecs
I0716 23:16:11.899421    6166 certs.go:474] validating certificate period for CA certificate
I0716 23:16:11.899476    6166 manifests.go:109] [control-plane] adding volume "ca-certs" for component "kube-apiserver"
I0716 23:16:11.899481    6166 manifests.go:109] [control-plane] adding volume "etc-pki" for component "kube-apiserver"
I0716 23:16:11.899484    6166 manifests.go:109] [control-plane] adding volume "k8s-certs" for component "kube-apiserver"
I0716 23:16:11.907571    6166 manifests.go:126] [control-plane] wrote static Pod manifest for component "kube-apiserver" to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
I0716 23:16:11.907594    6166 manifests.go:96] [control-plane] getting StaticPodSpecs
I0716 23:16:11.907969    6166 manifests.go:109] [control-plane] adding volume "ca-certs" for component "kube-controller-manager"
I0716 23:16:11.907982    6166 manifests.go:109] [control-plane] adding volume "etc-pki" for component "kube-controller-manager"
I0716 23:16:11.907985    6166 manifests.go:109] [control-plane] adding volume "flexvolume-dir" for component "kube-controller-manager"
I0716 23:16:11.907988    6166 manifests.go:109] [control-plane] adding volume "k8s-certs" for component "kube-controller-manager"
I0716 23:16:11.907990    6166 manifests.go:109] [control-plane] adding volume "kubeconfig" for component "kube-controller-manager"
I0716 23:16:11.909522    6166 manifests.go:126] [control-plane] wrote static Pod manifest for component "kube-controller-manager" to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[control-plane] Creating static Pod manifest for "kube-scheduler"
I0716 23:16:11.909541    6166 manifests.go:96] [control-plane] getting StaticPodSpecs
I0716 23:16:11.909781    6166 manifests.go:109] [control-plane] adding volume "kubeconfig" for component "kube-scheduler"
I0716 23:16:11.910495    6166 manifests.go:126] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0716 23:16:11.910988    6166 local.go:74] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml"
I0716 23:16:11.910999    6166 waitcontrolplane.go:87] [wait-control-plane] Waiting for the API server to be healthy
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 6.502556 seconds
I0716 23:16:18.414773    6166 uploadconfig.go:108] [upload-config] Uploading the kubeadm ClusterConfiguration to a ConfigMap
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
I0716 23:16:18.434288    6166 uploadconfig.go:122] [upload-config] Uploading the kubelet component config to a ConfigMap
[kubelet] Creating a ConfigMap "kubelet-config-1.21" in namespace kube-system with the configuration for the kubelets in the cluster
I0716 23:16:18.443537    6166 uploadconfig.go:127] [upload-config] Preserving the CRISocket information for the control-plane node
I0716 23:16:18.443554    6166 patchnode.go:30] [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "master" as an annotation
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)"
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: nohjt3.812p870l8p88xbwi
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
I0716 23:16:19.481680    6166 clusterinfo.go:45] [bootstrap-token] loading admin kubeconfig
I0716 23:16:19.481895    6166 clusterinfo.go:53] [bootstrap-token] copying the cluster from admin.conf to the bootstrap kubeconfig
I0716 23:16:19.482072    6166 clusterinfo.go:65] [bootstrap-token] creating/updating ConfigMap in kube-public namespace
I0716 23:16:19.485123    6166 clusterinfo.go:79] creating the RBAC rules for exposing the cluster-info ConfigMap in the kube-public namespace
I0716 23:16:19.488357    6166 kubeletfinalize.go:88] [kubelet-finalize] Assuming that kubelet client certificate rotation is enabled: found "/var/lib/kubelet/pki/kubelet-client-current.pem"
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
I0716 23:16:19.488912    6166 kubeletfinalize.go:132] [kubelet-finalize] Restarting the kubelet to enable client certificate rotation
[addons] Applied essential addon: CoreDNS
I0716 23:16:19.852418    6166 request.go:591] Throttling request took 196.767727ms, request: POST:https://172.16.31.21:6443/api/v1/namespaces/kube-system/configmaps?timeout=10s
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.16.31.21:6443 --token nohjt3.812p870l8p88xbwi \
    --discovery-token-ca-cert-hash sha256:642a40feae03609775fa79444ed73dc83684caca1400bb4ceaaba4d55b45b383

3.2 执行下面日志提示命令

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
#设置环境变量
export KUBECONFIG=/etc/kubernetes/admin.conf

3.3 安装calico（网络插件）（也是master执行）

curl https://calico-v3-25.netlify.app/archive/v3.25/manifests/calico.yaml -O

kubectl apply -f ./calico.yaml

3.4 获取集群所有的pod

kubectl get pod -A

3.5 查看集群机器的状态

#可以看到目前master节点状态Ready
kubectl get nodes

至此，大工告成。

3.6 把server01和server02加入集群

此步骤分别在其他2个节点，server01和server02上执行即可。

kubeadm join 172.16.31.21:6443 --token nohjt3.812p870l8p88xbwi \
    --discovery-token-ca-cert-hash sha256:642a40feae03609775fa79444ed73dc83684caca1400bb4ceaaba4d55b45b383

#查看所有pod
kubectl get pod -A

#查看节点状态，可以看到3个节点都是Ready状态，如果没就绪，等一等，跟机器性能有关。
kubectl gte node