环境
- CentOS 7.3
- docker-ce-17.11.0.ce-1.el7.centos.x86_64
- kubectl-1.8.4-0.x86_64
- kubeadm-1.8.4-0.x86_64
- kubelet-1.8.4-0.x86_64
- kubernetes-cni-0.5.1-1.x86_64
系统配置
单节点，静态主机名解析
[root@master ~]# cat /etc/hosts
172.19.0.14 master
禁用防火墙和selinux
[root@master ~]# systemctl stop firewalld
[root@master ~]# systemctl disable firewalld
[root@master ~]# setenforce 0
[root@master ~]# vim /etc/selinux/config
SELINUX=disabled
k8s从1.8版本开始kubelet要求关闭系统swap,不然无法启动；可以通过kubelet的启动参数–fail-swap-on=false更改这个限制。
[root@master ~]# swapoff -a # 关闭系统swap
编辑/etc/fstab,禁用swap分区的挂载
[root@master ~]# free -m # 可用free -m查看swap是否关闭
total used free shared buff/cache available
Mem: 992 564 72 1 355 255
Swap: 0 0 0
swappiness参数调整
[root@master ~]# vim /etc/sysctl.d/k8s.conf # 添加如下内容
vm.swappiness=0
[root@master ~]# sysctl -p /etc/sysctl.d/k8s.conf # 使之生效
安装docker
安装相关依赖包
[root@master ~]# yum install -y yum-utils device-mapper-persistent-data lvm2
添加docker stable版本的repo
root@master ~]# yum-config-manager --add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
如果想安装最新版docker的话，就采用这个
[root@master ~]# curl -fsSL "https://get.docker.com/" | sh
安装docker包
[root@master ~]# yum install -y docker-ce docker-ce-selinux
docker从1.13版本(docker-ce没有这个问题)开始禁用iptables filter表中的FOWARD链，这样会导致k8s集群跨Node的Pod无法通信
[root@master ~]# vim /usr/lib/systemd/system/docker.service # 编辑docker systemd文件，在ExecStart上面加入ExecStartPost命令
ExecStartPost=/usr/sbin/iptables -P FORWARD ACCEPT
ExecStart=......
启动docker服务
[root@master ~]# systemctl daemon-reload
[root@master ~]# systemctl enable docker
[root@master ~]# systemctl start docker
kubeadm工具部署
kubeadm介绍
kubeadm是一种工具，旨在提供kubeadm初始化和kubeadm联接作为创建Kubernetes集群的最佳实践"快速路径". kubeadm执行必要的操作，以使最小可行的群集启动并运行。从设计上讲，它只关心引导程序，而不关心配置机器。 同样，也不在安装各种功能强大的插件，例如Kubernetes仪表板，监视解决方案和特定于云的插件。
安装kubeadm、kubelet、kubectl
添加kubernetes repo源
[root@master ~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
测试k8s repo地址是否可用，不然可能需要想办法了。
[root@master ~]# curl https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
安装kubelet kubeadm kubectl包
[root@master ~]# yum install -y kubelet kubeadm kubectl
kubelet服务开机自启动
[root@master ~]# systemctl enable kubelet.service
注：kubelet启动时带的cgroup-driver参数(默认cgroup)和docker使用的cgroup-driver参数(默认cgroup)有所不同，会导致kubelet服务启动失败，因为kubeadm包装出来的配置文件中cgroup-driver为systemd，所以这里选择修改docker cgroup-driver为systemd
[root@master ~]# rpm -ql kubeadm-1.8.4-0.x86_64
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
/usr/bin/kubeadm
[root@master ~]# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
[Service]
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
[root@master ~]# vim /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"]
}
重启docker服务
[root@master ~]# systemctl restart docker
[root@master ~]# systemctl status docker
kubeadm创建集群
这里选用flannel网络插件，还需要改些bridge参数
[root@master ~]# vim /etc/sysctl.d/k8s.conf # 添加如下内容
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
[root@master ~]# sysctl -p /etc/sysctl.d/k8s.conf # 使之生效
初始化集群
[root@master ~]# kubeadm init --kubernetes-version=v1.8.4 \
--pod-network-cidr=10.244.0.0/16 \ --apiserver-advertise-address=172.19.0.14
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.8.4
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks
[preflight] WARNING: docker version is greater than the most recently validated version. Docker version: 17.11.0-ce. Max validated version: 17.03
[preflight] Starting the kubelet service
[kubeadm] WARNING: starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0)
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.19.0.14]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] This often takes around a minute; or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 30.001692 seconds
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node master as master by adding a label and a taint
[markmaster] Master master tainted and labelled with key/value: node-role.kubernetes.io/master=""
[bootstraptoken] Using token: 5b0855.b90b56759e07723e
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run (as a regular user):
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
http://kubernetes.io/docs/admin/addons/
You can now join any number of machines by running the following on each node
as root:
# 这个跟后续新增节点有关，务必记住
kubeadm join --token 5b0855.b90b56759e07723e 172.19.0.14:6443 --discovery-token-ca-cert-hash sha256:2cad4211f45f0d454f9a3ac7f59e997248c57497421a0df719a08fca9e385cc1
安装pod网络插件
[root@master ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml
如果节点有多个网卡的话，需要指定网卡名称，详情见这里：https://github.com/kubernetes/kubernetes/issues/39701
不同网络插件略有不同，其它网络插件配置详情见这里：
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network
按照初始化集群后的提示，进行如下操作
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
查看集群状态
[root@master ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}
kubeadm安装出来的k8s服务都是以容器形式运行，查看所有pod状态
[root@master ~]# kubectl get pod --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
kube-system etcd-master 1/1 Running 0 20h 172.19.0.14 master
kube-system kube-apiserver-master 1/1 Running 0 20h 172.19.0.14 master
kube-system kube-controller-manager-master 1/1 Running 0 20h 172.19.0.14 master
kube-system kube-dns-545bc4bfd4-nt7k6 3/3 Running 0 20h 10.244.0.2 master
kube-system kube-flannel-ds-p9n2n 1/1 Running 0 20h 172.19.0.14 master
kube-system kube-proxy-5j8h7 1/1 Running 0 20h 172.19.0.14 master
kube-system kube-scheduler-master 1/1 Running 0 20h 172.19.0.14 master
让master节点也参与调度
[root@master ~]# kubectl taint nodes master node-role.kubernetes.io/master-
node "master" untainted
集群重置
kubeadm reset
ifconfig cni0 down
ip link delete cni0
ifconfig flannel.1 down
ip link delete flannel.1
rm -rf /var/lib/cni/
Kind工具部署
Kind介绍
kinded是一种使用Docker容器"节点"运行本地Kubernetes集群的工具。 该类型主要用于测试Kubernetes本身，但可用于本地开发或CI。 默认情况下，Kind会先下载kindest/node镜像，该镜像包含kubernetes的主要组件， 当镜像节点准备好时，Kind使用kubeadm进行集群的创建，内部使用containerd跑组件容器。
部署kind
wget -O /usr/local/bin/kind https://github.com/kubernetes-sigs/kind/releases/download/v0.9.0/kind-linux-amd64 && chmod +x /usr/local/bin/kind
kind创建单集群
root@xiabingyao-LC0:~# kind create cluster --name test
Creating cluster "test" ...
✓ Ensuring node image (kindest/node:v1.19.1) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-test"
You can now use your cluster with:
kubectl cluster-info --context kind-test
Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂
默认集群kubeconfig配置自动加入到～/.kube/config
查看kind集群
查看kind集群
# kind get clusters
test
也可以通过kind get kubeconfig生成kubeconfig配置
# kind get kubeconfig --name=test
下面就跟操作k8s集群一样的操作了
# kubectl cluster-info
Kubernetes master is running at https://127.0.0.1:35747
KubeDNS is running at https://127.0.0.1:35747/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-f9fd979d6-444gm 1/1 Running 0 6m10s
kube-system coredns-f9fd979d6-m4hzb 1/1 Running 0 6m10s
kube-system etcd-test-control-plane 1/1 Running 0 6m21s
kube-system kindnet-ng5ht 1/1 Running 0 6m10s
kube-system kube-apiserver-test-control-plane 1/1 Running 0 6m21s
kube-system kube-controller-manager-test-control-plane 1/1 Running 0 6m21s
kube-system kube-proxy-bfln2 1/1 Running 0 6m10s
kube-system kube-scheduler-test-control-plane 1/1 Running 0 6m21s
local-path-storage local-path-provisioner-78776bfc44-pglpg 1/1 Running 0 6m10s
# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
test-control-plane Ready master 6m37s v1.19.1 172.18.0.2 <none> Ubuntu Groovy Gorilla (development branch) 5.4.0-52-generic containerd://1.4.0
指定配置创建集群
指定service cidr创建集群
cat > config.yaml << EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
serviceSubnet: "11.96.0.0/12"
EOF
# # kind create cluster --name test2 --config config.yaml
Creating cluster "test2" ...
✓ Ensuring node image (kindest/node:v1.19.1) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
⠊⠁ Starting control-plane 🕹️
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-test2"
You can now use your cluster with:
kubectl cluster-info --context kind-test2
Not sure what to do next? 😅 Check out https://kind.sigs.k8s.io/docs/user/quick-start/
嵌入kubeadm配置创建集群
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
metadata:
name: config
networking:
dnsDomain: kind.iceyao.com
apiServer:
extraArgs:
insecure-bind-address: 0.0.0.0
insecure-port: "6444"
certSANs:
- kind.iceyao.com
- 172.17.0.2
- 172.17.0.3
- 172.17.0.4
- 127.0.0.1
- localhost
- kubernetes
- kubernetes.default
- kubernetes.default.svc
- kubernetes.default.svc.kind.iceyao.com
- 10.96.0.1
启用featureGate
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
kubeadmConfigPatches:
- |
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
metadata:
name: config
apiServer:
extraArgs:
"feature-gates": "ServiceTopology=true,EndpointSlice=true"
scheduler:
extraArgs:
"feature-gates": "ServiceTopology=true,EndpointSlice=true"
controllerManager:
extraArgs:
"feature-gates": "ServiceTopology=true,EndpointSlice=true"
- |
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
metadata:
name: config
nodeRegistration:
kubeletExtraArgs:
"feature-gates": "ServiceTopology=true,EndpointSlice=true"
nodes:
- role: control-plane
kind创建HA集群
cat > config.yaml << EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: control-plane
- role: control-plane
- role: worker
- role: worker
- role: worker
EOF
# kind create cluster --name test2 --config config.yaml
Creating cluster "test2" ...
✓ Ensuring node image (kindest/node:v1.19.1) 🖼
⠈⠱ Preparing nodes 📦 📦 📦 📦 📦 📦
✓ Preparing nodes 📦 📦 📦 📦 📦 📦
✓ Configuring the external load balancer ⚖️
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
⠈⡱ Joining more control-plane nodes 🎮
✓ Joining more control-plane nodes 🎮
✓ Joining worker nodes 🚜
Set kubectl context to "kind-test2"
You can now use your cluster with:
kubectl cluster-info --context kind-test2
Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂
# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
test2-control-plane Ready master 2m42s v1.19.1 172.18.0.3 <none> Ubuntu Groovy Gorilla (development branch) 5.4.0-52-generic containerd://1.4.0
test2-control-plane2 Ready master 2m15s v1.19.1 172.18.0.5 <none> Ubuntu Groovy Gorilla (development branch) 5.4.0-52-generic containerd://1.4.0
test2-control-plane3 Ready master 83s v1.19.1 172.18.0.6 <none> Ubuntu Groovy Gorilla (development branch) 5.4.0-52-generic containerd://1.4.0
test2-worker Ready <none> 72s v1.19.1 172.18.0.8 <none> Ubuntu Groovy Gorilla (development branch) 5.4.0-52-generic containerd://1.4.0
test2-worker2 Ready <none> 71s v1.19.1 172.18.0.4 <none> Ubuntu Groovy Gorilla (development branch) 5.4.0-52-generic containerd://1.4.0
test2-worker3 Ready <none> 72s v1.19.1 172.18.0.7 <none> Ubuntu Groovy Gorilla (development branch) 5.4.0-52-generic containerd://1.4.0
原理
一个集群节点就会起一个kindest/node镜像的docker容器
# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d92cd30a54bf kindest/node:v1.19.1 "/usr/local/bin/entr…" 2 hours ago Up 2 hours 127.0.0.1:35747->6443/tcp test-control-plane
有kind专用的docker网络
# docker network ls
NETWORK ID NAME DRIVER SCOPE
e973e7d48f68 bridge bridge local
702681764ddb host host local
f8bb928a9027 kind bridge local
fd0209865631 none null local
进入docker容器内部，然后使用crictl ps可以看到containerd起的容器
root@xiabingyao-LC0:/tmp# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d92cd30a54bf kindest/node:v1.19.1 "/usr/local/bin/entr…" 2 hours ago Up 2 hours 127.0.0.1:35747->6443/tcp test-control-plane
root@xiabingyao-LC0:/tmp# docker exec -it test-control-plane bash
root@test-control-plane:/# crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
62ce2437ef606 bfe3a36ebd252 2 hours ago Running coredns 0 9ccdb7c1e81ff
b6066c7f49614 e422121c9c5f9 2 hours ago Running local-path-provisioner 0 7798f456e8d89
ab794130d4c40 bfe3a36ebd252 2 hours ago Running coredns 0 10b2c8fb74299
a4c7af3163057 b77790820d015 2 hours ago Running kindnet-cni 0 a2cdeba8acb32
122569f5c27b6 47e289e332426 2 hours ago Running kube-proxy 0 b5bb9550d32d6
3ecd9df8fb4f5 0369cf4303ffd 2 hours ago Running etcd 0 698193e891b67
4fb2ba0b97bfa 7dafbafe72c90 2 hours ago Running kube-controller-manager 0 9af310c3f18be
8868b5c0e0aef 4d648fc900179 2 hours ago Running kube-scheduler 0 a9d00df6801d0
e787f5ba43275 8cba89a89aaa8 2 hours ago Running kube-apiserver 0 72758bf4bc244
参考链接
- 使用kubeadm安装Kubernetes 1.8
- https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
- kind官网
- crtctl使用
「真诚赞赏，手留余香」
爱折腾的工程师真诚赞赏，手留余香
使用微信扫描二维码完成支付
