快速部署K8S单主集群

简介

  • KubernetesGoogle20146月开源的一个容器集群部署与管理系统,简称为K8S
  • KubernetesGoogle内部的Borg容器集群部署和管理系统衍生而来,使用Go语言开发。
  • Kubernetes是一个可以移植、可扩展的开源平台,使用声明式的配置并依据配置信息自动地执行容器化应用程序的管理。
  • Kubernetes主要用于自动化部署、扩展和管理容器应用,提供资源的调度、部署管理、服务发现、扩容缩容、运维监控等一整套功能。

介绍集群

文档

  • GitHub《传送门》
  • 版本策略:《传送门》
  • 单主集群:《传送门》
  • 升级集群:《传送门》

版本

  • OS: CentOS 7.7
  • Etcd3.4.3
  • Flannel0.12.0
  • CoreDNS1.6.5
  • Docker19.03.8
  • Kubernetes1.17.4

要求

  • 每个节点的操作系统版本为CentOS 7.7
  • 每个节点至少拥有2CPU2GRAM
  • 每个节点的主机名都是唯一的,并且不包含下划线、小数点、大写字母。
  • 每个节点都有固定的IP地址,并且节点之间能相互通信。
  • 每个节点都关闭了swap分区,此要求从K8S v1.8开始实施。
  • 每个Master节点需安装dockerkubeadmkubectlkubelet
  • 每个Node节点需安装dockerkubeadmkubelet

规划

IP 角色 组件
172.18.20.10 Master etcd、flannel、kube-apiserver、kube-controller-manager、kube-scheduler
172.18.20.11 Node flannel、kube-proxy、kubelet
172.18.20.12 Node flannel、kube-proxy、kubelet

部署集群

SetUP

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# 关闭SELinux
setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

# 关闭防火墙
systemctl stop firewalld.service
systemctl disable firewalld.service

# 禁用交换分区
swapoff -a && sysctl -w vm.swappiness=0
sed -i '/swap/d' /etc/fstab
echo "vm.swappiness = 0" > /etc/sysctl.d/99-kubernetes.conf

# 内部解析
echo '127.0.0.1 localhost' > /etc/hosts
echo '172.18.20.10 master' >> /etc/hosts
echo '172.18.20.11 node-1' >> /etc/hosts
echo '172.18.20.12 node-2' >> /etc/hosts

# 允许 iptables 查看 bridge 的流量
modprobe br_netfilter
echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.d/99-kubernetes.conf
echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.d/99-kubernetes.conf
sysctl -qp /etc/sysctl.d/99-kubernetes.conf

# 提示符
echo 'export PS1="\u@\[\e[1;93m\]\h\[\e[m\]:\w\\$\[\e[m\] "' >> /root/.bashrc

# 环境变量
echo 'export PATH="${PATH}:/etc/kubernetes/bin"' >> /root/.bashrc

# 免密通信
ssh-keygen -t rsa -P "" -C "K8S" -f '/root/.ssh/id_rsa' >/dev/null 2>&1
echo "" >> /root/.ssh/authorized_keys
cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys

# 打开文件数量限制
echo '* soft nofile 204800' >> /etc/security/limits.conf
echo '* hard nofile 204800' >> /etc/security/limits.conf
# 打开进程数量限制
echo '* soft nproc 204800' >> /etc/security/limits.conf
echo '* hard nproc 204800' >> /etc/security/limits.conf

Docker

1
2
mkdir -p /etc/docker
vim /etc/docker/daemon.json
1
2
3
4
5
6
7
8
9
10
11
12
{
"log-opts": {
"max-size": "100m"
},
"log-driver": "json-file",
"storage-opts": [
"overlay2.override_kernel_check=true"
],
"storage-driver": "overlay2",
"exec-opts": ["native.cgroupdriver=systemd"],
"registry-mirrors": ["https://9cs90h5l.mirror.aliyuncs.com"]
}
1
2
systemctl enable docker.service
systemctl restart docker.service

Kubernetes

Package

1
vim /etc/yum.repos.d/kubernetes.repo
1
2
3
4
5
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# 更新软件源
yum makecache

# 获取可用版本
yum list kubelet --showduplicates | grep '1.17'

# 安装指定版本
yum install -y ebtables ethtool kubeadm-1.17.4 kubelet-1.17.4 kubectl-1.17.4
systemctl enable kubelet.service

# 锁定版本
yum install -y yum-plugin-versionlock
yum versionlock kubeadm kubelet kubectl

# 解除锁定
yum versionlock delete kubeadm kubelet kubectl

# 自动补全
kubectl completion bash > /etc/bash_completion.d/kubectl

Master

  • 初始化集群:
1
2
3
4
5
6
kubeadm init \
--service-cidr 10.20.0.0/16 \
--kubernetes-version v1.17.5 \
--pod-network-cidr 10.10.0.0/16 \
--apiserver-advertise-address 172.18.20.10 \
--image-repository registry.aliyuncs.com/google_containers
  • 拷贝配置文件:
1
2
3
mkdir -p /root/.kube
cp -a /etc/kubernetes/admin.conf /root/.kube/config
chown root:root /root/.kube/config
  • 获取节点信息,Master节点状态为NotReady
1
kubectl get nodes
  • 因为没有为集群部署容器网络接口CNI,所以节点状态是NotReady,基于Pod的网络插件:CalicoFlannel等。

Flannel

  • 下载配置文件:
1
2
# GitHub: https://github.com/coreos/flannel/releases/
wget https://raw.githubusercontent.com/coreos/flannel/v0.12.0/Documentation/kube-flannel.yml
  • 修改配置文件:
1
vim kube-flannel.yml
1
2
3
4
5
6
7
8
9
10
# 将 net-conf.json 中的 Network 修改为 kubeadm init 时的 pod-network-cidr

data:
net-conf.json: |
{
"Network": "10.10.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
  • 部署Flannel
1
kubectl apply -f kube-flannel.yml
  • 获取节点信息,Master节点状态为Ready
1
kubectl get nodes
  • 为解决Cluster IP访问速度慢的问题,需要在所有节点添加Service网段的路由:
1
ip route add 10.10.0.0/16 dev flannel.1

Node

  • 由于令牌(Token)有效期为24小时,若过了有效期,还需添加Node节点,则需要重新创建令牌。
  • 将自身加入到集群中:
1
2
3
kubeadm join 172.18.20.10:6443 \
--token 5dewcr.7zf4dkfk2rwnaj0o \
--discovery-token-ca-cert-hash sha256:4d747694c765dd085b1eb87a4f3f2d45ac95729a99b40bc034879d0ec8a374f1

Master

  • 给节点设置Lable
1
2
kubectl label nodes node-1 node-role.kubernetes.io/node=
kubectl label nodes node-2 node-role.kubernetes.io/node=
  • 获取节点信息,总共有3个节点,节点状态都为Ready
1
kubectl get nodes

Dashboard

  • Master节点上为集群部署Dashboard

  • 下载配置文件:

1
2
3
# https://github.com/kubernetes/dashboard
# https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc7/aio/deploy/recommended.yaml
  • 修改配置文件:
1
vim recommended.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
# 新增
type: NodePort
ports:
- port: 443
targetPort: 8443
# 新增
nodePort: 31443
selector:
k8s-app: kubernetes-dashboard

# 避免Chrome浏览器无法访问: 移除以下内容,手动创建证书
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-certs
namespace: kube-system
type: Opaque
  • 部署Dashboard
1
kubectl apply -f recommended.yaml
  • 创建证书并导入:
1
2
3
4
5
6
openssl genrsa -out dashboard.key 2048
openssl req -days 36000 -new -out dashboard.csr -key dashboard.key -subj '/CN=192.168.1.1'
openssl x509 -req -in dashboard.csr -signkey dashboard.key -out dashboard.crt
kubectl delete secret kubernetes-dashboard-certs -n kubernetes-dashboard
kubectl create secret generic kubernetes-dashboard-certs --from-file=dashboard.key --from-file=dashboard.crt -n kubernetes-dashboard
kubectl get secret kubernetes-dashboard-certs -n kubernetes-dashboard
  • 查看服务状态:
1
kubectl get pods -n kubernetes-dashboard
  • 创建Server Account并绑定默认的cluster-admin角色:
1
2
3
kubectl create serviceaccount dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
kubectl describe secrets -n kube-system $(kubectl get secrets -n kube-system | awk '/dashboard-admin/{print $1}')
  • 服务状态都为Running,通过https://<Node IP>:31443进行访问,输入创建的Token进行登录。

测试集群

  • 部署一个NginxDeployment
1
2
# https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
kubectl create deployment nginx --image=nginx:stable-alpine
  • 获取服务状态:
1
2
3
kubectl get all
kubectl get services,deployment,replicaset,pods
kubectl get svc,deploy,rs,po
  • NodePort方式对外提供服务:
1
2
# https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/
kubectl expose deployment nginx --port=80 --type=NodePort --target-port=80 --name=nginx
  • 获取访问端口及所在的Node
1
kubectl get svc,pods -o wide
  • 观察到应用落到了node-1上,外部访问端口是30651
1
2
3
4
5
6
# 访问 Cluster IP
curl 10.10.242.147:80

# 访问 NodePort
curl 172.18.20.11:30651
curl 172.18.20.12:30651
  • 查看指定容器的日志:
1
kubectl logs nginx-745b4df97d-wkkqk
  • 故障排查,分析创建流程:
1
2
3
kubectl describe deployment nginx
kubectl describe replicasets nginx
kubectl describe pods nginx
  • 更新镜像:
1
2
3
4
5
# 方式一
kubectl edit deployment/nginx -o yaml

# 方式二
kubectl set image deployment/nginx nginx=nginx:1.16 --record
  • 回滚操作:
1
2
3
4
5
6
7
8
# 查看操作的历史
kubectl rollout history deployment/nginx

# 查看当前的状态
kubectl rollout status deployment/nginx

# 回滚之前的操作
kubectl rollout undo deployment/nginx
  • 集群扩容:
1
kubectl scale --replicas=3 deployment/nginx
  • 缩容服务:
1
kubectl scale --replicas=2 deployment/nginx
  • 删除服务:
1
2
kubectl delete deployment/nginx
kubectl delete svc/nginx

维护集群

常用命令

  • 获取令牌(Token)列表:
1
kubeadm token list
  • 创建令牌(Token):
1
2
kubeadm token create
kubeadm token create --print-join-command
  • 获取指定namespacespods
1
kubectl get pods -n kube-system
  • 获取所有的Service
1
kubectl get service --all-namespaces
  • 获取所有的Deployment
1
kubectl get deployment --all-namespaces

卸载集群

Dashboard

  • 卸载Dsahboard
1
kubectl delete -f recommended.yaml

Node

  • Master节点上删除Node,并确保此Node为空, 然后再将其关闭。
1
2
kubectl drain <Node Name> --delete-local-data --force --ignore-daemonsets
kubectl delete node <Node Name>
  • 在对应的Node节点上重置节点:
1
kubeadm reset

Master

  • Master节点上删除自身,并确保没有其他Node节点。
1
2
kubectl drain <Master Name> --delete-local-data --force --ignore-daemonsets
kubectl delete node <Master Name>
  • Master节点上重置节点:
1
kubeadm reset

请作者喝瓶肥宅快乐水