Kubernetes 1.20 快速安装方法
必看！！！！！！！！！！！！！！！！
必需注意集群网络规划：
主机规划：
| 10.7.0.61 | | etcd-01 | etcd
| 10.7.0.62 | | etcd-02 | etcd
| 10.7.0.63 | | etcd-03 | etcd
| 10.7.0.65 | | master-01 | VIP: 10.7.0.64 主(keepalived) api、controller、scheduler
| 10.7.0.66 | | master-02 | VIP: 10.7.0.64 备(keepalived) api、controller、scheduler
| 10.7.0.67 | | master-03 | api、controller、scheduler
| 10.7.0.68 | | 预留master |
| 10.7.0.69 | | 预留master |
| 10.7.0.70 | | 预留master |
| 10.7.0.71 | | worker | kubelet、flannel、kube-proxy
| 10.7.0.72 | |
cluster-cidr: 172.17.0.0/16 pod的网段，需要和etcd设置flannel获取的网段一致！
kubernetes cluster ip: 10.43.0.1 kubernetes svc的ip地址
service-cluster-ip-range: 10.43.0.0/16 svc的网段
clusterDNS: 10.43.0.10 DNS的SVC ip地址
以下集群内部证书为100年有效期，ca证书为5年
如果添加了新的master节点，不需要重新签发证书，因为从一开始的crs文件就提前预留了几个ip，签发的证书已包含了。
只需要重新生成各个组件kubeconfig文件，即是与api-server通信的集群参数，还有客户端认证参数、上下文参数、上下文绑定、角色绑定。
然后把相应的*.pem证书分发到相应节点和目录上，创建组件的配置文件，启动即可
下面etcd、api-server、controller-manager、scheduler组件均提前添加了3个master节点ip地址，下面操作已说明
必看！！！！！！！！！！！！！！！！！！！！！
master节点无需安装flannel插件
二进制K8s集群，服务启动顺序
启动Master节点–启动顺序
service keepalived start
service etcd start
service kube-scheduler start
service kube-controller-manager start
service kube-apiserver restart
kubectl get cs
启动worker节点
service flanneld start
service docker start
service kubelet start
service kube-proxy start
停止顺序：
停止worker节点
service kubelet stop
service kube-proxy stop
service docker stop
service flanneld stop
11.5.2 停止Master 节点
service kube-controller-manager stop
service kube-scheduler stop
service etcd stop
service keepalived stop
4系统初始化
4.1 初始化工具安装
#所有节点
[root@master-1 ~]# apt install net-tools vim wget lrzsz git conntrack-tools -y
4.2 关闭防火墙与Selinux
#所有节点
[root@master-1 ~]# systemctl stop firewalld
[root@master-1 ~]# systemctl disable firewalld
[root@master-1 ~]# sed -i “s/SELINUX=enforcing/SELINUX=disabled/g” /etc/selinux/config
[root@master-1 ~]# reboot
4.3设置时区
#所有节点
[root@master-1 ~]# timedatectl set-timezone Asia/Shanghai
#设置主机名
[root@master-1 ~]# hostnamectl set-hostname master-1
4.4关闭交换分区
#所有节点
[root@master-1 ~]# swapoff -a
[root@master-1 ~]# sed -i ‘/ swap / s/^(.*)$/#\1/g’ /etc/fstab
4.5设置系统时间同步
#所有节点
[root@master-1 ~]# apt install -y chrony
[root@master-1 ~]# systemctl start chronyd
[root@master-1 ~]# systemctl enable chronyd
[root@master-1 ~]# chronyc sources
4.6 设置主机名
#所有节点
[root@master-1 ~]# cat >> /etc/hosts <<EOF
10.17.1.47 crm-etcd-k8s-b1
10.17.1.48 crm-etcd-k8s-b2
10.17.1.49 crm-etcd-k8s-b3
10.17.1.51 crm-ma01-k8s-b1
10.17.1.52 crm-ma01-k8s-b2
10.17.1.53 crm-ma01-k8s-b3
10.17.1.57 crm-wrk01-k8s-b1
10.17.1.58 crm-wrk01-k8s-b2
10.17.1.59 crm-wrk01-k8s-b3
#预留master
10.17.1.54 crm-ma01-k8s-b4
10.17.1.55 crm-ma01-k8s-b5
10.17.1.56 crm-ma01-k8s-b6
EOF
4.7 设置免密码登录
#从任意Master节点分发配置到其他所有的节点(包括其他的Master与Node)
#本例中从master-01分发
[root@master-1 ~]# apt install -y expect
[root@master-1 ~]# ssh-keygen -t rsa -P “” -f /root/.ssh/id_rsa
#密码更换，自己的root密码
[root@master-1 ~]# export mypass=root_passwd
[root@master-1 ~]# name=(xds-ma01-etcd-p1 xds-ma01-etcd-p2 xds-ma01-etcd-p3 xds-ma01-k8s-p1 xds-ma01-k8s-p2 xds-ma01-k8s-p3 xds-wrk01-k8s-p1 xds-wrk01-k8s-p2 xds-wrk01-k8s-p3)
[root@master-1 ~]# for i in i
expect {
“yes/no” {send “yes\r”; exp_continue}
“password” {send “mypass\r\"; exp_continue}
\"*Password*\" {send \"mypass\r”;}
}"
done
#连接测试
[root@master-1 ~]# ssh master-02
4.8 优化内核参数
#所有节点
[root@master-1 ~]# cat >>/etc/sysctl.conf<<EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
vm.swappiness=0
fs.file-max=52706963
fs.nr_open=52706963
EOF
#应用内核配置
[root@master-2 ~]# sysctl -p
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory
net.ipv4.ip_forward = 1
vm.swappiness = 0
fs.file-max = 52706963
fs.nr_open = 52706963
#worker 节点执行
[root@demo ~]# cat <<EOF | tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
[root@demo ~]# modprobe overlay
[root@demo ~]# modprobe br_netfilter
[root@master-2 ~]# sysctl -p
4.9 高可用节点安装Keepalived
#10.7.0.13
[root@master-1 ~]# apt install -y keepalived
#注意修改网卡地址与SLAVE节点的优先级
[root@master-1 ~]# cat >/etc/keepalived/keepalived.conf <<EOL
global_defs {
router_id KUB_LVS
}
vrrp_script check_api {
script “/etc/keepalived/check_api.sh”
interval 5
}
vrrp_instance VI_1 {
state MASTER
interface ens192
virtual_router_id 61
priority 100
advert_int 1
nopreempt
authentication {
auth_type PASS
auth_pass 111111
}
unicast_src_ip 10.17.1.51
unicast_peer {
10.17.1.52
}
virtual_ipaddress {
10.17.1.50/22 dev ens192
}
track_script {
check_api
}
}
EOL
chmod 644 /etc/keepalived/keepalived.conf
#主的脚本
cat > /etc/keepalived/check_api.sh <<‘EOF’
#!/bin/bash
#如果apiserver进程挂了，则重启apiserver。重启失败则停止keepalived
API_STATUS=$(ps -ef|grep [k]ube-apiserver|wc -l)
if [ ${API_STATUS} == 0 ]
then
systemctl restart kube-apiserver
if [ $? == 1 ]
then
systemctl stop keepalived
fi
fi
EOF
chmod +x /etc/keepalived/check_api.sh
systemctl enable keepalived && systemctl restart keepalived
service keepalived status
#SLAVE
#修改state为slave, priority 为 90
#10.7.0.14，备节点执行
[root@master-02 ~]# apt install -y keepalived
#注意修改网卡地址与SLAVE节点的优先级
[root@master-02 ~]# cat >/etc/keepalived/keepalived.conf <<EOL
global_defs {
router_id KUB_LVS_SLAVE
}
vrrp_script check_vip {
script “/etc/keepalived/check_vip.sh”
interval 5
}
vrrp_instance VI_1 {
state SLAVE
interface ens192
virtual_router_id 61
priority 90
advert_int 1
nopreempt
authentication {
auth_type PASS
auth_pass 111111
}
unicast_src_ip 10.17.1.52
unicast_peer {
10.17.1.51
}
virtual_ipaddress {
10.17.1.50/22 dev ens192
}
track_script {
check_vip
}
}
EOL
chmod 644 /etc/keepalived/keepalived.conf
#从的脚本
cat > /etc/keepalived/check_vip.sh <<‘EOF’
#!/bin/bash
#50是VIP，51是master
#master自身有VIP，并且SLAVE自身也有VIP，则停止自身的keepalived
MASTER_VIP=(ip a|grep 10.17.1.50|wc -l)
if [ ${MASTER_VIP} == 1 -a ${MY_VIP} == 1 ]
then
systemctl stop keepalived
fi
EOF
chmod +x /etc/keepalived/check_vip.sh
4.10.启动keepalived
[root@master-1 ~]# systemctl enable keepalived && systemctl restart keepalived
[root@master-1 ~]# service keepalived status
4.11配置证书
4.12 下载自签名证书生成工具
#在分发机器Master-01上操作
[root@master-1 ~]# mkdir /soft && cd /soft
[root@master-1 ~]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
[root@master-1 ~]# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
[root@master-1 ~]# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
[root@master-1 ~]# chmod +x cf*
[root@master-1 ~]# mv cfssl /usr/local/bin/cfssl
[root@master-1 ~]# mv cfssljson /usr/local/bin/cfssljson
[root@master-1 ~]# mv cfssl-certinfo /usr/local/bin/cfssl-certinfo
5.2 生成ETCD证书
#创建目录（Master-1）
[root@master-1 ~]# mkdir /root/etcd && cd /root/etcd
5.2.1 CA 证书配置（Master-1）
[root@master-1 ~]# cat << EOF | tee ca-config.json
{
“signing”: {
“default”: {
“expiry”: “876000h”
},
“profiles”: {
“www”: {
“expiry”: “876000h”,
“usages”: [
“signing”,
“key encipherment”,
“server auth”,
“client auth”
]
}
}
}
}
EOF
5.2.2 创建CA证书请求文件（Master-1）
[root@master-1 ~]# cat << EOF | tee ca-csr.json
{
“CN”: “etcd CA”,
“key”: {
“algo”: “rsa”,
“size”: 2048
},
“names”: [
{
“C”: “CN”,
“L”: “Beijing”,
“ST”: “Beijing”
}
]
}
EOF
5.2.3 创建ETCD证书请求文件
#可以把所有的master IP 加入到csr文件中（Master-1）
#可以提前预多几个master IP填进去，方便后期扩容master
[root@master-1 ~]# cat << EOF | tee server-csr.json
{
“CN”: “etcd”,
“hosts”: [
“crm-ma01-k8s-b1”,
“crm-ma01-k8s-b2”,
“crm-ma01-k8s-b3”,
“crm-ma01-k8s-b4”,
“crm-ma01-k8s-b5”,
“crm-ma01-k8s-b6”,
“crm-etcd-k8s-b1”,
“crm-etcd-k8s-b2”,
“crm-etcd-k8s-b3”,
“10.17.1.51”,
“10.17.1.52”,
“10.17.1.53”,
“10.17.1.54”,
“10.17.1.55”,
“10.17.1.56”,
“10.17.1.47”,
“10.17.1.48”,
“10.17.1.49”,
“10.17.1.50”
],
“key”: {
“algo”: “rsa”,
“size”: 2048
},
“names”: [
{
“C”: “CN”,
“L”: “Beijing”,
“ST”: “Beijing”
}
]
}
EOF
5.2.4 生成 ETCD CA 证书和ETCD公私钥（Master-1）
[root@master-1 ~]# cd /root/etcd/
#生成ca证书（Master-1）
[root@master-1 ~]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca –
[root@master-1 etcd]# ll
total 24
-rw-r–r-- 1 root root 287 Mar 4 10:16 ca-config.json
-rw-r–r-- 1 root root 956 Mar 4 10:18 ca.csr
-rw-r–r-- 1 root root 209 Mar 4 10:17 ca-csr.json
-rw------- 1 root root 1679 Mar 4 10:18 ca-key.pem
-rw-r–r-- 1 root root 1265 Mar 4 10:18 ca.pem
-rw-r–r-- 1 root root 338 Mar 4 10:18 server-csr.json
#生成etcd证书（Master-1）
[root@master-1 etcd]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
[root@master-1 etcd]# ll
total 36
-rw-r–r-- 1 root root 287 Mar 4 10:16 ca-config.json
-rw-r–r-- 1 root root 956 Mar 4 10:18 ca.csr
-rw-r–r-- 1 root root 209 Mar 4 10:17 ca-csr.json
-rw------- 1 root root 1679 Mar 4 10:18 ca-key.pem
-rw-r–r-- 1 root root 1265 Mar 4 10:18 ca.pem
-rw-r–r-- 1 root root 1054 Mar 4 10:19 server.csr
-rw-r–r-- 1 root root 338 Mar 4 10:18 server-csr.json
-rw------- 1 root root 1679 Mar 4 10:19 server-key.pem
-rw-r–r-- 1 root root 1379 Mar 4 10:19 server.pem
#10 11 12这三个节点，部署etcd
下载etcd软件包
[root@etcd-01 soft]# cd /soft
[root@etcd-01 soft]# wget https://github.com/etcd-io/etcd/releases/download/v3.4.13/etcd-v3.4.13-linux-amd64.tar.gz
[root@etcd-01 soft]# tar -xvf etcd-v3.4.13-linux-amd64.tar.gz
[root@etcd-01 soft]# cp -p etcd-v3.4.13-linux-amd64/etcd* /usr/local/bin/
[root@etcd-01 soft]# for i in crm-etcd-k8s-b1 crm-etcd-k8s-b2 crm-etcd-k8s-b3;do scp /usr/local/bin/etc* $i:/usr/local/bin/;done
6.1 编辑etcd配置文件（所有master）
#注意修改每个节点的ETCD_NAME
#注意修改每个节点的监听地址
[root@etcd-01 ~]# mkdir -p /etc/etcd/cfg/
[root@etcd-01 ~]# cat >/etc/etcd/cfg/etcd.conf<<EOFL
#[Member]
ETCD_NAME=“crm-etcd-k8s-b1”
ETCD_DATA_DIR=“/var/lib/etcd/default.etcd”
ETCD_LISTEN_PEER_URLS=“https://10.17.1.47:2380”
ETCD_LISTEN_CLIENT_URLS=“https://10.17.1.47:2379,https://127.0.0.1:2379”
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS=“https://10.17.1.47:2380”
ETCD_ADVERTISE_CLIENT_URLS=“https://10.17.1.47:2379”
ETCD_INITIAL_CLUSTER=“crm-etcd-k8s-b1=https://10.17.1.47:2380,crm-etcd-k8s-b2=https://10.17.1.48:2380,crm-etcd-k8s-b3=https://10.17.1.49:2380”
ETCD_INITIAL_CLUSTER_TOKEN=“etcd-cluster”
ETCD_ENABLE_V2=“true”
EOFL
[root@etcd-02 ~]# mkdir -p /etc/etcd/cfg/
[root@etcd-02 ~]# cat >/etc/etcd/cfg/etcd.conf<<EOFL
#[Member]
ETCD_NAME=“crm-etcd-k8s-b2”
ETCD_DATA_DIR=“/var/lib/etcd/default.etcd”
ETCD_LISTEN_PEER_URLS=“https://10.17.1.48:2380”
ETCD_LISTEN_CLIENT_URLS=“https://10.17.1.48:2379,https://127.0.0.1:2379”
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS=“https://10.17.1.48:2380”
ETCD_ADVERTISE_CLIENT_URLS=“https://10.17.1.48:2379”
ETCD_INITIAL_CLUSTER=“crm-etcd-k8s-b1=https://10.17.1.47:2380,crm-etcd-k8s-b2=https://10.17.1.48:2380,crm-etcd-k8s-b3=https://10.17.1.49:2380”
ETCD_INITIAL_CLUSTER_TOKEN=“etcd-cluster”
ETCD_ENABLE_V2=“true”
EOFL
[root@etcd-03 ~]# mkdir -p /etc/etcd/cfg/
[root@etcd-03 ~]# cat >/etc/etcd/cfg/etcd.conf<<EOFL
#[Member]
ETCD_NAME=“crm-etcd-k8s-b3”
ETCD_DATA_DIR=“/var/lib/etcd/default.etcd”
ETCD_LISTEN_PEER_URLS=“https://10.17.1.49:2380”
ETCD_LISTEN_CLIENT_URLS=“https://10.17.1.49:2379,https://127.0.0.1:2379”
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS=“https://10.17.1.49:2380”
ETCD_ADVERTISE_CLIENT_URLS=“https://10.17.1.49:2379”
ETCD_INITIAL_CLUSTER=“crm-etcd-k8s-b1=https://10.17.1.47:2380,crm-etcd-k8s-b2=https://10.17.1.48:2380,crm-etcd-k8s-b3=https://10.17.1.49:2380”
ETCD_INITIAL_CLUSTER_TOKEN=“etcd-cluster”
ETCD_ENABLE_V2=“true”
EOFL
注：
#参数说明:
ETCD_NAME 节点名称, 如果有多个节点, 那么每个节点要修改为本节点的名称。
ETCD_DATA_DIR 数据目录
ETCD_LISTEN_PEER_URLS 集群通信监听地址
ETCD_LISTEN_CLIENT_URLS 客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEER_URLS 集群通告地址
ETCD_ADVERTISE_CLIENT_URLS 客户端通告地址
ETCD_INITIAL_CLUSTER 集群节点地址,如果多个节点那么逗号分隔
ETCD_INITIAL_CLUSTER=“master1=https://192.168.91.200:2380,master2=https://192.168.91.201:2380,master3=https://192.168.91.202:2380”
ETCD_INITIAL_CLUSTER_TOKEN 集群Token
ETCD_INITIAL_CLUSTER_STATE 加入集群的当前状态，new是新集群，existing表示加入已有集群
2.3.1 创建ETCD的系统启动服务(所有etcd节点)
报错:
ETCD3.4版本会自动读取环境变量的参数,所以EnvironmentFile文件中有的参数,不需要再次在ExecStart启动参数中添加,二选一,
如同时配置，会触发以下类似报错“etcd: conflicting environment variable “ETCD_NAME”
is shadowed by corresponding command-line flag (either unset environment variable or disable flag)”
[root@etcd-01 ~]# cat > /usr/lib/systemd/system/etcd.service<<EOFL
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/etc/etcd/cfg/etcd.conf
ExecStart=/usr/local/bin/etcd
–initial-cluster-state=new
–cert-file=/etc/etcd/ssl/server.pem
–key-file=/etc/etcd/ssl/server-key.pem
–peer-cert-file=/etc/etcd/ssl/server.pem
–peer-key-file=/etc/etcd/ssl/server-key.pem
–trusted-ca-file=/etc/etcd/ssl/ca.pem
–heartbeat-interval=100
–election-timeout=500
–quota-backend-bytes=8589934592
–peer-trusted-ca-file=/etc/etcd/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOFL
6.3 复制etcd证书到指定目录
[root@etcd-01 ~]# mkdir -p /etc/etcd/ssl/
[root@etcd-01 ~]# \cp /root/etcd/*pem /etc/etcd/ssl/ -rf
#复制etcd证书到每个节点,所有master、worker、etcd节点
[root@etcd-01 ~]# for i in crm-etcd-k8s-b1 crm-etcd-k8s-b2 crm-etcd-k8s-b3 crm-ma01-k8s-b2 crm-ma01-k8s-b3 crm-wrk01-k8s-b1 crm-wrk01-k8s-b2 crm-wrk01-k8s-b3;do ssh $i mkdir -p /etc/etcd/{cfg,ssl};done
[root@etcd-01 ~]# for i in crm-etcd-k8s-b1 crm-etcd-k8s-b2 crm-etcd-k8s-b3 crm-ma01-k8s-b2 crm-ma01-k8s-b3 crm-wrk01-k8s-b1 crm-wrk01-k8s-b2 crm-wrk01-k8s-b3;do scp /etc/etcd/ssl/* $i:/etc/etcd/ssl/;done
[root@etcd-01 ~]# for i in crm-etcd-k8s-b1 crm-etcd-k8s-b2 crm-etcd-k8s-b3 crm-ma01-k8s-b2 crm-ma01-k8s-b3 crm-wrk01-k8s-b1 crm-wrk01-k8s-b2 crm-wrk01-k8s-b3;do echo $i “------>”; ssh $i ls /etc/etcd/ssl;done
6.4 启动etcd (所有etcd节点)
[root@etcd-01 ~]# systemctl daemon-reload
[root@etcd-01 ~]# systemctl start etcd
[root@etcd-01 ~]# systemctl enable etcd
[root@etcd-01 ~]# service etcd status
2.3.5 #debug 模式启动测试
/usr/local/bin/etcd
–name=etcd-01
–data-dir=/var/lib/etcd/default.etcd
–listen-peer-urls=https://10.7.0.10:2380
–listen-client-urls=https://10.7.0.10:2379,http://10.7.0.10:2390
–advertise-client-urls=https://10.7.0.10:2379
–initial-advertise-peer-urls=https://10.7.0.10:2380
–initial-cluster=etcd-01=https://10.7.0.10:2380,etcd-02=https://10.7.0.11:2380,etcd-03=https://10.7.0.12:2380
–initial-cluster-token=etcd-cluster
–initial-cluster-state=new
–cert-file=/etc/etcd/ssl/server.pem
–key-file=/etc/etcd/ssl/server-key.pem
–peer-cert-file=/etc/etcd/ssl/server.pem
–peer-key-file=/etc/etcd/ssl/server-key.pem
–trusted-ca-file=/etc/etcd/ssl/ca.pem
–peer-trusted-ca-file=/etc/etcd/ssl/ca.pem
2.3.6 #集群状态检查
[root@master-2 ~]# etcdctl --cacert=/etc/etcd/ssl/ca.pem
–cert=/etc/etcd/ssl/server.pem
–key=/etc/etcd/ssl/server-key.pem
–endpoints=“https://10.17.1.47:2379,https://10.17.1.48:2379,https://10.17.1.49:2379”
endpoint health
#v3 方式
ETCDCTL_API=3 /usr/local/bin/etcdctl
–write-out=table --cacert=/etc/etcd/ssl/ca.pem
–cert=/etc/etcd/ssl/server.pem
–key=/etc/etcd/ssl/server-key.pem
–endpoints=“https://10.17.1.47:2379,https://10.17.1.48:2379,https://10.17.1.49:2379”
endpoint health
#结果展示
±---------------------------±-------±------------±------+
| ENDPOINT | HEALTH | TOOK | ERROR |
±---------------------------±-------±------------±------+
| https://10.7.0.10:2379 | true | 16.669768ms | |
| https://10.7.0.11:2379 | true | 23.794203ms | |
| https://10.7.0.12:2379 | true | 23.658683ms | |
±---------------------------±-------±------------±------+
2.3.7 设置flannel 网段，任意一个etcd节点执行
注意：这里set的网段需要和kube-controller-manager、kube-proxy配置里的cluster-cidr网段一致，也就是pod的网段
ETCDCTL_API=2 etcdctl
–endpoints=“https://10.17.1.47:2379,https://10.17.1.48:2379,https://10.17.1.49:2379”
–ca-file=/etc/etcd/ssl/ca.pem
–key-file=/etc/etcd/ssl/server-key.pem
–cert-file=/etc/etcd/ssl/server.pem
set /coreos.com/network/config ‘{ “Network”: “172.17.0.0/16”, “Backend”: {“Type”: “vxlan”}}’
#如果需要更改值，直接重置下（set）
#检查是否建立网段
[root@master-1 etcd-v3.3.10-linux-amd64]# ETCDCTL_API=2 etcdctl
–endpoints=“https://10.17.1.47:2379,https://10.17.1.48:2379,https://10.17.1.49:2379”
–ca-file=/etc/etcd/ssl/ca.pem
–cert-file=/etc/etcd/ssl/server.pem
–key-file=/etc/etcd/ssl/server-key.pem
get /coreos.com/network/config
#结果
#以上所有操作都是etcd节点
#以下操作，部署master 3个节点的组件
#分发二进制
[root@master-1 soft]# wget https://dl.k8s.io/v1.20.1/kubernetes-server-linux-amd64.tar.gz
[root@master-1 soft]# tar xvf kubernetes-server-linux-amd64.tar.gz
[root@master-1 soft]# cd kubernetes/server/bin/
[root@master-1 bin]# \cp kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/
[root@master-1 bin]# for i in crm-ma01-k8s-b2 crm-ma01-k8s-b3;do scp kube-apiserver kube-controller-manager kube-scheduler kubectl $i:/usr/local/bin/;done
[root@master-1 bin]# for i in crm-wrk01-k8s-b1 crm-wrk01-k8s-b2 crm-wrk01-k8s-b3;do scp kubelet kube-proxy $i:/usr/local/bin/;done
3.0 生成kubernetes 证书
#创建 Kubernetes 相关证书
#此证书用于Kubernetes节点直接的通信, 与之前的ETCD证书不同. （Master-1）
[root@master-1 ~]# mkdir /root/kubernetes/ && cd /root/kubernetes/
3.1 配置ca 文件（Master-1）
[root@master-1 ~]# cat << EOF | tee ca-config.json
{
“signing”: {
“default”: {
“expiry”: “876000h”
},
“profiles”: {
“kubernetes”: {
“expiry”: “876000h”,
“usages”: [
“signing”,
“key encipherment”,
“server auth”,
“client auth”
]
}
}
}
}
EOF
3.2 创建ca证书申请文件（Master-1）
[root@master-1 ~]# cat << EOF | tee ca-csr.json
{
“CN”: “kubernetes”,
“key”: {
“algo”: “rsa”,
“size”: 2048
},
“names”: [
{
“C”: “CN”,
“L”: “Beijing”,
“ST”: “Beijing”,
“O”: “k8s”,
“OU”: “System”
}
]
}
EOF
#证书字段意义
CN=commonName (网站域名)
OU=organizationUnit (组织部门名)
O=organizationName (组织名)
L=localityName (城市)
S=stateName (省份)
C=country (国家)
3.3 生成API SERVER证书申请文件（Master-1）
#注意要修改VIP的地址
#同样是提前预留3个ip位，给以后需要扩容的master节点使用
注意：此处填写master和node所有节点ip！！！
10.43.0.1是kubernetes的SVC地址
[root@master-1 ~]# cat << EOF | tee server-csr.json
{
“CN”: “kubernetes”,
“hosts”: [
“10.43.0.1”,
“127.0.0.1”,
“10.17.1.50”,
“10.17.1.51”,
“10.17.1.52”,
“10.17.1.53”,
“10.17.1.54”,
“10.17.1.55”,
“10.17.1.56”,
“10.17.1.57”,
“10.17.1.58”,
“10.17.1.59”,
“crm-ma01-k8s-b1”,
“crm-ma01-k8s-b2”,
“crm-ma01-k8s-b3”,
“crm-ma01-k8s-b4”,
“crm-ma01-k8s-b5”,
“crm-ma01-k8s-b6”,
“crm-wrk01-k8s-b1”,
“crm-wrk01-k8s-b2”,
“crm-wrk01-k8s-b3”,
“kubernetes”,
“kubernetes.default”,
“kubernetes.default.svc”,
“kubernetes.default.svc.cluster”,
“kubernetes.default.svc.cluster.local”
],
“key”: {
“algo”: “rsa”,
“size”: 2048
},
“names”: [
{
“C”: “CN”,
“L”: “Beijing”,
“ST”: “Beijing”,
“O”: “k8s”,
“OU”: “System”
}
]
}
EOF
3.4 创建 Kubernetes Proxy 证书申请文件（Master-1）
[root@master-1 ~]# cat << EOF | tee kube-proxy-csr.json
{
“CN”: “system:kube-proxy”,
“hosts”: [],
“key”: {
“algo”: “rsa”,
“size”: 2048
},
“names”: [
{
“C”: “CN”,
“L”: “Beijing”,
“ST”: “Beijing”,
“O”: “k8s”,
“OU”: “System”
}
]
}
EOF
3.5 生成 kubernetes CA 证书和公私钥
生成ca证书（Master-1）
[root@master-1 ~]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca –
3.5.1 # 生成 api-server 证书（Master-1）
[root@master-1 ~]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem
-config=ca-config.json
-profile=kubernetes server-csr.json | cfssljson -bare server
cfssl参数
gencert: 生成新的key(密钥)和签名证书
-initca：初始化一个新ca
-ca：指明ca的证书
-ca-key：指明ca的私钥文件
-config：指明请求证书的json文件
-profile：与-config中的profile对应，是指根据config中的profile段来生成证书的相关信息
3.5.2 # 生成 kube-proxy 证书（Master-1）
[root@master-1 ~]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json
-profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
4 安装 Docker
#安装CE版本(node)
[root@node-1 ~]# yum install -y yum-utils device-mapper-persistent-data lvm2
[root@node-1 ~]# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
[root@node-1 ~]# yum install -y docker-ce-19.03.6 docker-ce-cli-19.03.6 containerd.io
4.1 启动Docker服务
[root@node-1 ~]# systemctl restart docker
[root@node-1 ~]# systemctl enable docker
[root@node-1 ~]# service docker status
debian系统：
apt-get install apt-transport-https ca-certificates curl gnupg2 software-properties-common -y
curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
add-apt-repository “deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable”
apt-get update
apt-cache madison docker-ce #查找docker版本
apt install docker-ce=5:19.03.153-0debian-buster containerd.io -y
systemctl enable docker
systemctl status docker
docker info
#worker节点都要安装。etcd、master节点不需要安装
5 安装flannel
5.1 下载Flannel二进制包
#所有的worker节点,下载到master-1
[root@ master -1 ~]# mkdir /soft ; cd /soft
[root@ master -1 ~]# wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
[root@ master -1 ~]# tar xvf flannel-v0.11.0-linux-amd64.tar.gz
[root@ master -1 ~]# mv flanneld mk-docker-opts.sh /usr/local/bin/
#复制flanneld到其他的所有worker节点
[root@ master -1 ~]# for i in crm-wrk01-k8s-b1 crm-wrk01-k8s-b2 crm-wrk01-k8s-b3;do scp flannel-v0.11.0-linux-amd64.tar.gz $i:/root;done
worker节点操作：
tar xvf flannel-v0.11.0-linux-amd64.tar.gz
mv flanneld mk-docker-opts.sh /usr/local/bin/
5.2 配置Flannel (所有worker节点)
[root@node-1 ~]# mkdir -p /etc/flannel
[root@node-1 ~]# cat > /etc/flannel/flannel.cfg<<EOF
FLANNEL_OPTIONS=“-etcd-endpoints=https://10.17.1.47:2379,https://10.17.1.48:2379,https://10.17.1.49:2379 -etcd-cafile=/etc/etcd/ssl/ca.pem -etcd-certfile=/etc/etcd/ssl/server.pem -etcd-keyfile=/etc/etcd/ssl/server-key.pem”
EOF
#只填etcd节点ip
5.3 配置flannel 配置文件
cat > /usr/lib/systemd/system/flanneld.service <<EOF
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service
[Service]
Type=notify
EnvironmentFile=/etc/flannel/flannel.cfg
ExecStart=/usr/local/bin/flanneld --ip-masq $FLANNEL_OPTIONS
ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
5.4 所有worker节点启动Flannel
systemctl daemon-reload
service flanneld restart
systemctl enable flanneld
service flanneld status
#flannel能运行起来之后，停止，修改docker启动文件再运行flannel
5.5#所有worker节点停止flanneld
service flanneld stop
5.6 修改docker 配置文件(所有worker节点)
cat >/usr/lib/systemd/system/docker.service<<EOFL
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
EOFL
5.7 重启Docker服务
systemctl daemon-reload
service flanneld restart
service docker restart
6 安装Master组件
6.1 安装Api Server服务
6.1.1 下载Kubernetes二进制包(1.20.2)（master-1）
[root@master-1 soft]# cd /soft
[root@master-1 soft]# tar xvf kubernetes-server-linux-amd64.tar.gz
[root@master-1 soft]# cd kubernetes/server/bin/
[root@master-1 soft]# cp kube-scheduler kube-apiserver kube-controller-manager kubectl /usr/local/bin/
#复制执行文件到其他的master节点
[root@master-1 bin]# for i in crm-ma01-k8s-b2 crm-ma01-k8s-b3;do scp /usr/local/bin/kube* $i:/usr/local/bin/;done
6.1.2 配置Kubernetes证书
#Kubernetes各个组件之间通信需要证书,需要复制个每个master节点（master-1）
[root@master-1 soft]# mkdir -p /etc/kubernetes/{cfg,ssl}
[root@master-1 soft]# cp /root/kubernetes/*.pem /etc/kubernetes/ssl/
#复制到其他的节点
[root@master-1 soft]# for i in crm-ma01-k8s-b2 crm-ma01-k8s-b3 crm-wrk01-k8s-b1 crm-wrk01-k8s-b2 crm-wrk01-k8s-b3;do ssh $i mkdir -p /etc/kubernetes/{cfg,ssl};done
[root@master-1 soft]# for i in crm-ma01-k8s-b2 crm-ma01-k8s-b3 crm-wrk01-k8s-b1 crm-wrk01-k8s-b2 crm-wrk01-k8s-b3;do scp /etc/kubernetes/ssl/* $i:/etc/kubernetes/ssl/;done
[root@master-1 bin]# for i in crm-ma01-k8s-b2 crm-ma01-k8s-b3 crm-wrk01-k8s-b1 crm-wrk01-k8s-b2 crm-wrk01-k8s-b3;do echo $i “---------->”; ssh $i ls /etc/kubernetes/ssl;done
6.1.4 编辑Token 文件(master-1)
#f89a76f197526a0d4bc2bf9c86e871c3:随机字符串,自定义生成; kubelet-bootstrap:用户名; 10001:UID; system:kubelet-bootstrap：用户组
#可以不用修改，直接粘贴
[root@master-1 soft]# vim /etc/kubernetes/cfg/token.csv
f89a76f197526a0d4bc2bf9c86e871c3,kubelet-bootstrap,10001,“system:kubelet-bootstrap”
#复制到其他的master节点
[root@master-1 bin]# for i in crm-ma01-k8s-b2 crm-ma01-k8s-b3 ;do scp /etc/kubernetes/cfg/token.csv $i:/etc/kubernetes/cfg/token.csv;done
#配置api-server启动文件，每台master节点都需要执行
#可以自行修改集群网段和子网掩码
[root@master-1 soft]# cat >/etc/kubernetes/cfg/kube-apiserver.cfg <<EOFL
KUBE_APISERVER_OPTS=“–enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount
–anonymous-auth=false
–bind-address=0.0.0.0
–secure-port=6443
–advertise-address=0.0.0.0
–insecure-port=0
–authorization-mode=Node,RBAC
–runtime-config=api/all=true
–enable-bootstrap-token-auth
–service-cluster-ip-range=10.43.0.0/16
–token-auth-file=/etc/kubernetes/cfg/token.csv
–service-node-port-range=30000-32767
–tls-cert-file=/etc/kubernetes/ssl/server.pem
–tls-private-key-file=/etc/kubernetes/ssl/server-key.pem
–client-ca-file=/etc/kubernetes/ssl/ca.pem
–service-account-key-file=/etc/kubernetes/ssl/ca-key.pem
–service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem
–service-account-issuer=https://kubernetes.default.svc.cluster.local
–etcd-cafile=/etc/etcd/ssl/ca.pem
–etcd-certfile=/etc/etcd/ssl/server.pem
–etcd-keyfile=/etc/etcd/ssl/server-key.pem
–etcd-servers=https://10.17.1.47:2379,https://10.17.1.48:2379,https://10.17.1.49:2379
–enable-swagger-ui=true
–allow-privileged=true
–apiserver-count=3
–audit-log-maxage=30
–audit-log-maxbackup=3
–audit-log-maxsize=100
–audit-log-path=/var/log/kube-apiserver-audit.log
–event-ttl=1h
–alsologtostderr=true
–logtostderr=false
–log-dir=/var/log/kubernetes
–watch-cache=true
–default-watch-cache-size=1500
–event-ttl=1h0m0s
–max-requests-inflight=800
–max-mutating-requests-inflight=400
–feature-gates=RemoveSelfLink=false
–default-not-ready-toleration-seconds=60
–default-unreachable-toleration-seconds=60
–v=4”
EOFL
注：
–logtostderr：启用日志
–v：日志等级
–log-dir：日志目录
–etcd-servers：etcd集群地址
–bind-address：监听地址
–secure-port：https安全端口
–advertise-address：集群通告地址
–allow-privileged：启用授权
–service-cluster-ip-range：Service虚拟IP地址段
–enable-admission-plugins：准入控制模块
–authorization-mode：认证授权，启用RBAC授权和节点自管理
–enable-bootstrap-token-auth：启用TLS bootstrap机制
–token-auth-file：bootstrap token文件
–service-node-port-range：Service nodeport类型默认分配端口范围
–kubelet-client-xxx：apiserver访问kubelet客户端证书
–tls-xxx-file：apiserver https证书
–etcd-xxxfile：连接Etcd集群证书
–audit-log-xxx：审计日志
参数说明: https://kubernetes.io/zh/docs/reference/command-line-tools-reference/kube-apiserver/
#设置启动
cat >/usr/lib/systemd/system/kube-apiserver.service<<EOFL
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/etc/kubernetes/cfg/kube-apiserver.cfg
ExecStart=/usr/local/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOFL
#配置自动启动
systemctl daemon-reload
service kube-apiserver restart
systemctl enable kube-apiserver
service kube-apiserver status
#debug 模式
/usr/local/bin/kube-apiserver
–enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount
–anonymous-auth=false
–bind-address=0.0.0.0
–secure-port=6443
–advertise-address=0.0.0.0
–insecure-port=0
–authorization-mode=Node,RBAC
–runtime-config=api/all=true
–enable-bootstrap-token-auth
–service-cluster-ip-range=10.43.0.0/16
–token-auth-file=/etc/kubernetes/cfg/token.csv
–service-node-port-range=30000-50000
–tls-cert-file=/etc/kubernetes/ssl/server.pem
–tls-private-key-file=/etc/kubernetes/ssl/server-key.pem
–client-ca-file=/etc/kubernetes/ssl/ca.pem
–service-account-key-file=/etc/kubernetes/ssl/ca-key.pem
–service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem
–service-account-issuer=https://kubernetes.default.svc.cluster.local
–etcd-cafile=/etc/etcd/ssl/ca.pem
–etcd-certfile=/etc/etcd/ssl/server.pem
–etcd-keyfile=/etc/etcd/ssl/server-key.pem
–etcd-servers=https://10.7.0.10:2379,https://10.7.0.11:2379,https://10.7.0.12:2379
–enable-swagger-ui=true
–allow-privileged=true
–apiserver-count=3
–audit-log-maxage=30
–audit-log-maxbackup=3
–audit-log-maxsize=100
–audit-log-path=/var/log/kube-apiserver-audit.log
–event-ttl=1h
–alsologtostderr=true
–logtostderr=false
–log-dir=/var/log/kubernetes
–watch-cache=true
–default-watch-cache-size=1500
–event-ttl=1h0m0s
–max-requests-inflight=800
–max-mutating-requests-inflight=400
–feature-gates=RemoveSelfLink=false
–v=4
#验证地址,ip地址是VIP
[root@master-1 bin]# curl --insecure https://10.17.1.50:6443
#结果
{
“kind”: “Status”,
“apiVersion”: “v1”,
“metadata”: {
},
“status”: “Failure”,
“message”: “Unauthorized”,
“reason”: “Unauthorized”,
“code”: 401
}
#部署kubectl
创建csr请求文件
[root@master-1 ~]# cd /root/kubernetes
#配置客户端授权文件
[root@master-1 ~]# cat > admin-csr.json <<‘EOF’
{
“CN”: “admin”,
“hosts”: [],
“key”: {
“algo”: “rsa”,
“size”: 2048
},
“names”: [
{
“C”: “CN”,
“ST”: “BeiJing”,
“L”: “BeiJing”,
“O”: “system:masters”,
“OU”: “System”
}
]
}
EOF
说明：
后续 kube-apiserver 使用 RBAC 对客户端(如 kubelet、kube-proxy、Pod)请求进行授权；
kube-apiserver 预定义了一些 RBAC 使用的 RoleBindings，如 cluster-admin 将 Group system:masters 与 Role cluster-admin 绑定，该 Role 授予了调用kube-apiserver 的所有 API的权限；
O指定该证书的 Group 为 system:masters，kubelet 使用该证书访问 kube-apiserver 时 ，由于证书被 CA 签名，所以认证通过，同时由于证书用户组为经过预授权的 system:masters，所以被授予访问所有 API 的权限；
注：
这个admin 证书，是将来生成管理员用的kube config 配置文件用的，现在我们一般建议使用RBAC 来对kubernetes 进行角色权限控制， kubernetes 将证书中的CN 字段 作为User， O 字段作为 Group；
“O”: “system:masters”, 必须是system:masters，否则后面kubectl create clusterrolebinding报错。
#生成客户端证书
[root@master-1 kubernetes]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem
-config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
#复制证书
[root@master-1 kubernetes]# \cp /root/kubernetes/admin*.pem /etc/kubernetes/ssl/
[root@master-1 kubernetes]# for i in crm-ma01-k8s-b2 crm-ma01-k8s-b3;do
scp /etc/kubernetes/ssl/admin*.pem $i:/etc/kubernetes/ssl/;done
设置集群参数
[root@master-1 kubernetes]# export KUBE_APISERVER=“https://10.17.1.50:6443” #VIP地址
[root@master-1 kubernetes]# kubectl config set-cluster kubernetes
–certificate-authority=/etc/kubernetes/ssl/ca.pem
–embed-certs=true
–server=${KUBE_APISERVER}
–kubeconfig=kube.config
设置客户端认证参数
[root@master-1 kubernetes]# kubectl config set-credentials admin
–client-certificate=/etc/kubernetes/ssl/admin.pem
–client-key=/etc/kubernetes/ssl/admin-key.pem
–embed-certs=true --kubeconfig=kube.config
设置上下文参数
[root@master-1 kubernetes]# kubectl config set-context kubernetes
–cluster=kubernetes --user=admin --kubeconfig=kube.config
设置默认上下文
[root@master-1 kubernetes]# kubectl config use-context kubernetes --kubeconfig=kube.config
[root@master-1 kubernetes]# mkdir ~/.kube
[root@master-1 kubernetes]# \cp kube.config ~/.kube/config
授权kubernetes证书访问kubelet api权限
[root@master1 work]# kubectl create clusterrolebinding kube-apiserver:kubelet-apis
–clusterrole=system:kubelet-api-admin --user kubernetes
#获取集群信息
[root@master-1 kubernetes]# kubectl cluster-info
Kubernetes control plane is running at https://10.7.0.59:6443
To further debug and diagnose cluster problems, use ‘kubectl cluster-info dump’.
#获取组件信息
[root@master-1 ~]# kubectl get componentstatuses
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
scheduler Unhealthy Get “http://127.0.0.1:10251/healthz”: dial tcp 127.0.0.1:10251: connect: connection refused
controller-manager Unhealthy Get “http://127.0.0.1:10252/healthz”: dial tcp 127.0.0.1:10252: connect: connection refused
etcd-2 Healthy {“health”:“true”}
etcd-0 Healthy {“health”:“true”}
etcd-1 Healthy
配置kubectl子命令补全
curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.20.15/bin/linux/amd64/kubectl
chmod +x ./kubectl
mv ./kubectl /usr/local/bin/kubectl
mkdir -p ~/.local/bin/kubectl
kubectl version --client
apt install bash-completion -y
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
kubectl completion bash >/etc/bash_completion.d/kubectl
3.4.5 部署kube-controller-manager 所有master节点执行
创建csr请求文件
[root@master-1 ~]# cd /root/kubernetes
[root@master-1 kubernetes]# cat > kube-controller-manager-csr.json <<‘EOF’
{
“CN”: “system:kube-controller-manager”,
“key”: {
“algo”: “rsa”,
“size”: 2048
},
“hosts”: [
“127.0.0.1”,
“10.17.1.50”,
“10.17.1.51”,
“10.17.1.52”,
“10.17.1.53”,
“10.17.1.54”,
“10.17.1.55”,
“10.17.1.56”
],
“names”: [
{
“C”: “CN”,
“ST”: “BeiJing”,
“L”: “BeiJing”,
“O”: “system:kube-controller-manager”,
“OU”: “system”
}
]
}
EOF
注：
hosts 列表包含所有 kube-controller-manager 节点 IP；
CN 为 system:kube-controller-manager、O 为 system:kube-controller-manager，kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限
生成证书
[root@master-1 kubernetes]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem
-config=ca-config.json -profile=kubernetes
kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
#查看证书
[root@master-1 kubernetes]# ls kube-controller-manager*.pem
kube-controller-manager-key.pem kube-controller-manager.pem
#复制证书
[root@master-1 kubernetes]# \cp kube-controller-manager*.pem /etc/kubernetes/ssl/
[root@master-1 kubernetes]# for i in crm-ma01-k8s-b2 crm-ma01-k8s-b3;do
scp /etc/kubernetes/ssl/kube-controller-manager*.pem $i:/etc/kubernetes/ssl/;done
创建kube-controller-manager的kubeconfig
设置集群参数,设置VIP地址
[root@master-1 kubernetes]# kubectl config set-cluster kubernetes
–certificate-authority=/etc/kubernetes/ssl/ca.pem
–embed-certs=true
–server=https://10.17.1.50:6443
–kubeconfig=kube-controller-manager.kubeconfig
设置客户端认证参数
[root@master-1 kubernetes]# kubectl config set-credentials
system:kube-controller-manager
–client-certificate=/etc/kubernetes/ssl/kube-controller-manager.pem
–client-key=/etc/kubernetes/ssl/kube-controller-manager-key.pem
–embed-certs=true
–kubeconfig=kube-controller-manager.kubeconfig
设置上下文参数
[root@master-1 kubernetes]# kubectl config set-context
system:kube-controller-manager --cluster=kubernetes
–user=system:kube-controller-manager
–kubeconfig=kube-controller-manager.kubeconfig
设置默认上下文
[root@master-1 kubernetes]# kubectl config use-context
system:kube-controller-manager
–kubeconfig=kube-controller-manager.kubeconfig
#复制文件
[root@master-1 kubernetes]# \cp kube-controller-manager.kubeconfig /etc/kubernetes/cfg/
[root@master-1 kubernetes]# for i in crm-ma01-k8s-b2 crm-ma01-k8s-b3;do
scp /etc/kubernetes/cfg/kube-controller-manager.kubeconfig $i:/etc/kubernetes/cfg;done
#设置配置文件，master节点都要执行
注意：cluster-cidr是pod的网段，需要和上面etcd set的网段一致。
[root@master-1 bin]# cat >/etc/kubernetes/cfg/kube-controller-manager.cfg<<EOFL
KUBE_CONTROLLER_MANAGER_OPTS=“–bind-address=0.0.0.0
–secure-port=10257
–port=10252
–address=0.0.0.0
–kubeconfig=/etc/kubernetes/cfg/kube-controller-manager.kubeconfig
–service-cluster-ip-range=10.43.0.0/16
–cluster-name=kubernetes
–cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem
–cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem
–allocate-node-cidrs=true
–cluster-cidr=172.17.0.0/16
–experimental-cluster-signing-duration=876000h
–root-ca-file=/etc/kubernetes/ssl/ca.pem
–service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem
–leader-elect=true
–feature-gates=RotateKubeletServerCertificate=true
–controllers=*,bootstrapsigner,tokencleaner
–horizontal-pod-autoscaler-use-rest-clients=true
–horizontal-pod-autoscaler-sync-period=10s
–tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem
–tls-private-key-file=/etc/kubernetes/ssl/kube-controller-manager-key.pem
–use-service-account-credentials=true
–alsologtostderr=true
–logtostderr=false
–log-dir=/var/log/kubernetes
–node-monitor-period=2s
–node-monitor-grace-period=20s
–node-startup-grace-period=30s
–pod-eviction-timeout=1m
–v=2”
EOFL
#注
参数说明: https://kubernetes.io/zh/docs/reference/command-line-tools-reference/kube-controller-manager/
kube-proxy 根据 --cluster-cidr 判断集群内部和外部流量，指定 --cluster-cidr 或 --masquerade-all 选项后 kube-proxy 才会对访问 Service IP 的请求做 SNAT；
9.3.2 创建kube-controller-manager 启动文件
[root@master-1 bin]# cat >/usr/lib/systemd/system/kube-controller-manager.service<<EOFL
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/etc/kubernetes/cfg/kube-controller-manager.cfg
ExecStart=/usr/local/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOFL
9.3.3启动kube-controller-manager服务
[root@master-1 bin]# systemctl daemon-reload && systemctl enable kube-controller-manager
[root@master-1 bin]# service kube-controller-manager restart
[root@master-1 bin]# service kube-controller-manager status
#debug 启动
/usr/local/bin/kube-controller-manager
–bind-address=0.0.0.0
–secure-port=10257
–port=10252
–address=0.0.0.0
–kubeconfig=/etc/kubernetes/cfg/kube-controller-manager.kubeconfig
–service-cluster-ip-range=10.43.0.0/16
–cluster-name=kubernetes
–cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem
–cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem
–allocate-node-cidrs=true
–cluster-cidr=172.17.0.0/16
–experimental-cluster-signing-duration=876000h
–root-ca-file=/etc/kubernetes/ssl/ca.pem
–service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem
–leader-elect=true
–feature-gates=RotateKubeletServerCertificate=true
–controllers=*,bootstrapsigner,tokencleaner
–horizontal-pod-autoscaler-use-rest-clients=true
–horizontal-pod-autoscaler-sync-period=10s
–tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem
–tls-private-key-file=/etc/kubernetes/ssl/kube-controller-manager-key.pem
–use-service-account-credentials=true
–alsologtostderr=true
–logtostderr=false
–log-dir=/var/log/kubernetes
–node-monitor-grace-period=20s
–node-startup-grace-period=30s
–pod-eviction-timeout=1m
–v=2
详细版，请联系QQ：1043018380（备注来处）
