k8s基础应用一：macOS和linux环境docker安装

1 macOS安装docker

1.1 安装

macOS一般安装desktop桌面版，用于平时测试。从docker官网下载docker.dmg安装即可。安装后启动docker就可以使用了。

% docker version
Client:
 Cloud integration: v1.0.35+desktop.4
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:28:49 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.23.0 (120376)
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:36 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

1.2 docker dashboard和配置

docker.dmg安装完后，启动docker，打开docker的dashboard可以查看docker的运行情况，也可以在Settings中根据需要修改docker配置：

一般会把memory设置为大于4G，

同时由于国内从国外docker仓库下载镜像一般很慢，一般添加一些国内镜像地址进行加速

文本内容如下：

{
  "builder": {
    "gc": {
      "enabled": true,
      "defaultKeepStorage": "20GB"
    }
  },
  "registry-mirrors": [
    "https://registry.cn-hangzhou.aliyuncs.com",
    "https://mirror.ccs.tencentyun.com",
    "https://05f073ad3c0010ea0f4bc00b7105ec20.mirror.swr.myhuaweicloud.com",
    "https://registry.docker-cn.com",
    "http://hub-mirror.c.163.com",
    "http://f1361db2.m.daocloud.io"
  ],
  "debug": true,
  "experimental": false
}

查看docker-for-mac对应的kubernetes版本，并使能kubernetes，这样启动docker时会同时启动kubernetes

1.3 手动下载kubernetes镜像

docker desktop同时可以运行kubernetes，但是由于docker.dmg中并不包含kubernetes相关的组件，需要通过网络进行下载，又因为从国外下载的速度往往非常慢，导致kubernetes往往无法正常启动。这时，我们可以手动下载kubernetes相关镜像。github仓库k8s-for-docker-desktop已经维护了desktop不同kubernetes版本的下载脚本，找到相应版本的images.properties和load_images.sh文件下载到本地，然后执行脚本即可。

sh load_images.sh

load_images.sh脚本的内容如下：

#!/bin/bash

file="images.properties"

if [ -f "$file" ]
then
  echo "$file found."

  while IFS='=' read -r key value
  do
    #echo "${key}=${value}"
    docker pull ${value}
    docker tag ${value} ${key}
    docker rmi ${value}
  done < "$file"

else
  echo "$file not found."
fi

如果你的docker desktop对应的kubernetes版本比较新，k8s-for-docker-desktop仓库还没有维护，可以自己编辑images.properties文件，在文件中添加相应版本的kubernetes镜像，如上面docker对应的v1.27.2版本kubernetes可以使用如下内容的images.properties:

registry.k8s.io/pause:3.8=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.8
registry.k8s.io/kube-controller-manager:v1.27.2=registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.27.2
registry.k8s.io/kube-scheduler:v1.27.2=registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.27.2
registry.k8s.io/kube-proxy:v1.27.2=registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.2
registry.k8s.io/kube-apiserver:v1.27.2=registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.27.2
registry.k8s.io/etcd:3.5.5-0=registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.5-0
registry.k8s.io/coredns/coredns:v1.9.3=registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.9.3
registry.k8s.io/ingress-nginx/controller:v1.6.4=registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.6.4
registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.5.2=registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.5.2

1.4 docker和k8s测试

上面kubernetes的相关镜像成功下载后，k8s一般就正常启动了，我们查看一下已安装的本地镜像

% docker images
REPOSITORY                                                TAG                                                                          IMAGE ID       CREATED         SIZE
hubproxy.docker.internal:5555/docker/desktop-kubernetes   kubernetes-v1.27.2-cni-v1.2.0-critools-v1.27.0-cri-dockerd-v0.3.2-1-debian   8ba658ef36dd   3 months ago    398MB
registry.k8s.io/kube-apiserver                            v1.27.2                                                                      72c9df6be7f1   4 months ago    115MB
registry.k8s.io/kube-controller-manager                   v1.27.2                                                                      2ee705380c3c   4 months ago    107MB
registry.k8s.io/kube-scheduler                            v1.27.2                                                                      305d7ed1dae2   4 months ago    56.2MB
registry.k8s.io/kube-proxy                                v1.27.2                                                                      29921a084542   4 months ago    66.5MB
docker/desktop-vpnkit-controller                          dc331cb22850be0cdd97c84a9cfecaf44a1afb6e                                     3750dfec169f   4 months ago    35MB
registry.k8s.io/ingress-nginx/controller                  v1.6.4                                                                       bad06a492068   7 months ago    282MB
registry.k8s.io/coredns/coredns                           v1.10.1                                                                      97e04611ad43   7 months ago    51.4MB
sysrepo/sysrepo-netopeer2                                 latest                                                                       976d1dcf6c0c   7 months ago    622MB
registry.k8s.io/etcd                                      3.5.7-0                                                                      24bc64e91103   8 months ago    181MB
registry.k8s.io/pause                                     3.9                                                                          829e9de338bd   11 months ago   514kB
registry.k8s.io/etcd                                      3.5.5-0                                                                      b9a1dfeddea9   12 months ago   179MB
registry.k8s.io/pause                                     3.8                                                                          4e42fb3c9d90   15 months ago   514kB
registry.k8s.io/coredns/coredns                           v1.9.3                                                                       b19406328e70   16 months ago   47.7MB
docker/desktop-storage-provisioner                        v2.0                                                                         c027a58fa0bb   2 years ago     39.8MB
registry.k8s.io/ingress-nginx/kube-webhook-certgen        v1.5.2                                                                       9dadbac53d3b 

可以发现就是我们上面下载安装的k8s镜像。再看一下k8s系统已经启动的工作节点，只有一个，名字就叫做docker-desktop：

% kubectl get nodes
NAME             STATUS   ROLES           AGE   VERSION
docker-desktop   Ready    control-plane   11d   v1.27.2

查看一下k8s已创建的namespace：

% kubectl get namespaces
NAME              STATUS   AGE
default           Active   11d
kube-node-lease   Active   11d
kube-public       Active   11d
kube-system       Active   11d

查看k8s系统启动的pod，pod都放在kube-system这个namespace下，由此也发现虽然kubernetes本事是一个容器（pod）编排和管理系统，docker-for-mac的kubernetes系统的所有组件本身都是以pod的形式运行的

% kubectl get pod --all-namespaces
NAMESPACE     NAME                                     READY   STATUS    RESTARTS      AGE
kube-system   coredns-5d78c9869d-9f9wp                 1/1     Running   1 (11d ago)   11d
kube-system   coredns-5d78c9869d-kqxgl                 1/1     Running   1 (11d ago)   11d
kube-system   etcd-docker-desktop                      1/1     Running   1 (11d ago)   11d
kube-system   kube-apiserver-docker-desktop            1/1     Running   1 (11d ago)   11d
kube-system   kube-controller-manager-docker-desktop   1/1     Running   1 (11d ago)   11d
kube-system   kube-proxy-brjpm                         1/1     Running   1 (11d ago)   11d
kube-system   kube-scheduler-docker-desktop            1/1     Running   1 (11d ago)   11d
kube-system   storage-provisioner                      1/1     Running   2 (11d ago)   11d
kube-system   vpnkit-controller                        1/1     Running   1 (11d ago)   11d

2 linux安装docker

2.1 通过yum安装

2.1.1 下载docker

如果网络环境允许yum安装，可以直接通过yum -y install docker命令下载

$ sudo su -
# yum -y install docker

yum安装完后，可以在/usr/lib/systemd/system目录下看到以下跟docker相关的服务管理配置文件：

# cd /usr/lib/systemd/system
# ls | grep docker
docker-cleanup.service
docker-cleanup.timer
docker.service
docker-storage-setup.service

查看这些文件的内容，我们可以就可以知道docker涉及的二进制文件和配置文件位置了。

2.1.2 启动dockerd服务

启动docker：

# systemctl daemon-reload
# systemctl start docker

设置开机自启动：

# systemctl daemon-reload
# systemctl enable --now docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.

2.2 二进制包安装

2.2.1 安装

直接通过yum安装的docker，有可能安装的docker版本比较老旧，比如上面安装的docker版本为：

$ docker -v
Docker version 1.13.1, build 7d71120/1.13.1

我们也可以到docker官网自己下载二进制包，手动安装。先查看机器的CPU架构和操作系统版本：

# uname -a
Linux 4.14.0_1-0-0-45 #2 SMP Tue Oct 19 18:27:28 CST 2021 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/centos-release
CentOS Linux release 7.5.1804 (Core) 

然后从官网下载对应CPU架构和操作系统的二进制包docker-24.0.6.tgz，然后安装

$ sudo su -
# tar -xzf docker-24.0.6.tgz
# 安装包中的二进制文件
# ls docker
containerd  containerd-shim-runc-v2  ctr  docker  dockerd  docker-init  docker-proxy  runc
# mv docker/* /usr/bin/

然后编辑相应的service配置文件

# cd /usr/lib/systemd/system

# cat  docker.service 
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target
Requires=docker-cleanup.timer

[Service]
Type=notify
NotifyAccess=main
EnvironmentFile=-/run/containers/registries.conf
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
Environment=GOTRACEBACK=crash
Environment=DOCKER_HTTP_HOST_COMPAT=1
Environment=PATH=/usr/bin:/usr/sbin
ExecStart=/usr/bin/dockerd \
          $OPTIONS \
          $DOCKER_STORAGE_OPTIONS \
          $DOCKER_NETWORK_OPTIONS \
          $ADD_REGISTRY \
          $BLOCK_REGISTRY \
          $INSECURE_REGISTRY \
          $REGISTRIES
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0
Restart=on-abnormal
KillMode=process

[Install]
WantedBy=multi-user.target
EOF

# cat  docker-cleanup.timer 
[Unit]
Description=Run docker-cleanup every hour

[Timer]
OnCalendar=hourly
Persistent=true

[Install]
WantedBy=timers.target
EOF

# cat  docker-cleanup.service
[Unit]
Description=Docker Cleanup
Requires=docker.service


[Service]
Type=oneshot
ExecStart=/usr/bin/sh -c "DEAD=`docker ps -aq -f status=dead` && [ -n \"$DEAD\" ] && docker rm $DEAD; exit 0"

[Install]
WantedBy=multi-user.target
EOF

docker的镜像等数据默认存放在/var/lib/docker目录中，我们一般希望更改它的存储位置。我们也可能希望设置自己的docker仓库镜像地址，这些配置以及其他的一些相关配置可以在docker的配置文件/etc/docker/daemon.json中指定

# cd /etc/docker
# cat  daemon.json
{
  "insecure-registries": ["registry.cn-hangzhou.aliyuncs.com"],
  "registry-mirrors": ["https://registry.cn-hangzhou.aliyuncs.com"],
  "data-root":"/home/disk1/docker"
}
EOF

配置完成就可以启动了

# systemctl daemon-reload
# systemctl start docker
# systemctl enable --now docker

查看一下安装信息：

# docker version
Client:
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:30:51 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:32:17 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.3
  GitCommit:        7880925980b188f4c97b462f709d0db8e8962aff
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
# docker info
Client:
 Version:    24.0.6
 Context:    default
 Debug Mode: false

Server:
 Containers: 3
  Running: 0
  Paused: 0
  Stopped: 3
 Images: 2
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7880925980b188f4c97b462f709d0db8e8962aff
 runc version: v1.1.9-0-gccaecfc
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 4.14.0_1-0-0-45
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 48
 Total Memory: 125.3GiB
 Name: szm3b-szlab-dev20.szm3b.baidu.com
 ID: 6bca25b8-7482-421e-993f-93cf787c4d08
 Docker Root Dir: /home/disk1/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  registry.cn-hangzhou.aliyuncs.com
  127.0.0.0/8
 Registry Mirrors:
  https://registry.cn-hangzhou.aliyuncs.com/
 Live Restore Enabled: false
 Product License: Community Engine

2.3 问题

2.3.1 手动创建docker0网桥

如果启动时出现错误：dockerd-current: Error starting daemon: Error initializing network controller: list bridge addresses failed: no available network，则表示没有配置docker0网桥，执行以下命令添加docker0网桥，并配置网桥的ip地址网段：

# ip link add name docker0 type bridge
# ip addr add dev docker0 172.17.0.1/16

执行完后确认docker0网桥已安装

# ip addr
40: docker0:  mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:a5:ce:b1:a5 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
       
# ip link
40: docker0:  mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:a5:ce:b1:a5 brd ff:ff:ff:ff:ff:ff

2.3.2 手动挂载cgroups

如果启动时出现failed to start daemon: Devices cgroup isn‘t mounted错误，可以执行以下脚本进行挂载：

# cat  mount_cgroup.sh
#! /bin/sh
set -e # 这句是告诉bash如何有任何语句执行结果不为ture，就应该退出。

if grep -v '^#' /etc/fstab | grep -q cgroup; then
	echo 'cgroups mounted from fstab, not mounting /sys/fs/cgroup'
	exit 0
fi

# kernel provides cgroups?
if [ ! -e /proc/cgroups ]; then
	exit 0
fi

# 确保目录存在
if [ ! -d /sys/fs/cgroup ]; then
	exit 0
fi

# mount /sys/fs/cgroup if not already done
if ! mountpoint -q /sys/fs/cgroup; then
	mount -t tmpfs -o uid=0,gid=0,mode=0755 cgroup /sys/fs/cgroup
fi

cd /sys/fs/cgroup

# get/mount list of enabled cgroup controllers
for sys in $(awk '!/^#/ { if ($4 == 1) print $1 }' /proc/cgroups); do
	mkdir -p $sys
	if ! mountpoint -q $sys; then
		if ! mount -n -t cgroup -o $sys cgroup $sys; then rmdir $sys || true
		fi
	fi
done
exit 0
EOF

执行完后查看cgroup已经安装：

# df -h
文件系统	      容量  已用  可用 已用%% 挂载点
/dev/sda2              19G  6.8G   12G  38% /
tmpfs                  48G   11M   48G   1% /dev/shm
/dev/sda3              14G  5.1G  8.6G  38% /var
/dev/sda4              14G  1.1G   13G   8% /noah
/dev/sda5             4.6G   68M  4.5G   2% /matrix
/dev/sda6             1.9G   20K  1.9G   1% /has
/dev/sda7             6.4G  9.9M  6.4G   1% /tmp
/dev/sda8             392G  101G  272G  27% /home
/dev/sdb1             1.7T   68M  1.7T   1% /home/disk1
cgroup                 48G     0   48G   0% /sys/fs/cgroup