一个项目的master节点down了之后kubelet起不来，由于是远古时期的项目，早期未作任何监控手段。因此也没有监控，没有文档，非常棘手经过查看，master节点的kubelet未启动，查看kubelet日志: failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory
- 我重新梳理了问题的关键，请转到bootstrap-kubelet.conf: no such file or directory kubelet证书轮换失败
failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory随即在/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf中去掉--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf后重启
master的kubelet已经启动起来，但是还是有问题，status是NotReady
查看master状态也是NotReady，而后就暂停了master的调度SchedulingDisabled
[root@master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
master NotReady,SchedulingDisabled master 647d v1.16.3
worker1 Ready <none> 647d v1.16.3
worker2 Ready <none> 647d v1.16.3我们留意到worker是ready的，于是我们查看worker节点。发现worker节点调度正常，并且可以进行拉取镜像，并且此时业务也恢复了。我们使用kubectl describe master查看到master节点还是有问题
[root@master ~]# kubectl describe node master
Name: master
Roles: master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=master
kubernetes.io/os=linux
node-role.kubernetes.io/master=
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
projectcalico.org/IPv4Address: 192.168.133.6/25
projectcalico.org/IPv4IPIPTunnelAddr: 10.100.219.64
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Sat, 14 Mar 2020 11:37:43 +0800
Taints: node.kubernetes.io/unreachable:NoExecute
node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/unreachable:NoSchedule
node.kubernetes.io/unschedulable:NoSchedule
Unschedulable: true
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Thu, 04 Mar 2021 10:26:49 +0800 Thu, 04 Mar 2021 10:26:49 +0800 CalicoIsUp Calico is running on this node
MemoryPressure Unknown Tue, 21 Dec 2021 09:17:47 +0800 Tue, 21 Dec 2021 09:18:41 +0800 NodeStatusUnknown Kubelet stopped posting node status.
DiskPressure Unknown Tue, 21 Dec 2021 09:17:47 +0800 Tue, 21 Dec 2021 09:18:41 +0800 NodeStatusUnknown Kubelet stopped posting node status.
PIDPressure Unknown Tue, 21 Dec 2021 09:17:47 +0800 Tue, 21 Dec 2021 09:18:41 +0800 NodeStatusUnknown Kubelet stopped posting node status.
Ready Unknown Tue, 21 Dec 2021 09:17:47 +0800 Tue, 21 Dec 2021 09:18:41 +0800 NodeStatusUnknown Kubelet stopped posting node status.
Addresses:
InternalIP: 192.168.133.6
Hostname: master
Capacity:
cpu: 4
ephemeral-storage: 36805060Ki
hugepages-2Mi: 0
memory: 16259676Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 33919543240
hugepages-2Mi: 0
memory: 16157276Ki
pods: 110
System Info:
Machine ID: abfbcf67c72dca40affb0cd0c8debd6d
System UUID: 63B3A0C6-AEB7-3841-BA81-2724178E0890
Boot ID: 02424d77-4b14-45ea-a632-038dbc70a2c4
Kernel Version: 3.10.0-957.el7.x86_64
OS Image: CentOS Linux 7 (Core)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://18.9.7
Kubelet Version: v1.16.3
Kube-Proxy Version: v1.16.3
PodCIDR: 10.100.0.0/24
PodCIDRs: 10.100.0.0/24
Non-terminated Pods: (6 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system calico-node-v2rmv 250m (6%) 0 (0%) 0 (0%) 0 (0%) 4h15m
kube-system etcd-master 0 (0%) 0 (0%) 0 (0%) 0 (0%) 362d
kube-system kube-apiserver-master 250m (6%) 0 (0%) 0 (0%) 0 (0%) 362d
kube-system kube-controller-manager-master 200m (5%) 0 (0%) 0 (0%) 0 (0%) 362d
kube-system kube-proxy-k7shj 0 (0%) 0 (0%) 0 (0%) 0 (0%) 4h15m
kube-system kube-scheduler-master 100m (2%) 0 (0%) 0 (0%) 0 (0%) 362d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 800m (20%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
Events: <none>接着查看日志,提示node "master" not found
[root@master ~]# journalctl -u kubelet -f
-- Logs begin at Sun 2021-10-17 15:04:51 CST. --
Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.159602 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.259813 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.329948 34678 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: services is forbidden: User "system:anonymous" cannot list resource "services" in API group "" at the cluster scope
Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.360614 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.460822 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.531167 34678 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: nodes "master" is forbidden: User "system:anonymous" cannot list resource "nodes" in API group "" at the cluster scope
Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.560990 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.661174 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.731398 34678 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" at the cluster scope
Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.761342 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.861818 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.930789 34678 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User "system:anonymous" cannot list resource "csidrivers" in API group "storage.k8s.io" at the cluster scope
Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.962003 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.062175 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.131629 34678 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User "system:anonymous" cannot list resource "runtimeclasses" in API group "node.k8s.io" at the cluster scope
Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.162315 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.262472 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.331431 34678 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: services is forbidden: User "system:anonymous" cannot list resource "services" in API group "" at the cluster scope
Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.362644 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.462865 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.532574 34678 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: nodes "master" is forbidden: User "system:anonymous" cannot list resource "nodes" in API group "" at the cluster scope
Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.563028 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.663176 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.732705 34678 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" at the cluster scope
Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.763328 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.863481 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.932104 34678 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User "system:anonymous" cannot list resource "csidrivers" in API group "storage.k8s.io" at the cluster scope
Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.963637 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:20 master kubelet[34678]: E1221 14:56:20.063823 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:20 master kubelet[34678]: E1221 14:56:20.133218 34678 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User "system:anonymous" cannot list resource "runtimeclasses" in API group "node.k8s.io" at the cluster scope
Dec 21 14:56:20 master kubelet[34678]: E1221 14:56:20.164019 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:20 master kubelet[34678]: E1221 14:56:20.264632 34678 kubelet.go:2267] node "master" not found
Dec 21 14:56:20 master kubelet[34678]: E1221 14:56:20.332795 34678 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: services is forbidden: User "system:anonymous" cannot list resource "services" in API group "" at the cluster scope
Dec 21 14:56:20 master kubelet[34678]: E1221 14:56:20.364844 34678 kubelet.go:2267] node "master" not found然而此时master的kubelet 是runing的，但是没有状态还是没有ready,接着我重启了docker程序，仍然没有进展，反倒让proxy和coredns，calico Pending了。而etcd,controlle,api-server却是正常running的。
由于项目太旧，没有任何文档，半年前做过证书更新，于是查看证书
[root@master ]# kubeadm alpha certs check-expiration
CERTIFICATE EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
admin.conf Dec 22, 2030 06:19 UTC 9y no
apiserver Dec 22, 2030 06:19 UTC 9y no
apiserver-etcd-client Dec 22, 2030 06:19 UTC 9y no
apiserver-kubelet-client Dec 22, 2030 06:19 UTC 9y no
controller-manager.conf Dec 22, 2030 06:19 UTC 9y no
etcd-healthcheck-client Dec 22, 2030 06:19 UTC 9y no
etcd-peer Dec 22, 2030 06:19 UTC 9y no
etcd-server Dec 22, 2030 06:19 UTC 9y no
front-proxy-client Dec 22, 2030 06:19 UTC 9y no
scheduler.conf Dec 22, 2030 06:19 UTC 9y no发现证书也没有问题而后我们确认master是否配置正确
[root@master kubernetes]# cat kubelet.conf
apiVersion: v1
clusters:
- cluster:
certificate-authority-data:
server: https://master:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
user:
client-certificate-data:
client-key-data: L在查看hosts
[root@master kubernetes]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.133.6 master一切都没有问题，
现在怀疑是kubelet.conf的证书权限有问题，于是将node的kubelet拿到master尝试。发现还是没有解决
最后在尝试将master的admin.conf替换成kubelet.conf
[root@master kubernetes]# systemctl restart kubelet查看日志
[root@master kubernetes]# journalctl -u kubelet -f
-- Logs begin at Sun 2021-10-31 03:30:27 CST. --
Dec 21 15:15:48 master kubelet[39135]: I1221 15:15:48.782529 39135 reconciler.go:301] Volume detached for volume "calico-node-token-sfzfw" (UniqueName: "kubernetes.io/secret/b7039ad3-f568-47b7-a323-737898628333-calico-node-token-sfzfw") on node "master" DevicePath ""
Dec 21 15:15:48 master kubelet[39135]: I1221 15:15:48.782541 39135 reconciler.go:301] Volume detached for volume "coredns-token-6rmpw" (UniqueName: "kubernetes.io/secret/bf57f81b-911a-48b7-aa1e-624193cd7c51-coredns-token-6rmpw") on node "master" DevicePath ""
Dec 21 15:15:48 master kubelet[39135]: I1221 15:15:48.782552 39135 reconciler.go:301] Volume detached for volume "config-volume" (UniqueName: "kubernetes.io/configmap/f4d237c7-2212-4c91-8535-4cad2bb88644-config-volume") on node "master" DevicePath ""
Dec 21 15:15:48 master kubelet[39135]: I1221 15:15:48.782564 39135 reconciler.go:301] Volume detached for volume "coredns-token-6rmpw" (UniqueName: "kubernetes.io/secret/f4d237c7-2212-4c91-8535-4cad2bb88644-coredns-token-6rmpw") on node "master" DevicePath ""
Dec 21 15:15:48 master kubelet[39135]: W1221 15:15:48.993096 39135 kubelet_getters.go:292] Path "/var/lib/kubelet/pods/3114871e-b583-4e02-a8f3-b63a25b2dffa/volumes" does not exist
Dec 21 15:15:49 master kubelet[39135]: W1221 15:15:49.215461 39135 kubelet_getters.go:292] Path "/var/lib/kubelet/pods/02f1318a-3b06-48cf-a96e-83322f66c17b/volumes" does not exist
Dec 21 15:15:49 master kubelet[39135]: W1221 15:15:49.215524 39135 kubelet_getters.go:292] Path "/var/lib/kubelet/pods/b7039ad3-f568-47b7-a323-737898628333/volumes" does not exist
Dec 21 15:15:49 master kubelet[39135]: W1221 15:15:49.215548 39135 kubelet_getters.go:292] Path "/var/lib/kubelet/pods/f4d237c7-2212-4c91-8535-4cad2bb88644/volumes" does not exist
Dec 21 15:15:49 master kubelet[39135]: W1221 15:15:49.215570 39135 kubelet_getters.go:292] Path "/var/lib/kubelet/pods/83515e8e-8ba5-459d-9bc6-3d897a4f18e9/volumes" does not exist
Dec 21 15:15:49 master kubelet[39135]: W1221 15:15:49.215593 39135 kubelet_getters.go:292] Path "/var/lib/kubelet/pods/bf57f81b-911a-48b7-aa1e-624193cd7c51/volumes" does not exist
Dec 21 15:15:49 master kubelet[39135]: I1221 15:15:49.384045 39135 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "lib-modules" (UniqueName: "kubernetes.io/host-path/809554bc-4e5c-4645-b8dd-9963182a8196-lib-modules") pod "calico-node-k4vx7" (UID: "809554bc-4e5c-4645-b8dd-9963182a8196")
Dec 21 15:15:49 master kubelet[39135]: I1221 15:15:49.384101 39135 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/809554bc-4e5c-4645-b8dd-9963182a8196-xtables-lock") pod "calico-node-k4vx7" (UID: "809554bc-4e5c-4645-b8dd-9963182a8196")
Dec 21 15:15:49 master kubelet[39135]: I1221 15:15:49.384152 39135 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "flexvol-driver-host" (UniqueName: "kubernetes.io/host-path/809554bc-4e5c-4645-b8dd-9963182a8196-flexvol-driver-host") pod "calico-node-k4vx7" (UID: "809554bc-4e5c-4645-b8dd-9963182a8196")
Dec 21 15:15:49 master kubelet[39135]: I1221 15:15:49.384194 39135 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "cni-net-dir" (UniqueName: "kubernetes.io/host-path/809554bc-4e5c-4645-b8dd-9963182a8196-cni-net-dir") pod "calico-node-k4vx7" (UID: "809554bc-4e5c-4645-b8dd-9963182a8196")
Dec 21 15:15:49 master kubelet[39135]: I1221 15:15:49.384233 39135 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "host-local-net-dir" (UniqueName: "kubernetes.io/host-path/809554bc-4e5c-4645-b8dd-9963182a8196-host-local-net-dir") pod "calico-node-k4vx7" (UID: "809554bc-4e5c-4645-b8dd-9963182a8196")
Dec 21 15:15:49 master kubelet[39135]: I1221 15:15:49.384271 39135 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "policysync" (UniqueName: "kubernetes.io/host-path/809554bc-4e5c-4645-b8dd-9963182a8196-policysync") pod "calico-node-k4vx7" (UID: "809554bc-4e5c-4645-b8dd-9963182a8196")
Dec 21 15:15:49 master kubelet[39135]: I1221 15:15:49.384305 39135 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "calico-node-token-sfzfw" (UniqueName: "kubernetes.io/secret/809554bc-4e5c-4645-b8dd-9963182a8196-calico-node-token-sfzfw") pod "calico-node-k4vx7" (UID: "809554bc-4e5c-4645-b8dd-9963182a8196")
Dec 21 15:15:49 master kubelet[39135]: I1221 15:15:49.384332 39135 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "cni-bin-dir" (UniqueName: "kubernetes.io/host-path/809554bc-4e5c-4645-b8dd-9963182a8196-cni-bin-dir") pod "calico-node-k4vx7" (UID: "809554bc-4e5c-4645-b8dd-9963182a8196")
Dec 21 15:15:49 master kubelet[39135]: I1221 15:15:49.384369 39135 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "var-run-calico" (UniqueName: "kubernetes.io/host-path/809554bc-4e5c-4645-b8dd-9963182a8196-var-run-calico") pod "calico-node-k4vx7" (UID: "809554bc-4e5c-4645-b8dd-9963182a8196")
Dec 21 15:15:49 master kubelet[39135]: I1221 15:15:49.384398 39135 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "var-lib-calico" (UniqueName: "kubernetes.io/host-path/809554bc-4e5c-4645-b8dd-9963182a8196-var-lib-calico") pod "calico-node-k4vx7" (UID: "809554bc-4e5c-4645-b8dd-9963182a8196")
Dec 21 15:15:49 master kubelet[39135]: W1221 15:15:49.986228 39135 kubelet_getters.go:292] Path "/var/lib/kubelet/pods/754ff5ae-8b4f-4f67-9027-68b5f59a8b90/volumes" does not exist
Dec 21 15:15:50 master kubelet[39135]: I1221 15:15:50.387442 39135 reconciler.go:207]问题暂时得到解决
[root@master kubernetes]# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-dc4d48847-mmwdk 1/1 Running 0 4h30m
kube-system calico-node-9z59b 1/1 Running 0 21m
kube-system calico-node-k4vx7 1/1 Running 0 22m
kube-system calico-node-xpktb 1/1 Running 0 21m
kube-system coredns-dffb59cff-4tnrn 1/1 Running 0 4h30m
kube-system coredns-dffb59cff-hjt7v 1/1 Running 0 4h30m
kube-system eip-nfs-minio-6c44f59b76-nq8fm 1/1 Running 4 237d
kube-system etcd-master 1/1 Running 5 362d
kube-system kube-apiserver-master 1/1 Running 3 362d
kube-system kube-controller-manager-master 1/1 Running 244 362d
kube-system kube-proxy-7d5qb 1/1 Running 0 22m
kube-system kube-proxy-k755k 1/1 Running 0 21m
kube-system kube-proxy-pxhb6 1/1 Running 0 22m
kube-system kube-scheduler-master 1/1 Running 241 362d
kube-system kuboard-7d745b566-wwp8w 1/1 Running 0 42d
kube-system metrics-server-7dcd7b44c-dvpp4 1/1 Running 0 87d[root@master kubernetes]# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready,SchedulingDisabled master 647d v1.16.3
worker1 Ready <none> 647d v1.16.3
worker2 Ready <none> 647d v1.16.3[root@master kubernetes]# kubectl describe node master
Name: master
Roles: master
...
Unschedulable: true
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Tue, 21 Dec 2021 15:15:52 +0800 Tue, 21 Dec 2021 15:15:52 +0800 CalicoIsUp Calico is running on this node
MemoryPressure False Tue, 21 Dec 2021 15:46:58 +0800 Tue, 21 Dec 2021 15:15:47 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Tue, 21 Dec 2021 15:46:58 +0800 Tue, 21 Dec 2021 15:15:47 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Tue, 21 Dec 2021 15:46:58 +0800 Tue, 21 Dec 2021 15:15:47 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Tue, 21 Dec 2021 15:46:58 +0800 Tue, 21 Dec 2021 15:15:47 +0800 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 192.168.133.6
Hostname: master
Capacity:
cpu: 4
ephemeral-storage: 36805060Ki
hugepages-2Mi: 0
memory: 16259676Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 33919543240
hugepages-2Mi: 0
memory: 15211100Ki
pods: 110
System Info:
Machine ID: abfbcf67c72dca40affb0cd0c8debd6d
System UUID: 63B3A0C6-AEB7-3841-BA81-2724178E0890
Boot ID: 02424d77-4b14-45ea-a632-038dbc70a2c4
Kernel Version: 3.10.0-957.el7.x86_64
OS Image: CentOS Linux 7 (Core)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://18.9.7
Kubelet Version: v1.16.3
Kube-Proxy Version: v1.16.3
PodCIDR: 10.100.0.0/24
PodCIDRs: 10.100.0.0/24
Non-terminated Pods: (6 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system calico-node-k4vx7 250m (6%) 0 (0%) 0 (0%) 0 (0%) 31m
kube-system etcd-master 0 (0%) 0 (0%) 0 (0%) 0 (0%) 362d
kube-system kube-apiserver-master 250m (6%) 0 (0%) 0 (0%) 0 (0%) 362d
kube-system kube-controller-manager-master 200m (5%) 0 (0%) 0 (0%) 0 (0%) 362d
kube-system kube-proxy-pxhb6 0 (0%) 0 (0%) 0 (0%) 0 (0%) 31m
kube-system kube-scheduler-master 100m (2%) 0 (0%) 0 (0%) 0 (0%) 362d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 800m (20%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 31m kubelet, master Starting kubelet.
Normal NodeHasSufficientMemory 31m (x2 over 31m) kubelet, master Node master status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 31m (x2 over 31m) kubelet, master Node master status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 31m (x2 over 31m) kubelet, master Node master status is now: NodeHasSufficientPID
Normal NodeNotReady 31m kubelet, master Node master status is now: NodeNotReady
Normal NodeNotSchedulable 31m kubelet, master Node master status is now: NodeNotSchedulable
Normal NodeAllocatableEnforced 31m kubelet, master Updated Node Allocatable limit across pods
Normal NodeReady 31m kubelet, master Node master status is now: NodeReady
Normal Starting 31m kube-proxy, master Starting kube-proxy.延伸阅读
bootstrap-kubelet.conf: no such file or directory kubelet证书轮换失败
