2021/01/12
今回できたこと
未解決
今後の作業
お約束
文章化作業
定例作業
参考文献
K3sクラスタTimeoutの原因追求
code:shell
pi@master-rpi4$ journalctl -a --no-pager -u k3s
code:shell
# 正常終了時
master-rpi4$ sudo kubectl get node; echo $?
NAME STATUS ROLES AGE VERSION
worker-rpi4-2 NotReady <none> 4d2h v1.20.0+k3s2
worker-rpi4-1 Ready <none> 6d v1.20.0+k3s2
master-rpi4 Ready control-plane,master 6d2h v1.20.0+k3s2
0
# 異常終了時
master-rpi4$ sudo kubectl get node; echo $?
Unable to connect to the server: context deadline exceeded
1
code:shell
master-rpi4-132807$ export PS1="\h-\$(date +"%H%M%S")\$ "
master-rpi4-141630$ sudo systemctl restart k3s
master-rpi4-141757$ while true; do date;sudo kubectl get node; if $? != 0 ;then break; fi;sleep 1; done 2021年 1月 12日 火曜日 14:18:02 JST
NAME STATUS ROLES AGE VERSION
worker-rpi4-2 NotReady <none> 4d3h v1.20.0+k3s2
master-rpi4 Ready control-plane,master 6d3h v1.20.0+k3s2
worker-rpi4-1 Ready <none> 6d1h v1.20.0+k3s2
(snip)
2021年 1月 12日 火曜日 14:18:58 JST
NAME STATUS ROLES AGE VERSION
worker-rpi4-1 Ready <none> 6d1h v1.20.0+k3s2
worker-rpi4-2 NotReady <none> 4d3h v1.20.0+k3s2
master-rpi4 Ready control-plane,master 6d3h v1.20.0+k3s2
2021年 1月 12日 火曜日 14:19:00 JST
Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get nodes)
master-rpi4-142001$
code:log
code:shell
Tasks: 151 total, 2 running, 149 sleeping, 0 stopped, 0 zombie
%Cpu(s): 74.2 us, 17.4 sy, 0.1 ni, 8.2 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st
MiB Mem : 3827.9 total, 1771.7 free, 1385.9 used, 670.3 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 2310.7 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
615 root 20 0 1489312 1.2g 77428 S 335.2 31.6 1321:27 k3s-server
886 root 20 0 892244 94708 60628 S 0.7 2.4 3:27.40 containerd
2067 root 20 0 829584 35252 25564 S 0.7 0.9 1:43.78 coredns
10 root 20 0 0 0 0 I 0.3 0.0 0:19.25 rcu_sched
1901 root 20 0 802812 10344 5624 S 0.3 0.3 0:10.91 container+
2037 root 20 0 865284 48324 36344 S 0.3 1.2 1:28.05 traefik
5002 pi 20 0 6980 3024 2484 R 0.3 0.1 1:15.56 top
1 root 20 0 34032 8472 6540 S 0.0 0.2 0:19.90 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.04 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp
8 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_percpu+
9 root 20 0 0 0 0 S 0.0 0.0 0:01.84 ksoftirqd+
11 root rt 0 0 0 0 S 0.0 0.0 0:00.17 migration+
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/1
14 root rt 0 0 0 0 S 0.0 0.0 0:00.16 migration+
15 root 20 0 0 0 0 S 0.0 0.0 0:01.69 ksoftirqd+
18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/2
19 root rt 0 0 0 0 S 0.0 0.0 0:00.17 migration+
20 root 20 0 0 0 0 S 0.0 0.0 0:01.57 ksoftirqd+
23 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/3
24 root rt 0 0 0 0 S 0.0 0.0 0:00.16 migration+
25 root 20 0 0 0 0 S 0.0 0.0 0:01.70 ksoftirqd+
28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
29 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 netns oup_pidlist_d
32 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kauditd
33 root 20 0 0 0 0 S 0.0 0.0 0:00.01 khungtaskd
34 root 20 0 0 0 0 S 0.0 0.0 0:00.00 oom_reaper
35 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 writeback
36 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kcompactd0
54 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kblockd
55 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 blkcg_pun+
58 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdogd
59 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rpciod
60 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/u+
packet_write_wait: Connection to fe80::eba6:a598:54a2:c1e6%en1 port 22: Broken pipe 0:top* 1:bash- "master-rpi4" 21:17 12- 1月-2111111111211112121
mutoubushinokonpyuta-2 ~ %