CraftLab Management

Gracefully turn off CraftLab

For single systems, from the machine itself or over SSH:

# immediate power off
sudo poweroff

# or
sudo shutdown -h now

# schedule in N minutes (and announce)
sudo shutdown -h +5 "Homelab shutting down in 5 minutes"

# cancel a scheduled shutdown
sudo shutdown -c

Gracefully drain your Kubernetes nodes

• On your kube-admin box:

# See nodes and roles
kubectl get nodes -o wide

# (a) Cordon & drain workers first (replace with your worker names)
kubectl cordon worker-1 worker-2
kubectl drain  worker-1 worker-2 --ignore-daemonsets --delete-emptydir-data --grace-period=60 --timeout=10m

# (b) Then the control-plane last (if you’re shutting EVERYTHING down)
kubectl cordon control-1
kubectl drain  control-1 --ignore-daemonsets --delete-emptydir-data --grace-period=60 --timeout=10m

• If you’ve got Longhorn/NFS storage, draining first ensures pods unmount volumes cleanly. If you’re just turning everything off, draining is still good practice—clean logs, clean termination.

After drains complete, power off the ThinkCentres:

ssh control-1 'sudo poweroff'
ssh worker-1  'sudo poweroff'
ssh worker-2  'sudo poweroff'

AdGuard/Pi-hole/Docker will stop cleanly with a normal shutdown:

ssh pi-dns1 'sudo poweroff'
ssh pi-dns2 'sudo poweroff'

Pi tip: wait ~20–30s until they stop responding to ping (or the green ACT LED is solid/off) before unplugging power—prevents SD corruption.

(Optional) Ping until down

Ping until down (watch one box go down cleanly):

watch -n1 'ping -c1 -W1 <host-or-ip> >/dev/null && echo up || echo down'

Waking things back up

ThinkCentres: enable Wake-on-LAN in BIOS/UEFI + NIC settings. From your PC:

brew install wakeonlan
wakeonlan <MAC-ADDRESS>

Pis: no WOL; you just plug power back in.

Uncordon both nodes (let the cluster schedule again)

# replace names as shown by `kubectl get nodes`
kubectl uncordon <control-node-name>
kubectl uncordon <worker-node-name>

kubectl get nodes -o wide

Your control node should now be Ready (without SchedulingDisabled). The worker will still say NotReady—we’ll fix that next.

Fix the worker’s NotReady (most common post-reboot causes)

SSH into the worker and run:

# A) swap must be OFF
sudo swapon --summary            # should show nothing
sudo swapoff -a                  # disable now
# if swap appears in /etc/fstab, comment that line and save:
sudo nano /etc/fstab             # put a '#' at the start of the swap line

# B) restart runtime + kubelet
sudo systemctl restart containerd
sudo systemctl restart kubelet

# C) quick health checks
sudo systemctl --no-pager --full status containerd | sed -n '1,25p'
sudo systemctl --no-pager --full status kubelet     | sed -n '1,25p'
timedatectl      # time should be correct, NTP service: yes
df -h / /var /var/lib/containerd   # ensure no disk pressure