改めてだが、k3sの単体サーバを１台作ってみる。k3sはライトウェイトなKubernetesで、kindみたいなめんどくさいこともない。ただし、Loadbalancerとかはそのままだと難しいかもしれない。
今回は、ストレージとして、Longhornも立ててみる。
k3sで作成されるものは以下。
CNI:funnel
Loadbalancer:servicelb(Klipper Service Load Balancer)
Ingress: traefik
前提条件
ハードウェアとしては、AMD64あるいはARM64の環境
OS
Ubuntu Server 20.04でクリーンインストールのもの。前提パッケージや事前のOS設定も一切不要。
ホスト名とIPアドレスが設定されていればOK
リソース
1 CPU / 1GB RAM以上。（もっとあったほうがいい）
HDDディスクは、/dev/sdaとして、20GB＋コンテナストレージ容量 (Longhornはノードのローカルパスを使う)
Network
1 NIC (1IP) 適当な構成でいいということ。
k3sインストール
sudo -i
curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=v1.23 K3S_KUBECONFIG_MODE="644" sh -
実行結果（こっそりARM64で構築してみた。)
[INFO] Finding release for channel v1.23
[INFO] Using v1.23.9+k3s1 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.23.9+k3s1/sha256sum-arm64.txt
[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.23.9+k3s1/k3s-arm64
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Skipping installation of SELinux RPM
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
確認
1分くらい待って、以下を実行。
kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-test Ready control-plane,master,worker 53m v1.23.9+k3s1 172.16.155.129 <none> Ubuntu 20.04.4 LTS 5.4.0-122-generic containerd://1.5.13-k3s1
RunningあるいはCompletedになっていることを確認
kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system local-path-provisioner-6c79684f77-67mr4 1/1 Running 0 44s
kube-system coredns-d76bd69b-xl4zd 1/1 Running 0 44s
kube-system helm-install-traefik-crd-vd2vp 0/1 Completed 0 44s
kube-system helm-install-traefik-x248v 0/1 Completed 1 44s
kube-system svclb-traefik-3c6c1b61-c62pj 2/2 Running 0 20s
kube-system metrics-server-7cd5fcb6b7-84srv 1/1 Running 0 44s
kube-system traefik-df4ff85d6-wfckp 1/1 Running 0 20s
kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 66s
kube-system kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 62s
kube-system metrics-server ClusterIP 10.43.192.17 <none> 443/TCP 61s
kube-system traefik LoadBalancer 10.43.191.148 192.168.1.69 80:30717/TCP,443:30638/TCP 27s
kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 66s
追加設定
補完設定やhelmの設定をする。
mkdir -p ~/.kube
cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
chmod 600 ~/.kube/config
kubectl label node `hostname` node-role.kubernetes.io/worker=worker
kubectl completion bash > /etc/bash_completion.d/kubectl
source /etc/bash_completion.d/kubectl
crictl completion bash > /etc/bash_completion.d/crictl
source /etc/bash_completion.d/crictl
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
rm ./get_helm.sh
helm completion bash > /etc/bash_completion.d/helm
source /etc/bash_completion.d/helm
Longhornのインストール
External Snapshotterのインストール
CSIスナップショットも使いたいのでExternal Snapshotterを導入する。
SNAPSHOTTERVER=5.0.1
# Apply VolumeSnapshot CRDs
kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v${SNAPSHOTTERVER}/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v${SNAPSHOTTERVER}/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
kubectl create -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v${SNAPSHOTTERVER}/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml
# Create Snapshot Controller
curl --retry 10 --retry-delay 3 --retry-connrefused -sSOL https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v${SNAPSHOTTERVER}/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml
sed -i -e "s/namespace: default/namespace: kube-system/g" rbac-snapshot-controller.yaml
kubectl create -f rbac-snapshot-controller.yaml
rm -rf rbac-snapshot-controller.yaml
curl --retry 10 --retry-delay 3 --retry-connrefused -sSOL https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v${SNAPSHOTTERVER}/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml
sed -i -e "s/namespace: default/namespace: kube-system/g" setup-snapshot-controller.yaml
kubectl create -f setup-snapshot-controller.yaml
rm setup-snapshot-controller.yaml
kubectl -n kube-system wait pod -l app=snapshot-controller --for condition=Ready --timeout 60s
確認
30秒くらい待って、以下を実行。すべてがRunningになることを確認
kubectl get pods -A | grep snapshot
kube-system snapshot-controller-8496df95fc-58vlj 1/1 Running 0 75s
kube-system snapshot-controller-8496df95fc-9hfvz 1/1 Running 0 75s
kubectl get crd | grep volumesnapshot
volumesnapshotclasses.snapshot.storage.k8s.io 2022-08-06T11:45:04Z
volumesnapshotcontents.snapshot.storage.k8s.io 2022-08-06T11:45:06Z
volumesnapshots.snapshot.storage.k8s.io 2022-08-06T11:45:07Z
Longhornのインストール（本体）
事前準備
sed -i -e "s/debian/debian.$(hostname)/g" /etc/iscsi/initiatorname.iscsi
systemctl restart iscsid.service
apt -y install nfs-common jq
Longhornのインストール
curl -sSfL https://raw.githubusercontent.com/longhorn/longhorn/master/scripts/environment_check.sh | bash
LONGHORNVER=1.3.1
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v${LONGHORNVER}/deploy/longhorn.yaml
cat <<EOF | kubectl apply -f -
kind: VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1
metadata:
annotations:
snapshot.storage.kubernetes.io/is-default-class: "true"
name: longhorn
driver: driver.longhorn.io
deletionPolicy: Delete
EOF
kubectl create -f https://raw.githubusercontent.com/longhorn/longhorn/v${LONGHORNVER}/deploy/backupstores/minio-backupstore.yaml
kubectl -n longhorn-system wait pod -l app=longhorn-manager --for condition=Ready --timeout 180s
kubectl -n longhorn-system wait pod -l app=longhorn-conversion-webhook --for condition=Ready --timeout 180s
kubectl -n longhorn-system wait pod -l app=longhorn-driver-deployer --for condition=Ready --timeout 180s
kubectl wait pod -l app=longhorn-test-minio --for condition=Ready --timeout 180s
kubectl -n longhorn-system wait pod -l app=longhorn-ui --for condition=Ready --timeout 180s
確認
３分くらい待って、以下を実行。すべてがRunningになることを確認
kubectl -n longhorn-system get pods
NAME READY STATUS RESTARTS AGE
longhorn-ui-759876b84b-bjrr6 1/1 Running 0 2m26s
longhorn-conversion-webhook-f47668987-678hv 1/1 Running 0 2m26s
longhorn-conversion-webhook-f47668987-99bpz 1/1 Running 0 2m26s
longhorn-admission-webhook-6f4f8467d9-22dmn 1/1 Running 0 2m26s
longhorn-manager-4lmfc 1/1 Running 0 2m26s
longhorn-driver-deployer-5db8697569-w9n7h 1/1 Running 0 2m26s
longhorn-admission-webhook-6f4f8467d9-2rrf7 1/1 Running 0 2m26s
instance-manager-r-2c344cae 1/1 Running 0 80s
instance-manager-e-ff337275 1/1 Running 0 80s
csi-attacher-7bf4b7f996-49wkf 1/1 Running 0 60s
csi-provisioner-869bdc4b79-6sthc 1/1 Running 0 60s
csi-attacher-7bf4b7f996-6bw2d 1/1 Running 0 60s
csi-resizer-6d8cf5f99f-4pd5b 1/1 Running 0 60s
csi-provisioner-869bdc4b79-lstgc 1/1 Running 0 60s
csi-provisioner-869bdc4b79-t46nt 1/1 Running 0 60s
csi-attacher-7bf4b7f996-zn7np 1/1 Running 0 60s
csi-snapshotter-588457fcdf-n5r2x 1/1 Running 0 59s
csi-resizer-6d8cf5f99f-nxd2c 1/1 Running 0 60s
csi-snapshotter-588457fcdf-7ddlq 1/1 Running 0 59s
csi-resizer-6d8cf5f99f-sfsxm 1/1 Running 0 60s
csi-snapshotter-588457fcdf-vtdw7 1/1 Running 0 59s
longhorn-csi-plugin-5798c 2/2 Running 0 59s
engine-image-ei-df38d2e5-n6gfq 1/1 Running 0 80s
デフォルトストレージの変更
kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
確認
kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-path rancher.io/local-path Delete WaitForFirstConsumer false 14m
longhorn (default) driver.longhorn.io Delete Immediate true 2m41s
LonghornのCSI Snapshotの設定
kubectl port-forward deployment/longhorn-ui –address 0.0.0.0 7000:8000 -n longhorn-system
ブラウザで以下を開く
http://<Node IP>:7000/#/setting
以下の値を入力して画面の一番下にあるSaveをクリック
Backup Target : s3://backupbucket@us-east-1/
Backup Target Credential Secret : minio-secret
まず、基本的な構築はこんな感じ。