catalin/fukuops
1
0
Fork 1
Code Issues 9 Pull requests Projects 2 Releases Packages 10 Activity
fukuops/METALLB.md
cătălin a3eb40f772
 feat: add metallb
2026-08-24 13:18:52 +02:00

7.2 KiB
Raw Blame History

Initial problem

the k3s cluster iss reversed proxied by an nginx in a small lxc (fuku.nginx) which acts as the exit point to the internet, so the isp router port forwards the ports 80 and 443 to nginx which in turn has a config file for each service in k8s (and other hosts, but those are just a few). I'd like to sundown nginx since I don't have a gitops way of managing each service nginx config file. I have three master nodes (high availability) but my isp control panel does't allow me to put three different nodes and I can't do dns load-balancing either since I can't change the router hosts file or anything.

Install MetalLB in the cluster

Solution: MetalLB in L2 mode: MetalLB elects one node to own the VIP via ARP/NDP (gratuitous ARP) and answers for it on the LAN. If that node dies, another node takes over the VIP within a few seconds — it's essentially a Kubernetes-native keepalived. You port-forward your ISP router to that single VIP, and MetalLB handles which physical node is actually answering at any given time.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: ingress-pool
  namespace: metallb-system
spec:
  addresses:
    - 192.168.1.240/32   # single reserved IP for Traefik's LB
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: ingress-l2
  namespace: metallb-system
spec:
  ipAddressPools: [ingress-pool]

Installation

  • https://artifacthub.io/packages/helm/metallb/metallb
  • Install it in L2 mode in an argo-app but don't touch Traefik's existing Service yet. At this point MetalLB is running but idle — nothing's using it.
  • Test MetalLB on a throwaway Service (a temporary nginx LoadBalancer Service with a test IP from your pool) to confirm ARP/failover actually works on your LAN before touching anything real
  • Cut Traefik over: since ServiceLB only manages Services that don't already have an IP it recognizes, the actual switch happens when you either (a) delete and recreate Traefik's Service so MetalLB's webhook claims it, or (b) explicitly disable ServiceLB at that point so MetalLB becomes the only controller answering LoadBalancer requests, then bounce Traefik's Service. This step causes a brief blip (seconds) as the Service gets a new external IP — which is also when you update your router's port-forward target to the new MetalLB VIP.

Implementation runbook

Generated during the build session. Status is updated as steps complete. Branch: feat/metallb (local only, not pushed). Method: Argo app created with kubectl apply; synced with argocd app sync --local. Target VIP: 192.168.1.126/32. Test IP: 192.168.1.125/32.

Pre-execution decisions

  • Single Traefik VIP architecture.
  • Nginx already proxies to Traefik.
  • Local argocd app sync --local workflow.

Steps and status

  • 1. Update this file (METALLB.md) with the runbook.
  • 2. Create local feat/metallb branch.
  • 3. SSH connectivity check: fuku.master1, fuku.master2, fuku.k3m3.
  • 4. Add MetalLB Helm repo to management AppProject sourceRepos.
  • 5. Create MetalLB Argo app + manifests (test pool 192.168.1.125/32, autoAssign: false).
  • 6. Apply MetalLB Argo app and sync. argocd CLI token expired; used helm upgrade --install as fallback. Argo app remains in place.
  • 7. Test MetalLB with throwaway nginx LoadBalancer service. Finding: ServiceLB and MetalLB fight over the Service (EXTERNAL-IP flaps between 192.168.1.125 and <pending>), confirming they cannot coexist. Must disable ServiceLB before cutover.
  • 8. Prepare production manifests: ingress-pool (192.168.1.126/32, autoAssign: false), converted web apps to ClusterIP/NodePort.
  • 9. Disabled k3s ServiceLB on fuku.master1, fuku.master2, fuku.k3m3 and restarted k3s. All nodes Ready; ServiceLB pods terminated.
  • 10. Applied production Argo changes. Traefik Service assigned 192.168.1.126. Web apps converted to ClusterIP; AdGuard DNS and Portainer changed to NodePort.
  • 11. Validated HTTPS ingress via VIP (argo.fuku, auth.fukurokuju.dev, git.roboces.dev all respond). Failover test deferred to final verification.
  • 12. Ask user to update ISP router port-forward 80/443 → 192.168.1.126.
  • 13. Fix AdGuard DNS by giving it dedicated MetalLB IP 192.168.1.125. DNS resolution and web UI (dns.fuku) verified working.
  • 14. Final verification after router update + nginx decommission (user handled).

Rollback commands (keep handy)

If MetalLB test fails:

kubectl delete svc nginx-metallb-test -n default
kubectl delete -f k8s/argo-apps/metallb.yaml
kubectl delete ns metallb-system

If k3s ServiceLB disable causes problems (run on each master):

# Remove --disable servicelb from k3s server flags, then:
sudo systemctl restart k3s

If production cutover breaks:

# Re-apply main branch manifests to restore ServiceLB LoadBalancers
git checkout main
kubectl apply -f k8s/services/kube-system/
# Re-enable ServiceLB if it was disabled

Post-cutover state

What works now

  • Traefik is reachable on the MetalLB VIP 192.168.1.126.
  • HTTPS ingress for argo.fuku, auth.fukurokuju.dev, git.roboces.dev responds via the VIP.
  • ServiceLB (svclb-* pods) has been disabled on all three master nodes.
  • Web apps in this repo were converted from LoadBalancer to ClusterIP and route through Traefik.
  • Portainer was switched to NodePort to keep the Edge port available without stealing the ingress VIP.
  • AdGuard DNS was given its own MetalLB IP (192.168.1.125) and is resolving queries again.

IP allocation

Service IP Pool Notes
Traefik ingress 192.168.1.126 ingress-pool Router forwards 80/443 here
AdGuard DNS 192.168.1.125 adguard-dns-pool LAN DNS resolver on standard port 53

Known leftovers (not managed by this repo)

These services are still LoadBalancer and will stay <pending> until they are migrated separately (different repos / operators / manual installs):

  • apps-fuku/ak-outpost-proxy (Authentik outpost)
  • apps-fuku/factorio-factorio-server-charts-rcon
  • apps-fuku/psql15-postgres
  • apps-roboces/huesoporro (chart lives in git.roboces.dev/catalin/huesoporro.git)
  • apps-roboces/valkey-valkey-cluster
  • default/prometheus-main

Because both pools have autoAssign: false and only Traefik/AdGuard carry the matching pool annotations, these leftover services cannot steal either IP.

Deviation from original plan

  • argocd CLI token was expired and port-forward login failed, so the MetalLB chart was installed with helm upgrade --install instead of argocd app sync. The Argo app metallb was still created and will take over once ArgoCD login is restored.
  • Service conversions were applied by updating the Argo app manifests with kubectl apply and letting ArgoCD auto-sync, rather than argocd app sync --local.

Next step for the user

Update the ISP router to port-forward TCP 80 and TCP 443 to 192.168.1.126. After that, nginx can be decommissioned at your convenience.