diff --git a/METALLB.md b/METALLB.md new file mode 100644 index 0000000..29105d1 --- /dev/null +++ b/METALLB.md @@ -0,0 +1,128 @@ +## Initial problem +the k3s cluster iss reversed proxied by an nginx in a small lxc (fuku.nginx) which acts as the exit point to the internet, +so the isp router port forwards the ports 80 and 443 to nginx which in turn has a config file for each service in k8s (and other hosts, but those are just a few). +I'd like to sundown nginx since I don't have a gitops way of managing each service nginx config file. I have three master nodes (high availability) but my isp +control panel does't allow me to put three different nodes and I can't do dns load-balancing either since I can't change the router hosts file or anything. + + +## Install MetalLB in the cluster +Solution: MetalLB in L2 mode: MetalLB elects one node to own the VIP via ARP/NDP (gratuitous ARP) and answers for it on the LAN. +If that node dies, another node takes over the VIP within a few seconds — it's essentially a Kubernetes-native keepalived. +You port-forward your ISP router to that single VIP, and MetalLB handles which physical node is actually answering at any given time. + +```yaml +apiVersion: metallb.io/v1beta1 +kind: IPAddressPool +metadata: + name: ingress-pool + namespace: metallb-system +spec: + addresses: + - 192.168.1.240/32 # single reserved IP for Traefik's LB +--- +apiVersion: metallb.io/v1beta1 +kind: L2Advertisement +metadata: + name: ingress-l2 + namespace: metallb-system +spec: + ipAddressPools: [ingress-pool] +``` +### Installation + +- https://artifacthub.io/packages/helm/metallb/metallb +- Install it in L2 mode in an argo-app but don't touch Traefik's existing Service yet. At this point MetalLB is running but idle — nothing's using it. +- Test MetalLB on a throwaway Service (a temporary nginx LoadBalancer Service with a test IP from your pool) to confirm ARP/failover actually works on your LAN before touching anything real +- Cut Traefik over: since ServiceLB only manages Services that don't already have an IP it recognizes, the actual switch happens when you either (a) delete and recreate +Traefik's Service so MetalLB's webhook claims it, or (b) explicitly disable ServiceLB at that point so MetalLB becomes the only controller answering LoadBalancer requests, then bounce Traefik's Service. +This step causes a brief blip (seconds) as the Service gets a new external IP — which is also when you update your router's port-forward target to the new MetalLB VIP. + +--- + +## Implementation runbook + +> Generated during the build session. Status is updated as steps complete. +> Branch: `feat/metallb` (local only, not pushed). +> Method: Argo app created with `kubectl apply`; synced with `argocd app sync --local`. +> Target VIP: `192.168.1.126/32`. +> Test IP: `192.168.1.125/32`. + +### Pre-execution decisions +- [x] Single Traefik VIP architecture. +- [x] Nginx already proxies to Traefik. +- [x] Local `argocd app sync --local` workflow. + +### Steps and status +- [x] 1. Update this file (`METALLB.md`) with the runbook. +- [x] 2. Create local `feat/metallb` branch. +- [x] 3. SSH connectivity check: `fuku.master1`, `fuku.master2`, `fuku.k3m3`. +- [x] 4. Add MetalLB Helm repo to `management` AppProject sourceRepos. +- [x] 5. Create MetalLB Argo app + manifests (test pool `192.168.1.125/32`, `autoAssign: false`). +- [x] 6. Apply MetalLB Argo app and sync. `argocd` CLI token expired; used `helm upgrade --install` as fallback. Argo app remains in place. +- [x] 7. Test MetalLB with throwaway nginx LoadBalancer service. **Finding:** ServiceLB and MetalLB fight over the Service (`EXTERNAL-IP` flaps between `192.168.1.125` and ``), confirming they cannot coexist. Must disable ServiceLB before cutover. +- [x] 8. Prepare production manifests: `ingress-pool` (`192.168.1.126/32`, `autoAssign: false`), converted web apps to `ClusterIP`/NodePort. +- [x] 9. Disabled k3s ServiceLB on `fuku.master1`, `fuku.master2`, `fuku.k3m3` and restarted k3s. All nodes Ready; ServiceLB pods terminated. +- [x] 10. Applied production Argo changes. Traefik Service assigned `192.168.1.126`. Web apps converted to `ClusterIP`; AdGuard DNS and Portainer changed to `NodePort`. +- [x] 11. Validated HTTPS ingress via VIP (`argo.fuku`, `auth.fukurokuju.dev`, `git.roboces.dev` all respond). Failover test deferred to final verification. +- [ ] 12. Ask user to update ISP router port-forward 80/443 → `192.168.1.126`. +- [x] 13. Fix AdGuard DNS by giving it dedicated MetalLB IP `192.168.1.125`. DNS resolution and web UI (`dns.fuku`) verified working. +- [ ] 14. Final verification after router update + nginx decommission (user handled). + +### Rollback commands (keep handy) + +If MetalLB test fails: +```bash +kubectl delete svc nginx-metallb-test -n default +kubectl delete -f k8s/argo-apps/metallb.yaml +kubectl delete ns metallb-system +``` + +If k3s ServiceLB disable causes problems (run on each master): +```bash +# Remove --disable servicelb from k3s server flags, then: +sudo systemctl restart k3s +``` + +If production cutover breaks: +```bash +# Re-apply main branch manifests to restore ServiceLB LoadBalancers +git checkout main +kubectl apply -f k8s/services/kube-system/ +# Re-enable ServiceLB if it was disabled +``` + +--- + +## Post-cutover state + +### What works now +- Traefik is reachable on the MetalLB VIP `192.168.1.126`. +- HTTPS ingress for `argo.fuku`, `auth.fukurokuju.dev`, `git.roboces.dev` responds via the VIP. +- ServiceLB (`svclb-*` pods) has been disabled on all three master nodes. +- Web apps in this repo were converted from `LoadBalancer` to `ClusterIP` and route through Traefik. +- Portainer was switched to `NodePort` to keep the Edge port available without stealing the ingress VIP. +- AdGuard DNS was given its own MetalLB IP (`192.168.1.125`) and is resolving queries again. + +### IP allocation +| Service | IP | Pool | Notes | +|--------------------|----------------|---------------------|------------------------------------| +| Traefik ingress | 192.168.1.126 | `ingress-pool` | Router forwards 80/443 here | +| AdGuard DNS | 192.168.1.125 | `adguard-dns-pool` | LAN DNS resolver on standard port 53 | + +### Known leftovers (not managed by this repo) +These services are still `LoadBalancer` and will stay `` until they are migrated separately (different repos / operators / manual installs): +- `apps-fuku/ak-outpost-proxy` (Authentik outpost) +- `apps-fuku/factorio-factorio-server-charts-rcon` +- `apps-fuku/psql15-postgres` +- `apps-roboces/huesoporro` (chart lives in `git.roboces.dev/catalin/huesoporro.git`) +- `apps-roboces/valkey-valkey-cluster` +- `default/prometheus-main` + +Because both pools have `autoAssign: false` and only Traefik/AdGuard carry the matching pool annotations, these leftover services cannot steal either IP. + +### Deviation from original plan +- `argocd` CLI token was expired and port-forward login failed, so the MetalLB chart was installed with `helm upgrade --install` instead of `argocd app sync`. The Argo app `metallb` was still created and will take over once ArgoCD login is restored. +- Service conversions were applied by updating the Argo app manifests with `kubectl apply` and letting ArgoCD auto-sync, rather than `argocd app sync --local`. + +### Next step for the user +Update the ISP router to port-forward TCP 80 and TCP 443 to `192.168.1.126`. After that, nginx can be decommissioned at your convenience. diff --git a/README.md b/README.md index aff389b..35a621b 100644 --- a/README.md +++ b/README.md @@ -19,3 +19,5 @@ helm upgrade argo-cd argo/argo-cd -n argo-cd -f k8s/helm/argo-cd/values.yaml ```bash scripts/users.py add -u catalin -n cătălin -e catalin@roboces.dev ``` + +## MetalLB and reserved VIPs diff --git a/k8s/argo-apps/adguard.yaml b/k8s/argo-apps/adguard.yaml index de1880e..d7611da 100644 --- a/k8s/argo-apps/adguard.yaml +++ b/k8s/argo-apps/adguard.yaml @@ -41,6 +41,10 @@ spec: service: dns: type: LoadBalancer + port: 53 + annotations: + metallb.universe.tf/address-pool: adguard-dns-pool + loadBalancerIP: 192.168.1.125 web: - type: LoadBalancer + type: ClusterIP port: 8006 diff --git a/k8s/argo-apps/audiobookshelf.yaml b/k8s/argo-apps/audiobookshelf.yaml index c1863e2..d52b97a 100644 --- a/k8s/argo-apps/audiobookshelf.yaml +++ b/k8s/argo-apps/audiobookshelf.yaml @@ -16,7 +16,7 @@ spec: helm: valuesObject: service: - type: LoadBalancer + type: ClusterIP port: 8081 ingress: enabled: true diff --git a/k8s/argo-apps/authelia.yaml b/k8s/argo-apps/authelia.yaml index c3096c2..0bd9ddc 100644 --- a/k8s/argo-apps/authelia.yaml +++ b/k8s/argo-apps/authelia.yaml @@ -28,7 +28,7 @@ spec: cpu: 1 memory: 1024Mi service: - type: LoadBalancer + type: ClusterIP port: 9091 ingress: enabled: true diff --git a/k8s/argo-apps/authentik.yaml b/k8s/argo-apps/authentik.yaml index 9479e43..e421fa8 100644 --- a/k8s/argo-apps/authentik.yaml +++ b/k8s/argo-apps/authentik.yaml @@ -63,7 +63,7 @@ spec: enabled: true minAvailable: 1 service: - type: LoadBalancer + type: ClusterIP servicePortHttp: 9001 servicePortHttps: 9444 annotations: @@ -72,7 +72,7 @@ spec: metrics: enabled: true service: - type: LoadBalancer + type: ClusterIP serviceMonitor: enabled: false ingress: diff --git a/k8s/argo-apps/dokuwiki.yaml b/k8s/argo-apps/dokuwiki.yaml index 6e5216d..83e2daf 100644 --- a/k8s/argo-apps/dokuwiki.yaml +++ b/k8s/argo-apps/dokuwiki.yaml @@ -26,7 +26,7 @@ spec: drop: - all service: - type: LoadBalancer + type: ClusterIP port: 8004 ingress: enabled: true diff --git a/k8s/argo-apps/forgejo.yaml b/k8s/argo-apps/forgejo.yaml index 5f234e6..3ceecd3 100644 --- a/k8s/argo-apps/forgejo.yaml +++ b/k8s/argo-apps/forgejo.yaml @@ -20,7 +20,7 @@ spec: replicaCount: 1 service: http: - type: LoadBalancer + type: ClusterIP ssh: type: NodePort ingress: diff --git a/k8s/argo-apps/helm-rustical.yaml b/k8s/argo-apps/helm-rustical.yaml index 7fcba18..2962404 100644 --- a/k8s/argo-apps/helm-rustical.yaml +++ b/k8s/argo-apps/helm-rustical.yaml @@ -25,7 +25,7 @@ spec: targetCPUUtilizationPercentage: 80 targetMemoryUtilizationPercentage: 80 service: - type: LoadBalancer + type: ClusterIP port: 8001 targetPort: 4000 ingress: diff --git a/k8s/argo-apps/immich.yaml b/k8s/argo-apps/immich.yaml index 31e0c8b..65cca62 100644 --- a/k8s/argo-apps/immich.yaml +++ b/k8s/argo-apps/immich.yaml @@ -27,7 +27,7 @@ spec: drop: - all service: - type: LoadBalancer + type: ClusterIP config: database: hostname: "192.168.1.3" @@ -96,7 +96,7 @@ spec: machineLearning: enabled: true service: - type: LoadBalancer + type: ClusterIP secret: enabled: true existingSecretName: "immich-secrets" diff --git a/k8s/argo-apps/lldap.yaml b/k8s/argo-apps/lldap.yaml index c2218b2..577dfd3 100644 --- a/k8s/argo-apps/lldap.yaml +++ b/k8s/argo-apps/lldap.yaml @@ -17,7 +17,7 @@ spec: valuesObject: lldap: service: - type: LoadBalancer + type: ClusterIP httpPort: 17170 ldapPort: 3890 ingress: diff --git a/k8s/argo-apps/metallb.yaml b/k8s/argo-apps/metallb.yaml new file mode 100644 index 0000000..deb24b6 --- /dev/null +++ b/k8s/argo-apps/metallb.yaml @@ -0,0 +1,25 @@ +--- +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: metallb + namespace: argo-cd +spec: + destination: + name: '' + namespace: metallb-system + server: https://kubernetes.default.svc + sources: + - repoURL: https://metallb.github.io/metallb + chart: metallb + targetRevision: 0.16.* + helm: + valuesObject: + crds: + enabled: true + - repoURL: https://git.roboces.dev/catalin/fukuops.git + path: k8s/services/metallb + targetRevision: feat/metallb + project: management + syncPolicy: + automated: {} diff --git a/k8s/argo-apps/miniflux.yaml b/k8s/argo-apps/miniflux.yaml index 2339c6f..af22dc8 100644 --- a/k8s/argo-apps/miniflux.yaml +++ b/k8s/argo-apps/miniflux.yaml @@ -27,7 +27,7 @@ spec: drop: - all service: - type: LoadBalancer + type: ClusterIP ingress: enabled: true className: "traefik" diff --git a/k8s/argo-apps/oxicloud.yaml b/k8s/argo-apps/oxicloud.yaml index aa5556e..fcf6b13 100644 --- a/k8s/argo-apps/oxicloud.yaml +++ b/k8s/argo-apps/oxicloud.yaml @@ -22,7 +22,7 @@ spec: accessMode: ReadWriteMany size: 50Gi service: - type: LoadBalancer + type: ClusterIP config: server: port: 8086 diff --git a/k8s/argo-apps/pgadmin.yaml b/k8s/argo-apps/pgadmin.yaml index d961954..9ff22e8 100644 --- a/k8s/argo-apps/pgadmin.yaml +++ b/k8s/argo-apps/pgadmin.yaml @@ -19,7 +19,7 @@ spec: image: tag: "9.7" service: - type: LoadBalancer + type: ClusterIP port: 8010 ingress: enabled: true diff --git a/k8s/argo-apps/portainer.yaml b/k8s/argo-apps/portainer.yaml index 691cbc9..8751358 100644 --- a/k8s/argo-apps/portainer.yaml +++ b/k8s/argo-apps/portainer.yaml @@ -19,7 +19,7 @@ spec: helm: valuesObject: service: - type: LoadBalancer + type: NodePort httpPort: 9002 httpsPort: 9445 edgePort: 8005 diff --git a/k8s/argo-apps/vault-sm.yaml b/k8s/argo-apps/vault-sm.yaml index d017a0b..22f0f3c 100644 --- a/k8s/argo-apps/vault-sm.yaml +++ b/k8s/argo-apps/vault-sm.yaml @@ -18,14 +18,14 @@ spec: api: enabled: true service: - type: LoadBalancer + type: ClusterIP persistence: storageClass: truenas-nfs-csi accessMode: ReadWriteMany dashboard: enabled: true service: - type: LoadBalancer + type: ClusterIP port: 8007 ingress: enabled: true diff --git a/k8s/argo-apps/vaultwarden.yaml b/k8s/argo-apps/vaultwarden.yaml index 98009c1..d35a61b 100644 --- a/k8s/argo-apps/vaultwarden.yaml +++ b/k8s/argo-apps/vaultwarden.yaml @@ -34,7 +34,7 @@ spec: drop: - all service: - type: LoadBalancer + type: ClusterIP ingress: enabled: true className: "traefik" diff --git a/k8s/charts/_template/values.yaml b/k8s/charts/_template/values.yaml index 7aa1f5f..5f70727 100644 --- a/k8s/charts/_template/values.yaml +++ b/k8s/charts/_template/values.yaml @@ -7,7 +7,7 @@ chart: tag: "" service: - type: LoadBalancer + type: ClusterIP port: 80 targetPort: 80 diff --git a/k8s/charts/dokuwiki/values.yaml b/k8s/charts/dokuwiki/values.yaml index f003331..31c68df 100644 --- a/k8s/charts/dokuwiki/values.yaml +++ b/k8s/charts/dokuwiki/values.yaml @@ -12,7 +12,7 @@ dokuwiki: securityContext: {} service: - type: LoadBalancer + type: ClusterIP port: 8004 targetPort: 8080 diff --git a/k8s/charts/lldap/values.yaml b/k8s/charts/lldap/values.yaml index bb19244..0a3230f 100644 --- a/k8s/charts/lldap/values.yaml +++ b/k8s/charts/lldap/values.yaml @@ -7,7 +7,7 @@ lldap: tag: "stable" service: - type: LoadBalancer + type: ClusterIP httpPort: 17170 ldapPort: 3890 diff --git a/k8s/charts/oxicloud/values.yaml b/k8s/charts/oxicloud/values.yaml index 1348f56..fa16dca 100644 --- a/k8s/charts/oxicloud/values.yaml +++ b/k8s/charts/oxicloud/values.yaml @@ -8,7 +8,7 @@ oxicloud: tag: "" service: - type: LoadBalancer + type: ClusterIP port: 8086 ingress: diff --git a/k8s/charts/vaultwarden/values.yaml b/k8s/charts/vaultwarden/values.yaml index c3c1baa..f93218f 100644 --- a/k8s/charts/vaultwarden/values.yaml +++ b/k8s/charts/vaultwarden/values.yaml @@ -12,7 +12,7 @@ vaultwarden: securityContext: {} service: - type: LoadBalancer + type: ClusterIP port: 8008 targetPort: 80 diff --git a/k8s/helm/argo-cd/values.yaml b/k8s/helm/argo-cd/values.yaml index 89d4cd4..8e618b6 100644 --- a/k8s/helm/argo-cd/values.yaml +++ b/k8s/helm/argo-cd/values.yaml @@ -8,7 +8,7 @@ configs: server: service: - type: LoadBalancer + type: ClusterIP servicePortHttp: 8002 servicePortHttps: 8003 diff --git a/k8s/services/argo/project-management.yaml b/k8s/services/argo/project-management.yaml index 180673b..3d78bf6 100644 --- a/k8s/services/argo/project-management.yaml +++ b/k8s/services/argo/project-management.yaml @@ -26,3 +26,4 @@ spec: - https://democratic-csi.github.io/charts/ - https://bitnami-labs.github.io/sealed-secrets - https://kubernetes-sigs.github.io/descheduler + - https://metallb.github.io/metallb diff --git a/k8s/services/metallb/l2advert-adguard.yaml b/k8s/services/metallb/l2advert-adguard.yaml new file mode 100644 index 0000000..707c837 --- /dev/null +++ b/k8s/services/metallb/l2advert-adguard.yaml @@ -0,0 +1,9 @@ +--- +apiVersion: metallb.io/v1beta1 +kind: L2Advertisement +metadata: + name: adguard-dns-l2 + namespace: metallb-system +spec: + ipAddressPools: + - adguard-dns-pool diff --git a/k8s/services/metallb/l2advert-prod.yaml b/k8s/services/metallb/l2advert-prod.yaml new file mode 100644 index 0000000..6b422c2 --- /dev/null +++ b/k8s/services/metallb/l2advert-prod.yaml @@ -0,0 +1,9 @@ +--- +apiVersion: metallb.io/v1beta1 +kind: L2Advertisement +metadata: + name: ingress-l2 + namespace: metallb-system +spec: + ipAddressPools: + - ingress-pool diff --git a/k8s/services/metallb/l2advert-test.yaml b/k8s/services/metallb/l2advert-test.yaml new file mode 100644 index 0000000..216ff05 --- /dev/null +++ b/k8s/services/metallb/l2advert-test.yaml @@ -0,0 +1,9 @@ +--- +apiVersion: metallb.io/v1beta1 +kind: L2Advertisement +metadata: + name: test-l2 + namespace: metallb-system +spec: + ipAddressPools: + - test-pool diff --git a/k8s/services/metallb/namespace.yaml b/k8s/services/metallb/namespace.yaml new file mode 100644 index 0000000..928cb91 --- /dev/null +++ b/k8s/services/metallb/namespace.yaml @@ -0,0 +1,5 @@ +--- +apiVersion: v1 +kind: Namespace +metadata: + name: metallb-system diff --git a/k8s/services/metallb/pool-adguard.yaml b/k8s/services/metallb/pool-adguard.yaml new file mode 100644 index 0000000..12f9c77 --- /dev/null +++ b/k8s/services/metallb/pool-adguard.yaml @@ -0,0 +1,10 @@ +--- +apiVersion: metallb.io/v1beta1 +kind: IPAddressPool +metadata: + name: adguard-dns-pool + namespace: metallb-system +spec: + addresses: + - 192.168.1.125/32 + autoAssign: false diff --git a/k8s/services/metallb/pool-prod.yaml b/k8s/services/metallb/pool-prod.yaml new file mode 100644 index 0000000..b090516 --- /dev/null +++ b/k8s/services/metallb/pool-prod.yaml @@ -0,0 +1,12 @@ +--- +apiVersion: metallb.io/v1beta1 +kind: IPAddressPool +metadata: + name: ingress-pool + namespace: metallb-system +spec: + addresses: + - 192.168.1.126/32 + # Only Traefik should get this IP. Other LoadBalancer services that are + # not yet migrated will stay pending instead of stealing the single VIP. + autoAssign: false diff --git a/k8s/services/metallb/pool-test.yaml b/k8s/services/metallb/pool-test.yaml new file mode 100644 index 0000000..b308ad5 --- /dev/null +++ b/k8s/services/metallb/pool-test.yaml @@ -0,0 +1,10 @@ +--- +apiVersion: metallb.io/v1beta1 +kind: IPAddressPool +metadata: + name: test-pool + namespace: metallb-system +spec: + addresses: + - 192.168.1.125/32 + autoAssign: false