catalin/fukuops
1
0
Fork 1
Code Issues 9 Pull requests Projects 2 Releases Packages 10 Activity

feat: add metallb

Browse source
This commit is contained in:
cătălin 2026-08-24 13:21:44 +02:00
commit 45aa1154e1
No known key found for this signature in database
30 changed files with 231 additions and 24 deletions
Download patch file Download diff file Expand all files Collapse all files

128
METALLB.md Normal file
View file

@ -0,0 +1,128 @@
## Initial problem
the k3s cluster iss reversed proxied by an nginx in a small lxc (fuku.nginx) which acts as the exit point to the internet,
so the isp router port forwards the ports 80 and 443 to nginx which in turn has a config file for each service in k8s (and other hosts, but those are just a few).
I'd like to sundown nginx since I don't have a gitops way of managing each service nginx config file. I have three master nodes (high availability) but my isp
control panel does't allow me to put three different nodes and I can't do dns load-balancing either since I can't change the router hosts file or anything.
## Install MetalLB in the cluster
Solution: MetalLB in L2 mode: MetalLB elects one node to own the VIP via ARP/NDP (gratuitous ARP) and answers for it on the LAN.
If that node dies, another node takes over the VIP within a few seconds — it's essentially a Kubernetes-native keepalived.
You port-forward your ISP router to that single VIP, and MetalLB handles which physical node is actually answering at any given time.
```yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: ingress-pool
namespace: metallb-system
spec:
addresses:
- 192.168.1.240/32 # single reserved IP for Traefik's LB
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: ingress-l2
namespace: metallb-system
spec:
ipAddressPools: [ingress-pool]
```
### Installation
- https://artifacthub.io/packages/helm/metallb/metallb
- Install it in L2 mode in an argo-app but don't touch Traefik's existing Service yet. At this point MetalLB is running but idle — nothing's using it.
- Test MetalLB on a throwaway Service (a temporary nginx LoadBalancer Service with a test IP from your pool) to confirm ARP/failover actually works on your LAN before touching anything real
- Cut Traefik over: since ServiceLB only manages Services that don't already have an IP it recognizes, the actual switch happens when you either (a) delete and recreate
Traefik's Service so MetalLB's webhook claims it, or (b) explicitly disable ServiceLB at that point so MetalLB becomes the only controller answering LoadBalancer requests, then bounce Traefik's Service.
This step causes a brief blip (seconds) as the Service gets a new external IP — which is also when you update your router's port-forward target to the new MetalLB VIP.
---
## Implementation runbook
> Generated during the build session. Status is updated as steps complete.
> Branch: `feat/metallb` (local only, not pushed).
> Method: Argo app created with `kubectl apply`; synced with `argocd app sync --local`.
> Target VIP: `192.168.1.126/32`.
> Test IP: `192.168.1.125/32`.
### Pre-execution decisions
- [x] Single Traefik VIP architecture.
- [x] Nginx already proxies to Traefik.
- [x] Local `argocd app sync --local` workflow.
### Steps and status
- [x] 1. Update this file (`METALLB.md`) with the runbook.
- [x] 2. Create local `feat/metallb` branch.
- [x] 3. SSH connectivity check: `fuku.master1`, `fuku.master2`, `fuku.k3m3`.
- [x] 4. Add MetalLB Helm repo to `management` AppProject sourceRepos.
- [x] 5. Create MetalLB Argo app + manifests (test pool `192.168.1.125/32`, `autoAssign: false`).
- [x] 6. Apply MetalLB Argo app and sync. `argocd` CLI token expired; used `helm upgrade --install` as fallback. Argo app remains in place.
- [x] 7. Test MetalLB with throwaway nginx LoadBalancer service. **Finding:** ServiceLB and MetalLB fight over the Service (`EXTERNAL-IP` flaps between `192.168.1.125` and `<pending>`), confirming they cannot coexist. Must disable ServiceLB before cutover.
- [x] 8. Prepare production manifests: `ingress-pool` (`192.168.1.126/32`, `autoAssign: false`), converted web apps to `ClusterIP`/NodePort.
- [x] 9. Disabled k3s ServiceLB on `fuku.master1`, `fuku.master2`, `fuku.k3m3` and restarted k3s. All nodes Ready; ServiceLB pods terminated.
- [x] 10. Applied production Argo changes. Traefik Service assigned `192.168.1.126`. Web apps converted to `ClusterIP`; AdGuard DNS and Portainer changed to `NodePort`.
- [x] 11. Validated HTTPS ingress via VIP (`argo.fuku`, `auth.fukurokuju.dev`, `git.roboces.dev` all respond). Failover test deferred to final verification.
- [ ] 12. Ask user to update ISP router port-forward 80/443 → `192.168.1.126`.
- [x] 13. Fix AdGuard DNS by giving it dedicated MetalLB IP `192.168.1.125`. DNS resolution and web UI (`dns.fuku`) verified working.
- [ ] 14. Final verification after router update + nginx decommission (user handled).
### Rollback commands (keep handy)
If MetalLB test fails:
```bash
kubectl delete svc nginx-metallb-test -n default
kubectl delete -f k8s/argo-apps/metallb.yaml
kubectl delete ns metallb-system
```
If k3s ServiceLB disable causes problems (run on each master):
```bash
# Remove --disable servicelb from k3s server flags, then:
sudo systemctl restart k3s
```
If production cutover breaks:
```bash
# Re-apply main branch manifests to restore ServiceLB LoadBalancers
git checkout main
kubectl apply -f k8s/services/kube-system/
# Re-enable ServiceLB if it was disabled
```
---
## Post-cutover state
### What works now
- Traefik is reachable on the MetalLB VIP `192.168.1.126`.
- HTTPS ingress for `argo.fuku`, `auth.fukurokuju.dev`, `git.roboces.dev` responds via the VIP.
- ServiceLB (`svclb-*` pods) has been disabled on all three master nodes.
- Web apps in this repo were converted from `LoadBalancer` to `ClusterIP` and route through Traefik.
- Portainer was switched to `NodePort` to keep the Edge port available without stealing the ingress VIP.
- AdGuard DNS was given its own MetalLB IP (`192.168.1.125`) and is resolving queries again.
### IP allocation
| Service | IP | Pool | Notes |
|--------------------|----------------|---------------------|------------------------------------|
| Traefik ingress | 192.168.1.126 | `ingress-pool` | Router forwards 80/443 here |
| AdGuard DNS | 192.168.1.125 | `adguard-dns-pool` | LAN DNS resolver on standard port 53 |
### Known leftovers (not managed by this repo)
These services are still `LoadBalancer` and will stay `<pending>` until they are migrated separately (different repos / operators / manual installs):
- `apps-fuku/ak-outpost-proxy` (Authentik outpost)
- `apps-fuku/factorio-factorio-server-charts-rcon`
- `apps-fuku/psql15-postgres`
- `apps-roboces/huesoporro` (chart lives in `git.roboces.dev/catalin/huesoporro.git`)
- `apps-roboces/valkey-valkey-cluster`
- `default/prometheus-main`
Because both pools have `autoAssign: false` and only Traefik/AdGuard carry the matching pool annotations, these leftover services cannot steal either IP.
### Deviation from original plan
- `argocd` CLI token was expired and port-forward login failed, so the MetalLB chart was installed with `helm upgrade --install` instead of `argocd app sync`. The Argo app `metallb` was still created and will take over once ArgoCD login is restored.
- Service conversions were applied by updating the Argo app manifests with `kubectl apply` and letting ArgoCD auto-sync, rather than `argocd app sync --local`.
### Next step for the user
Update the ISP router to port-forward TCP 80 and TCP 443 to `192.168.1.126`. After that, nginx can be decommissioned at your convenience.

2
README.md
View file

@ -19,3 +19,5 @@ helm upgrade argo-cd argo/argo-cd -n argo-cd -f k8s/helm/argo-cd/values.yaml
```bash
scripts/users.py add -u catalin -n cătălin -e catalin@roboces.dev
```
## MetalLB and reserved VIPs

6
k8s/argo-apps/adguard.yaml
View file

@ -41,6 +41,10 @@ spec:
service:
dns:
type: LoadBalancer
port: 53
annotations:
metallb.universe.tf/address-pool: adguard-dns-pool
loadBalancerIP: 192.168.1.125
web:
type: LoadBalancer
type: ClusterIP
port: 8006

2
k8s/argo-apps/audiobookshelf.yaml
View file

@ -16,7 +16,7 @@ spec:
helm:
valuesObject:
service:
type: LoadBalancer
type: ClusterIP
port: 8081
ingress:
enabled: true

2
k8s/argo-apps/authelia.yaml
View file

@ -28,7 +28,7 @@ spec:
cpu: 1
memory: 1024Mi
service:
type: LoadBalancer
type: ClusterIP
port: 9091
ingress:
enabled: true

4
k8s/argo-apps/authentik.yaml
View file

@ -63,7 +63,7 @@ spec:
enabled: true
minAvailable: 1
service:
type: LoadBalancer
type: ClusterIP
servicePortHttp: 9001
servicePortHttps: 9444
annotations:
@ -72,7 +72,7 @@ spec:
metrics:
enabled: true
service:
type: LoadBalancer
type: ClusterIP
serviceMonitor:
enabled: false
ingress:

2
k8s/argo-apps/dokuwiki.yaml
View file

@ -26,7 +26,7 @@ spec:
drop:
- all
service:
type: LoadBalancer
type: ClusterIP
port: 8004
ingress:
enabled: true

2
k8s/argo-apps/forgejo.yaml
View file

@ -20,7 +20,7 @@ spec:
replicaCount: 1
service:
http:
type: LoadBalancer
type: ClusterIP
ssh:
type: NodePort
ingress:

2
k8s/argo-apps/helm-rustical.yaml
View file

@ -25,7 +25,7 @@ spec:
targetCPUUtilizationPercentage: 80
targetMemoryUtilizationPercentage: 80
service:
type: LoadBalancer
type: ClusterIP
port: 8001
targetPort: 4000
ingress:

4
k8s/argo-apps/immich.yaml
View file

@ -27,7 +27,7 @@ spec:
drop:
- all
service:
type: LoadBalancer
type: ClusterIP
config:
database:
hostname: "192.168.1.3"
@ -96,7 +96,7 @@ spec:
machineLearning:
enabled: true
service:
type: LoadBalancer
type: ClusterIP
secret:
enabled: true
existingSecretName: "immich-secrets"

2
k8s/argo-apps/lldap.yaml
View file

@ -17,7 +17,7 @@ spec:
valuesObject:
lldap:
service:
type: LoadBalancer
type: ClusterIP
httpPort: 17170
ldapPort: 3890
ingress:

25
k8s/argo-apps/metallb.yaml Normal file
View file

@ -0,0 +1,25 @@
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: metallb
namespace: argo-cd
spec:
destination:
name: ''
namespace: metallb-system
server: https://kubernetes.default.svc
sources:
- repoURL: https://metallb.github.io/metallb
chart: metallb
targetRevision: 0.16.*
helm:
valuesObject:
crds:
enabled: true
- repoURL: https://git.roboces.dev/catalin/fukuops.git
path: k8s/services/metallb
targetRevision: feat/metallb
project: management
syncPolicy:
automated: {}

2
k8s/argo-apps/miniflux.yaml
View file

@ -27,7 +27,7 @@ spec:
drop:
- all
service:
type: LoadBalancer
type: ClusterIP
ingress:
enabled: true
className: "traefik"

2
k8s/argo-apps/oxicloud.yaml
View file

@ -22,7 +22,7 @@ spec:
accessMode: ReadWriteMany
size: 50Gi
service:
type: LoadBalancer
type: ClusterIP
config:
server:
port: 8086

2
k8s/argo-apps/pgadmin.yaml
View file

@ -19,7 +19,7 @@ spec:
image:
tag: "9.7"
service:
type: LoadBalancer
type: ClusterIP
port: 8010
ingress:
enabled: true

2
k8s/argo-apps/portainer.yaml
View file

@ -19,7 +19,7 @@ spec:
helm:
valuesObject:
service:
type: LoadBalancer
type: NodePort
httpPort: 9002
httpsPort: 9445
edgePort: 8005

4
k8s/argo-apps/vault-sm.yaml
View file

@ -18,14 +18,14 @@ spec:
api:
enabled: true
service:
type: LoadBalancer
type: ClusterIP
persistence:
storageClass: truenas-nfs-csi
accessMode: ReadWriteMany
dashboard:
enabled: true
service:
type: LoadBalancer
type: ClusterIP
port: 8007
ingress:
enabled: true

2
k8s/argo-apps/vaultwarden.yaml
View file

@ -34,7 +34,7 @@ spec:
drop:
- all
service:
type: LoadBalancer
type: ClusterIP
ingress:
enabled: true
className: "traefik"

2
k8s/charts/_template/values.yaml
View file

@ -7,7 +7,7 @@ chart:
tag: ""
service:
type: LoadBalancer
type: ClusterIP
port: 80
targetPort: 80

2
k8s/charts/dokuwiki/values.yaml
View file

@ -12,7 +12,7 @@ dokuwiki:
securityContext: {}
service:
type: LoadBalancer
type: ClusterIP
port: 8004
targetPort: 8080

2
k8s/charts/lldap/values.yaml
View file

@ -7,7 +7,7 @@ lldap:
tag: "stable"
service:
type: LoadBalancer
type: ClusterIP
httpPort: 17170
ldapPort: 3890

2
k8s/charts/oxicloud/values.yaml
View file

@ -8,7 +8,7 @@ oxicloud:
tag: ""
service:
type: LoadBalancer
type: ClusterIP
port: 8086
ingress:

2
k8s/charts/vaultwarden/values.yaml
View file

@ -12,7 +12,7 @@ vaultwarden:
securityContext: {}
service:
type: LoadBalancer
type: ClusterIP
port: 8008
targetPort: 80

2
k8s/helm/argo-cd/values.yaml
View file

@ -8,7 +8,7 @@ configs:
server:
service:
type: LoadBalancer
type: ClusterIP
servicePortHttp: 8002
servicePortHttps: 8003

3
k8s/services/argo/project-management.yaml
View file

@ -18,6 +18,8 @@ spec:
server: https://kubernetes.default.svc
- namespace: argo-workflows
server: https://kubernetes.default.svc
- namespace: metallb-system
server: https://kubernetes.default.svc
clusterResourceWhitelist:
- group: '*'
kind: '*'
@ -26,3 +28,4 @@ spec:
- https://democratic-csi.github.io/charts/
- https://bitnami-labs.github.io/sealed-secrets
- https://kubernetes-sigs.github.io/descheduler
- https://metallb.github.io/metallb

9
k8s/services/metallb/l2advert-adguard.yaml Normal file
View file

@ -0,0 +1,9 @@
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: adguard-dns-l2
namespace: metallb-system
spec:
ipAddressPools:
- adguard-dns-pool

9
k8s/services/metallb/l2advert-prod.yaml Normal file
View file

@ -0,0 +1,9 @@
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: ingress-l2
namespace: metallb-system
spec:
ipAddressPools:
- ingress-pool

5
k8s/services/metallb/namespace.yaml Normal file
View file

@ -0,0 +1,5 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: metallb-system

10
k8s/services/metallb/pool-adguard.yaml Normal file
View file

@ -0,0 +1,10 @@
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: adguard-dns-pool
namespace: metallb-system
spec:
addresses:
- 192.168.1.125/32
autoAssign: false

12
k8s/services/metallb/pool-prod.yaml Normal file
View file

@ -0,0 +1,12 @@
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: ingress-pool
namespace: metallb-system
spec:
addresses:
- 192.168.1.126/32
# Only Traefik should get this IP. Other LoadBalancer services that are
# not yet migrated will stay pending instead of stealing the single VIP.
autoAssign: false