mikluko/kuberex

Watch

Fork 0

Deploy kuberex on saraksh via Flux HelmRelease #8

New issue

Closed

opened 2026-10-31 13:09:05 +00:00 by mikluko · 5 comments

mikluko commented

2026-10-31 13:09:05 +00:00

Owner

Map: #1

Question

Deploy kuberex in saraksh/kube through Flux: a HelmRelease of the kuberex OCI chart in namespace agents, ServiceAccount rex, pinned to bulgakov with tolerations for sarak.sh/baremetal and sarak.sh/workstation, hostPath /srv/rex at /var/lib/rex, egress policy from the pod-shape ticket, and an ExternalSecret for the Tailscale join key (kv/k8s/saraksh/agents/rex-tailscale).

Open here: how /srv/rex gets created with uid/gid 999 on bulgakov. Steps for me: mint a one-time pre-authorized Tailscale key, put it in OpenBao, disable key expiry for kuberex after it enrolls. Confirm TS_CONTROL_URL reaches tsnet when set. Resolved when the pod is Ready and kuberex is on the tailnet.

Map: #1 ## Question Deploy kuberex in `saraksh/kube` through Flux: a HelmRelease of the kuberex OCI chart in namespace `agents`, ServiceAccount `rex`, pinned to `bulgakov` with tolerations for `sarak.sh/baremetal` and `sarak.sh/workstation`, hostPath `/srv/rex` at `/var/lib/rex`, egress policy from the pod-shape ticket, and an ExternalSecret for the Tailscale join key (`kv/k8s/saraksh/agents/rex-tailscale`). Open here: how `/srv/rex` gets created with uid/gid 999 on `bulgakov`. Steps for me: mint a one-time pre-authorized Tailscale key, put it in OpenBao, disable key expiry for `kuberex` after it enrolls. Confirm `TS_CONTROL_URL` reaches tsnet when set. Resolved when the pod is Ready and `kuberex` is on the tailnet.

mikluko added the

wayfinder:task

label

2026-10-31 13:09:16 +00:00

mikluko added a new dependency

2026-10-31 13:09:18 +00:00

#3 Exposure: joining the tailnet

mikluko added a new dependency

2026-10-31 13:09:18 +00:00

#4 Pod shape: namespace, pool, sandbox, egress

mikluko added a new dependency

2026-10-31 13:09:18 +00:00

#6 OpenBao: identity agent-rex

mikluko added a new dependency

2026-10-31 13:09:18 +00:00

#7 Image: Debian base, multi-arch, Rex and Claude Code pinned in the image

mikluko added a new dependency

2026-10-31 13:09:19 +00:00

#9 End-to-end check from my devices

mikluko added a new dependency

2026-10-31 15:49:30 +00:00

#10 Prototype: startup install through a shared overlay

mikluko commented

2026-10-31 16:08:12 +00:00

Author

Owner

From the overlay prototype (#10): the pod needs appArmorProfile: Unconfined besides hostUsers: false and CAP_SYS_ADMIN; overlays use userxattr. Open here: whether hostPath /srv/rex works under the user namespace (idmapped mount on bulgakov's filesystem), and its ownership against the image's fixed rex uid. kuberex-taw exists via Flux for trying it.

From the overlay prototype (#10): the pod needs `appArmorProfile: Unconfined` besides `hostUsers: false` and `CAP_SYS_ADMIN`; overlays use `userxattr`. Open here: whether hostPath `/srv/rex` works under the user namespace (idmapped mount on `bulgakov`'s filesystem), and its ownership against the image's fixed `rex` uid. `kuberex-taw` exists via Flux for trying it.

mikluko commented

2026-10-31 16:19:32 +00:00

Author

Owner

Exposure decided (#3): --tailscale with --tailscale-hostname ${TS_HOSTNAME:-kuberex}, --tailscale-auth-key-file $TS_AUTHKEY_FILE from Secret rex-tailscale (ExternalSecret, OpenBao kv/k8s/agents/rex-tailscale), TS_CONTROL_URL passed through when set. Steps for me here: mint a one-time pre-authorized key, put it in OpenBao, and disable key expiry for kuberex after it enrolls. Confirm TS_CONTROL_URL reaches tsnet.

Exposure decided (#3): `--tailscale` with `--tailscale-hostname ${TS_HOSTNAME:-kuberex}`, `--tailscale-auth-key-file $TS_AUTHKEY_FILE` from Secret `rex-tailscale` (ExternalSecret, OpenBao `kv/k8s/agents/rex-tailscale`), `TS_CONTROL_URL` passed through when set. Steps for me here: mint a one-time pre-authorized key, put it in OpenBao, and disable key expiry for `kuberex` after it enrolls. Confirm `TS_CONTROL_URL` reaches tsnet.

mikluko added a new dependency

2026-10-31 16:37:56 +00:00

#12 Helm chart for kuberex

mikluko changed title from ~~Flux manifests for Rex in saraksh/kube~~ to Deploy kuberex on saraksh via Flux HelmRelease

2026-10-31 16:37:57 +00:00

mikluko commented

2026-11-02 20:20:53 +00:00

Author

Owner

Chart decided (Helm chart for kuberex): oci://sarak.sh/mikluko/charts/kuberex, 0.x. The HelmRelease here sets stateVolume.hostPath (/srv/rex), tailscale.authKey.existingSecret: rex-tailscale, serviceAccount.name: rex, networkPolicy.enabled: true, plus nodeSelector, tolerations and resources from the pod-shape ticket.

Chart decided ([Helm chart for kuberex](https://sarak.sh/mikluko/kuberex/issues/12)): `oci://sarak.sh/mikluko/charts/kuberex`, `0.x`. The HelmRelease here sets `stateVolume.hostPath` (`/srv/rex`), `tailscale.authKey.existingSecret: rex-tailscale`, `serviceAccount.name: rex`, `networkPolicy.enabled: true`, plus `nodeSelector`, `tolerations` and `resources` from the pod-shape ticket.

mikluko self-assigned this

2026-11-02 20:21:51 +00:00

mikluko commented

2026-11-02 20:25:34 +00:00

Author

Owner

Decided here: /srv/rex is a dataset rpool/state/rex on bulgakov, like /srv/jellyfin and /srv/buildkit: encrypted with the node state, mounted by zfs-unlock-state before k0sworker, owned 999:999 (kubelet applies no fsGroup to a hostPath). The HelmRelease mounts it as hostPath type Directory, so an unmounted or missing dataset fails the pod instead of writing into the root filesystem.

Manifests drafted and validated (kubeconform -strict, chart 0.4.0 rendered with the values): namespace agents with its openbao SecretStore, Flux Kustomization kuberex, k8s/apps/kuberex with OCIRepository kuberex (0.x), HelmRelease rex, ExternalSecret rex-tailscale from k8s/saraksh/agents/rex-tailscale (key authkey). The empty kuberex-taw namespace goes. TS_CONTROL_URL is wired through the chart and the entrypoint; unset for Tailscale proper.

Waiting on: the dataset on bulgakov, the join key in OpenBao, then the push to saraksh/kube.

Decided here: `/srv/rex` is a dataset `rpool/state/rex` on `bulgakov`, like `/srv/jellyfin` and `/srv/buildkit`: encrypted with the node state, mounted by `zfs-unlock-state` before `k0sworker`, owned `999:999` (kubelet applies no `fsGroup` to a hostPath). The HelmRelease mounts it as `hostPath` type `Directory`, so an unmounted or missing dataset fails the pod instead of writing into the root filesystem. Manifests drafted and validated (`kubeconform -strict`, chart 0.4.0 rendered with the values): namespace `agents` with its `openbao` SecretStore, Flux Kustomization `kuberex`, `k8s/apps/kuberex` with OCIRepository `kuberex` (`0.x`), HelmRelease `rex`, ExternalSecret `rex-tailscale` from `k8s/saraksh/agents/rex-tailscale` (key `authkey`). The empty `kuberex-taw` namespace goes. `TS_CONTROL_URL` is wired through the chart and the entrypoint; unset for Tailscale proper. Waiting on: the dataset on `bulgakov`, the join key in OpenBao, then the push to `saraksh/kube`.

mikluko commented

2026-11-02 20:34:00 +00:00

Author

Owner

Answer

Deployed by saraksh/kube@2e79b1e: HelmRelease rex (chart kuberex 0.4.0) in agents, pod Running on bulgakov, ExternalSecret rex-tailscale synced, kuberex on the tailnet at 100.116.215.51.

/srv/rex is rpool/state/rex, owned 999:999, mode 700, in bulgakov.setup.sh and zfs-unlock-state; hostPath type Directory.
Join key in OpenBao kv/k8s/saraksh/agents/rex-tailscale, key authkey.
OCIRepository follows 0.x hourly, and the chart is released with the image at its version, so every kuberex release rolls out by itself; no image automation or bump PR.
kuberex-taw removed; /srv/rex listed in docs/backups.md as not backed up.

Left for me: disable key expiry for kuberex in the Tailscale admin.

## Answer Deployed by saraksh/kube@2e79b1e: HelmRelease `rex` (chart `kuberex` 0.4.0) in `agents`, pod Running on `bulgakov`, ExternalSecret `rex-tailscale` synced, `kuberex` on the tailnet at 100.116.215.51. - `/srv/rex` is `rpool/state/rex`, owned `999:999`, mode 700, in `bulgakov.setup.sh` and `zfs-unlock-state`; hostPath type `Directory`. - Join key in OpenBao `kv/k8s/saraksh/agents/rex-tailscale`, key `authkey`. - OCIRepository follows `0.x` hourly, and the chart is released with the image at its version, so every kuberex release rolls out by itself; no image automation or bump PR. - `kuberex-taw` removed; `/srv/rex` listed in docs/backups.md as not backed up. Left for me: disable key expiry for `kuberex` in the Tailscale admin.

mikluko closed this issue

2026-11-02 20:34:01 +00:00

mikluko referenced this issue

2026-11-02 20:34:22 +00:00

Map: Rex server on saraksh #1