mikluko/kuberex

Watch

Fork 0

Prototype: startup install through a shared overlay #10

New issue

Closed

opened 2026-10-31 15:49:23 +00:00 by mikluko · 1 comment

mikluko commented

2026-10-31 15:49:23 +00:00

Owner

Map: #1

Question

Does a startup install through a shared overlay work on bulgakov? Shape to prove, in a throwaway pod:

Pod with hostUsers: false; init and main containers from the same image, each with namespaced CAP_SYS_ADMIN.
Init mounts overlayfs per directory (/usr, /etc, /var/lib/dpkg and whatever else apt writes) with upper dirs on an emptyDir, runs apt-get install for rex (netrc credential from a mounted Secret) and Claude Code, and unmounts.
Main mounts the same overlays over its identical lower layer, drops to rex with setpriv, and runs rex --version and claude --version.

Unverified going in: user namespaces enabled on this k0s/containerd, RuntimeDefault seccomp allowing mount with the capability, apt working as namespaced root. The answer is the working manifest snippet and entrypoint, or the point it breaks.

Map: #1 ## Question Does a startup install through a shared overlay work on `bulgakov`? Shape to prove, in a throwaway pod: - Pod with `hostUsers: false`; init and main containers from the same image, each with namespaced `CAP_SYS_ADMIN`. - Init mounts overlayfs per directory (`/usr`, `/etc`, `/var/lib/dpkg` and whatever else apt writes) with upper dirs on an emptyDir, runs `apt-get install` for rex (netrc credential from a mounted Secret) and Claude Code, and unmounts. - Main mounts the same overlays over its identical lower layer, drops to `rex` with `setpriv`, and runs `rex --version` and `claude --version`. Unverified going in: user namespaces enabled on this k0s/containerd, `RuntimeDefault` seccomp allowing `mount` with the capability, apt working as namespaced root. The answer is the working manifest snippet and entrypoint, or the point it breaks.

mikluko added the

wayfinder:prototype

label

2026-10-31 15:49:29 +00:00

mikluko added a new dependency

2026-10-31 15:49:29 +00:00

#7 Image: Debian base, multi-arch, Rex and Claude Code pinned in the image

mikluko added a new dependency

2026-10-31 15:49:30 +00:00

#8 Deploy kuberex on saraksh via Flux HelmRelease

mikluko referenced this issue

2026-10-31 15:49:30 +00:00

Image toolchain for agents in Rex terminals #5

mikluko referenced this issue

2026-10-31 15:52:56 +00:00

Image toolchain for agents in Rex terminals #5

mikluko self-assigned this

2026-10-31 15:56:19 +00:00

mikluko commented

2026-10-31 16:08:11 +00:00

Author

Owner

Answer

Works on bulgakov. Init container installed rex 0.0.1793420730-g6d0d9a and claude-code 2.1.309 with a real apt-get install; the main container mounted the same overlays, dropped to rex (uid 999) with every capability set empty, and ran both.

What it takes:

hostUsers: false; root in the pod maps to an unprivileged host range.
CAP_SYS_ADMIN added in both containers.
AppArmor Unconfined: containerd's default profile denies mount. seccomp RuntimeDefault is fine.
Overlay option userxattr: without it, removing a lower directory fails with EIO and dpkg fails with EXDEV.
Overlays per directory: /usr, /etc, /var. Re-bind kubelet's /etc/resolv.conf, hosts, hostname after mounting over /etc.
The credential stays out of the overlay: apt reads it from the Secret mount via -o Dir::Etc::netrcparts=/auth.

Carry into the image and manifests:

Create the rex user with a fixed uid/gid in the image before install; postinst reuses it, and /srv/rex ownership depends on it.
kubectl exec lands as pod root with the capability; only processes started through the entrypoint run as rex.
Unverified: hostPath /srv/rex under a user namespace (idmapped mount on the host filesystem).

Prototype: branch prototype/overlay-install. Ran in kuberex-taw (namespace via Flux, saraksh/kube 1f233f8); prototype objects deleted.

## Answer Works on `bulgakov`. Init container installed `rex 0.0.1793420730-g6d0d9a` and `claude-code 2.1.309` with a real `apt-get install`; the main container mounted the same overlays, dropped to `rex` (uid 999) with every capability set empty, and ran both. What it takes: - `hostUsers: false`; root in the pod maps to an unprivileged host range. - `CAP_SYS_ADMIN` added in both containers. - **AppArmor `Unconfined`**: containerd's default profile denies `mount`. seccomp `RuntimeDefault` is fine. - Overlay option **`userxattr`**: without it, removing a lower directory fails with EIO and dpkg fails with EXDEV. - Overlays per directory: `/usr`, `/etc`, `/var`. Re-bind kubelet's `/etc/resolv.conf`, `hosts`, `hostname` after mounting over `/etc`. - The credential stays out of the overlay: apt reads it from the Secret mount via `-o Dir::Etc::netrcparts=/auth`. Carry into the image and manifests: - Create the `rex` user with a fixed uid/gid in the image before install; postinst reuses it, and `/srv/rex` ownership depends on it. - `kubectl exec` lands as pod root with the capability; only processes started through the entrypoint run as `rex`. - Unverified: hostPath `/srv/rex` under a user namespace (idmapped mount on the host filesystem). Prototype: branch [`prototype/overlay-install`](https://sarak.sh/mikluko/kuberex/src/branch/prototype/overlay-install/prototype/overlay-install). Ran in `kuberex-taw` (namespace via Flux, `saraksh/kube` `1f233f8`); prototype objects deleted.

mikluko closed this issue

2026-10-31 16:08:12 +00:00

mikluko referenced this issue

2026-10-31 16:08:12 +00:00

Map: Rex server on saraksh #1

mikluko referenced this issue

2026-10-31 16:08:12 +00:00

Deploy kuberex on saraksh via Flux HelmRelease #8

mikluko referenced this issue

2026-10-31 16:08:12 +00:00

Image: Debian base, multi-arch, Rex and Claude Code pinned in the image #7

mikluko referenced this issue from a commit