1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
|
# Worker recovery — FreeBSD + Linuxulator podman
This host runs the buildbot **worker**; the **master** is 192.168.3.110
(Debian armv7l, "dangofactory.net"). The worker is a container, and two host
level facts decide how it must be started. Both were measured, not guessed.
## Fact 1 — `pf` must be enabled
Without it the container will not start at all:
```
Error: unable to start container: plugin type="bridge" failed (add):
cni plugin bridge failed: The pf kernel module must be loaded to support
ipMasq networks
```
`/etc/pf.conf` already carries the matching NAT rule for podman's subnet, so
enabling is all that is needed. The ruleset is NAT-only (no `block` rules), so
this does not filter traffic:
```
nat on em0 inet from 10.88.0.0/16 to any -> (em0) # podman
nat on em0 inet from 10.0.0.0/24 to any -> (em0)
```
```sh
pfctl -n -f /etc/pf.conf # dry run the ruleset first
pfctl -f /etc/pf.conf -e # load + enable
```
## Fact 2 — the CNI bridge path does not work; use `--network=host`
Every outbound `connect()` from a bridge-networked container fails with
**EACCES**, including to the container's own gateway. Measured from a *fresh*
container with no volumes, so it is not a per-container problem:
| destination | `--network=podman` (bridge) | `--network=host` |
|---|---|---|
| `10.88.0.1:22` (gateway) | EACCES (13) | OK |
| `192.168.3.110:9989` (master) | EACCES (13) | OK |
| `8.8.8.8:53` | EACCES (13) | OK |
| `1.1.1.1:443` | EACCES (13) | OK |
The container's route table is correct, so this is ocijail refusing the
connect, not a routing problem. `--network=host` loses nothing here: the worker
only makes **outbound** connections (master `:9989`, github, the FTP deploy).
`--network=slirp4netns` is not an alternative — it fails outright with
`not implemented GetSlirp4netnsIP`.
## The caches are in the container's writable layer — extract them first
`GRADLE_USER_HOME=/var/jenkins_home/.gradle` (6.3 GB) and `.konan` (1.7 GB)
live in the container's **writable layer**. There is no named volume
(`podman volume ls` is empty, and `run-worker.sh`'s `-v jenkins-data:...` was
never real). So **`podman rm` destroys them**, and the next build re-downloads
everything.
`podman commit` is the obvious way to preserve them and **it fails**:
```
Error: copying layers and metadata for container "6b6421f1...":
on copy: write /var/tmp/buildah3539187385/layer: no space left on device
```
buildah stages the layer into `/var/tmp`, which is on the root zfs pool —
1.5 GB free against a 6 GB layer. Point its tmpdir at `/ROM` (451 GB):
```sh
podman --tmpdir /ROM/tmp_probe commit ... # still needs 6+GB of scratch
```
Copying just the cache directories out is smaller and is what
`extract-caches.sh` does:
```sh
sh deploy/buildbot/worker/extract-caches.sh
# -> /ROM/worker-cache/{.gradle,.konan,.kotlin}
```
## The recovery
```sh
sh deploy/buildbot/worker/extract-caches.sh # once, before any rm
sh deploy/buildbot/worker/recreate-worker-hostnet.sh
```
Verify with the **master**, not the container — a container that is merely
`Up` may still be retrying:
```sh
curl -s http://192.168.3.110:8010/api/v2/workers | python3 -m json.tool
# "connected_to": [{"masterid": 1}] <- attached
tail -3 /ROM/worker/twistd.log
# [Broker,client] Connected to buildmaster; worker is ready
```
## Traps on this host
- **Background jobs stop.** `terminal(background=true)` wrappers land in `Ts`
and silently do nothing. A `podman commit` reported "running" for 5 minutes
while frozen; CPU time stayed at 0:00.03. Check `ps -o stat=` and
`kill -CONT <pid>` (loop — one CONT is often not enough, and the *child*
needs it too, not just the wrapper).
- **The root pool is nearly full** (23 G, 1.5 GB free). `podman commit` and
anything that stages into `/var` will die with ENOSPC. Use
`--tmpdir /ROM/tmp_probe` and `export TMPDIR=/ROM/tmp_probe`.
- **Secrets live in two places and they differ.** The container mounts
`/root/buildbot/secrets/*` — that is the correct set. The similarly named
`/ROM/wiiu/secrets/` has a **different** `worker_pass` and an **empty**
`git_pass`; copying from there produces a worker that cannot authenticate.
Container uid 1000 must be able to *read* the secrets, so they are mode 644.
- **The master's secrets must not be echoed.** `master.cfg` interpolates
`$(cat /run/secrets/...)` inside single-quoted step bodies; tooling that
redacts `Authorization:` may show you the redacted form and you can mistake it
for file corruption. Check the bytes, not the rendered output.
## What is NOT wrong here
Checked and confirmed healthy, so do not spend time on them:
- the master (`/api/v2/masters.last_active` tracks the clock; poller is current)
- git HTTP auth (`/root/buildbot/secrets/git_pass` → `info/refs` = 200; no auth = 401)
- the worker passphrase (`buildbot.tac` matches the mounted secret)
- `github_token` (`GET /user` → 200, login `ketsuban152`)
- OOM / crash / network loss — the earlier exit was `143` after a clean
`SIGTERM`, `OOMKilled:false`, with normal keepalives up to 8 s prior
|
