| Age | Commit message (Collapse) | Author |
|---|
|
build 83's assemble ended with "command finished with signal 9". I had assumed
that was wasm-opt's -Oz pass list and tried to change it to -O2, which broke the
detekt gate instead (detekt parses build.gradle.kts as Kotlin source, and the
`optimizations { wasm-opt { ... } }` accessor does not resolve there -- 9
"Unresolved reference" errors). That DSL block is gone again in 4f00e13.
With the block removed, wasm-opt still runs at -Oz and both of the steps that
were dying now pass on this host:
./gradlew --max-workers=1 assemble BUILD SUCCESSFUL in 7m38s
./gradlew --max-workers=1 bundleWeb BUILD SUCCESSFUL in 3m11s
with no signal 9. So the killer was concurrency, not the optimizer level: several
JVMs (Gradle, the Kotlin daemon, wasm-opt) resident at once is what overran the
11.9 GB. The budgets are lowered to match (Gradle 3g -> 2g, Kotlin 2g -> 1.5g)
and --max-workers=1 is applied to assemble, bundles and bundle-web so only one
compile is resident at a time. The wall clock goes up, which is the trade that has
to be made for a bundle that exists.
|
|
builds 76 and 77 both died in assemble with 'Gradle build daemon disappeared
unexpectedly'. I had guessed --no-daemon would help; it did not, it just moved
the whole build into the client JVM and that died the same way. The useful
evidence was ./gradlew --status reporting every daemon as 'STOPPED (by user or
operating system)' while vmstat showed 91% of 11.9 GB idle, and the build log
complaining that 9 stopped daemons could not be reused.
What actually fixes it is the worker count. assemble with --max-workers=2
completed in 5m33s from a clean daemon registry. So:
org.gradle.parallel=false project-level parallelism off
org.gradle.jvmargs -Xmx1536m -> 3g (holds Kotlin now that it is in-process)
kotlin.daemon.jvmargs 4096m -> 2g
kotlin.compiler.execution.strategy=in-process one fewer 4 GB JVM
registry.maximum-size=1, idletimeout 2 min the pool stops accumulating
assemble step passes --max-workers=2 explicitly
bundle-web drops --no-daemon again; with the pool bounded, a normal reused daemon
is correct and faster for a ~1 min task.
|
|
build 73 failed in bundle-web. Re-running the task by hand reproduced the
message 'Starting a Gradle Daemon, 7 busy and 1 stopped Daemons could not be
used', and ./gradlew --status showed a pool that had accumulated across the
repeated forced builds. The invocation lost that race and died; a later run with
'--stop' followed by --no-daemon completed the same task in 1m12s and produced
the zip.
--no-daemon gives the build its own JVM and tears it down afterwards, so a
wedged or crashed previous build cannot poison the next one. That matters more
than the lost daemon reuse, because bundleWeb is only ~1 min here.
|
|
build 69 died in 'bundles' with no wasm-web zip in distributions, while the same
task run by hand took 8m45s and produced the zip. wasm-opt -O3/-Oz plus webpack
is by far the slowest thing in this build, and six bundle tasks in one
invocation pushed the step past the worker's timeout.
Splitting bundleWeb into its own step with timeout=3600 keeps the fast bundles
where they are and stops the web bundle from riding on their clock. The deploy
step already asserts every expected zip exists before uploading, so a silent
missing bundle is now reported by name rather than as a bare FTP exit 3.
|
|
fetch-github-ios derived the version with
sed "s/.*-\([0-9][^/]*\)\.zip/\1/"
but that step lives inside a Python triple-quoted string, where \1 is an
OCTAL escape. Python consumed it as chr(1) and handed the shell a replacement
character, so every build staged the file as
kukuri-ios-\x01.zip
which then failed the FTP upload. Build 60 surfaced it as
'FTP FAILED: kukuri-ios-\x01.zip' -- the artifact assertion added in the
previous commit is what made this visible at all.
Use parameter expansion instead (no backreference to escape), and reject a
non-numeric version rather than carrying it into a filename. Verified:
'kukuri-wasm-web-0.1.0.zip' -> 'kukuri-ios-0.1.0.zip', and 'abc' is rejected.
|
|
The previous commit wrote the shell line-continuations as a literal double
backslash, so the embedded script failed 'bash -n' with 'Syntax error: word
unexpected' and the deploy step would have aborted on every run.
Now verified with 'bash -n' on the extracted script, and the clean-wasm-output
step re-checked the same way.
|
|
Build 58 exited 3 in deploy-to-lighttpd with no usable message. The FTP loop
had no error handling, and the underlying fault was upstream and silent: a Zip
task whose from(...) directory does not exist archives nothing and does NOT
fail, so a bundles step that produced no kukuri-wasm-web zip left the loop
uploading a short list and dying partway on curl's exit code 3.
Check for all five expected artifacts before uploading anything, name the
missing one, and report which file failed on FTP instead of surfacing a bare
exit code.
Diagnosed by reproducing the CI sequence exactly (clean outputs, then all
bundle tasks in one invocation) and confirming bundleWeb alone succeeds in
6m07s -- the earlier missing-zip results came from the harness killing the
build at its timeout, not from a defect in the task.
|
|
The build cache alone was not enough. A reused configuration-cache entry made
the detekt gate fail with
ClassNotFoundException: io.gitlab.arturbosch.detekt.cli.Main
Reusing configuration cache.
...while the same task passed locally with --no-configuration-cache, so the
cached task graph held a classpath that no longer resolved. That cache is also
what pins the Wasm compile UP-TO-DATE, which is how a stale bundle shipped
past a green build.
Note: master.cfg lives in git AND on the master at
/var/lib/buildbot/masters/default/master.cfg. A change here does nothing until
it is copied across and 'buildbot reconfig' runs -- the first attempt at this
fix appeared to work because the step list still lacked clean-wasm-output.
|
|
The served web bundle was stale while every build reported SUCCESS.
Measured from the deployed pixels, not inferred: the nav rail rendered at
~232px and the pie disc at exactly 132x132px -- both are the PRE-change
values, while the source has had 280dp and diameterDp=200 since bd86609.
Cause: the Kotlin/Wasm compile stayed UP-TO-DATE against build/ outputs
left in the tree by an earlier local build, so `bundles` repackaged the old
wasm. Clearing the Gradle build cache alone does not help, because the
up-to-date check also keys on the compile output already being present.
Add a clean-wasm-output step ahead of `bundles` that removes the Wasm
compile/dist/webpack outputs and the build cache. Only build directories are
touched, never source.
Note the correction to an earlier claim in this repo's history: the
content-addressed wasm filename was taken as proof that the served artifact
matched the source. It does not -- the address only changes when the compile
actually runs, so an unchanged address meant "nothing recompiled", not
"already up to date". Measuring the rendered pixels is what caught it.
|
|
The bundles step asked for `bundleWasm`, which is not a task. Gradle's
camelCase abbreviation matching resolves it to `bundleWasmNode`, so the
build reports SUCCESS while building the node bundle again and leaving the
web bundle to `bundleWeb` alone. Confirmed:
$ ./gradlew bundleWasm --dry-run
:modules:orgflow-domain:bundleWasmNode SKIPPED
BUILD SUCCESSFUL
Harmless today, but it silently retargets if another bundleWasm* task ever
appears, and it misrepresents what the pipeline does.
Separately, while chasing this: the deployed web wasm looked stale because
its bytes never changed across builds. The cause was the Gradle build cache
serving pre-change entries, not a broken pipeline. With the cache removed,
the same source edit produces a new content-addressed wasm name and
orgflow-ui.js picks it up. The wasm filename is content-addressed, which
makes it a cheap oracle: 38dp produced 5e232b18..., and restoring 39dp
reproduced 8bc1b48e... exactly -- so the served artifact does contain the
current PieMenu change. The 2026-09-27 mtime inside the zip is Gradle
preserving timestamps for unchanged entries.
|
|
master.cfg checks out with method='clean' and buildbot builds inside this
same tree, so every build silently reverts uncommitted changes here. That
cost a finished PieMenu edit, which had to be reconstructed by hand.
Add a script that snapshots staged+unstaged diffs to
/ROM/worker-cache/uncommitted-<stamp>.patch. Run it before gradlew, before
pushing, and before any build not initiated here.
|
|
The worker container could not start and, once started, could not reach the
master. Two independent host-level causes, both measured rather than assumed:
1. pf was disabled, so podman refused to start the container at all
("The pf kernel module must be loaded to support ipMasq networks").
/etc/pf.conf already carries the matching NAT rule for podman's
10.88.0.0/16, and the ruleset has no block rules, so enabling it changes
no filtering.
2. The CNI bridge path returns EACCES on every outbound connect, including to
the container's own gateway -- verified from a fresh container with no
volumes, against 10.88.0.1, 192.168.3.110, 8.8.8.8 and 1.1.1.1. The
route table inside the container is correct, so this is ocijail refusing
the connect. --network=host was measured to work for all four and costs
nothing here: the worker only makes outbound connections.
Also record, because both cost real time:
- GRADLE_USER_HOME (6.3G) and .konan (1.7G) live in the container's writable
layer with no named volume, so `podman rm` destroys them.
- `podman commit` does not preserve them: buildah stages into /var/tmp on the
root zfs pool, which has 1.5G free against a 6G layer (ENOSPC). Extracting
the two directories with --tmpdir pointed at /ROM works.
- Secrets exist in two locations and they DIFFER. /root/buildbot/secrets is
the set the container mounts; /ROM/wiiu/secrets has a different
worker_pass and an empty git_pass.
- `master.cfg`'s TOKEN lines look corrupted when a tool redacts
"Authorization:" in rendered output. They are intact.
extract-caches.sh + recreate-worker-hostnet.sh make the recovery rerunnable;
RECOVERY.txt documents the evidence and the traps. Named .txt because
.gitignore excludes *.md as local-only.
Verified after the change: master reports connected_to [{masterid: 1}],
twistd logs "Connected to buildmaster; worker is ready", and the toolchain
inside the container is intact (JDK 21.0.12, node 20.20.2, gradle 6.3G,
konan prebuilt, Android build-tools 35/36).
|
|
audio daemon, and pre-create Gradle output dirs
Three fixes found by actually building the image and then running the pipeline
against it, rather than by reading the Dockerfile:
1. Android SDK levels. apps/android, modules/orgflow-domain, modules/orgflow-ui
and modules/zero-bridge-android all set compileSdk/targetSdk = 35, but the
image only shipped platform 36 + build-tools 36. The first assemble died with
"Failed to install the following SDK components: build-tools;35.0.0,
platforms;android-35 / The SDK directory is not writable". Install 35 and 36
both, and chown /opt/android-sdk to the build user, since AGP writes into the
SDK itself (it installs missing components and maintains .knownPackages).
2. No audio daemon. The pulseaudio *server* package pulls in systemd, and
systemd's postinst blocks forever when the image is built under a runtime
without systemd - observed under FreeBSD's podman + Linuxulator, where
"systemd.postinst configure" hung the build indefinitely. Only the client
libraries (libasound2, libpulse0) are needed so webrtc-java natives can link;
the worker never plays media back. start-worker.sh no longer launches the
daemon either.
3. prepare-build-dirs.sh (new). Gradle tasks create their output trees at
execution time, and on Linuxulator a Gradle-internal directory creation can
fail with "Failed to create parent directory X when creating directory X/Y"
even though the same mkdir -p from a shell succeeds - and the created parents
must be owned by the build user or the next run dies with "Unable to delete
directory". Pre-creating the usual build/reports, build/outputs, build/tmp,
build/intermediates, ... trees as the build user side-steps both. On a real
Linux kernel this script is a harmless no-op.
The apt cache-size bump (APT::Cache-Start) is retained from the earlier revision:
the default 24MB mmap cache cannot parse the jammy package lists and fails with
"Dynamic MMap ran out of room" on hosts where the apt cache is expanded.
|
|
eclipse-temurin:21-jdk plus an explicit toolchain
WHY: the base image the worker used to inherit from (jenkins-kotlin:latest) was
built interactively during the Jenkins era and its Dockerfile was never
committed. It is now nowhere to be found - verified by searching this repo's
full history on both main and backup/full-pre-reset (only *references* to the
name appear, never a definition), the cgit server's repository list (no
jenkins/docker repo; the name 404s), and the build host's filesystem. The
worker therefore could not be rebuilt on any host, which is exactly what
blocked rebuilding it here on FreeBSD.
WHAT: the toolchain that image provided is now reproduced explicitly so the
image builds from source anywhere. Parent is eclipse-temurin:21-jdk-jammy, and
everything master.cfg's pipeline needs is installed on top:
- JDK 21 (modules target Java 17 via jvmTarget=17, so 21 as the toolchain is
correct and matches the master's own comment)
- Gradle 8.14.3 arrives through the committed wrapper (services.gradle.org),
so no Gradle is baked in
- node 20 (NodeSource) for wasmJsNodeTest and the web bundle
- Android SDK fully provisioned non-interactively: cmdline-tools 13114758
(verified HTTP 200 before pinning), then sdkmanager installs platform-tools,
platforms;android-36 and build-tools;36.0.0 - the same platform/build-tools
pair found installed on the known-good Ubuntu host, and the ones AGP 8.13.2
needs
- X11/xcb/xext/xrender/xi/xtst + ALSA + PulseAudio for the webrtc-java native
library
- python3 + buildbot-worker 4.1.0, git/curl/unzip for checkout and deploy
Compatibility: the image still runs as user "jenkins" with HOME=/var/jenkins_home
and GRADLE_USER_HOME=/var/jenkins_home/.gradle, so the existing run-worker.sh
docker run line (including -v jenkins-data:/var/jenkins_home) works unchanged and
the Gradle/Konan caches still persist across rebuilds. The one real regression is
the loss of the old image's pre-warmed caches, so the first build after this
change re-downloads dependencies; that is stated in the file's header comment.
|
|
its artifact to lighttpd; run-worker.sh: mount git_pass+github_token
|
|
androidTarget
|
|
Full pre-reset history preserved on branch backup/full-pre-reset (2ccab79).
|
