spark-folia-patched

Patched build of the spark profiling plugin for Folia, with our JfrReader DirectByteBuffer leak fix applied. Built nightly + on push by Forgejo Actions, released as a downloadable jar under Releases.

The bug

me.lucko.spark.common.sampler.async.jfr.JfrReader allocates DirectByteBuffer in two places (constructor + ensureBytes() resize) and never explicitly frees them — close() only closes the FileChannel. Off-heap memory is reclaimed only when the GC's PhantomReference Cleaner eventually fires.

When backgroundProfiler: enabled is true (default), AsyncSampler.rotateProfilerJob() opens a new JfrReader every ~10s. On a JVM with -XX:+DisableExplicitGC (legacy Aikar's flag), the JVM's own self-rescue via System.gc() is blocked when MaxDirectMemorySize is hit → OOM thrown even though GBs of release-able direct memory are pending.

Reproduced in production on 2026-05-04 (twice within 5 hours, identical stack):

java.lang.OutOfMemoryError: Cannot reserve 2097152 bytes of direct buffer memory
  (allocated: 2146629621, limit: 2147483648)
        at me.lucko.spark.common.sampler.async.jfr.JfrReader.<init>(JfrReader.java:89)
        at me.lucko.spark.common.sampler.async.AsyncProfilerJob.aggregate(AsyncProfilerJob.java:213)
        at me.lucko.spark.common.sampler.async.AsyncSampler.rotateProfilerJob(AsyncSampler.java:156)

This was first hit during a production incident: two OOMs within five hours from the same stack trace on a Folia server running spark's background profiler.

The fix

jfrreader-direct-buffer-leak.patch adds a freeDirectBuffer(ByteBuffer) helper that calls sun.misc.Unsafe.invokeCleaner(buffer) via reflection (JDK 9+ standard, no --add-opens needed because sun.misc lives in the always-open jdk.unsupported module). The helper is invoked:

1. In close() after ch.close(), on this.buf
2. In ensureBytes() after the resize, on the abandoned oldBuf

Reflection is cached at static-init time; if it fails (weird JDK with restricted access), the helper silently falls back to GC-based cleanup. No new failure mode introduced — worst case is "behaves like upstream".

This bug is not yet reported upstream (verified via GitHub API search 2026-05-04). We'll file an issue + PR after 1 week of clean production runtime.

Architecture: why the build is two-step

spark-folia is not in lucko/spark upstream. The official lucko/spark repo publishes 9 platform variants (bukkit, paper, velocity, fabric, forge, neoforge, sponge, bungeecord, standalone-agent) — no folia. The Folia variant lives in the community-maintained lucko/spark-extra-platforms repo, alongside Hytale/Geyser/Minestom/Nukkit variants.

spark-extra-platforms/spark-folia/ depends on me.lucko:spark-common-SNAPSHOT which it pulls from repo.lucko.me. So to ship our patch in the final folia jar, we have to:

1. Patch lucko/spark source (only JfrReader.java in spark-common)
2. Build + publish patched spark-common to local Maven cache (~/.m2)
3. Build spark-extra-platforms/spark-folia with mavenLocal() listed first in repositories — otherwise gradle resolves spark-common from repo.lucko.me and silently produces a jar without our patch

This trap caught us once during initial development (built jar had freeDirectBuffer field absent from the bytecode — caught by a javap check before deploy). The CI workflow now enforces the marker check + fails fast if any of the 3 patch markers are missing in the final jar's bytecode.

Build pipeline (CI)

.forgejo/workflows/build.yml runs:

• On push when the patch file or this workflow itself changes
• Nightly at 02:30 UTC (rebuilds against latest upstream HEAD of both repos)
• Manual dispatch via Forgejo UI / API

Steps:

1. Checkout this repo (for the .patch file)
2. Set up JDK 21 (Temurin, via actions/setup-java@v4)
3. Clone lucko/spark master + git fetch --tags (needed for git-describe versioning)
4. git apply our patch + grep-verify source markers
5. ./gradlew :spark-common:publishToMavenLocal + javap-verify class in published jar
6. Clone lucko/spark-extra-platforms master
7. Prepend mavenLocal() to repositories block in build.gradle (idempotent Python edit)
8. ./gradlew :spark-folia:shadowJar (NO --refresh-dependencies — would bypass mavenLocal)
9. javap verify freeDirectBuffer + INVOKE_CLEANER + UNSAFE markers AND close() bytecode invokes freeDirectBuffer. Build fails if any check missing.
10. Compute SHA-256 + tag = build-YYYYMMDD-HHMM-<sha12>
11. Create Forgejo release via API + upload jar as asset

The workflow uploads the jar as a workflow artifact too (30-day retention), so even if the release-upload step fails (e.g. missing FJ_TOKEN secret), the jar is still downloadable from the run page.

Setup: required Forgejo secret

For releases, the repo needs a secret called FJ_TOKEN (Forgejo blocks names prefixed with FORGEJO_* as reserved). Add via repo Settings → Actions → Secrets, value = a Forgejo PAT with write:repository scope.

Without this secret, the build still works and the jar is available as a workflow artifact — only the auto-release step is skipped (with a warning).

Manual build (no CI)

# 1. Build patched spark-common
git clone https://github.com/lucko/spark.git /tmp/spark
cd /tmp/spark
git fetch --tags                                # for git-describe versioning
git apply /path/to/jfrreader-direct-buffer-leak.patch
./gradlew :spark-common:publishToMavenLocal

# 2. Build spark-folia
git clone https://github.com/lucko/spark-extra-platforms.git /tmp/spark-extra
cd /tmp/spark-extra
# Prepend mavenLocal() — CRITICAL
sed -i 's|mavenCentral()|mavenLocal()\n        mavenCentral()|' build.gradle
./gradlew :spark-folia:shadowJar
ls spark-folia/build/libs/spark-*-folia.jar

# 3. Verify
JAR=spark-folia/build/libs/spark-*-folia.jar
unzip -p $JAR me/lucko/spark/common/sampler/async/jfr/JfrReader.class \
    | javap -p | grep -E 'freeDirectBuffer|INVOKE_CLEANER|UNSAFE'
# Must print 3 lines. If empty → gradle picked the unpatched spark-common from repo.lucko.me.

Deploy

Drop the jar into your Folia plugins directory:

# Grab the built jar from your CI's releases page (or build it yourself, below)
curl -L -o spark.jar \
    "$RELEASE_URL/spark-X.Y.Z-SNAPSHOT-folia-jfrreader-leakfix.jar"

# Backup current + replace
mv plugins/spark-*.jar plugins/.bak/
mv spark.jar plugins/

# Restart MC
docker restart <mc-container>

# Verify in container after boot
docker exec <mc-container> bash -c 'ls /data/plugins/spark*'

Recommended companion JVM flags (covers the same failure class for any plugin that allocates DirectByteBuffer, not just spark):

-XX:-DisableExplicitGC                 # was +, allow System.gc()
-XX:+ExplicitGCInvokesConcurrent       # but route through G1 concurrent (no STW)
-XX:MaxDirectMemorySize=6G             # if currently 2G, was too tight for spark
-XX:+ExitOnOutOfMemoryError            # graceful exit (Paper save world hook runs)

Do NOT add -XX:+CrashOnOutOfMemoryError or -XX:OnOutOfMemoryError="kill -9 %p" — both bypass Paper's shutdown hook → no world save → chunk corruption likely.

Upstream

• spark: https://github.com/lucko/spark
• spark-extra-platforms: https://github.com/lucko/spark-extra-platforms
• our patch: jfrreader-direct-buffer-leak.patch (in this repo)