tlog-patched

Two unrelated fixes that ekaii needs to run tlog reliably:

1. watchdog.patch — fixes the tlog-rec-session spin loop bug observed during the 2026-05-06 incident on the affected host (Ubuntu 24, see below).
2. ubuntu26-utmp.conf + install-utmp-ubuntu26.sh — restores /run/utmp on Ubuntu 26+ so that tlog-rec can complete its I/O tap setup. Discovered 2026-05-23 on the affected host. Required prerequisite on Ubuntu 26+ hosts — without it, every recorded SSH session closes instantly with "Failed setting up the I/O tap".

---

Fix 1 — spin loop watchdog (Ubuntu 24, tlog 14)

What this fixes

When a tlog-recorded shell runs a pipeline like cmd | head -N, after head exits the wrapped tlog process can enter an infinite user-space loop in libtlog.so : tlog_rec → tlog_sink_write → tlog_pkt_pos_cmp, burning 1 CPU core forever with 0 syscalls (verified by strace -c -p $PID).

Root cause: the position state machine in lib/tlog/rec.c:tlog_rec_transfer never reaches a terminal state when the child has exited but buffered packets remain. None of the existing break conditions (EIO, tlog_pkt_is_eof, write-error) fire, and tlog_sink_write returns OK without advancing the position cursor.

The kernel can't help: pure user-space loop, no wchan, scheduler thinks the process is doing legitimate compute.

The patch

watchdog.patch adds a 30-second watchdog inside the main loop:

• After every successful tlog_sink_write or tlog_source_read, record time(NULL) as watchdog_last_progress.
• At each loop iteration, if tlog_rec_child_exited is true AND time(NULL) - watchdog_last_progress > 30, force exit with ETIMEDOUT and log "Watchdog: child exited and no flush progress in 30s, forcing exit".

Total: 21 lines added, 0 removed. No behavior change for healthy sessions — the watchdog only fires when the bug condition is hit.

Status

• Patch written
• Builds cleanly against tlog 14 source (Ubuntu 24)
• Patched binary runs without crashing in non-spin scenarios (smoke test)
• Spin reproduction test (the bug requires a specific TTY+buffer condition that's hard to reproduce in a test rig — relying on production observation with LTSP-0.1 watchdog as fallback)
• Upstream PR to Scribery/tlog
• Production deploy (waiting on reproduction test + 24h soak in non-prod)

Mitigation in place while patch isn't deployed

The affected host's security-report/systemd/tlog-orphan-killer.{sh,service,timer} (LTSP-0.1 v2, commit b076464) is an external watchdog that kills any tlog-rec-session process matching:

• CPU > 90%
• age > 5 min
• utime/etime ratio > 90% (sustained CPU burn signal)

This catches the bug in <5 min worst case, vs the patched binary which catches it in <30 s.

DEPLOYMENT GUARDRAILS

• DO NOT install on a separate production without:
  1. Separate reproduction test on a a separate production host
  2. 24h soak on one another environment host before fleet-wide rollout
• DO NOT replace /usr/lib/x86_64-linux-gnu/libtlog.so.0.0.0 on the affected host without:
  1. Backup the current binary
  2. Smoke test: open a fresh SSH session, verify recording still works
  3. Verify journalctl / rsyslog still receives session logs
• The current LTSP-0.1 v2 mitigation is sufficient for the affected host — this patch is an upstream fix candidate, not an emergency deploy

How to build

git clone https://github.com/Scribery/tlog.git
cd tlog
git apply path/to/watchdog.patch
autoreconf -i
./configure --prefix=/usr --enable-journal
make -j$(nproc)
# Patched library:    lib/tlog/.libs/libtlog.so.0.0.0
# Patched binary:     src/tlog/.libs/tlog-rec-session

How to verify the patch is compiled in

strings lib/tlog/.libs/libtlog.so.0.0.0 | grep -i watchdog
# Expected:
#   Watchdog: child exited and no flush progress in 30s, forcing exit
#   watchdog_last_progress

---

Fix 2 — Ubuntu 26 missing utmp (tlog 14-2.1 from resolute/universe)

Symptom

$ ssh user@the-affected-host
Welcome to Ubuntu 26.04 LTS ...
Failed adding a utmp record
Success
Failed removing utmp record
Failed setting up the I/O tap
Shared connection to the-affected-host closed.
$

The connection closes immediately. tlog-rec exits with 0 but never actually records anything. If you exec tlog-rec ... from /etc/profile.d/01-tlog.sh the user has no shell to fall back to and is kicked out.

Root cause

Ubuntu 26.04 (resolute) dropped the tmpfiles entry that creates /run/utmp at boot. systemd-logind itself no longer needs the flat file (it tracks sessions via D-Bus), but tlog-rec calls libutempter to register the pty in utmp during tap setup. libutempter opens /run/utmp, gets ENOENT, returns failure → tlog treats this as a fatal tap setup failure and aborts the recording entirely.

Confirmed via strace from a pty context:

openat(AT_FDCWD, "/dev/ptmx", O_RDWR) = 4                      # pty fine
newfstatat(AT_FDCWD, "/dev/pts/7", ...) = 0                    # slave fine
access("/var/run/utmpx", F_OK) = -1 ENOENT                     # ← here
openat(AT_FDCWD, "/var/run/utmp", O_RDONLY|O_CLOEXEC) = -1 ENOENT  # ← here

Fix

Add the missing systemd-tmpfiles config and create the files immediately:

sudo bash install-utmp-ubuntu26.sh

This is non-invasive — it only restores the file layout that earlier Ubuntu releases had by default. Side benefit: who(1), w(1), last(1) start working again.

Verify

ls -la /run/utmp /var/log/wtmp /var/log/btmp
# /run/utmp     should be 0664 root utmp
# /var/log/wtmp should exist (may already, untouched if so)

# Then test recording:
sudo -u someuser script -q -f \
  -c "/usr/bin/tlog-rec --writer=file --file-path=/tmp/t.jsonl /bin/echo OK" \
  /dev/null
# Expected: prints "OK", no "Failed setting up the I/O tap"
# /tmp/t.jsonl should be ~250 bytes with a valid JSONL record.

Status

• Confirmed root cause on the affected host (Ubuntu 26.04, tlog 14-2.1)
• Workaround tested and deployed on the affected host 2026-05-23
• File upstream bug against tlog (consider utmp failure non-fatal)
• File upstream bug against Ubuntu (restore tmpfiles utmp entry, or remove libutempter dep from anything that links against it)

---