Trelix v2.7 to v2.9: Fixing the Pipeline Debt

Most developers treat the CI/CD pipeline as a "set it and forget it" utility, but the jump from Trelix v2.7 to v2.9 proved that the shipping infrastructure is actually a product surface that can fail just as spectacularly as the core logic. I hit a wall when I realized my GitHub Release page was missing assets. The release.yml was building macOS and Linux PyInstaller binaries both as dist/trelix, and since softprops/action-gh-release uploads by basename, the files collided. I had no way of knowing which OS actually survived the upload. It wasn't a bug in the retrieval logic—it was a failure in the delivery mechanism that passed CI because the workflow turned green, but the output was broken.

The Infrastructure Debt

Fixing the binary collision in v2.7.1 was the start of a two-week audit. I discovered that the PR-time CI workflow had never actually built a Linux binary, despite the release workflow doing so at tag time. I had to manually add the missing matrix entry and a verification step to ensure parity.

The most frustrating find was in trelix-mcp. The test suite wasn't even wired into CI. This allowed a regression to sit undetected where a test asserted the MCP server had "exactly 6 tools," while it actually had 8 (due to subscription tools added in v2.5.0). A test that passes for the wrong reason is worse than having no test at all.

To fix the deployment pipeline and avoid future collisions, I updated the build script to rename binaries uniquely before the upload step:

# Example fix for binary naming collision in GitHub Actions
for binary in dist/trelix*; do
  os_name=$(echo "$binary" | cut -d'-' -f1) # Assuming naming convention like macos-trelix
  mv "$binary" "dist/trelix-${os_name}"
done

Concurrency and Race Conditions

v2.7.2 was where I stopped pretending check_same_thread=False made my code thread-safe. While I added Qdrant Cloud readiness and parallel BM25 read pools, the real value was in the five concurrency bugs I unearthed via stress tests.

I found a TOCTOU (Time-of-Check to Time-of-Use) race in the sparse embedder's lazy-load. Two threads could check if a model was loaded, both see "False," and both attempt to initialize the model simultaneously. I solved this using double-checked locking.

I also dealt with an MCP stdout write race where concurrent notification writes interleaved partial JSON-RPC lines, corrupting the client's output. The fix required a strict lock around the write-and-flush sequence:

import threading

# Fix for MCP stdout interleaving
_write_lock = threading.Lock()

def safe_write(data):
    with _write_lock:
        sys.stdout.write(data)
        sys.stdout.flush()

Other critical fixes included implementing a max-subscriber cap and a TTL sweep for the subscription registry to prevent unbounded memory growth from misbehaving clients.

The Silent Data Corruption Bug

The most unsettling issue was silent foreign-key corruption during partial re-indexing. In the database schema, parent-symbol, call-callee, and type-edge columns were set to null on delete. While this seems safe, deleting a changed symbol's old row silently nulled those links on every row that referenced it—including unchanged rows.

There was no error message; the graph edges were just quietly disappearing as files changed. To stop this "silent rot," I had to implement a snapshot-and-reverify mechanism to ensure referential integrity during the re-index process.

If you're building an LLM agent or a complex retrieval system, don't just test the "happy path" of your AI workflow. Stress test the concurrency and actually check your release assets. The pipeline is part of the product.