The Crash Diary: What Eighteen Months of Live Operation Leaves Behind

There is a specific kind of code that you only write after something breaks.

Not a defensive comment. Not a try-catch around something that might theoretically fail. A specific environment variable set to a specific value, or a specific NumPy operation replaced with its slower but safer equivalent, or a specific thread count cap that is not round and not documented in any configuration guide — because it was not chosen as a preference. It was discovered as a constraint.

This kind of code is a forensic record.

Every environment variable in the startup configuration of a live system is a tombstone for a crash that happened, was diagnosed, and was fixed. The fix is in the variable. The crash is in the absence of documentation — because the crash happened in production and the diagnosis came from reading a core dump or a faulthandler trace rather than from reading a tutorial.

What a Live Supervisor Looks Like

The startup script for the market intelligence live engine manages twenty-two concurrent processes. It is not a Docker compose file and not a cloud service manifest. It is a Python script that launches each service, monitors its health, and handles graceful restart when a process fails.

The first thirty lines of that script are not infrastructure. They are a crash diary.

Single-threaded BLAS: the environment forces all numerical compute libraries — OpenBLAS, MKL, and the platform’s native threading layer — to operate in single-thread mode. This is not a performance choice. Multi-threaded BLAS on Windows, running under Python’s GIL, on a machine that also runs an order-book feed and a WebSocket connection, produces thread contention that manifests as silent deadlock. The process does not crash. It stops producing output. The symptom looks like a hung signal. The cause is in the threading model of a precompiled wheel. The fix is the environment variable.

AVX2 instruction set restriction: one environment variable disables NumExpr’s AVX2 code path; another disables its VML path. These are not conservative defaults. NumExpr ships precompiled against the AVX2 instruction set. When deployed on a machine where the Python environment and the system compiler made different assumptions about available CPU features, NumExpr selects the AVX2 path, executes an unsupported instruction, and the process exits with a SIGILL — an illegal instruction signal. No error message. No traceback. The faulthandler output is the only evidence of what happened.

You do not write those two environment variables unless you received that exact failure, read the crash output, identified NumExpr as the source, and confirmed the fix. The variables are the forensic remainder.

Faulthandler on all threads: Python’s faulthandler module dumps a C-level stack trace on crash. Enabling it on all threads, not just the main thread, is not default behaviour. It is the configuration that makes a crash in a background worker — the WebSocket reader, the database writer, the signal router — visible rather than silent. This line exists because a background thread crashed silently first, and the investigation that followed found no trace of what happened. Enabling faulthandler on all threads was the fix for that specific unknown.

Worker pool cap: the thread pool for a background compute task is capped at a number that corresponds exactly to the size of a watchlist. This is not a round number. It was determined by observing that increasing the pool size beyond the number of concurrent tasks produced queue contention that cascaded into request timeout failures in a separate service. The cap is the number at which the failure stopped occurring.

Why This Record Exists

Academic code runs once. A model is trained, evaluated, and submitted. If it works in the training environment, the work is complete. The failure modes of long-running processes — memory leaks, thread contention, state corruption from interrupted writes, dependency conflicts introduced by a package update — are not encountered because the code is not alive long enough to encounter them.

A live system accumulates its crash diary through operation. Each entry in the diary represents a failure that the developers did not anticipate, encountered in production, diagnosed using the available tools, and fixed. The fix is typically one line — one environment variable, one configuration parameter, one thread count. The line is trivial to write. The knowledge of which line to write took the crash to acquire.

The market intelligence live engine has been running under load for eighteen months. It manages approximately twenty-two services: raw data feeds, order book reconstruction, feature computation, model inference, signal routing, database writes, health monitoring, and notification delivery. Each of those services is a potential failure surface. Each failure that was encountered, diagnosed, and fixed is embedded in the code as a constraint that will never be explained in a comment — because the constraint is self-evident to anyone who has encountered the failure it prevents, and opaque to anyone who hasn’t.

The Connection to TVS Client Work

The forensic record of a live system is the most honest possible representation of operational maturity. It cannot be fabricated. A developer who has never run a multi-threaded Python process under sustained load does not know which environment variables to set, because they have never encountered the failures those variables prevent.

When TVS builds analytical systems — the market intelligence dataset, the TKAD knowledge pipeline, the client-facing reporting infrastructure — the same operational discipline applies. Systems that run once and produce a result are engineering projects. Systems that run continuously, under load, against live data, over eighteen months, accumulate a different kind of knowledge.

That knowledge is not in documentation. It is in the specific constraints embedded in the code: the thread count that is not round, the environment variable that is not explained, the initialisation sequence that waits for a health check before processing. Each constraint is a crash that happened, was diagnosed, and will not happen again.

This is what production engineering actually looks like. Not the absence of failure, but the systematic incorporation of everything that failed.

---

The research programme that runs on this infrastructure is described at The Most Valuable Thing We Built Wasn’t a Signal. The specialist committee architecture the live engine deploys is at The Specialist Committee. The market physics reconstruction that produced the named episode states is at Market Physics: Why We Model Behaviour Before We Model Price.