After the Discovery: The Architecture of a System That Actually Remembers

The previous article in this series described the staged correlation discovery process we built for high-frequency financial market microstructure data — event labelling, timeline fingerprinting, stability scanning, interaction discovery, sequence mining, genetic optimisation, neural specialist committees, chronological freeze.

Discovery is one phase. What comes after discovery is harder, takes longer, and is where most analytical efforts fail.

The harder phase is turning discovered signals into a production system that operates reliably, generates new evidence every day, and grows more capable as the dataset deepens. This is what we built next.

The golden dataset problem

Early in any quantitative research programme, the data landscape fragments. One analysis joins three sources to answer one question. A different analysis joins two different sources to answer a different question. The time alignment differs slightly. The row definition differs slightly. The results are not directly comparable. Each new finding is a standalone study, not an increment on a shared foundation. The research cannot compound.

Every analytical programme eventually reaches a decision point: continue building disposable, per-question joins, or build the shared foundation that makes all future questions cheaper to answer.

The architectural decision is to build a single, authoritative, episode-level dataset. One row per episode. Every measurable attribute of that episode — from the raw market microstructure readings through every derived and composite signal — as a named column. Every outcome variant — directional result at 30 seconds, 2 minutes, 10 minutes, 30 minutes; magnitude bucket; branching classification — as additional named columns.

We call this the golden dataset. It is not built once and frozen. It is maintained as the authoritative record of all research history. Every new analytical finding is not a separate study with its own data join. It is a new set of columns appended to the same record. Research compounds. Every signal discovered in phase one is queryable alongside every signal discovered in phase four. Findings from different discovery runs are directly comparable because they share a common row definition and a common time alignment.

The difference in practice is this: in a system without a golden dataset, answering a new question requires reconstructing the conditions that produced the original data. In a system with one, answering a new question is a query.

The live signal layer

The most technically distinctive component of the production system was not the collection infrastructure, though that is substantial. It was the integration of a large language model as a real-time signal generator — not as a text interface, and not as an offline model, but as a live analyst generating quantitative scores at every interval.

Standard quantitative practice treats machine learning as a batch process: train a model on historical data, deploy the weights, feed it features at inference time. The model is static. It encodes what was true during training. It does not read, interpret, or reason about the current state — it computes.

We took a different approach. We integrated a language model as a running analyst that consumed the current market state as structured context at every observation interval and produced a proprietary composite signal as output. That signal was not a classification label. It was a directional certainty score — a filtered interpretation of a high-dimensional, high-noise stream, compressed into a single scalar value.

The results of this integration were the most significant finding of the entire research programme. The language model-generated signal became the dominant predictor in the dataset. Its information coefficient correlation with the primary directional outcome measure at lag zero was materially stronger than any feature constructed from raw market data alone — by a margin significant enough that the signal was not one contributor among many. It was the signal. It was not a supplementary feature or a sanity check. It was the strongest signal the system had discovered, and it was being generated live, in real time, by a language model reading structured market context as input.

What this means technically: a language model, given well-structured context and a clear analytical task, functions as a noise filter. It reduces the information in a complex, high-dimensional stream to a scalar score that carries more predictive signal than the raw stream itself. This is not how language models are typically described or deployed in quantitative settings. The standard framing is that language models handle unstructured text and quantitative models handle structured data. We found that the framing is wrong. A language model with a well-specified context schema operates as a quantitative signal generator that outperforms hand-crafted features on the same data.

A second language model integration extended this further: live generation of two additional proprietary delta signals — a large-participant behaviour index and a structural velocity overlay — using a different context schema designed to surface information that the primary signal did not capture. These signals were evaluated as independent columns in the golden dataset, compared against the primary signal, and confirmed to carry non-redundant information.

The two-engine architecture

The production system runs on two processing layers operating in sequence.

The structural layer operates continuously. It maintains real-time state across multiple concurrent data streams — order book depth, trade flow, large-participant activity, momentum structure, and supplementary market signals. It runs the pattern detectors produced by the correlation discovery phase. When a structural candidate condition is identified — the multi-stage temporal signature that the discovery process found to be predictive — it raises an event.

The context layer evaluates events raised by the structural layer. When an event arrives, the context layer assesses the broader market environment: current regime classification, cross-market lead-lag state, data freshness indicators, and contextual conditions under which the structural pattern has historically been reliable versus historically been unreliable. Candidates that are structurally valid but contextually unsupported are suppressed. Only candidates that pass both layers propagate.

The two-engine design exists because structural pattern matching and contextual evaluation are different problems. The structural layer is a shape detector — it identifies when the data looks like a known candidate pattern. The context layer is an epistemological check — it asks whether the conditions under which that pattern has been reliable are currently present. Conflating these two functions produces a system that acts on every structural candidate regardless of whether the broader environment supports it. Separating them produces a system that acts selectively, with the contextual filter acting as a second opinion on every structural finding.

The data collection infrastructure

Supporting both layers is a collection infrastructure running as multiple concurrent services with dedicated responsibilities, independent restart policies, and health-check logic.

Real-time data streams maintain continuous state without polling. Supplementary data sources are fetched on schedule. A document-level historical store preserves every raw input from every stream in a format that permits retrospective analysis of any prior period without reconstructing the original collection run.

Shared data artefacts — the derived state files read by both the structural and context layers — are protected by named file locks, preventing data races between asynchronous services operating on the same files. A shared event queue replaces file-based polling for inter-service communication. When the structural layer raises a candidate event, it is placed in a priority queue with a time-to-live constraint. The context layer consumes from the queue. Transit from structural detection to context evaluation is sub-second.

The warm-start governance layer deserves particular attention. The context evaluation engine does not begin processing incoming candidates until the quality of its input data is verified. If a supplementary data source is stale or unavailable on startup, the context engine waits. It does not operate on degraded inputs and pass the degradation downstream.

This is an architectural discipline that is easy to state and difficult to maintain. The pressure in any live system is always toward availability — toward returning a result, any result, rather than declining to operate when inputs are inadequate. The discipline of refusing to operate on degraded inputs is the difference between a system that is reliably accurate when conditions are good and a system that generates plausible-looking output under all conditions, including conditions where that output should not be trusted.

The compounding research effect

After eighteen months of operation, the golden dataset contains hundreds of thousands of rows, each with dozens of signal columns accumulated across multiple research phases. Every research question asked since the first discovery run is answerable by querying the same table. Signals discovered in phase one can be evaluated against signals discovered in phase six on the same dataset, with the same time alignment, without reconstructing any joins.

The effect of this architecture over time is that the research programme accelerates. Each new discovery phase builds on the full signal history of every prior phase. Redundancy between signals is detectable because all signals exist in the same table. Interaction effects between signals discovered at different times are analysable because the temporal alignment is shared.

Research that does not compound tends to plateau. Research built on a golden dataset architecture tends to deepen. The infrastructure determines whether the work accumulates or restarts.

What this means for TVS

The TVS market intelligence system is building the same architectural foundations, in a different domain.

Every day the system adds a new observation to each tracked business record — search position for each monitored query, content publication activity, schema implementation changes, technical quality score, review prominence, competitor position changes in the same query space. The dataset grows by one row per business per day across 249 tracked businesses and 18 industries.

That structure is, architecturally, identical to the golden dataset problem described above. One row per episode — here, one business on one day. Every measurable attribute as a column. Every position change as an outcome column.

The research questions the TVS dataset will eventually support are the same class of question the correlation discovery engine was built to answer. Which combination of attributes, in which temporal sequence, reliably precedes a position improvement for a given query type in a given industry? Which content activities have lead times of days versus weeks before their position effect is measurable? What cross-business lead-lag relationships exist — does a competitor’s content activity predict a position shift in other businesses competing for the same queries, and at what temporal offset?

These questions cannot be answered by looking at a single business’s data in isolation. They require the kind of cross-entity, time-series discovery that the correlation engine was built to perform. They also require the kind of data discipline — golden dataset architecture, chronological freeze, branch discriminator audit — that prevents finding patterns that are real in the current dataset but do not generalise.

The TVS dataset is not yet deep enough to run this analysis at the level of reliability that the financial market research reached. Markets generate hundreds of thousands of observations per day; search ranking data generates one observation per business per day. The convergence of statistical power takes longer.

But the architecture — the decision to build data the right way from the beginning, to maintain a single authoritative record, to apply validated discovery methods only when the data warrants them — is in place. The financial market research was, in retrospect, preparation for this. The methods are domain-agnostic. The discipline is the same. The difference is the data maturation timeline.

---

The discovery methodology this dataset was built to support is at The Correlation Engine. The Signal 1 retraction — where the validation infrastructure found a leak the methodology would have missed — is at The Most Valuable Thing We Built Wasn’t a Signal. If you want to understand specifically where your business stands in the KZN search market and what the data suggests would move it, speak to us.