The Correlation Engine: How We Learned to Find What Actually Matters in Data

The claim that a company understands data is easy to make and almost impossible to verify from the outside. The tools are commoditised. The dashboards look the same. Every analytics platform promises insight.

What separates genuine analytical capability from a subscription to someone else’s reports is the ability to build the discovery infrastructure yourself — and to know which methods to apply when, and why.

We built that infrastructure for a domain far more demanding than search ranking. And the architecture we developed there is directly informing how we intend to analyse the TVS market intelligence dataset as it matures.

The domain

Over an extended research period we built a correlation discovery engine applied to high-frequency financial market microstructure data — the kind of data generated at the level of individual order book events, measured in five-second intervals, producing hundreds of thousands of labelled data points per day.

The research question was: given 445 candidate predictor features spanning order book structure, trade flow, market participant behaviour, timing overlays, momentum conditions, and constructed composite signals, which combinations of features — in which temporal configurations — actually predict directional movement? And critically: which ones hold up out of sample?

That question is not answered by running a regression. It requires a staged, structured discovery process designed from the ground up to prevent the most common failure mode in data research: finding correlations that are real in the sample, and absent in production.

The methods

Event library construction. The first step was building a structured catalogue of tagged outcome events — directional moves classified by type (long, short, neutral), temporal horizon (from 30 seconds to 30 minutes), and magnitude bucket (from 3 to 40 basis points). A cooldown mechanism prevented adjacent events from dominating the sample. This produced a clean, representative set of outcomes to use as labels for all subsequent discovery work.

Timeline fingerprinting. For every event in the library, we computed 12 statistical metrics across 9 backward-looking time windows — from 90 minutes before the event down to the final 15 seconds. The metrics: latest value, mean, minimum, maximum, range, slope, rate of change per step, positive persistence, negative persistence, zero-cross count, mean absolute value, and late deviation from background. Applied across 445 features, this produced a high-dimensional characterisation of what each feature was doing in the structured run-up to each outcome. The goal was not to find features that were high or low at the moment of the event — it was to find features with specific behavioural signatures in the hours, minutes, and seconds before.

Stability scanning. Before interaction discovery, every feature was tested individually. Long outcomes versus short outcomes. Long versus neutral. Short versus neutral. Across every horizon and magnitude bucket combination. This separated three meaningfully different feature types: directional features (which distinguish long from short), move-readiness features (which distinguish moves from flat, regardless of direction), and magnitude features (which scale with how large a move becomes). Features that were stable across this scan became candidates. Features that were not were excluded from the combinatorial search.

Pairwise interaction screening. This is where the scale of the work becomes significant. The engine tested all meaningful combinations of feature pairs in six conditional interaction forms: A elevated while B elevated, A elevated while B suppressed, A accelerating while B persistent, A rate-of-change spike after B has maintained a state, A flip-rate regime coinciding with B confirmation, and long-window state in A acting as context for a short-window trigger in B. The last form — slow-building context plus fast-moving trigger — turned out to be the most predictive structure in the dataset. Liquidity structure that builds over 10 to 90 minutes, followed by a specific short-window event in the final 15 to 30 seconds, was a materially stronger predictor than any single feature examined in isolation.

Sequence and onset mining. Beyond pairwise interaction, the engine searched for three-stage temporal structures: a background condition active over long windows, a formation condition that persists in the intermediate window, and a trigger condition that accelerates in the final seconds. For each candidate triplet, we measured onset gap (how far before the event the background became active), overlap duration, and the stability of the sequence across days and sessions.

Genetic algorithm optimisation. Rather than manually specifying feature thresholds and interaction windows, the engine used evolutionary search to explore the parameter space. Candidate feature combinations were evaluated, crossed, and mutated across generations — allowing the discovery process to find threshold combinations that would not have been obvious from inspection alone.

GRU specialist committee. A parallel research track applied Gated Recurrent Unit neural networks — a sequence-learning architecture well-suited to time-series data — across six specialist domains (market structure, aggression, order flow, large-participant activity, momentum, and context). Each specialist was a two-layer GRU trained on 346,753 five-second bars across 22 in-sample days, then evaluated on unseen out-of-sample dates using Spearman rank correlation information coefficient matrices. The cross-horizon evaluation was particularly revealing: a model trained on 30-second prediction horizons still produced meaningful information coefficient at three-minute evaluation horizons for some specialists, confirming that certain market structure signals genuinely carry information across timescales. Others collapsed almost immediately — an important finding that ruled out certain features from multi-horizon deployment.

Cross-symbol lead-lag analysis. We extended the engine to test whether one asset’s order flow predicted another’s directional movement — and at what temporal offset. The strongest measurable lead-lag relationship appeared at the 10-second window. By the 60-second window it had largely collapsed. This kind of temporal decay characterisation — knowing not just whether a relationship exists but over exactly what window it is reliable and where it degrades — is the difference between a finding you can act on and a finding you cannot.

Chronological freeze and branch discriminator audit. Every candidate that survived the discovery phases was subject to a strict chronological replay. No future data could be used. Every occurrence of the candidate pattern was identified in time order and its outcome recorded. Outcomes were split into expected, inverse, and neutral branches. The branch audit then searched for discriminating features inside each branch — to understand not just that a signal had edge, but under what conditions it produced expected outcomes versus inverse or failed ones. This step alone eliminated a significant proportion of candidates that had looked promising in discovery but did not hold up under strict temporal validation.

What this produces

A correlation discovery process of this depth does not produce a list of correlations. It produces a validated, falsifiable understanding of which features are directional, which are move-readiness detectors, which are magnitude predictors, and which are genuine multi-stage interaction structures.

It also produces a methodology that is domain-agnostic. The stages — event labelling, timeline fingerprinting, stability scanning, interaction discovery, sequence mining, genetic optimisation, recurrent network evaluation, chronological validation, branch discrimination — apply to any problem where you have time-series data and measurable outcomes.

The connection to TVS market intelligence

The TVS market intelligence system is currently tracking 249 businesses across 18 industries in KwaZulu-Natal. Every day it adds another layer of data: search position for each tracked query, blog publishing activity, schema implementation status, technical scores, review prominence. Over time, the dataset will include social media signals, competitor position changes, and cross-industry patterns.

That dataset is, structurally, the same kind of problem the correlation engine was built to solve. The outcomes are search positions — measurable, daily, directional (moving up or down). The features are the observable attributes of each business: review count and recency, content cadence, schema types, technical quality, link profile, social media publishing frequency, competitor position changes in the same query space.

The question — which combinations of business attributes, in which temporal configurations, actually predict position change — is the same question the engine was built to answer. And the failure mode to avoid is the same: finding correlations that look real in the current dataset and do not generalise as the market evolves.

We are not at the stage of running that analysis on the TVS dataset yet. The dataset is still building. But the infrastructure — the discovery protocol, the validation discipline, the understanding of which methods surface stable signals versus artefacts — is already in place.

When the data is deep enough, the analysis will not be a spreadsheet correlation. It will be a staged, validated, chronologically frozen discovery process applied to one of the least-understood datasets in South African digital marketing: a daily, multi-year record of what every significant business in this market is doing, and where they rank because of it.

---

The Signal 1 retraction — the moment this methodology proved its worth — is at The Most Valuable Thing We Built Wasn’t a Signal. The specialist committee architecture that the discovery process produced is at The Specialist Committee. The evolutionary search method that finds formulas no analyst would design is at How We Search for Signals We Haven’t Imagined Yet. If you want to understand what the data currently shows about your market, speak to us.