How We Search for Signals We Haven't Imagined Yet

The standard approach to quantitative feature engineering starts with a human hypothesis. The analyst observes the market, forms a belief about what a relevant signal might look like — order book imbalance at a certain timescale, the ratio of aggressive buying to resting liquidity, the rate of change of spread — and constructs a feature to test it.

This approach has a structural constraint. It discovers signals that the analyst was already in a position to imagine. The search space is bounded by the analyst’s prior knowledge.

Evolutionary symbolic regression removes that constraint.

What a Formula Tree Is

A formula tree is an expression built from primitives and operators, structured as a tree. The leaves of the tree are base measurements — raw observations from the market data: trade volume by direction, order counts, size distributions, bid and ask quantities, spread, mid-price, directional tick sequences. The nodes are transformations applied to those leaves: rolling means, standard deviations, z-scores, slopes, deltas, non-linear activations, arithmetic combinations.

An example formula — not one that exists in the research, but illustrative — might be: the ratio of the 60-second z-score of order book imbalance to the 900-second slope of net aggressive flow. This is a formula no analyst sat down and wrote. It emerged from a process that combined primitives and operators without human guidance.

The space of possible trees given a set of primitives, operators, and time windows is enormous. The evolutionary engine searches it.

The Evolutionary Process

The search begins with a population of three hundred formula trees, generated randomly from the available building blocks. Each tree is evaluated against a target variable — the directional outcome of the instrument at a defined horizon — and assigned a fitness score.

The fitness function is not simply the information coefficient. It includes a complexity penalty: every node in the tree subtracts a small fixed amount from the score. A formula with twelve nodes must demonstrate substantially better predictive correlation than a formula with three nodes to achieve the same fitness. The penalty actively selects for parsimony. Overfitted formulas with elaborate structure but marginal predictive improvement are pushed out of the population.

The top ten percent of the population — the elite — are preserved unchanged into the next generation. The remaining population is rebuilt through tournament selection: five trees compete, the best advances to reproduce, the others are discarded. Reproduction creates new trees through crossover — subtrees are exchanged between parent formulas — and mutation — individual nodes are replaced or extended.

This process runs for sixty to one hundred and twenty generations. The population at the end has been shaped by selection pressure toward formulas that are both predictive and parsimonious.

The Null Control

Survival into the final generation is not sufficient to be counted as a real discovery.

An identical evolutionary run executes in parallel against a version of the data where the target labels have been randomly permuted. The shuffled targets have no genuine relationship to any feature. A formula tree that achieves high fitness against shuffled targets has overfit to noise — the evolutionary process found random structure in the random permutation.

The null population evolves under identical conditions: same primitives, same operators, same fitness function, same number of generations. It produces its own survivors with its own best fitness scores.

A formula from the real population counts as a discovery only if its fitness exceeds the null population’s best result. If the null can evolve an equivalently predictive formula against random data, then the real formula’s performance is not evidence of a genuine edge — it is evidence of the search process’s capacity to find patterns whether they are real or not.

This is the correct statistical test for symbolic regression. It is rarely applied.

The Train-Test Discipline

The evolutionary process runs on training data from one period. A formula that survives the null-control test in training is then evaluated on test data from a completely separate period — not a rolling window, not a holdout at the end of the same dataset, but a distinct and temporally distant market era.

Markets are non-stationary. A formula that overfit to a specific period’s noise will not generalise to a different period. The train-test split across months rather than within a single continuous dataset is the mechanism that catches this. A formula that survives training, beats the null, and then holds its predictive performance on a temporally separate test set has demonstrated a degree of robustness that in-sample testing cannot provide.

Most candidate formulas fail at this stage. The ones that don’t have earned the status of candidates, not discoveries.

Why This Matters Beyond Finance

The symbolic regression engine is a concrete instantiation of a general principle: search in a space defined by first principles, not by prior expectations.

The primitives — the leaves of the formula trees — are the raw measurements the domain makes available. The operators are the transformations that can be applied to any measurement. The search explores the combinations. The formulas that survive are shapes that the data selected, not the analyst.

The same principle governs how TVS approaches search ranking intelligence. The observable signals from search results — rankings, visibility patterns, local pack presence, content characteristics, engagement signals — are the primitives. The question is not what we expect to matter but what the data demonstrates actually predicts ranking outcomes. The disciplined search structure, the null control, and the train-test separation apply directly.

The difference between a hypothesis confirmed and a pattern discovered is the direction of the search. In the genetic engine, the formula comes from the data. In the null control, the data decides whether the formula counts.

---

The leak detection infrastructure that validates candidate signals is described at The Most Valuable Thing We Built Wasn’t a Signal. The specialist committee architecture that receives validated signals is at The Specialist Committee. The correlation discovery methodology is at The Correlation Engine.