Methods

The standing methodological contract. Written before the analysis, and binding on it.

This site exists because it is easy to produce a confident statistic that is worthless. The defences below are structural rather than aspirational: most are enforced by tests, and the ones that are not are stated so a reader can hold the work to them.

They are also subject-independent. A subject brings its own data sources, its own literature and its own thresholds — those live with the subject, and the football ones are in the football methods. What follows is what does not change.

1. The tri-anchor rule

No claim is published unless three things are present and named in the report:

A finding with two legs is a watchlist candidate, not a publication.

2. Metric provenance

Every metric carries one of three tags, rendered visibly wherever the metric appears:

• LITERATURE — defined or used in peer-reviewed work, cited.
• INDUSTRY — an established convention. Sourced, but explicitly not peer-reviewed.
• COINED — ours. Labelled as ours, every time, with the reasoning and the caveats.

A coined metric is never described as established. This rule exists because the transcript that seeded this project presented an invented metric as though it came from the literature.

3. Uncertainty before ranking

Raw ratios on small samples are noise. Therefore:

• Estimates are partially pooled toward the population mean (empirical Bayes / hierarchical model). Rankings use the shrunken posterior, never the raw value.
• Every point estimate is published with an interval.
• Below a metric's reliability threshold, nothing is ranked at all.

The shrinkage weight B = (α+β)/(α+β+n) → 1 as n → 0, so a thin-sample estimate collapses to the population mean on its own. That self-silencing property is the primary defence, and it is free.

Known implementation trap. The method-of-moments prior must subtract sampling variance: s2_true = max(s2_obs − s2_samp, ε). Omitting s2_samp overestimates between-unit variance, weakens shrinkage, and manufactures exactly the outliers this pipeline exists to suppress. This has a dedicated regression test against synthetic data with known τ.

4. Stabilisation

For a beta-binomial, reliability is r(n) = n/(n + n₀) where n₀ = α+β — so the stabilisation point is the prior sample size, obtained free from the model already fitted.

A general rule this once cost us, recorded because it generalises past the subject that taught it: absence of evidence in your own search is not evidence of absence in the literature. An earlier version of this document claimed no peer-reviewed work existed on a question. It did, it was findable, and any reviewer in the field would have known it. A novelty claim needs a positive search that found the neighbours, not a negative one that found nothing.

5. Multiple comparisons

Scanning many units across many metrics manufactures findings by chance. Controls:

• Partial pooling handles units-within-metric.
• The metric set is pre-registered and frozen; adding one after seeing the data is a pipeline failure with a test to catch it.
• One publication per cycle maximum, selected by a pre-registered priority order.
• Bayesian FDR E[FD] = Σ(1−S) logged, published set capped at 0.10.
• A shadow Benjamini–Hochberg screen is computed and published alongside. Where it disagrees with the pooled result, the disagreement is the story.

6. Claim levels

Language is bound to evidence, and the binding is linted.

7. Interpretability

A statistical outlier is only publishable if a substantive reading is plausible. Candidates are tested cheapest-first, and each stage can reject: data artifact → sampling artifact → byproduct of another property → deliberate behaviour. Failing to reject the first three is not evidence for the fourth; positive evidence is required, and an L3 claim additionally requires external corroboration with a resolvable URL.

8. What we will not claim

• No causal language from observational aggregates.
• No "best/worst" framing.
• No narrative from a single period's movement.
• No pooling of ratios across populations where Simpson's paradox is documented on that ratio.

9. Pre-registration

The question and the analysis choices are written into the report file before the data is pulled. This prevents fishing, and it means any paper drawn from this work has a genuine pre-registration trail rather than a reconstructed one.

10. Null results, and why there is no cadence promise

The format must be able to return nothing, and does so by design. The expected null rate is pre-registered so a quiet period reads as the method working rather than the author failing. A running counter is published.

Publication is gated on evidence, not on the calendar. No frequency commitment is made, because a schedule and a standard eventually conflict, and the schedule wins — quietly, by lowering the bar on a slow week. Every safeguard in this document exists to stop exactly that, so promising a cadence would undermine all of them at once.

A format that can only ever say "look at this" is indistinguishable from the thing this site was built to correct.

11. Pressure-testing: try to kill your own finding first

A result is not ready because it is significant. It is ready when a genuine attempt to destroy it has failed. Before publication, every finding is attacked on six fronts, and every one is declared in a table naming it with a status of run, skipped or not-applicable.

The table is the point. An attack nobody tried must not read like an attack that was tried and survived, and without a row per attack it does.

Specification. What does the model assume that the data might not support? Assumptions that look like arithmetic are the dangerous ones — a proportional model through the origin is an assumption, not a definition, and a non-zero intercept will bend the result in whichever direction the units happen to differ.

Aggregation. At what level does the effect actually live? A unit-level average will happily attribute a pairwise regularity to one of the two participants. The test is cheap: compute the same index for the counterparties. If they move with it, the effect is not the unit's.

Adjustment coarseness. Does a coarse control leave a gradient inside its own bins? Banded controls under-correct at the extremes. Redo it continuous; the point is that it has to be tried.

Prior work. Has someone already found this? Rediscovering a known result on more data is a contribution; presenting it as new is not.

Baseline sufficiency — what does a model with no unit property already predict? Before asking whether a pattern reflects something about a unit, compute what it would look like if units differed only in the means already known. Twice this has been the attack that killed a finding, and in both cases the earlier four attacks had all been passed.

The generalisation worth carrying: any "share of X in period P" metric is algebraically exposed whenever the period split is near-constant across units, and any distributional-shape metric is exposed to the moments it is built from. Report the excess over the matched null, never the raw correlation. A cheap standing screen — resplit each unit's totals under the population constant, or simulate from the fitted means, and compare — catches both.

Note also which attacks fail. Stratifying on a post-treatment outcome can manufacture the appearance of a strong effect where the mechanical null reproduces the within-stratum values exactly, so a stratified result needs its own null.

The same failure can live in the pipeline rather than in the finding, and passing the finding-level attack does not protect you from it. A project can compute a matched null correctly, report the excess over it, and still have the subject under investigation sitting inside the baseline it is measured against — because the baseline is fitted elsewhere in the codebase, by code nobody audits when the finding is checked.

That happened here. Several scripts fitted their reference model on the full history plus the period in progress, and the period in progress was the anomaly under investigation, so the anomalous subject contributed to its own expectation and shrank itself. Nothing failed, because the boundary between historical and in-progress was spelled three different ways across the codebase and omitted entirely in two places. The published figures moved in the third decimal and no conclusion moved, which is exactly why it survived: a contaminated baseline that changed a sign would have been caught immediately.

The operational form, which is the portable part: the subject of a study must not appear in the study's own baseline. Ask of every fitted nuisance model which rows went into it, not only which rows the estimate is reported over. Those are different questions and only the second is usually checked.

Cross-sectional associations across few units. A correlation computed across a handful of aggregate units is a correlation between things that differ in many ways at once, and it will often not survive being asked at a finer grain. Test it within units before believing it. The cheapest route to asking whether an effect is real is often not a statistical objection at all: if a covariate varies over time, ask why it is being treated as a fixed property.

Independent replication is the strongest form of this. Where a finding matters, it is re-derived by a separate pipeline on a different subsample before publication. Twice in this project that process reversed the conclusion — and a reversal found before publishing is a success of the method, not a failure of it.

Worked examples of each attack, including the two that killed findings, are in the football methods.

12. An association can be an artifact of its own aggregation level

The most instructive way to be wrong is to compute the same association at three levels of aggregation and watch it change sign.

A worked case, with the subject left in the repository that produced it: an association between two properties across eleven aggregate units correlates at +0.78. It has every mark of a result — a plausible mechanism, stability under leave-one-out, and survival of a multiplicity correction across the seven covariates tried.

Splitting the identical data into time blocks, and asking whether a unit's own value of the first property moves when its own value of the second moves, gives −0.09 — sloping the other way in five of the six units with enough blocks to fit a line, against a test able to detect 0.35 at 80% power. Pooled at block level it is +0.24.

One dataset, three levels, and the association is manufactured by the coarsest one. The between-unit correlation is real as a description of how units differ. It is not evidence that moving one property moves the other, which is what it will be read as.

What generalises. When units differ in many ways at once, a correlation across their averages measures the bundle, not the mechanism. Before believing one:

• Ask whether the covariate varies within a unit over time. If it does, it is not a fixed unit property, and treating it as one is the error. This is the cheapest available check, and in this case it was the one that exposed the result — the objection that found it was not statistical.
• Compute the association at the finest grain the data supports, and report every level rather than the most favourable one.
• State the power of the finer test. A null at a fine grain is only informative if it could have detected the effect claimed at the coarse one.

The same arithmetic and the same error apply to cohorts, segments, cost centres and model slices.

13. Corrections

Errata are published, never silently edited. Reports are versioned; the tracker is append-only and git-diffable; a correction appends a row and marks the superseded one. Retracting a finding in public is treated as a credibility asset.