Statistics & Experimentation

Call-centre tutorial: service level (fraction answered within a threshold) must be read together with abandonment, arrival-rate variability (doubly stochastic, overdispersed arrivals with strong time-of-day/day-of-week seasonality) and staffing.

He et al. (ISSRE 2016) evaluate supervised and unsupervised log anomaly detection methods on production system logs, showing that accurate log parsing and fixed-window/sliding-window event count matrices are prerequisite foundations before any downstream anomaly scoring.

Establishes the Quality-and-Efficiency-Driven (QED) many-server heavy-traffic regime: with c ≈ R + β√R servers, P(wait) converges to a nontrivial limit. Relevance: square-root staffing applies to large pools;

Empirical study of an Israeli bank call center: arrivals are time-inhomogeneous Poisson with overdispersed day-level rates; service times are approximately lognormal, not exponential; patience/abandonment estimated with survival methods for censored data.

Common Java performance methodologies (best-of, single run) mislead; use confidence intervals across multiple VM invocations and separate startup from steady state. For profiler agents: report CIs, not just spread, and keep cold vs warm distinct.

Product-limit estimator for censored time-to-event data; use for time from a universe's adoption of a release to its first ticket or crash, treating the 320 upgrade units excluded as too recent as censored instead of dropped.

Researcher degrees of freedom: undisclosed flexibility in thresholds, exclusions and windows produces false positives. Threshold perturbations of +-25% show which gallery numbers are threshold-sensitive (crash-loop rate about -38% at >=5 starts per 60 min).

Introduces the proportional hazards regression model and partial likelihood estimation for right-censored failure-time data without parametric baseline hazard assumptions.

gallery council 9 (dataviz/communication), 2026-10-28

Low prior odds, small effects, flexible designs and many tested relationships make many claimed findings false. Frames the gallery-wide multiplicity count (about 1,850 explicit tests, mostly in sigrel/sigjira under BH).

Applied as the Wave C outcome figure for the Bundle Insights gallery: segmented regression of the monthly bundled-minus-unbundled gap in median TTR (KM) and 30-day CE rate around the bot start (2026-06-15), phase-in month excluded, Newey-West SEs.

Probabilistic (Monte Carlo) bias analysis: place prior distributions on bias parameters that the data cannot identify (here: session gap, per-comment minutes, unlogged-work multiplier, missing-channel fraction), draw them jointly, recompute the estimate, and add sampling error by bootstrap. Exactly the right tool when there is no ground truth;

BCG field experiment: GPT-4 raised consultant quality and speed on tasks inside the AI frontier but lowered correctness by about 19 pp on a task outside it. Warns that AI analysis notes can mislead engineers on out-of-distribution cases; the gallery needs a correctness score on the note, stratified by case type, not only speed.

Defines negative-control exposures and outcomes: variables that share the confounding structure of the main analysis but cannot be causally affected, so any association signals bias.

Review of prospective outbreak detection (regression, time series, SPC, multiple streams) with emphasis on false-alarm control and evaluation against known outbreaks. Supplies the evaluation frame used in the YB common-good backtest: known events, lead time, false-alarm rate, minimum signal size.

Sensitivity of DiD/event-study to bounded parallel-trend violations

Quasi-Poisson regression on comparable past weeks gives a threshold for this week's count; alarms when the count exceeds it, downweighting past outbreaks. For YB fleet surveillance the analogue is per-template org counts per month;

KS-based tests show call-center and ED arrivals are consistent with NHPP only after accounting for rounding, intra-interval rate variation, and overdispersion from pooling hours across days.

Foundational epsilon-differential privacy for aggregate telemetry and histogram release

Re-analysed 190 published interrupted time series with six estimation methods: OLS, OLS with Newey-West, Prais-Winsten, REML, ARIMA and others. The choice of method changed the point estimates, the standard errors and often the statistical conclusion, especially in short series.

Erlang-A adds exponential patience; staffing at offered load plus a square-root safety term is robust once abandonment is modelled. In async B2B support the abandonment analogue is the customer going quiet when asked for a support bundle (44% of agent bundle requests in a May-Oct 2026 Yugabyte cohort never produced a downloaded bundle), so…

Common-good lens: randomized GenAI assistance at Alibaba sped work and raised ratings but did not improve objective quality, and top performers got worse.

Multiverse analysis: enumerate every reasonable processing choice and report the whole distribution of results. Applied in the Bundle Insights chaos council by re-running 15 gallery headlines under case exclusions (year-bug, stale _derived, shared bundles, top-3 orgs, LOO-org, last 6 months), threshold and window shifts of +-25% and bootstrap seeds.

Survey of agreement coefficients for coding: favours single-distribution coefficients (alpha, pi) for reliability, weighted coefficients for ordinal scales, and rejects Landis-Koch adjectives as thresholds.

Explains kappa's dependence on prevalence and bias, and gives sample-size tables for testing kappa against a null or estimating it to a given precision. Read across: with 61 doubly-labelled items and a binary label, a true kappa of 0.6 has a 95% interval of roughly 0.4-0.8 (our simulation).

Controls in ITS: a control series that did not receive the intervention removes co-interventions and secular shocks common to both; model either the difference series or a group x post interaction. Requires the control to be unaffected by the intervention and to share time-varying confounders.

Sharp bounds on a treatment effect when the outcome is observed only for a selected subsample whose size the treatment changes, under monotonicity of selection. Trim the arm with the larger selected share by the excess fraction from the top or bottom of its outcome distribution.

The RAND Health Insurance Experiment two-part model: P(any use) times E[spend | use], with a log-scale second part retransformed by smearing. It beat one-equation models on split-sample forecast error. This is the template for a staged ticket-effort model, P(CE escalation) times E[hours | CE] plus (1 - P) times E[hours | no CE].

Reviews strategies for diagnostic-accuracy studies when the reference standard is imperfect or missing: correcting with known reference error rates, latent-class models, composite reference standards, panel or consensus diagnosis, and validating the reference itself.

Replicates Deep-SE story-point estimation on the original and new data with proper baselines and finds it is not better than simple baselines (median or mean of past points) on most projects; earlier gains came from weak baselines and evaluation choices.

Five experiments: people who see an algorithm err lose confidence in it faster than in an erring human and choose the human forecaster even when the algorithm is more accurate. For support AI notes: one visible fabrication can drive engineers to disuse the whole note, so fabrication must be measured and reported separately from accuracy.

Counting-process extension of the Cox model for recurrent events with time-varying covariates and delayed entry. The right model for tickets per universe as a recurrent event with release (and release age) as time-varying covariates and universes entering exposure late (left truncation).

Funnel-plot control limits built from the binomial alone assume the only between-unit variation is sampling noise. Real institutional (here: error-signature) data are over-dispersed, so too many units fall outside the limits.

PM seat (council 5): Bundles are a nonprobability sample of the YB fleet. Elliott and Valliant set out quasi-randomisation (pseudo-inclusion weights from a reference probability sample) and superpopulation modelling as the two routes to fleet-level estimates.

29 teams, one dataset, effect estimates from OR 0.89 to 2.93: analyst choices alone generate wide dispersion. Motivates reporting card headlines with their perturbation range, not one number.

Principal stratification on post-treatment variables

Segmented regression needs enough points per segment (commonly about 8-12) and autocorrelation handling; with very short post periods slope changes are unstable and level changes dominate. Justifies a level-only model labelled early and underpowered when only 3 post months exist.

Specification curve: rank estimates over all defensible specifications and test jointly against the null.

PM seat (council 5): Data defect correlation: a huge non-random sample can be worth less than a small random one. The bundle fleet (about 1,185 universes) cannot stand in for the installed base without a selection model; call-home or a small random audit is worth more for adoption shares.

Hazard ratios are period-averaged and subject to built-in selection, so they do not translate into 'how much faster'. Recommends standardised survival curves / risk differences. Used to justify reporting G-computed RMST and median differences, not only Cox HRs, for intake grade vs TTR.

Target-trial emulation: specify eligibility, strategies, assignment, time zero, follow-up and outcome, then emulate; misaligned time zero yields immortal-time and selection bias. Used to diagnose relrate (series label from the quarter's last snapshot, possibly after the ticket) and the upgrade before/after card (upgrades triggered by tickets).

Practical guide to synthetic control: feasibility conditions (a good pre-period fit, no anticipation, no interference, a donor pool that is not itself treated), placebo-in-space and placebo-in-time tests, leave-one-out donors, and the ratio of post- to pre-period RMSPE for inference.

Meta-analysis of inter-coder agreement: agreement rises with annotator domain expertise, intensive training, fewer categories, and scheme maturity. Supports SE (expert) labellers, a calibration round, and collapsing fine scales when alpha is low.

Maps or tables of raw unit-level rates are dominated at both extremes by the smallest units, so the best and worst look the most extreme purely through sampling noise; partial pooling (empirical/hierarchical Bayes) shrinks small units toward the mean.

Improved Farrington: better dispersion handling, longer baselines, reweighting of past outbreaks; fewer false alarms across thousands of series. Relevant to one threshold per log template: multiplicity across about 4,000 ERROR/FATAL templates needs calibrated per-series baselines or an FDR step.

Begg & Greenes show that when the reference standard is verified only for a subset selected on the test result, naive sensitivity/specificity are biased, and that the bias is correctable when verification depends only on observed covariates (MAR) by inverse-verification-probability reweighting.

Vibration of effects: re-estimate an association across all covariate subsets; a 'Janus effect' (sign flips across specs) marks instability. Used as the sign-stability criterion: tabletskew CE gap flips sign (-11 to +10 pp) across buckets and min-node choices.

Manheim and Garrabrant split Goodhart's law into four mechanisms: regressional (optimizing a noisy proxy selects for noise), extremal (the proxy-goal relation breaks at the tails), causal (intervening on a non-causal correlate), and adversarial (agents game the measure).

Establishes that unpaired or sequential cloud benchmarks fail under multi-tenant drift and introduces interleaved single-instance trial schedules, providing the methodological foundation for our single-VM palindromic ABBA and ABCDDCBA schedules in what_the_log.

Introduces CUSUM: accumulate deviations from a target and alarm when the cumulative sum crosses a decision interval, trading average run length to false alarm against detection delay. Suits slow rises in per-release FATAL rates per 1,000 node-days (e.g. clock-skew loops), but only the glog plane has exposure denominators today.

Deng and Shi codify metric development as a data-driven process: metrics are judged on directionality (moves the right way when the product improves, validated on a corpus of past experiments with known outcomes) and sensitivity (probability of detecting a real effect, the product of how often a treatment moves it and statistical power).

Council 5 check: fs_bands now implements Poisson-gamma EB by org (shared frailty across bands, alpha by moments, EM-style updates) with org-bootstrap intervals. Remaining gaps: a case-level binary per node-day saturates in large clusters, and 'no trend' rests on overlapping intervals.

Reference text on classical vs Berkson error, regression calibration, SIMEX and Bayesian measurement-error models, including designs with a validation subsample. The log-derived hours are a surrogate for true effort;

73 teams on one hypothesis: results spanned significant negative to significant positive, and measured analyst choices explained little of the variance (hidden universe of uncertainty). Supports treating single-spec fleet claims (e.g. 'imbalance does not predict CE') as uninformative unless the spec curve is tight.

About half of duplicate bug reports carry inconsistent human severity labels across OpenOffice, Mozilla and Eclipse, so severity labels are noisy ground truth. Implication: our impact score should use few, rule-based, checkable fields (priority, production, page flag) and report label noise, not rely on free human grading.

ESI v3 reliability and validity; ESI lower-acuity levels are defined by the number of predicted resource types. Template for an ESI-style ex-post ticket-complexity grade (count of distinct resource types consumed) validated with weighted kappa.

Reliability conditions for content analysis and alpha thresholds (>=.800 for conclusions; .667-.800 tentative only).

Three RCTs (4,867 developers): +26% completed tasks pooled, noisy per site, gains larger for juniors. Operational translation: even thousands of randomized users gave noisy per-site estimates; ~80 bundled non-P1 tickets a month cannot power a CE-rate holdout of the AI note, so measure engineer-level process outcomes instead.

Reviews bifactor models: a general factor plus orthogonal group (source or method) factors. Omega-hierarchical tells how much of the score variance is general. For a multi-source effort latent, a general effort factor plus Slack-specific and Zendesk-specific method factors separates real work from source-specific traces.

Establishes the formal one-to-one theorem between ROC curves and Precision-Recall (PR) curves (a curve dominates in ROC space iff it dominates in PR space) and proves that linear interpolation in PR space is erroneous; directly underpins hagen-ai's step-wise interpolation-free AUPRC-error metric (forensic_metrics.py) under class imbalance.

The wild bootstrap is valid with a small number of large clusters under homogeneity conditions, via an approximate randomization-test argument.

Propagates every construction choice (weights, normalisation, aggregation, imputation) through Monte Carlo, then uses variance-based sensitivity indices to attribute the resulting rank uncertainty to each choice. A composite is robust only if the ranks it reports stay stable across plausible choices.

Classic M/G/1 with service interruptions (server breakdowns). The completion time of a job grows with the second moment of interruption length, so for a fixed fraction of time interrupted, a few long interruptions inflate waiting far more than many short ones.

Adjusted survival curves by inverse-probability-of-exposure weighting. The natural next step for the outcomes card (evidence-ready vs not), with org-cluster bootstrap; only as good as the measured confounders, which the bias card's failed CE check says are insufficient.

Shows that annotators who edit parser pre-annotations anchor on them: agreement with the pre-annotation is inflated and human-human agreement is overestimated versus annotation from scratch. Direct evidence that showing AI pre-labels to human raters contaminates human-AI agreement estimates.

Shows the Wald interval performs badly and the Clopper-Pearson 'exact' interval is overly conservative; the score (Wilson) interval or the 'add two successes and two failures' adjusted Wald interval gives near-nominal coverage even for small n and p near 0 or 1. Directly applicable to SLA attainment tiles near 100% with small n (e.g.

Extends the self-controlled case series to exposures whose occurrence depends on prior events (censored, perturbed or curtailed post-event exposure). Relevant for upgrades as exposure and tickets as events: tickets prompt upgrades, violating standard SCCS.

Stepped-wedge cluster randomization: all clusters start in control and cross to treatment at randomized steps; analysis is a mixed model with fixed period effects and cluster random intercepts, so secular trends are separated from treatment.

Sensitivity and specificity are not fixed properties of a test. They shift with prevalence and with the case spectrum, through mechanisms such as patient mix, referral filters and the reference standard.

Proposes landmarking for dynamic prediction: at each landmark time, fit a model on subjects still at risk using information available up to that time. Basis for our re-scoring design: at 24 h and 72 h the expected-effort estimate is refitted on still-open tickets with activity so far, raising AUC from about 0.72 to 0.82 and 0.85.

Entropy balancing finds unit weights that exactly match chosen covariate moments between groups, while staying as close as possible to uniform weights (minimum KL divergence). Unlike logistic propensity weighting, no balance iteration is needed, and extreme weights are less common.

Shows with generative copula models that 15 demographic attributes re-identify 99.98% of Americans and that sampling/incompleteness gives weak protection.

OECD/JRC Handbook on Constructing Composite Indicators (Nardo, Saisana, Saltelli, Tarantola et al., 2008). A ten-step checklist: theoretical framework, data selection, imputation, multivariate analysis, normalisation, weighting and aggregation, uncertainty and sensitivity analysis, back to the data, links to other indicators, and visualisation.

Competing risks: 1-KM overstates cumulative incidence when a competing event (e.g. ticket closed without bundle) precludes the event; use Aalen-Johansen CIF

Establishes representativeness (insensitivity to prior probabilities / base-rate neglect) and anchoring-and-adjustment in judgment under uncertainty.

Review of longitudinal data with irregular, possibly outcome-dependent observation times: ignorable, conditionally ignorable via inverse-intensity weighting, or informative needing joint models.

Proves that the normal-ogive two-parameter IRT model (and the graded response model) is equivalent to a factor analysis of discretised variables with tetrachoric or polychoric correlations.

Musa's operational profile is the set of operations and their occurrence probabilities in the field; test effort is allocated in proportion to it so that reliability measured in test predicts field reliability.

Hierarchical variance parameters with few groups are poorly identified; moment and ML estimates collapse to zero; weakly informative half-t priors recommended. Relevant to fs_bands (1/alpha by moments, clamped at 0) and the collapsed funnel shrinkage.

Sleep-time compute is literally the Security-1 daemon economics: pre-computing inferences over known context before queries arrive cut test-time compute ~5x at equal accuracy and amortized 2.5x across related queries; gains scale with query predictability.

Practical guide to SPC in healthcare: choose chart by data type (c/u for counts, p for proportions, XmR for continuous or when distributional assumptions fail), use about 20-25 subgroups to set limits, and read special-cause rules rather than reacting to single points.

Operational translation: a surrogate (here the North Star 'evidence-ready in 24h') can stand in for the true outcome only if the treatment effect on the outcome is captured through the surrogate.

PM seat (council 5): NPMLE for interval-censored event times. Bundle snapshots bracket upgrades to a median 21 days, so time-to-adopt a release series is interval-censored; Turnbull is the right estimator for adoption curves on the fleet frame.

Meta-analysis (70 samples, 233 effects): electronic monitoring slightly lowers job satisfaction (r=-.10), raises stress (r=.11) and counterproductive behaviour (r=.09), with no performance gain (r=-.01); targets and feedback attached to monitoring worsen the effects.

NHPP reliability growth model with mean value function a(1-exp(-bt)): field failure intensity decays with exposure time after release as defects are found and fixed. Implies a newer release always looks worse in calendar-pooled windows; release comparisons must be day-0 aligned or model the growth curve.