Log & Failure Analysis Classics

Fu, Lou et al. (ICDM 2009) convert unstructured distributed system logs into template keys and learn finite-state execution automata and temporal transition invariants across log sequences.

Chen et al. (LiDAR, FSE'20) identify linked incidents across services using text plus component-dependency features; motivated by duplicate/related incidents wasting engineer effort. For Yugabyte CE: many disk-stall CEs are linked (same customer, same AZ, same cloud volume);

Microsoft LogCluster: cluster failure log sequences, keep one representative per cluster, and match new clusters against a knowledge base of previously diagnosed clusters so engineers only inspect novel ones.

Multi-disciplinary review of call-centre OM: capacity planning, routing, skill-based staffing, and the human-resource side (agent burnout, turnover, learning).

SCWarn flags bad software changes via multimodal anomaly detection (metrics + logs) around each change. A YB analogue anchors on declared YBA tasks (GFlagsUpgrade, SoftwareUpgrade, RestartUniverse) and compares minute signals before/after; feasible only on the ~100 cases whose task log reaches at least 1 day before the ticket.

Incident data from 20 large Microsoft online services (20K+ teams): 4.11% to 91.58% of incidents per service were reassigned at least once, and reassignment increased triage time by up to 10.16x on average, with heavier impact on high-severity incidents. Gives a range for hand-off prevalence and shows that each hand-off adds substantial elapsed time;

Le and Zhang (ICSE 2023, LogPPT) formulate log parsing as prompt-based few-shot token classification using RoBERTa with virtual label tokens to identify keywords vs parameters from 32 adaptive few-shot training examples.

fsync T5: frequency-based longest-common-pattern parsing would also mask the S1 op phrase when it varies; catalog plus downstream typing is more stable across vintages.

Dai et al. (IEEE TSE 2020/2022, Logram) observe that static template words form high-frequency 2-grams and 3-grams across a log corpus whereas dynamic parameters form low-frequency n-grams, enabling fast dictionary-based template extraction without tree traversal.

Du and Li (ICDM 2016, Spell) formulate online log parsing as Longest Common Subsequence (LCS) matching, maintaining a prefix tree of LCSObjects alongside an inverted index of tokens so candidate templates can be pruned before running O(|m| * |t|) dynamic programming LCS alignment.

3000+ cloud-system issues: upgrade, configuration and hardware-failure handling bugs are major classes with severe outcomes; supports keeping upgrade and environment-triggered crashes separate in release QA metrics.

fsync T5: over-generalised templates (a constant token treated as variable) silently merge distinct events. WTL's 'Time spent <*>: real...' merges Fsync log, Append to log and Log roll; the op is recoverable from the slot value, so the minimal fix is downstream classification on the slot, avoiding a template-id change that would split silver vintages.

fsync T5: SemParser recovers parameter semantics (concept vs instance) that syntax-only parsers lose, improving downstream detection. Supports treating the S1 op string as a semantic concept, recovered from the slot downstream.

Practice-oriented review of the WFM chain: forecast arrivals and handle time per interval, convert to required staff with a queueing model (Erlang C/A or simulation), then schedule and roster, with feedback between the steps. It is honest about which methods work in practice and which theory does not transfer.

103 crash-recovery bugs in 4 distributed systems: most stem from incorrect handling of crash/reboot timing and persisted state; many need only one crash. Relevant to YB FATAL and ERROR buckets around tablet bootstrap / remote bootstrap ('Failed to destroy regular DB', 'Not changing its role ... remote bootstrap').

GRLIA: incident aggregation by learning a graph of cascading-failure relations among alert types, evaluated on Huawei Cloud with linked-incident labels. A template for multi-plane fingerprinting over bundles, but its labels came from rich on-call link records;

He et al. (Log3C, FSE'18, Microsoft) cluster log sequences cascadingly and correlate cluster counts with KPI degradation to surface impactful problems.

Chu et al. (ICDE 2021, Prefix-Graph) address the rigid token-length and prefix assumptions of prefix-tree parsers (like Drain) by merging a prefix trie with a probabilistic token transition graph and iteratively finding densest subgraphs (cliques/branches) to separate templates that share prefixes or diverge mid-message.

Five supercomputer syslogs. Alerts dominated by a few categories; logs heavily redundant and must be filtered spatially and temporally; timestamps unreliable (missing years, inter-node drift, corrupted messages). Directly relevant to yearless glog/syslog parsing and to deduping repeated FATAL lines before counting crashes.

Dhaliwal et al. (ICSM 2011) analyze Mozilla Firefox Socorro field crash reports and demonstrate that single-level bucketing by the top crashing signature frequently mixes distinct bugs (when crashes terminate inside generic memory/string/RPC routines) and that sub-clustering buckets using Levenshtein distance over the top 10 stack frames dramatically…

Microsoft, 18 online services: a large share of incidents are 'incidental', meaning engineers deprioritise them after diagnosis. DeepIP ranks incidents by the predicted probability of being incidental (AUC 0.808 vs 0.624). For CE escalation analytics: not every CE link is a real defect escalation.

Mozilla and Eclipse bug reports: a large fraction of bugs (reported as roughly 37-44%) are 'tossed' to another developer at least once, and tossing lengthens time-to-fix. Analogue for support escalations: re-routing among engineers is common and each reassignment adds latency and re-reading effort; tossing counts are a usable proxy for coordination load.

Survey of arrival-forecasting methods (time series, Poisson mixtures, Bayesian, distributional forecasts) plus case study; stresses overdispersion and the need for distributional (not point) forecasts for staffing. A support suite with overdispersed inflow and event surges (BFCM) needs interval forecasts of pages/tickets by shift to size event staffing.

ReBucket clusters crash reports by stack similarity. Council-5 DocDB lens: WTL's termination plane now records check_site (file:line) and raw FATAL text per pid, so the tests crash panel can key buckets on check_site plus status text;

Rodrigues et al. (EMSE 2022) unify position-discounted stack depth weighting (from ReBucket) with inverse-document-frequency frame specificity (from Brodie et al. and CrashLocator) into a single global alignment algorithm (TraceSim) for stack trace deduplication.

Schipper, Aniche, and van Deursen (MSR 2019) evaluate static source-to-log tracing in enterprise Java systems (using Eclipse JDT AST analysis to convert SLF4J/Log4j statements into regular expressions) and document five real-world failure modes that break naive AST extractors: (1) wrapper logging methods and custom facades, (2) string concatenation…

AlertRank ranks alerts by severity from textual, temporal and metric-anomaly features (XGBoost ranker; F1 about 0.89 on a bank's data). Framing for YBA alert review: measure precision and recall of default YBA alert policies against ticketed incidents rather than counting alerts fired.

Kim et al. (DSN 2011) introduce Crash Graphs, aggregating multiple stack traces within a crash bucket into a directed call-transition graph where nodes are functions and edges represent caller-callee adjacency across crashes.

Wu et al. (ISSTA 2014) propose CrashLocator, which ranks candidate faulty functions from crash stack traces using an inverse-frequency weighting scheme (analogous to TF-IDF: functions that appear frequently in a specific crash bucket's stacks but rarely across unrelated buckets receive high blame scores, weighted by inverse distance from the top of the…

Mines open-source repositories to measure how long bugs take to fix per file; fix times are long and highly skewed. Supports reporting customer-escalation (CE Jira) time-to-fix as a distribution/survival curve with open items censored, not as a mean.

Yuan et al. (ASPLOS 2011/2012, LogEnhancer) demonstrate that production log statements often omit critical causal variables along failure paths and use static data-flow slicing to automatically identify and append causally relevant in-scope variables to each log call site.

Shima (arXiv:1611.03213, 2016, LenMa) proposes clustering system log messages by representing each whitespace-tokenized message as a vector of word character-lengths and computing cosine similarity against existing clusters with the same token count.

Yuan et al. (ASPLOS 2010, SherLog) combine static source-code log template extraction with backward symbolic execution and constraint solving (SAT/SMT) to infer the exact control-flow paths and variable values that must have executed to produce a post-mortem runtime log sequence.

Cowic pioneers column-wise independent compression for log streams so individual log entries or columns (timestamps, IPs, template bodies) can be decompressed and queried independently without inflating entire multi-megabyte archive blocks.

Montgomery & Damian (IBM Canada, ~2.5M support tickets) engineer customer- and ticket-level features (support analysts' knowledge of customer: number of open tickets, recent escalations, days open, priority changes) and predict escalation (recall ~0.79).

Derives asymptotically optimal staffing (square-root safety staffing) trading off staffing cost against delay cost, with ED/QD/QED regimes. Relevance: formalizes why very tight response targets (e.g., 5-min P1) imply over-provisioning relative to load in small systems; supports framing 5 vs 15 min P1 targets as cost-of-capacity decisions.

Xu et al. (ICSE 2024, DivLog) apply in-context learning (ICL) with LLMs for log parsing by sampling a diverse candidate set via Determinantal Point Processes (DPP) and retrieving k-nearest neighbor demonstration examples per query log line.

Nezha converts heterogeneous observability data (log templates, metric alarms, trace spans) into a common event representation, mines event patterns, and localizes root causes by contrasting patterns between a fault-free and a faulty phase, giving interpretable code-region/resource-level answers without supervision.

Introduces LSM-Bush and the Wacky continuum, setting doubly-exponential capacity ratios across smaller LSM levels so write amplification scales as O(log log N) rather than O(log N) while keeping point-read and range-read amplification navigable.

CockroachDB states transient multi-second disk stalls are common in cloud, warns at 10 s and terminates the node at 20 s of WAL sync stall, and recommends WAL failover to a second disk. A vendor precedent for a bounded-stall policy (T1) rather than leaving leadership on the stalled node.

CoLA (small local model + frontier collaboration for log anomaly detection): the small model handles routine events locally and routes uncertain/anomalous events to the LRM, preserving detection while cutting cost/latency.

Panel study of a firm's call center after web self-service launched. The web substitutes for simple, information-type calls but not for problem-type calls, and self-service can generate new contacts. For a B2B database vendor this means 'deflection' cannot be read off portal views; it must be measured per issue type as change in assisted contacts.

Eadro jointly learns anomaly detection and root-cause localization from three modalities (logs as event counts, KPIs/metrics, and traces) with modality-specific encoders fused through a dependency graph, and shows that multi-source fusion beats single-source methods on microservice benchmarks.

ExplainIt! (SIGMOD 2019) introduces a declarative SQL-like DSL for specifying root-cause hypotheses and causal dependency graphs over massive operational time-series telemetry.

Optimistic crash consistency: separates ordering from durability (osync vs dsync) to avoid cache flushes; shows flushes are the dominant cost of journaling file systems. Background for why fsync cost is device-cache-flush cost.

Commercial (CA Technologies) projects: bug-fixing time is long-tailed; early report attributes carry signal for slow vs fast fixes (Markov-based and kNN models). Industrial evidence that initial-report attributes relate to resolution time, the hypothesis the intake seat tests with survival models.

Text-mining classifier predicts bug severity (severe vs non-severe) from report summaries on Mozilla, Eclipse and GNOME, precision and recall about 0.65-0.85 with roughly 500 reports per class. Shows severity is predictable at intake from text alone, so an impact score can be ex-ante. Severity is the defect's technical effect; it is not fix effort.

Top-crash prediction at Mozilla: few crash classes dominate. Council-5 DocDB lens: in YugabyteDB bundles a single field crash class (intents-DB 'Unexpected write id' Corruption, 2025.2.x before 2025.2.7) is a known-fixed cluster; the crash panel should rank by distinct events and versions-with-fix, so support can say 'upgrade' instead of 'unclassified'.

Crash-inducing commits in Firefox: links crash buckets to inducing changes/releases; basis for per-release signature emergence

Builds per-role profiles of SQL access from query logs (command, relations, attributes) and flags deviations with naive Bayes. A direct template for per-universe workload fingerprints from audit/pgaudit statement classes, and for drift detection between bundles of one universe.

Mines service correlations from historical incidents to group alerts into outages quickly in Azure. Linking/aggregation is a first-class step before RCA: without it one systemic fault is counted as many independent escalations, inflating both recurrence and per-org counts.

Predicts bug-fix effort from the most similar earlier reports; weak on novel reports. Reused in the triage audit as a TF-IDF nearest-closed-neighbour grade feature, with low maximum similarity as a novelty flag.

Practical SLO/error-budget design: SLIs at the user boundary, windowed and burn-rate alerting, separate cumulative budget from acute paging. For fsync stalls: GF1 = blocked-seconds budget with burn-rate alerts (catches chronic moderate stalls), GF2 = single-stall magnitude page (catches the rare long stall before threshold crossings).

Defines a set of resource-behaviour indicators (skills, utilisation, preferences, productivity, collaboration) computable from event logs and evaluates them on real logs.

Demonstrates that parsing structured logs and selectively protecting only sensitive fields (rather than scanning or encrypting entire log records) improves ingestion throughput by 22.4%–54.0% and reduces query latency by up to 42.3%.

Microsoft ASE 2013 experience report on software analytics for incident management: on-call engineers need fast retrieval of similar historical incidents plus the suspicious signals for the current one, presented together. Supports a case-open panel pairing vital signs with history.

For a support tool consumed by vendors, the SoK's 'constrained-audience transparency' designs (hashed commitments + out-of-band checkpointing) are the realistic target, not public logs, because bundles carry customer PII.

Template for the design we lack: estimate workload from system event logs (Epic, 142 physicians, 3 years) and calibrate against a small direct time-motion observation subset (14 physicians). Found 5.9 h/day in the EHR of an 11.4 h day, 1.4 h of it after hours.

EARS: simple CUSUM-style aberration detection on syndromic counts, judged by timeliness and false-alarm rate. Analogy for support: treat a cross-customer signature (Jira key, log template, differential label) as a syndrome; measure detection lead time and false alarms against the date engineering first linked the bug.

Peerally et al. list RCA failure modes: hindsight bias, the quest for a single root cause, weak recommendations (training, reminders) that are never implemented, and no tracking of whether actions reduced recurrence.

L = lambda W holds for any stationary queue. Used as a consistency check across three separately derived dashboard numbers: 55.8 customer tickets/week x mean 18.5 days to resolve implies ~147 open, against 124 observed open now, a ~16% gap that points at the estimated solve time (created + summed status durations) or reopen handling.

Empirically validates across 357 real faults in 5 open-source systems (321k LOC) that mutant detection is statistically correlated with real fault detection independently of code coverage, justifying cargo-mutants as a mandatory CI adequacy gate for Yugabyte diagnostic rulepacks.

Formalizes burst detection in continuous event streams using an infinite-state automaton where burst states correspond to elevated emission rates and state transitions incur cost proportional to the state jump. Establishes the theoretical foundation for separating baseline arrival rates from multi-scale burst onset and peak intervals.

ConfValley: declarative validation of cloud configuration against specs and inferred constraints. Supports encoding named-flag hazards as checks over YBA platform_universe snapshots rather than mining drift after incidents.

Cook's 18 points: complex systems run degraded, catastrophe needs multiple failures, post-accident attribution to a 'root cause' is fundamentally wrong, hindsight biases assessment.

Salesforce ICSE-SEIP 2022 (ICA): mines past incident investigations into a root-cause knowledge graph and retrieves similar past incidents for a new one. Relevance: the similar-past-cases panel should match on structured symptoms (error templates, subsystem, cluster/customer) and resolution, not trigram subject text;

Opprentice ensembles many cheap detectors under supervised operator labels.

Survey of queues with server vacations. Key result used for fsync stalls: stochastic decomposition, waiting time = ordinary M/G/1 wait + residual vacation time E[V^2]/(2E[V]). A long stall leaves a backlog that drains at rate (mu - lambda), so the after-effect lasts B*rho/(1-rho);

VOID report on thousands of public incident reports: MTTR and incident counts are heavy-tailed and unreliable as performance metrics; recommends learning-oriented measures. Warns against reading CE recurrence or RCA coverage as a team KPI.

Systematic review of studies that turn EHR audit logs into time-on-task. Its central methodological point for us: inactivity timeouts vary widely across studies and are often unreported, few studies validate against observation, and results are sensitive to the timeout choice.

Review of the safety literature defining learning from incidents as a cycle: report, analyse, plan and implement actions, then evaluate whether the change worked. Most organisations stop at analysis. For CE: an RCA document is the analysis step only;