Systems & Infrastructure

Large-scale empirical study of schema changes in ten popular open-source database applications and how application code co-changes with them. Builds a taxonomy of atomic schema change types (table/column add/drop/rename, type change, key/index/constraint changes) and measures frequency and the code impact.

Studies co-tuning memory allocation between write buffers (MemTables) and read caches (block cache + Bloom filters) in LSM storage systems, showing that static memory partitioning causes up to 2-5x suboptimal I/O bytes per operation under shifting read/write workloads.

Innocuous setup (env size, link order) biases measurements enough to flip conclusions; randomize setup and use causal analysis. For Python agents: multi-process pyperf runs, report distributions, not single before/after deltas.

Apache Arrow DataFusion demonstrates how a modular Rust analytical engine built natively on the `arrow-rs` and `parquet` crates can serve as the shared execution core across language ecosystems (including `datafusion-python` via PyO3 and the Arrow C Data Interface / PyCapsule zero-copy protocol).

PebblesDB introduces Fragmented Log-Structured Merge Trees (FLSM), using skip-list-inspired guards to partition SSTs across levels so data is never rewritten within the same level. Reduces write amplification by 2.4-3x and increases write throughput by 3-6.7x compared to RocksDB.

Deequ: declarative unit tests for data (completeness, uniqueness, consistency) computed incrementally at scale with anomaly detection on metric histories.

Taxonomy of database-access performance anti-patterns mined from real issues; distinguishes query-level from access-pattern-level problems. Useful as the target vocabulary for query_event.anti_patterns, which today covers only 6 text-regex query-level patterns.

Multi-level repetition design (iterations, processes, builds) with CIs on effect size, optimizing repetition allocation. Replaces ad hoc '3-5 runs' with multi-process repetition and a CI on the ratio of means.

Spooky demonstrates that partial compaction in RocksDB suffers from boundary-skewed overlapping files (inflating WAF) and uncompacted dead data in cold ranges (inflating SAF).

Reviews staffing and scheduling under time-varying demand (SIPP, lagged/offered-load methods). For follow-the-sun support with shift handoffs, staffing should track offered load per hour band; P1 misses clustering at handoffs and APJ taking most night pages point to schedule design, not individual performance.

Snowflake analysis of 667M production BI-tool queries over one week (Oct 2024, one region). Statement mix: SELECT 47%, SHOW 31%, SESSION (parameter-setting) 16%, DDL 4%, DML 2%, transaction control 0.5%. Joins in 46% of queries, CTEs 25%, window functions 11%; 21% of statements exceed 10KB.

Cloud microbenchmark variability differs widely by benchmark/provider; randomized multiple interleaved trials on the same instance reduce false positives. For profiler agents on shared hosts: interleave A/B and report the minimum detectable effect.

Formalizes planet-scale time-series data modeling with explicit cumulative counter reset semantics and series-scoped delta aggregation prior to cross-series spatial rollups.

Accordion applies LSM principles to the in-memory MemTable layer via proactive in-memory flushes and compactions into flat arrays, shrinking memory footprint, allocator fragmentation, and flush frequency (reducing Disk Write Amplification by 20-50%).

Formalizes workload compression with two goals, representativity (feature-distribution match) and coverage (keep common queries and outliers), proves the optimum hard and gives a greedy algorithm with approximation guarantees, beating sampling and clustering baselines.

Mytkowicz et al. (ASPLOS '14) formalize how to execute finite-state machines (such as multi-line log assemblers, tokenizers, and Aho-Corasick/DFA matchers) in data-parallel fashion across chunks of a single input stream by breaking state dependencies across chunk boundaries via small speculative state-transition functions or lookback/convergence properties.

Graefe's foundational survey (ACM CSUR '06) formalizes external merge sort invariants in database engines: replacement selection (generating runs ~2x memory buffer size on partially ordered inputs), optimal merge fan-in $F$ bounded by per-run read buffer size so seek/syscall overhead is amortized without thrashing memory, and forecasting/prefetching…

Instant Loading (VLDB '13) achieves wire-speed multi-core bulk loading of variable-length text/CSV files by partitioning raw byte ranges across threads (`pread`/mmap slices) and synchronizing each worker to the first valid record boundary in its slice.

Analyzes in-memory and disk-based OLTP engines using Intel Top-Down Microarchitecture Analysis Method (TMAM) and hardware PMU counters, showing that even when 100% of data fits in memory, OLTP engines spend >50-70% of execution cycles stalled (primarily L1-I/ITLB Front-End Bound and L3/DRAM Back-End Memory Bound pointer-chasing stalls) and retire only…

Extended VLDBJ study revisiting OLTP microarchitectural behavior across modern multi-socket Xeon processors, demonstrating how L3/DRAM bandwidth saturation and cross-core coherency stalls govern multi-tenant scaling even when single-thread instruction count barely changes.

DSN 2014: mines historical issue repositories (signatures plus healing actions) to suggest mitigation for new incidents; retrieval by symptom signature beats free text. Supports keying past-case lookup on WTL template ids and resolution actions.

Extracting the 13 Prometheus label fields (json_extract_string) in the inner subquery s before unnest(s.vals) reduces JSON object traversal from O(N_series * M_samples) (~190 samples/series) to O(N_series).

Morrison (JACM 1968, PATRICIA) introduces path-compressed radix trees where single-child internal chains are collapsed into multi-byte edge labels (or skip offsets), reducing tree depth to the number of branching points and enabling fast prefix lookup and longest-prefix matching.

Taxonomy of 104 distributed concurrency bugs in Cassandra/HBase/Hadoop/ZooKeeper: timing (message ordering, races with faults/reboots), triggering inputs and fixes. Useful to classify YB field FATALs such as 'Unexpected new leader in same term' that look like Raft message-ordering races rather than environment faults.

Availability = f(MTBF, MTTR, impact fraction); for a fixed target, frequent short outages and rare long ones are interchangeable in the budget but not in user perception or dependency chains.

Separates event time from processing time and defines the watermark: the point in event time up to which the system believes it has seen all input. For platform health, watermark lag (now minus watermark) is the right per-source freshness signal for incremental syncs.

PM seat (council 5): Vendor fleet telemetry (Redshift) shows real workloads differ from benchmarks: precedent for installed-base usage telemetry (call-home) as the broad lens, with bundles as the deep, biased complement.

Foundational sub-linear string searching algorithm proving that right-to-left pattern comparison with bad-character shifts skips large fractions of text bytes.

Survey classifying provenance by purpose, granularity (workflow vs tuple level) and capture mode. Workflow provenance (code version, parameters, input versions) reproduces aggregates only if inputs are immutable or versioned.

Taxonomy of DBMS extensibility across PostgreSQL, MySQL, MariaDB, SQLite, Redis, DuckDB plus static/dynamic analysis of 400+ PostgreSQL extensions: about 16.8% conflict with at least one other extension and can cause failures, correlated with extension complexity and hook usage.

Alibaba EBS production: tail spikes (>10 ms) are frequent (649 in ~100 clusters/week), P99999 reaches ~109 ms, caused by short bursts from a few major-client virtual disks that load-balancing spreads across all storage nodes; proxy event-loop scheduling dominates tail latency even underloaded.

Chhugani et al. (VLDB '08) show that sorting on multi-core SIMD CPUs becomes memory-bandwidth-bound once data exceeds L2/LLC cache unless blocks are sized to $C / (2E)$ and merged via register-resident SIMD bitonic merge networks and multi-way merge trees.

Studies performance anti-patterns in 12 popular open-source Rails (ORM) applications, classifying causes such as inefficient queries, N+1 / redundant per-row queries, missing indexes, unnecessary data retrieval and view-layer issues, and shows that simple fixes yield large speedups.

KVell demonstrates that on modern multi-queue NVMe SSDs, traditional LSM/B-tree storage engines become CPU-, synchronization-, and memory-copy-bound rather than disk-I/O-bound, motivating zero-copy shared-nothing I/O and CPU-per-op minimization.

PM seat (council 5): Windowed user-uptime separates one long outage from many short ones; directly supports VOLUME.md's GF1 reporting recommendation.

Demonstrates that analytical queries over nested JSON spend the majority of CPU cycles materializing unused fields and array elements unless projection pushdown and structural filtering precede unnesting.

Argues availability nines hide the distribution of outage durations; customers care how often and how long. Supports reporting fsync stall cost per length bucket rather than one stalled-seconds total: in the YB bundle fleet 80% of >=1 s stalls are 1-5 s but carry only 18% of write-unavailable seconds and 0% of >=5 s timeouts.

SQLCheck ranks anti-patterns by impact using query, schema and data context. Our regex flags on WTL templates lack schema/data context, so they are candidate markers, not diagnoses; ORDER BY without LIMIT and SELECT * fire on most OLTP shapes.

DSN 2006 (trunk year field wrongly says 2026). Nine years of LANL HPC failure records; root causes split into hardware, software, network, environment, human, unknown. Failure rate scales with system size and is roughly per-node constant; time between failures is Weibull with decreasing hazard.

Demonstrates that standard Top-Down Microarchitecture Analysis (TMAM) underestimates memory stalls in pointer-chasing OLTP workloads by up to 50% due to out-of-order overlap accounting, and provides a corrected OLTP PMU stall decomposition.

Describes AWS Cedar (OOPSLA 2024), the flagship industrial deployment of Verification-Guided Development pairing an executable Lean 4 formal model and machine-checked soundness proofs with differential testing against a production Rust implementation.

Probe semantics: timeoutSeconds default 1 s; failureThreshold x periodSeconds bounds detection; one success resets the counter. Explains why the YB helm liveness probe misses grey disk stalls (Comply tservers stalled 34-97 s never restarted) and over-fires on throttled volumes (fsync ticket T8).

Introduces Discretized Streams (D-Streams), modeling continuous computation as a series of deterministic, stateless micro-batch tasks over immutable RDDs. While operator-at-a-time streaming achieves sub-100ms latency, Parquet's PAX columnar format is physically a micro-batch format requiring 64k-256k row groups;

Raasveldt and Mühleisen prove empirically that traditional row/object-at-a-time serialization across client/language boundaries dominates analytical query latency by 10x–100x compared to columnar bulk vector transfer.

Weld quantifies the 'data movement tax' when composing multi-library Python analytics pipelines (Pandas, NumPy, SciPy): even when individual C/C++/Rust kernels are fast, materializing intermediate objects and crossing FFI boundaries repeatedly degrades performance by up to 23x.

Studies the lives of tables across open-source schema histories. Most tables change rarely or never after birth (rigidity); change concentrates in few tables and early lifetimes, and wide tables tend to survive longer.

Akidau et al. (PVLDB 2013) MillWheel: Fault-Tolerant Stream Processing at Internet Scale

Shankar et al. argue that production data pipelines fail silently through stale, partial or rewritten data rather than crashes, and that teams need cheap, always-on checks on freshness, completeness and schema drift.

Balkesen et al. (VLDB '13) demonstrate that multi-core sort-merge pipelines only scale with core count when both the partitioning/run-generation phase and the final multi-way merge phase are range-partitioned across threads with NUMA/cache-aware buffers.

pugz (Kerbiriou & Chikhi, IPDPSW '19) breaks the single-core serial ceiling of single-member gzip decompression (`MultiGzDecoder`) via a two-stage parallel algorithm: Stage 1 partitions the compressed bit-stream into chunks, locates a valid Deflate dynamic Huffman block header in each chunk via bit-level trial decoding, and inflates symbols while…

Introduces the Boyer-Moore-Horspool bad-character shift heuristic that powers CPython's C-level fastsearch (str.find and the 'in' operator).

Early EC2 measurement: CPU, memory, disk and network performance vary widely across instances and over time (coefficients of variation far above local clusters), and instance type/zone matter. Baseline evidence that cloud disk performance is not a fixed property of the instance type.

SSI detects dangerous rw-antidependency structures and aborts with serialization failures. For bundle data: 40001 is an expected, design-level outcome of optimistic concurrency, so a 40001 count is only interpretable as a rate per transaction and alongside retry handling; raw presence (56 of 76 audited cases) says little.

Inoue & Taura (VLDB '15) prove that when sorting an Array of Structures (AoS) such as compile-time log parsing `Vec<Row>` (`sizeof(Row) ~ 300-500B`), the dominant bottleneck is NOT key comparison—it is the random-access cache/TLB thrashing of rearranging large records after sorting packed `(key, index)` pairs (`take_record_batch` in the log parser).

Armbrust et al. (SIGMOD 2018) Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark

TFDV: schema inferred from data, each batch validated for missing features, type/domain drift and span skew.

Azure caps uncached I/O at both the disk tier and the VM uncached limit; whichever is lower throttles and raises latency. Metrics to request: VM Uncached IOPS/Bandwidth Consumed Percentage and Data Disk IOPS/Bandwidth Consumed Percentage. Host caching ReadWrite acknowledges writes at the host cache; fsync/FUA forces flush.

Distinguishes why-provenance (which inputs justify an output) from where-provenance (where a value was copied from). The _derived marker objects list is where-provenance at file granularity; gallery numbers lack why-provenance (which bundle set and file generations produced a card).

Analyzes why Python/process-boundary data pipelines pay heavy hidden serialization and memory-copy taxes even when claiming 'Arrow compatibility', and shows how true zero-copy Arrow RecordBatch pipelines eliminate double-buffering and materialization overhead.

Modification-request data at Lucent: work items spanning sites took about 2.5x longer than single-site items, and the slowdown was explained largely by the larger number of people involved, not by size or complexity alone.

Cost-as-proxy-for-need algorithm under-referred Black patients. Warning for resource-based ticket-complexity scores: they encode past allocation (calls, escalations, senior engineers), not need.

etcd: consensus is very sensitive to disk write latency; slow fsync times out heartbeats and causes elections. The vendor guidance assumes the slow node loses leadership, the opposite of YB's lease-holding leader on a stalled disk.

Coordinated omission: monitors that wait on a stalled system under-record the stall, turning one long stall into one slow sample. For fsync stalls this means per-request latency percentiles and 5-minute means hide long stalls; a stall must be measured as an interval (duration of the in-flight sync), not as sampled latencies.

Microsoft interview/design study: developers calibrate trust in AI code tools through community evidence (peers' experiences) and in-tool cues of reliability.

Argues for one open-format lake (Parquet plus a transactional metadata layer) serving BI and ML, with ACID commits, versioning and quality enforcement in the table layer. For the bundle lake: silver and _derived are plain Hive-partitioned Parquet with no table log, so a glob reader cannot tell a current from a stale snapshot;

Demonstrates Fowler's Strangler Fig pattern combined with Domain-Driven Design bounded contexts and an Anti-Corruption Layer (ACL): legacy callers remain untouched while individual bounded modules are extracted behind a routing facade and validated quantitatively against the unmigrated baseline.

Introduces free-list sharding per mimalloc page to eliminate multi-threaded allocator lock contention and arena fragmentation. In WTL-P2, switching from glibc malloc to mimalloc reduced wall time to 0.895x (95% CI [0.866, 0.924]), RSS to 0.746x [0.592, 0.957] across 11 bundles, and collapsed Case 19698 (33.6 GB) peak RSS from 52.9 GiB to 12.8 GiB (-75.8%).

Li et al. (PVLDB 2008) Out-of-Order Processing: A New Architecture for High-Performance Stream Systems

rapidgzip (Knespel & Brunst, HPDC '23) generalizes pugz's two-stage speculative parallel gzip decompression from ASCII-only text to arbitrary binary/UTF-8 gzip streams using a 16-bit LUT prefilter for Deflate block-header discovery, two-step marker resolution with cache prefetching, and on-the-fly seek-point indexing.

Blelloch (1989/1990) establishes that any sequential recurrence driven by an associative binary operator (including segmented accumulations, stream compaction, and first-order recurrences) reduces to a work-efficient parallel prefix scan in O(N) work and O(log N) depth.

Premium SSD v1 performance is tied to size tier (P10 500 IOPS/100 MB/s; P30 5,000/200; P70 18,000/750) with single-digit ms latency; Premium SSD v2 gives 3,000/125 free and sub-ms; Ultra sub-ms. In our fleet YBA Azure universes are Premium v1 (10/11 with device info), consistent with Azure's higher mean log_sync p50 (6.7 ms).