Calibration & Conformal Prediction

Papers drawn in Calibration & Conformal Prediction (10)

Every paper drawn in this area, sorted by curated degree. Click a title to open its ego graph on the canvas above, or follow the DOI / arXiv link to the publisher record.

Curated semantic relation notes touching Calibration (18)

• operationalizes → Breaking down memory walls → On Calibration of Modern Neural Networks

  Combines LSM-tree storage engine telemetry indicators with calibrated log-odds likelihood-ratio scoring.

  Drafted by: agent draft (gemini)

• complements ↔ MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents ↔ Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control

  Pairs sentence-level fact-checking entailment verification (MiniCheck) with distribution-free risk-controlling abstention gates (Learn Then Test).

  Drafted by: agent draft (gemini)

• refutes → Putting an Agent on the Incident Queue: What I Measured Before I Trusted It. → Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs

  Across n=368 incidents and 787 production runs, self-reported verbalized LLM confidence carries no usable discriminative signal (AUROC 0.460, p=0.315; and 0.518, p=0.661, 95% CI 0.447-0.580), refuting reliance on verbalized confidence for operational gating.

  Drafted by: agent draft (gemini)

• uses method → Putting an Agent on the Incident Queue: What I Measured Before I Trusted It. → A New Vector Partition of the Probability Score

  Applies Murphy (1973) exact Brier partition (MCB - DSC + UNC) to prove verbalized LLM confidence has DSC=0.0038

  Drafted by: seed curator

• extends → Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control → On Calibration of Modern Neural Networks

  Extends post-hoc temperature scaling to finite-sample, distribution-free family-wise error control (Learn Then Test) for risk-controlling abstention gates.

  Drafted by: agent draft (gemini)

• complements ↔ Drain: An Online Log Parsing Approach with Fixed Depth Tree ↔ On Calibration of Modern Neural Networks

  Combines fixed-depth parse-tree log template clustering with post-hoc probability calibration for log-based incident classification.

  Drafted by: agent draft (gemini)

• extends → Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control → Conformal Prediction Under Covariate Shift

  Extends split-conformal distribution-free guarantees to arbitrary non-monotone risk functions via Learn-Then-Test

  Drafted by: seed curator

• complements ↔ Prediction-Powered Inference ↔ Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control

  Combines small gold human labels (n=45) with large model predictions (N=253) for unbiased variance reduction

  Drafted by: seed curator

• complements ↔ Optimal Detection of Changepoints With a Linear Computational Cost ↔ On Calibration of Modern Neural Networks

  Combines exact linear-time PELT regime changepoint detection with post-hoc probability calibration.

  Drafted by: agent draft (gemini)

• complements ↔ Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions ↔ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

  Pezeshkpour & Hruschka (2024) quantify multiple-choice option-order sensitivity alongside Zheng et al.'s (2023) pairwise position bias in LLM-as-a-Judge.

  Drafted by: agent draft (gemini)

• contrasts with ↔ Large Language Models Must Be Taught to Know What They Don't Know ↔ Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback

  Tian et al. find verbalized confidence can be better calibrated than RLHF logits; Kapoor et al. find prompting-only uncertainty is unreliable and requires fine-tuning on ~1k examples.

  Drafted by: agent draft (claude)

• extends → Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences → Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

  Zheng et al. establish LLM judges agree with humans on aggregate; Shankar et al. show the judge criteria themselves must be aligned to human grades and drift during grading.

  Drafted by: agent draft (claude)

• operationalizes → Learning Selective LLM Autonomy from Copilot Feedback in Enterprise Customer Support Workflows → Selective Classification for Deep Neural Networks

  Production support system uses a learned abstention critic (selective prediction) trained on operator accept/correct feedback.

  Drafted by: agent draft (claude)

• contrasts with ↔ Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs ↔ Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure

  Contrasts uncalibrated verbalized self-confidence elicited from LLMs against likelihood-based EM prior-shift correction on class-imbalanced distributions.

  Drafted by: agent draft (gemini)

• complements ↔ Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision ↔ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

  Weak supervision of a strong model quantifies the ceiling that weak judges/raters impose

  Drafted by: agent draft (qwen)

• complements ↔ Automatic Root Cause Analysis via Large Language Models for Cloud Incidents ↔ Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control

  Pairs RCACopilot's automated cloud incident handler matching and root-cause category prediction with distribution-free conformal risk control.

  Drafted by: agent draft (gemini)

• extends → Towards a Theory of Longitudinal Trust Calibration in Human–Robot Teams → Trust in Automation: Designing for Appropriate Reliance

  Adds trust violation and repair dynamics over time to Lee and See's calibration framework

  Drafted by: agent draft (gemini)

• complements ↔ "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust ↔ Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making

  Both show uncertainty/confidence display calibrates reliance; Kim adds natural-language first-person hedges for LLMs.

  Drafted by: agent draft (claude)