研究库 论文知识库

内容库 / 论文

arXiv:2609.26489 · 评测基准

Calibration as a First-Class Criterion in LLM Evaluation

Calibration as a First-Class Criterion in LLM Evaluation

类型：arxiv
标识：2609.26489
链接：https://arxiv.org/abs/2609.26489
主分类：evaluation
形态：benchmark
TLDR：Calibration of language models -- the alignment between expressed or implicit confidence and empirical correctness -- is a well-studied subfield within NLP. Methods to measure it already exist. The problem is adoption: outside this subfield, NLP research regularly introduces new models, datasets, and benchmarks without checking whether the model's confidence scores are meaningful. We argue that this adoption gap is a major obstacle to trustworthy LLM evaluation. Miscalibration causes problems in two distinct areas: at deployment, where overconfident mistakes cause real harm, and inside the res
待LLM分类：否
来源文件：
/inbox/tom/_candidates/2026-09-24-agent-rag-longcontext-candidates.json

相关 · 按语义匹配

笔记精读笔记：Characterizing State Space Model and Hybrid Language Model Performance with Long Context

笔记 KVAE Tokenizers arXiv:2608.05798 · multimodal · 精读与批判 v2

文档 Stephen 自测 · R64 · 2026-09-03

仓库 deekshith318/AI-language-models-impact

笔记精读与批判 v2 · Where Reliability Lives in Vision-Language Models · 2026-08-31 22:50 (重写覆盖)

仓库 ELM-Research/ECG-Language-Models

arXiv

2609.26489

TLDR

Calibration of language models -- the alignment between expressed or implicit confidence and empirical correctness -- is a well-studied subfield within NLP. Methods to measure it already exist. The problem is adoption: outside this subfield, NLP research regularly introduces new models, datasets, and benchmarks without checking whether the model's confidence scores are meaningful. We argue that this adoption gap is a major obstacle to trustworthy LLM evaluation. Miscalibration causes problems in two distinct areas: at deployment, where overconfident mistakes cause real harm, and inside the res

arXiv ↗ 中文全文