研究库 论文知识库

内容库 / 论文

arXiv:2609.26550 · 评测基准

JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

JEV-as-a-Judge：自信则接受，不确定则升级

JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

类型：arxiv
标识：2609.26550
链接：https://arxiv.org/abs/2609.26550
主分类：evaluation
形态：method
TLDR：LLM-as-a-judge enables evaluation across diverse tasks, but inference cost and confidence reliability become critical at scale. We study whether a decision-only judge can provide an economical first pass and identify when stronger evaluation is needed. Comparing jev-as-a-judge with sixteen generative and reward-model judges, with blinded human adjudication, we find it within three percentage points of a state-of-the-art LLM judge, our strongest comparator, on ordinary preference and evidence-grounded factuality at 0.36% of the comparator's fee. Larger gaps arise when judgments require checking
待LLM分类：否
标题中文：JEV-as-a-Judge：自信则接受，不确定则升级
TLDR中文：LLM-as-a-judge 支持跨任务评估，但在大规模场景下推理成本与置信度可靠性成为关键问题。本文研究仅做判断的 judge 能否提供经济高效的首轮判别，并识别何时需要更强的评估。在与十六种生成式与奖励模型 judge 的对比中（采用盲法人类裁定作为参照），我们发现 jev-as-a-judge 在普通偏好与有证据支撑的事实性任务上，与作为最强对照的 SOTA LLM judge 仅相差 3 个百分点，成本仅为后者的 0.36%。在需要核查
来源文件：
/inbox/tom/_candidates/2026-09-23-agent-rag-longcontext-candidates.json

相关 · 按语义匹配

笔记 FLAT · Flexible-Length Aligned Transmodal Tokens · 批判性精读

笔记精读与批判 v2 · SpecPV · 自投机部分验证 (arXiv:2512.02337v2) · 2026-09-03 22:50 (重写覆盖)

笔记 flyP 精读与批判 · 2026-08-01 09:50 (Asia/Shanghai) · v2 覆盖

笔记 flyP 精读与反方审稿 · SpectraReward · 2026-07-15 15:50

笔记 SSR: Self-Speculation for Faster Reasoning Models — flyP 重写精读与批判

文档 Tom 评 flyP · 2026-08-29 批判性精读棒 · LongVQUBench 轻量精读

arXiv

2609.26550

TLDR

LLM-as-a-judge 支持跨任务评估，但在大规模场景下推理成本与置信度可靠性成为关键问题。本文研究仅做判断的 judge 能否提供经济高效的首轮判别，并识别何时需要更强的评估。在与十六种生成式与奖励模型 judge 的对比中（采用盲法人类裁定作为参照），我们发现 jev-as-a-judge 在普通偏好与有证据支撑的事实性任务上，与作为最强对照的 SOTA LLM judge 仅相差 3 个百分点，成本仅为后者的 0.36%。在需要核查LLM-as-a-judge enables evaluation across diverse tasks, but inference cost and confidence reliability become critical at scale. We study whether a decision-only judge can provide an economical first pass and identify when stronger evaluation is needed. Comparing jev-as-a-judge with sixteen generative and reward-model judges, with blinded human adjudication, we find it within three percentage points of a state-of-the-art LLM judge, our strongest comparator, on ordinary preference and evidence-grounded factuality at 0.36% of the comparator's fee. Larger gaps arise when judgments require checking

arXiv ↗ 中文全文