A Zeroth-Order Paradigm for LLM Preference Alignment

类型：arxiv
标识：2609.19144
链接：https://arxiv.org/abs/2609.19144
主分类：risk
形态：method
TLDR：Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins. In this paper, we propose and analyze Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles. ComPO extracts directional information from these pairs without directly optimizing a differentiable preference loss on them. We establish a converg
待LLM分类：否
来源文件：
/inbox/tom/_candidates/2026-09-17-agent-rag-longcontext-candidates.json

2026-08-31

笔记精读笔记：Characterizing State Space Model and Hybrid Language Model Performance with Long Context

2026-08-30

笔记 KVAE Tokenizers arXiv:2608.05798 · multimodal · 精读与批判 v2

2026-08-09

笔记精读与批判 · From 128K to 4M: Efficient Training of Ultra-Long Context LLMs

2026-08-06

仓库 Dusto-AI/llm-situational-awareness-paper-list

2026-08-11

笔记精读与批判 v2 · Where Reliability Lives in Vision-Language Models · 2026-08-31 22:50 (重写覆盖)

2026-08-31

arXiv

2609.19144

TLDR

Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins. In this paper, we propose and analyze Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles. ComPO extracts directional information from these pairs without directly optimizing a differentiable preference loss on them. We establish a converg

arXiv ↗ 中文全文

A Zeroth-Order Paradigm for LLM Preference Alignment

相关 · 按语义匹配