Transforming LLMs into Efficient Cross-Encoders via Knowledge Distillation for RAG Reranking

• 类型：arxiv
• 标识：2607.11933
• 链接：http://arxiv.org/abs/2607.11933v1
• 主分类：rag
• 形态：application
• 被引：0
• 被引来源：Semantic Scholar
• S2被引：0
• 影响力被引：0
• TLDR：Cross-encoders achieve high reranking accuracy in Retrieval-Augmented Generation (RAG) pipelines but impose quadratic inference costs that limit real-time deployment. We address this by fine-tuning LLaMA 3 (8B) as a drop-in reranker using a two-stage pipeline: supervised fine-tuning on a custom query-document relevance dataset via the Unsloth framework with LoRA adapters, followed by 4-bit quantization for efficient inference. The resulting model replaces the cross-encoder in a dual-retriever RAG pipeline combining BM25 and dense vector search. Evaluated on a domain-specific question-answering
• 副分类：llm-infra
• 待LLM分类：否
• 标题中文：通过知识蒸馏将 LLM 转化为高效的 RAG 重排序 Cross-Encoder
• TLDR中文：Cross-encoder 在 RAG 流水线中具有较高的重排序准确率，但推理成本随序列长度呈二次增长，难以实时部署。本文通过两阶段流水线解决该问题：使用 Unsloth 框架与 LoRA 适配器，在自定义的查询-文档相关性数据集上对 LLaMA 3 (8B) 进行监督微调，随后进行 4-bit 量化以提升推理效率。该模型可替换双路检索 RAG 流水线中结合 BM25 与稠密向量检索的 cross-encoder，并在特定领域问答……
• 来源文件：
• /inbox/tom/_candidates/2026-07-21-rag-retrieval-reranking-candidates.json
• [S2 enrich]