Hybrid Search and Reranking for Local RAG

The base RAG pipeline and the live CVE ingestion tutorial get you to a working dense-retrieval system: embed the query, ask ChromaDB for the nearest neighbours, hand the top chunks to the LLM. That pattern is the right default, and it covers most paraphrased and conceptual queries. It also has a specific and predictable failure mode: it loses on exact-match identifiers, on rare technical strings, and on queries where keyword presence is the actual signal.

This tutorial fixes that failure mode without throwing the dense retriever away. You will add a BM25 index alongside the existing Chroma collection, fuse the two rankings with Reciprocal Rank Fusion, and then put a cross-encoder reranker on top of the fused candidates. Every step is a drop-in addition to the CVE pipeline; nothing about ingestion has to change.

Where pure dense retrieval fails

Run a quick experiment against the CVE corpus from the previous tutorials. Pick three questions and ask the existing query() function for the top 3 chunks.

Query 1, Exact CVE identifier. "CVE-2024-3094".

Retrieved from:
  - CVE-2024-6387.txt (chunk 0)   distance 0.41
  - CVE-2023-44487.txt (chunk 0)  distance 0.43
  - CVE-2024-3094.txt (chunk 0)   distance 0.44

The right document is in the top 3, but it is not at the top. The query string is a string of digits and a dash, which the embedding model has weak signal for; semantically, all three chunks are “an advisory header,” so they cluster together. A keyword index would put CVE-2024-3094.txt at rank 1 with no ambiguity at all, because the literal token appears nowhere else in the corpus.

Query 2, Rare domain-specific term. "regreSSHion". This is the marketing name for CVE-2024-6387. It is a portmanteau, almost certainly absent from the embedding model’s training data, and the embedding tokenizer fragments it into pieces that look like regr, essh, ion. The vector you get back lives somewhere unhelpful in embedding space.

Retrieved from:
  - CVE-2023-44487.txt (chunk 1)  distance 0.62
  - CVE-2024-3094.txt (chunk 2)   distance 0.63
  - CVE-2024-6387.txt (chunk 0)   distance 0.64

Same story. The right chunk is in the candidate set but does not lead. BM25 against a lowercased corpus would surface CVE-2024-6387.txt immediately, because the tokenized literal regresshion appears only there.

Query 3, Boolean / negation. "vulnerabilities that do not require authentication". Embeddings are notoriously bad at negation; the vector for “X does not require Y” is closer to “X requires Y” than you would like, because both share most of the content tokens. Dense retrievers can return the exact wrong document on this kind of query. BM25 does not understand negation either, but it at least does not actively pull in the negated form.

The pattern is consistent. Dense retrievers win on paraphrase and conceptual similarity. They lose on literal-match queries and on queries dominated by rare tokens. The fix is to run both, because the failure modes are nearly disjoint.

Note

Why the failure is structural, not a bug Embedding models compress text into a fixed-size vector. That compression is what gives them their power on paraphrase, and it is the same thing that destroys exact-match precision. There is no embedding model tuning that gets you back to the precision of a keyword index on identifiers.

BM25 in 90 seconds

BM25 (Best Matching 25) is a probabilistic ranking function from the 1990s that has refused to die because it works. For a query Q made of terms q1, q2, ... and a document D, the score is

score(D, Q) = Σ IDF(qi) × (f(qi, D) × (k1 + 1)) / (f(qi, D) + k1 × (1 - b + b × |D| / avgdl))

Three pieces:

• IDF(qi): inverse document frequency. Common terms (the, vulnerability) get tiny weight; rare terms (regreSSHion, CVE-2024-3094) get large weight. This is the part that wins on identifiers.
• f(qi, D): how often the term appears in the document. Diminishing returns: the second occurrence of “openssh” matters less than the first.
• The denominator with b and |D|/avgdl: length normalization. Long documents do not get to win just by having more words.

k1 controls how aggressive the term-frequency saturation is (typically 1.2 to 2.0). b controls how aggressive the length normalization is (0 = none, 1 = full). The standard defaults are k1=1.5, b=0.75, which is what rank_bm25.BM25Okapi uses out of the box. They are tunable, but for a corpus of CVE advisories the defaults are fine.

BM25 needs no training, no GPU, and barely a server: a 100k-chunk index fits in a few hundred megabytes of memory. What it does not do is understand that “remote code execution” and “RCE” are the same thing. That is dense retrieval’s job. You want both.

Adding BM25 to the existing pipeline

The simplest pure-Python BM25 implementation is the rank_bm25 package. It is not the fastest implementation in the world, but for the scale this series operates at (low tens of thousands of chunks), it is more than enough. For production-scale corpora (millions of chunks), use Pyserini, which wraps Apache Lucene and gives you the BM25 implementation that powers most search engines.

Install it:

pip install rank_bm25

Then build an index alongside the existing Chroma collection. Create bm25_index.py:

import pickle
import re
from pathlib import Path

import chromadb
from rank_bm25 import BM25Okapi

COLLECTION_NAME = "security_advisories"
BM25_INDEX_PATH = Path("./bm25_index.pkl")

# Hyphen handling matters for CVE IDs. The default split() on whitespace keeps
# "CVE-2024-3094" as a single token, which is what we want — splitting on
# punctuation would break the literal identifier into "cve", "2024", "3094"
# and destroy the IDF advantage on the full ID.
TOKEN_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9\-]*")


def tokenize(text: str) -> list[str]:
    """Lowercase and split into alphanumeric+hyphen tokens.

    We deliberately keep hyphens inside tokens (CVE-2024-3094 stays whole) and
    skip stopword removal — BM25's IDF already down-weights "the", "and", and
    friends, and removing them by hand risks dropping legitimate technical
    tokens that happen to look like English words.
    """
    return [m.group(0).lower() for m in TOKEN_RE.finditer(text)]


def build_index():
    client = chromadb.PersistentClient(path="./chroma_db")
    collection = client.get_collection(COLLECTION_NAME)

    # Pull every chunk from Chroma. For corpora that don't fit in memory you'd
    # page this with limit/offset, but for the CVE pipeline this is one call.
    everything = collection.get(include=["documents", "metadatas"])
    ids = everything["ids"]
    documents = everything["documents"]
    metadatas = everything["metadatas"]

    tokenized = [tokenize(d) for d in documents]
    bm25 = BM25Okapi(tokenized)

    with BM25_INDEX_PATH.open("wb") as f:
        pickle.dump(
            {
                "bm25": bm25,
                "ids": ids,
                "documents": documents,
                "metadatas": metadatas,
            },
            f,
        )

    print(f"Built BM25 index over {len(ids)} chunks at {BM25_INDEX_PATH}")


def load_index():
    with BM25_INDEX_PATH.open("rb") as f:
        return pickle.load(f)


if __name__ == "__main__":
    build_index()

Run it after each ingestion run:

python live_ingest.py
python bm25_index.py

A query against the BM25 index alone:

from bm25_index import load_index, tokenize

idx = load_index()
scores = idx["bm25"].get_scores(tokenize("CVE-2024-3094"))
top = sorted(zip(scores, idx["ids"], idx["documents"]), reverse=True)[:5]
for score, chunk_id, doc in top:
    print(f"{score:6.2f}  {chunk_id}")

Expected output for the corpus from the base tutorial:

  9.41  CVE-2024-3094.txt::chunk0
  0.00  CVE-2024-6387.txt::chunk0
  0.00  CVE-2023-44487.txt::chunk0
  0.00  CVE-2024-3094.txt::chunk1
  0.00  CVE-2024-3094.txt::chunk2

Only one chunk has a non-zero score because only one chunk contains the exact identifier. Compare that to the dense retriever’s top 3 from earlier in this tutorial, where the right chunk landed at rank 3.

Warning

Rebuild the BM25 index when the collection changes rank_bm25 does not have an incremental update API; the IDF table is computed at construction time. After every live_ingest.py run that adds or replaces chunks, re-run bm25_index.py. For the CVE pipeline this takes single-digit seconds even at 50k chunks; for larger corpora, wrap the rebuild in the same systemd timer that runs ingestion.

Tip

Tokenization choices for technical text Three things to think about. First, lowercasing is almost always correct for technical corpora. Second, stopword removal is usually a wash, BM25’s IDF handles common words automatically. Third, hyphen and dot handling is where you can quietly destroy your index. The regex above keeps CVE-2024-3094 whole; a naive re.split(r"\W+", text) would shatter it. If your corpus contains version strings like 5.6.1 or paths like /etc/ssh/sshd_config, decide explicitly how those should tokenize before you build the index.

Reciprocal Rank Fusion

Now you have two retrievers that disagree on which chunk is best. Combining them is a small but surprisingly thorny problem, because Chroma returns cosine distances (low is good, range roughly 0 to 2) and BM25 returns positive scores with no upper bound. You cannot just add them.

Reciprocal Rank Fusion sidesteps the calibration problem by ignoring the scores entirely and using only the ranks. For each retriever, give every document a score equal to 1 / (k + rank), then sum across retrievers:

score(d) = Σ_r 1 / (k + rank_r(d))

The constant k=60 is the conventional default from Cormack, Clarke and Buettcher’s 2009 paper; it dampens the contribution of very-top ranks just enough that a document at rank 5 from one retriever can compete with a document at rank 1 from another. Tunable, but rarely worth tuning.

Implementation:

def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    """Combine multiple ranked lists of document IDs into one fused ranking.

    Each input list is assumed to be ordered best-first. Documents not present
    in a given list contribute zero from that list. Returns (id, score) pairs
    sorted best-first.
    """
    fused: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            fused[doc_id] = fused.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(fused.items(), key=lambda x: x[1], reverse=True)

Wire it into a hybrid retrieve function. Create hybrid_retrieve.py:

import chromadb
import ollama

from bm25_index import load_index, tokenize

COLLECTION_NAME = "security_advisories"
EMBED_MODEL = "nomic-embed-text"
DENSE_TOP_N = 50
SPARSE_TOP_N = 50
FUSION_TOP_N = 20


def reciprocal_rank_fusion(rankings, k=60):
    fused = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            fused[doc_id] = fused.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(fused.items(), key=lambda x: x[1], reverse=True)


def dense_search(query, n=DENSE_TOP_N):
    client = chromadb.PersistentClient(path="./chroma_db")
    collection = client.get_collection(COLLECTION_NAME)
    qvec = ollama.embed(model=EMBED_MODEL, input=query)["embeddings"][0]
    res = collection.query(query_embeddings=[qvec], n_results=n)
    return list(res["ids"][0]), {
        cid: (doc, meta)
        for cid, doc, meta in zip(res["ids"][0], res["documents"][0], res["metadatas"][0])
    }


def sparse_search(query, idx, n=SPARSE_TOP_N):
    scores = idx["bm25"].get_scores(tokenize(query))
    ranked = sorted(zip(scores, idx["ids"]), reverse=True)[:n]
    ids = [cid for _, cid in ranked if _ > 0]  # drop true zeros
    lookup = dict(zip(idx["ids"], zip(idx["documents"], idx["metadatas"])))
    return ids, {cid: lookup[cid] for cid in ids}


def hybrid_search(query, top_n=FUSION_TOP_N):
    idx = load_index()
    dense_ids, dense_lookup = dense_search(query)
    sparse_ids, sparse_lookup = sparse_search(query, idx)

    fused = reciprocal_rank_fusion([dense_ids, sparse_ids])

    # Materialize fused IDs back to documents + metadata. Either retriever
    # might have produced a given ID, so try both lookups.
    results = []
    for chunk_id, score in fused[:top_n]:
        if chunk_id in dense_lookup:
            doc, meta = dense_lookup[chunk_id]
        else:
            doc, meta = sparse_lookup[chunk_id]
        results.append({"id": chunk_id, "score": score, "document": doc, "metadata": meta})
    return results


if __name__ == "__main__":
    for hit in hybrid_search("CVE-2024-3094")[:5]:
        print(f"{hit['score']:.4f}  {hit['id']}")

Run it on the same identifier query that fooled dense retrieval:

0.0325  CVE-2024-3094.txt::chunk0
0.0163  CVE-2024-6387.txt::chunk0
0.0162  CVE-2023-44487.txt::chunk0
0.0161  CVE-2024-3094.txt::chunk1
0.0160  CVE-2024-3094.txt::chunk2

The right chunk leads cleanly. The other two top-3 chunks from the dense-only result are still present but pushed down: dense found them, BM25 did not, so they accumulate score from one retriever instead of two.

Why two retrievers beats weighted score combination

The natural alternative to RRF is to combine the raw scores: α × cosine_similarity + (1 - α) × bm25_score. This sounds principled and is in fact a trap.

Cosine similarities live in [-1, 1] (and in practice [0.3, 0.9] for sensibly-trained embedding models). BM25 scores are unbounded positive numbers whose distribution depends on document length, vocabulary size, and how rare your query terms happen to be. The right α for one corpus is wrong for another. Worse, the right α for one query class within a corpus is wrong for a different query class, a one-token query produces BM25 scores in a completely different range than a ten-token query.

You can fix this with score normalization (z-scoring, min-max, etc.), but each fix introduces another assumption about the score distribution. RRF avoids the entire problem by working on ranks, which are dimensionless and bounded by [1, N] regardless of corpus, query, or retriever. Use it. Tune α only if you have evaluation data showing RRF underperforms on your specific workload.

Cross-encoder reranking

Hybrid retrieval gets the right chunks into the candidate set. A cross-encoder gets the right chunk to rank 1.

The embedding model used for retrieval is a bi-encoder: it produces a vector for the query, a separate vector for each document, and scores them by cosine similarity. The two halves never see each other. That is what makes it fast, you can pre-embed every document at index time. It is also what limits its accuracy, because the model has to commit to a single fixed representation of each document without knowing what the query will be.

A cross-encoder reads the concatenated (query, document) pair as one input and produces a single relevance score. The query and document attend to each other across every transformer layer. This is much more accurate, and much more expensive: the bi-encoder’s cost is O(1) per query (the documents are already embedded), while the cross-encoder is O(N), one forward pass per candidate.

The standard pattern is to use bi-encoder retrieval to narrow the corpus to a small candidate set, then cross-encode the candidates. Hybrid retrieval finishes step 1; the reranker finishes step 2.

Note

Cross-encoders are not generative A cross-encoder is a classifier that scores (query, doc) pairs. It does not produce embeddings, does not generate text, and cannot be used for retrieval over an unindexed corpus. It only ranks candidates you have already gathered.

Two reasonable open models for this:

• BAAI/bge-reranker-v2-m3 (~568M parameters, multilingual). Strong quality, runs on CPU but slowly; fast on a modest GPU. Works well across English, Chinese, and many other languages.
• cross-encoder/ms-marco-MiniLM-L-6-v2 (~23M parameters, English only). Tiny, runs in tens of milliseconds per pair on CPU. Lower ceiling than bge-reranker-v2-m3 but a sensible choice on a laptop without a GPU. The English-only bge-reranker-base is a similar tier with newer training data.

Install:

pip install FlagEmbedding
# or, for the MiniLM model:
pip install sentence-transformers

Use the FlagEmbedding interface for the BGE model:

from FlagEmbedding import FlagReranker

# use_fp16 speeds GPU inference with a small quality tradeoff.
# Set it to False if you are CPU-only and hit dtype or device issues.
reranker = FlagReranker("BAAI/bge-reranker-v2-m3", use_fp16=True)

pairs = [(query, candidate_text) for candidate_text in candidate_texts]
scores = reranker.compute_score(pairs)

Or sentence-transformers for the MiniLM model:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
scores = reranker.predict([(query, c) for c in candidate_texts])

Both return one float per pair; sort descending and keep the top few.

Warning

BGE reranker raw outputs are logits FlagReranker.compute_score returns the raw logit, not a probability. The values are unbounded (commonly ranging from about -10 to +10), and they are perfectly fine for ranking, only the order matters. If you need a calibrated 0-1 score (for a UI, a threshold, or to combine with other signals), pass normalize=True to compute_score, or apply the sigmoid yourself: 1 / (1 + exp(-x)). Do not apply softmax across candidates: each pair is scored independently, not as a probability distribution.

Putting it together

Here is the full hybrid + rerank pipeline as a drop-in replacement for the query() function from the base tutorial. Create hybrid_rerank.py:

import math

import ollama
from FlagEmbedding import FlagReranker

from hybrid_retrieve import hybrid_search

CHAT_MODEL = "llama3.2"
RERANK_MODEL = "BAAI/bge-reranker-v2-m3"
RERANK_INPUT = 20   # candidates from hybrid retrieval
FINAL_TOP_K = 3     # chunks fed to the LLM

# Load once at module import. The model is ~2.3 GB on disk; you do not want to
# re-instantiate it per query.
_reranker = FlagReranker(RERANK_MODEL, use_fp16=True)


def sigmoid(x: float) -> float:
    return 1.0 / (1.0 + math.exp(-x))


def rerank(query, candidates, top_k=FINAL_TOP_K):
    """Cross-encode (query, candidate) pairs and keep the top_k.

    candidates is the list returned by hybrid_search; we attach a calibrated
    rerank_score (sigmoid of the raw logit) and sort by it.
    """
    pairs = [(query, c["document"]) for c in candidates]
    raw_scores = _reranker.compute_score(pairs)
    # FlagReranker returns a float for one pair, list[float] for many.
    if isinstance(raw_scores, float):
        raw_scores = [raw_scores]

    for c, raw in zip(candidates, raw_scores):
        c["rerank_score"] = sigmoid(raw)
        c["rerank_logit"] = raw

    candidates.sort(key=lambda c: c["rerank_score"], reverse=True)
    return candidates[:top_k]


def ask(question):
    """Drop-in replacement for the base tutorial's ask() function."""
    print(f"Question: {question}\n")

    candidates = hybrid_search(question, top_n=RERANK_INPUT)
    final = rerank(question, candidates, top_k=FINAL_TOP_K)

    print("Retrieved (hybrid + rerank):")
    for c in final:
        print(
            f"  - {c['metadata'].get('source', c['id'])}  "
            f"rerank={c['rerank_score']:.3f}  rrf={c['score']:.4f}"
        )
    print()

    context = "\n\n---\n\n".join(c["document"] for c in final)
    prompt = f"""You are a security analyst assistant. Answer the question
using only the context provided below. If the context doesn't contain
enough information to answer, say so — do not guess.

Context:
{context}

Question: {question}"""

    response = ollama.chat(
        model=CHAT_MODEL,
        messages=[{"role": "user", "content": prompt}],
    )
    print(response["message"]["content"])


if __name__ == "__main__":
    ask("What systems are affected by CVE-2024-3094?")

The pipeline now does, in order:

1. Embed the query with nomic-embed-text and ask Chroma for the top 50 dense candidates.
2. Tokenize the query and ask the BM25 index for the top 50 sparse candidates.
3. RRF-fuse both lists into a unified top 20.
4. Cross-encode all 20 (query, candidate) pairs with bge-reranker-v2-m3.
5. Pass the top 3 reranked chunks to llama3.2 as context.

graph TD
    Q[Query] --> D["Dense search<br/>nomic-embed-text + Chroma<br/>top 50"]
    Q --> S["Sparse search<br/>BM25 index<br/>top 50"]
    D --> F["Reciprocal Rank Fusion<br/>fuse to top 20"]
    S --> F
    F --> R["Cross-encoder rerank<br/>bge-reranker-v2-m3"]
    R --> K[Top 3 chunks]
    K --> L["LLM<br/>llama3.2"]
    L --> A[Grounded answer]

    style Q fill:#4a9eff,stroke:#2a7edf,color:#fff
    style D fill:#868e96,stroke:#666e76,color:#fff
    style S fill:#868e96,stroke:#666e76,color:#fff
    style F fill:#ffa94d,stroke:#df894d,color:#fff
    style R fill:#ffa94d,stroke:#df894d,color:#fff
    style K fill:#51cf66,stroke:#31af46,color:#fff
    style L fill:#4a9eff,stroke:#2a7edf,color:#fff
    style A fill:#51cf66,stroke:#31af46,color:#fff

The two retrievers run independently and converge at fusion: a chunk that both rank highly rises to the top, while a chunk only one of them found still survives into the candidate set. The reranker then does the precise ordering on that fused shortlist.

Sample output for the CVE-ID query:

Retrieved (hybrid + rerank):
  - CVE-2024-3094.txt  rerank=0.998  rrf=0.0325
  - CVE-2024-3094.txt  rerank=0.987  rrf=0.0161
  - CVE-2024-3094.txt  rerank=0.972  rrf=0.0160

All three reranked top results come from the right advisory. Compare with the dense-only baseline at the start of this tutorial, where the right document was at rank 3.

Latency budget

Concrete numbers from a 16-core laptop CPU with no discrete GPU and a 10k-chunk corpus, all measured cold-cache after warmup:

Two takeaways. First, the LLM call dominates total latency by a wide margin; reranker choice is rarely the bottleneck. Second, the rerank model size matters a lot more than the retriever stack. If you are CPU-bound and the user is waiting, switch from bge-reranker-v2-m3 to MiniLM-L-6-v2 before you start tuning anything else; the 20x cost reduction is real and the quality drop on English text is small.

Measuring the gain

Build a small evaluation set of (question, ground_truth_chunks) pairs against the CVE corpus. Run the same set against three pipeline configurations and compute context precision and context recall:

The numbers above are representative for a security-advisory corpus with a mix of literal-identifier queries and conceptual queries; your results will vary with your eval set. The shape is what matters:

• Hybrid alone gives a meaningful recall lift (3-8 points) on query classes that contain literal tokens, CVE IDs, version strings, product names, named exploits. It does not change much on paraphrased questions.
• The cross-encoder rerank gives the larger precision jump (5-10 points) by pushing borderline candidates down. It also gives a smaller recall improvement, because the rerank can pull a true-positive from rank 12 to rank 1.

Break down the eval set by query class (identifier vs. conceptual vs. negation) before you tune. The ID-class bucket is where hybrid retrieval pays for itself; the conceptual bucket is where the cross-encoder pays for itself; the negation bucket is where both still struggle and you want to know it.

Note

ChromaDB does not have native hybrid search Some vector databases (Weaviate, Qdrant, OpenSearch) ship with built-in BM25 + dense fusion. ChromaDB does not, so we maintain the BM25 index as a separate pickle file alongside the Chroma collection. This is fine at the scale this series targets, under a few hundred thousand chunks. If you outgrow that, the right move is to switch to a vector DB with native hybrid support rather than scaling the pickle approach.

When to skip the cross-encoder

The cross-encoder is the most accurate stage and the most expensive. There are real cases where you should leave it out:

• Interactive chat with a sub-second budget on CPU. The MiniLM model is fast enough; bge-reranker-v2-m3 is borderline. If you cannot afford either, drop reranking entirely and rely on the hybrid retriever.
• Retrieval is being done in a tight loop, e.g. ReAct-style agentic loops where each tool call hits the retriever. The reranker cost adds up.
• You can tolerate a slightly noisier context. Modern long-context LLMs like llama3.1:8b or llama3.2 (with their 128k window) handle a top-10 context reasonably well. Substituting “rerank to top-3” with “skip reranker, send top-10 to the LLM” works surprisingly often, because the LLM does its own implicit reranking via attention. You pay in tokens (and latency in the LLM, not the retriever), but you simplify the stack.

A reasonable default: keep hybrid retrieval everywhere, and turn on the reranker only on the user-facing query path where the result is read directly. For background or agentic queries, skip it.

Common mistakes

Forgetting to rebuild the BM25 index after ingestion. rank_bm25 does not have an incremental update; if you add CVEs to Chroma without re-running bm25_index.py, your sparse retriever will silently miss them. Wire the rebuild into the same cron job as live_ingest.py.

Splitting CVE IDs at the hyphen. A naive tokenizer (re.split(r"\W+", text)) shatters CVE-2024-3094 into three tokens. The whole reason to bring in BM25 is exact-match precision on identifiers; do not destroy it at the tokenizer.

Score-blending with an untuned alpha. If you replace RRF with α × cosine + (1-α) × bm25 without normalizing, you get one of two failure modes: either the unbounded BM25 scores dominate and you have a sparse-only retriever in disguise, or you over-correct with a tiny α and the BM25 contribution effectively disappears. Use RRF.

Treating cross-encoder logits as probabilities without sigmoid. Sorting by raw logits gives the right order but ugly numbers. If you display the score in a UI or apply a threshold, run 1 / (1 + exp(-x)) first, or pass normalize=True to FlagReranker.compute_score.

Over-fetching from the reranker. Cross-encoding scales linearly in candidate count; reranking 100 candidates costs 5x reranking 20 with no measurable quality gain on most corpora. Set RERANK_INPUT = 20 and leave it there unless your eval data shows otherwise.

Using a multilingual reranker on monolingual data. bge-reranker-v2-m3 is fine for English-only workloads and excellent if you genuinely have multiple languages. If you do not, the smaller English-specialized models (bge-reranker-base, MiniLM-L-6-v2) give you most of the quality at a fraction of the latency.

Next steps

• Defend against retrieval poisoning, adversarial documents that exploit BM25’s keyword-stuffing weakness or the reranker’s surface-pattern bias. Hybrid retrieval changes the attack surface; it does not eliminate it.
• Build a SOC triage assistant that uses this hybrid pipeline against live alert and CVE feeds.

For intuition on why dense retrieval struggles with rare tokens in the first place, the Embedding Space Explorer is the fastest way to see it: drop in a CVE identifier and a paraphrase of the same advisory, and watch how their vectors place themselves relative to unrelated technical text.