Understanding and Using the Rerank Package

What is Reranking?

Reranking is a retrieval stage that takes one query and an ordered set of candidate documents and scores them with a cross-encoder model. Unlike embedding retrieval (where query and document vectors are produced independently), a cross-encoder scores the query and each document together, producing a relevance score per document. This typically yields higher-quality ordering than first-stage vector or BM25 retrieval.

Reranking is Geppetto's third model-service primitive, alongside inference (pkg/inference/engine) and embeddings (pkg/embeddings).

Use cases:

• RAG — reorder retrieved chunks before generation
• Search — improve first-stage retrieval precision
• Evidence selection — pick the most relevant passages

Core Concepts

Caller-owned document identity

The rerank request carries only a caller-controlled unique ID and exact text per document. Application metadata and first-stage retrieval scores stay outside the provider request. This keeps the interface reusable and reduces accidental disclosure.

Provider index mapping

HTTP reranker protocols identify results by the zero-based position of the submitted document. Array position is transport identity, not application identity. The adapter maps:

request.Documents[i].ID
    -> provider request documents[i]
    -> provider response result.index == i
    -> response.Results[*].DocumentID

A caller must never infer application identity from response order alone.

TopN

TopN controls response cardinality. Some providers return only the highest-scoring TopN documents. A caller that needs one score per input must set TopN == len(Documents). The package validates the actual response against the requested cardinality. TopN is never defaulted: explicit cardinality lets complete-score callers prove they requested all candidates.

Scores

Scores are provider- and model-specific. They may be negative and must not be interpreted as probabilities. The package accepts any finite score and sorts descending.

Usage and cost

Reranking usually consumes input tokens but produces no generated output tokens. The Usage record carries input and total tokens when the provider reports them. A nil Cost means pricing is unknown; a pointer to zero means the provider is explicitly free/local under the selected pricing policy. nil and zero are intentionally distinguishable.

Quick Start (Go)

import (
    "github.com/go-go-golems/geppetto/pkg/rerank"
    "github.com/go-go-golems/geppetto/pkg/rerank/llamacpp"
    "github.com/go-go-golems/geppetto/pkg/security"
)

provider, err := llamacpp.New(llamacpp.Options{
    BaseURL: "http://127.0.0.1:18012",
    Model:   "qllama/bge-reranker-v2-m3:q4_k_m",
    OutboundURL: security.OutboundURLOptions{
        AllowHTTP:          true,
        AllowLocalNetworks: true,
    },
})
if err != nil { panic(err) }

resp, err := provider.Rerank(ctx, rerank.Request{
    Query: "How does TTC calculate a payroll adjustment?",
    Documents: []rerank.Document{
        {ID: "chunk-001", Text: "A payroll adjustment corrects wages or deductions."},
        {ID: "chunk-002", Text: "Cypress trees tolerate dry conditions."},
    },
    TopN: 2,
})
if err != nil { panic(err) }

for _, r := range resp.Results {
    fmt.Println(r.Rank, r.DocumentID, r.Score)
}

Profile-based Construction (Go)

Reranking integrates with Geppetto's engine profile system. A rerank-only profile stacks a base API profile:

inference_settings:
  api:
    base_urls:
      rerank-base-url: http://127.0.0.1:18012
    allow_http:
      rerank: true
    allow_local_networks:
      rerank: true
  rerank:
    type: llamacpp
    engine: qllama/bge-reranker-v2-m3:q4_k_m
    max_request_bytes: 2097152
    max_response_bytes: 1048576

Construct from resolved InferenceSettings:

import (
    rerankfactory "github.com/go-go-golems/geppetto/pkg/rerank/factory"
)

factory, err := rerankfactory.NewSettingsFactoryFromInferenceSettings(resolvedSettings)
if err != nil { panic(err) }
provider, err := factory.NewProvider()
if err != nil { panic(err) }

ValidateInferenceSettingsForRerank gives profile-oriented diagnostics before construction.

Hosted provider: Cohere

A Cohere profile needs no base URL — the adapter defaults to the canonical https://api.cohere.com endpoint — only an API key and a model:

inference_settings:
  api:
    api_keys:
      cohere-api-key: ${COHERE_API_KEY}
  rerank:
    type: cohere
    engine: rerank-v3.5

Notes:

• cohere-base-url under api.base_urls is an optional override for proxies or tests. Plain-HTTP or local-network overrides are rejected unless the allow_http.rerank / allow_local_networks.rerank flags are set, exactly like llama.cpp.
• Cohere bills rerank calls in search units, not tokens (meta.billed_units.search_units). Response.Usage therefore stays nil (the provider did not report token usage), and Response.Cost stays nil unless a per-search rate is configured (GEPPETTO-RERANKER-002, DR-3).
• The request ID comes from the response body (id), not a header.
• The wire option max_tokens_per_doc is intentionally not exposed; callers pre-truncate Document.Text (DR-4).

JavaScript API

The require("geppetto") module exposes reranker(settings), consistent with embeddings(settings):

const gp = require("geppetto");
const settings = gp.inferenceProfiles
  .load("~/.config/pinocchio/profiles.yaml")
  .resolve("bge-reranker-local");

const reranker = gp.reranker(settings);

// Synchronous (for bounded scripts):
const response = reranker.rerank(
  "How does TTC calculate a payroll adjustment?",
  [
    {id: "chunk-001", text: "A payroll adjustment corrects wages or deductions."},
    {id: "chunk-002", text: "Cypress trees tolerate dry conditions."}
  ],
  {topN: 2}
);

for (const result of response.results) {
  console.log(result.rank, result.documentId, result.score);
}

Asynchronous execution

For event-loop applications, use rerankAsync to avoid blocking the runtime owner thread:

const handle = reranker.rerankAsync(query, documents, {topN: documents.length});
try {
  const response = await handle.promise;
  // ...
} finally {
  handle.close();
}

The handle exposes cancel() and close() for cancellation and runtime shutdown. The provider goroutine touches no JavaScript value; Promise settlement happens on the runtime owner thread.

Security constraints

JavaScript cannot supply an endpoint, credential, HTTP client, local-network exception, or provider implementation directly. Those capabilities remain in host/profile configuration. The settings argument must be the hidden-reference InferenceSettings wrapper returned by inferenceProfiles.resolve.

Security

The llama.cpp adapter enforces:

• Bounded IO: request and response bodies are size-limited (MaxRequestBytes, MaxResponseBytes).
• Strict decoding: unknown JSON fields and trailing data are rejected.
• Outbound URL policy: scheme, host, userinfo, query, and fragment are validated; HTTP and local networks are denied by default.
• Redirect rejection: local model endpoints should not redirect; redirects are rejected to prevent validated-endpoint escape.
• Safe errors: query text, document text, credentials, and response bodies never enter error messages.
• Cancellation: context cancellation terminates active requests.

Supported Providers

Two providers are supported, both constructed through the same factory and usable from Go profiles and gp.reranker(settings) without any provider-specific JavaScript:

• llamacpp — a self-hosted llama.cpp /v1/rerank server. Local HTTP and local networks must be explicitly allowed in the profile.
• cohere — the hosted Cohere v2 /rerank API (e.g. rerank-v3.5). Authenticates with api_keys.cohere-api-key, defaults to https://api.cohere.com, and requires no allow flags.

The core package is transport-neutral; future adapters (Jina, voyage) can be added without changing the Provider interface.

Live Qualification

Use the runnable example for interactive qualification. It accepts either a resolved rerank profile or complete inline rerank settings and emits one row per ranked document, so both JSON and table output remain readable:

go run ./cmd/examples/rerank-profile-smoke run \
  --rerank-type llamacpp \
  --rerank-engine qllama/bge-reranker-v2-m3:q4_k_m \
  --rerank-base-url http://127.0.0.1:18012 \
  --output json

The opt-in test remains the automation guard:

GEPPETTO_LIVE_RERANK=1 \
GEPPETTO_RERANK_BASE_URL=http://127.0.0.1:18012 \
GEPPETTO_RERANK_MODEL=qllama/bge-reranker-v2-m3:q4_k_m \
go test ./pkg/rerank/llamacpp -run TestLive -v -count=1

It skips unless GEPPETTO_LIVE_RERANK=1 is set exactly, never falls back to a fixture, and never starts external services itself.

The equivalent opt-in test exists for Cohere and requires a real API key:

GEPPETTO_LIVE_RERANK=1 \
COHERE_API_KEY=<your-key> \
GEPPETTO_RERANK_MODEL=rerank-v3.5 \
go test ./pkg/rerank/cohere -run TestLive -v -count=1

Downstream Integration

Applications (such as a RAG system) adapt Geppetto reranking through a thin domain adapter. The dependency direction is:

application -> geppetto/pkg/rerank
geppetto/pkg/rerank -X-> any application package

The application owns evidence IDs, manifests, truncation, complete-score requirements, traces, citations, and evaluation. Geppetto owns the provider transport.