The chunking module provides the boundary and packing primitives needed before embedding or indexing a document. It keeps each result connected to the original UTF-8 source, making it possible to inspect chunk text, reproduce generation, and cite exact byte ranges.

This guide builds a Markdown chunking pipeline in stages. The stages remain separate because boundary quality, budget policy, tokenizer choice, and retrieval metadata change for different applications.

Start with a source partition

A segmenter decides where complete units begin and end. Start with Markdown blocks when the source is Markdown and code fences, lists, block quotes, and headings should remain recognizable.

const fs = require("fs");
const chunking = require("chunking");

const source = fs.readFileSync(
  "examples/markdown/chunking-sample.md",
  "utf-8"
);

const blocks = chunking.markdownBlocks(source, {
  atomic: ["fencedCodeBlock", "codeBlock", "htmlBlock"],
});

Immediately verify the preservation invariant while developing a new pipeline:

const reconstructed = blocks.Spans
  .map((span) => span.Text)
  .join("");

if (reconstructed !== source) {
  throw new Error("segmenter did not preserve the source");
}

Built-in segmenters perform this validation in Go. Keeping the assertion in an exploratory script makes the intended contract visible when custom transformations are added later.

Inspect spans before choosing a budget

Span inspection shows which structures will be indivisible during ordinary packing. This matters because a single fenced code block may be larger than the planned chunk budget.

for (const span of blocks.Spans) {
  console.log({
    ordinal: span.Ordinal,
    kind: span.Kind,
    bytes: span.EndByte - span.StartByte,
    runes: span.EndRune - span.StartRune,
    atomic: span.Atomic,
    language: span.Language,
  });
}

The repository command packages the same inspection as structured Glazed output:

./dist/goja-text chunking blocks \
  examples/markdown/chunking-sample.md \
  --output table

Do this before tuning retrieval results. If the primitive spans do not match the document structures the application needs, changing only the chunk budget will not correct the boundary policy.

Pack complete spans

Packing groups consecutive spans while their measured size remains within a budget. The packer is greedy and deterministic.

const packed = chunking.pack(blocks.Spans, {
  maxUnits: 220,
  measure: "runes",
  overlap: { unit: "spans", value: 1 },
  oversized: "allow",
});

The selected measurement answers a specific operational question:

Overlap repeats complete trailing spans. It supplies local context without creating partial source ranges. It also increases total indexed text, so record the overlap setting with an index generation.

Handle oversized structures explicitly

No complete-span packer can satisfy a budget smaller than one of its input spans. The oversized policy makes that condition visible.

With allow, the chunk is emitted with Oversized: true and a diagnostic:

for (const chunk of packed.Chunks) {
  if (chunk.Oversized) {
    console.error(
      "oversized chunk",
      chunk.Ordinal,
      chunk.Diagnostics.map((item) => item.Code)
    );
  }
}

With error, packing stops instead. Use error in a pipeline that must never publish an index with budget violations. Use allow in an exploration tool that needs to show the problematic source and let an operator revise the strategy.

Use recursive fallback when coarse units are too large

Recursive chunking begins with strong document boundaries and refines only oversized ranges. This preserves sections and blocks when they fit while still making progress on long prose.

const recursive = chunking.recursive(source, {
  maxUnits: 140,
  measure: "runes",
  levels: [
    "markdownSections",
    "markdownBlocks",
    "paragraphs",
    "lines",
    "runes",
  ],
  atomic: ["fencedCodeBlock", "codeBlock", "htmlBlock"],
  overlap: { unit: "spans", value: 0 },
  oversized: "allow",
});

The order is policy. Put stronger boundaries first and mechanical boundaries last. The final runes level creates fixed windows when section, block, paragraph, and line boundaries are still too large.

Atomic kinds stop refinement. An atomic fence larger than the budget therefore remains oversized. Remove that kind from atomic only when splitting its exact source text is acceptable to the application.

Run the bundled command to compare the fallback output:

./dist/goja-text chunking recursive \
  examples/markdown/chunking-sample.md \
  --max-units 140 \
  --measure runes \
  --output json

Integrate an actual model tokenizer

The library intentionally does not select an embedding model or tokenizer. A model-specific pipeline computes one weight for each span and passes those weights to packWeighted.

const weighted = chunking.packWeighted(
  blocks.Spans.map((span) => ({
    span,
    weight: tokenizer.count(span.Text),
  })),
  {
    maxWeight: 512,
    overlapWeight: 48,
    oversized: "allow",
  }
);

The tokenizer function is application code. Record its model name, vocabulary revision, normalization settings, special-token policy, and library version beside the chunking strategy. Without that metadata, the same source may produce different weights after a model or tokenizer upgrade.

Store retrieval metadata outside the text primitive

The chunking result provides ranges and structural metadata, but an indexing application still owns document identity and generation metadata. A practical indexed record can include:

const record = {
  documentId,
  generationId,
  chunkOrdinal: chunk.Ordinal,
  sourceRange: [chunk.StartByte, chunk.EndByte],
  headingPath: chunk.HeadingPath,
  strategy: recursive.Spec,
  embeddingModel,
  text: chunk.Text,
  vector,
};

Do not use StartByte alone as a global chunk identifier. It is meaningful only together with the source document and the exact document revision.

Compare strategies in an exploration loop

A useful exploration script keeps segmentation, weighting, packing, and evaluation as replaceable functions:

source
  -> segment(strategy)
  -> inspect spans and preservation
  -> measure(model or deterministic unit)
  -> pack(budget and overlap)
  -> embed
  -> search evaluation queries
  -> compare retrieval evidence

Pseudocode for a strategy sweep:

for each segmentation strategy:
    spans = segment(source, strategy)
    assert join(spans.text) == source

    for each budget and overlap:
        chunks = pack(spans, budget, overlap)
        vectors = embed(chunks.text)
        scores = evaluate(search(vectors, queries))
        save(strategy, chunks, scores, diagnostics)

The module covers the first three operations. Embedding, vector storage, search, and evaluation remain application concerns so they can evolve independently.

Run the complete example

The included script prints block, packed, and recursive summaries:

make build-xgoja
./dist/goja-text run examples/js/chunking-demo.js

Read examples/js/chunking-demo.js before extending it. It is deliberately small enough to copy into an experiment while still checking source preservation and reporting exact coordinates.

Troubleshooting

See Also

• goja-text help goja-text-chunking-api-reference
• goja-text help goja-text-markdown-user-guide
• goja-text chunking blocks --help
• goja-text chunking pack --help
• goja-text chunking recursive --help