Methodology

Each source text is chunked and passed through a large language model for structured named entity recognition. The LLM identifies entities and classifies them into a controlled taxonomy of ten categories, each with specific subcategories.

Chunking strategy

Source texts are split into 2,500-character chunks with 200-character overlap. Breaks are paragraph-aware: the chunker looks for paragraph boundaries near the target split point to avoid cutting mid-sentence. EEBO-specific artifacts (page markers, marginal references) and OCR noise are cleaned before chunking.

LLM extraction

Each chunk is sent to Gemini 2.5 Flash Lite with a structured prompt asking for entities in JSON format. The model operates at temperature 0.1 for near-deterministic output.

Entity taxonomy

Each category also includes a catch-all “Other” subcategory for entities that don’t fit neatly into a specific subcategory. The original six-book corpus primarily produced the first eight categories; Anatomy and Organization emerged as distinct categories after adding nineteenth-century scientific texts (Darwin, James) with their richer anatomical and institutional vocabularies.

Per-entity output

For each entity the model returns: the surface form name, category, subcategory, a short context string (up to 10 words), and variant spellings found in the same chunk. After extraction, a separate script locates full-text excerpts (150 characters of surrounding context per mention) using regex-based string matching across the source text.

LLM extraction produces many duplicate or near-duplicate entities within a single text (e.g. “Galeno,” “galeno,” “GALENO”). These are merged using embedding similarity and graph-based clustering.

Process

1. Embed all entities within a book using the fine-tuned BGE-M3 model with category context appended to each name
2. Compute pairwise cosine similarity between all entity embeddings
3. Build a graph where edges connect entities above the merge threshold
4. Find connected components via BFS — each component becomes a merged entity
5. Validate each component: check that every member has sufficient similarity to the primary entity (highest-count member)

Off-the-shelf multilingual embeddings struggle with early modern naming conventions, archaic spellings, and the non-obvious conceptual linkages central to historical concordance-building. We fine-tune BAAI/bge-m3 on curated training pairs using contrastive learning, iterating through multiple rounds of data curation and evaluation.

Training data curation

The training dataset was built through a multi-stage process. An initial automated extraction from verified concordance clusters produced ~6,800 positive pairs, but quality review revealed a ~14% error rate and heavy skew toward trivial cognates. Strict filtering (Levenshtein distance scoring, per-cluster caps, blocked pair lists) reduced this to ~800 pairs, which were then manually reviewed and supplemented with expert-curated batches targeting specific domains.

The final training file uses an additive batch format — each batch is independently contributed and can be appended without modifying existing entries. The current dataset comprises five batches totalling 889 positive pairs and 154 hard negatives:

Positive pairs teach the model that surface forms refer to the same entity, ranging from straightforward cross-lingual matches (canela ↔ cinnamon) to non-obvious conceptual concordances (Palo santo ↔ Guaiacum, Falling sickness ↔ epilepsy, vibratiuncles ↔ memory traces). Hard negatives teach the model to distinguish confusable terms: canfora ≠ canela, phrenology ≠ phenology, vis viva ≠ vis vitalis, hystérie ≠ hystérèse.

Training configuration

Evaluation: A/B testing v2 vs. v3

To verify the fine-tuned model generalizes beyond its training data and does not overfit, we ran a comprehensive A/B test comparing the previous fine-tune (v2, trained on ~500 pairs) against the new fine-tune (v3, trained on 889 pairs + 154 hard negatives). The test used six evaluation suites comprising ~430 pair comparisons, including held-out concordance pairs explicitly excluded from training, novel cross-lingual pairs, and randomly sampled same-category negative pairs.

The critical metric is separation: the gap between average positive similarity and average negative similarity on held-out data. V3 achieved a separation of 0.438 compared to v2’s 0.406, confirming that the new model generalizes well and is not overfitting to training data.

Before and after: base model vs. fine-tuned

Comparing the unmodified BGE-M3 base model against the v3 fine-tune on the full evaluation set shows the scale of improvement:

The base BGE-M3 model had negative separation on our data — it rated confusable non-matches (canfora/canela, phrenology/phenology) higher than genuine cross-lingual equivalences (cão danado/Rabies, effluvium/radiation). After fine-tuning, the model correctly separates these distributions. Notable spot-check improvements include:

With entities deduplicated and the embedding model fine-tuned, every entity across all books is embedded and compared pairwise. This is the core operation that discovers cross-lingual correspondences.

Matching rules

• Same category only: PERSON matches PERSON, PLANT matches PLANT — no cross-category matches
• Subcategory compatibility: when both entities have valid subcategories, they must match
• One-to-one constraint: each entity can match at most one entity per book, preventing “attractor” entities
• String similarity integration: lexical similarity provides a ±0.03 bonus/penalty to the embedding score

Link classification

Each match is classified into one of five types:

Pairwise matches are assembled into concordance clusters using connected component analysis. Each cluster represents a single real-world referent as it appears across multiple books and languages.

Cluster construction

1. All cross-book matches form a graph; connected components become candidate clusters
2. The primary entity (highest mention count) becomes the canonical name
3. Every other member must have a direct edge (similarity ≥ 0.84) to the primary — this prevents chaining artifacts where A→B→C creates a false A→C link
4. Substring matching serves as an additional confirmation signal (one name containing the other)
5. Members failing validation are removed; clusters shrinking to a single book are dissolved

Near-duplicate merging

A post-processing step merges clusters that were split due to subcategory noise or minor orthographic differences. This catches splits like “cheiro” (categorized as QUALITY) vs “cheyro” (categorized as OTHER_CONCEPT) that should be one cluster.

Embedding-based matching inevitably produces some false positives. A verification pass uses an LLM to review “suspicious” clusters and remove members that don’t belong.

Suspicion heuristics

Rather than reviewing all 1,500+ clusters, the system flags those that exhibit patterns correlated with false matches:

LLM review

Flagged clusters are sent to Gemini 2.5 Flash Lite, which receives the cluster’s canonical name, all members with their book of origin and context excerpts, and must return a verdict: which members to keep, which to remove, and a brief justification. When the LLM identifies sub-groups within a cluster, it can split the cluster rather than simply removing members.

Verified clusters are enriched with modern identifications. Currently, an LLM identification pass provides modern names, descriptions, and Linnaean binomials. Wikidata entity linking is planned for a future iteration.

LLM identification

Clusters are batched (8 per API call) and sent to Gemini with their canonical name, category, top members, and context excerpts. The model returns a structured identification including: modern name, Linnaean binomial (for biological entities), type classification, temporal data, geographic associations, and a confidence level.

Wikidata linking (planned)

A planned enrichment step will have the LLM suggest a Wikidata search term for each cluster. This term will be used to query the Wikidata API, with results scored by domain relevance:

Semantic glosses (planned)

A planned enrichment pass will generate 2–3 sentence “semantic glosses” for each cluster — thematic descriptions grounded in how the entity appears in the source texts. Unlike encyclopedic descriptions, these glosses will capture historical context: e.g. “Venomous snakes considered extremely dangerous in early modern medicine. Associated with poison, antidotes, theriac preparations, and fear.”

Enrichment coverage

Wikidata linking and Wikipedia URL resolution are planned but not yet integrated into the current dataset. Semantic glosses are similarly pending a future enrichment pass.

The final stage generates a semantic search index for the web interface. Each cluster’s identity is compressed into a rich text representation and embedded using a fast, general-purpose model.

Embedding text composition

The embedding text is constructed from multiple fields to maximize search recall:

1. Canonical name (highest weight)
2. Category and subcategory
3. Semantic gloss (thematic description)
4. Variant names from all members (up to 20)
5. Modern name from ground truth
6. Linnaean binomial
7. Wikidata description
8. Botanical family
9. Source text context excerpts

Hybrid search

The web search combines semantic similarity (cosine distance between query and cluster embeddings) with lexical matching (Levenshtein distance, substring matching, fuzzy matching against canonical names, variant names, modern names, and semantic glosses). This ensures that both conceptual queries (“exotic spice”) and exact-name queries (“Galeno”) return good results.

Edge Cases & Known Challenges

Early modern texts present problems that rarely arise in modern NLP. Below are the most significant challenges and how the pipeline addresses them.

Models & APIs