Baradb/docs/en/search.md

Unified Search Module

Overview

The UnifiedSearchEngine is the main entry point for all search operations in BarabaDB. It combines multiple search capabilities into a single, cohesive API:

• Full-Text Search (FTS) — BM25-ranked retrieval over segmented inverted indexes.
• Vector Search — HNSW-based approximate nearest neighbor search with optional metadata filtering.
• Phrase Search — Exact or slop-aware phrase matching.
• Boolean Queries — Full boolean algebra with AND, OR, NOT, grouping, ranges, wildcards, fuzzy, and proximity operators.
• Faceted Search — Categorical filtering with per-field facet counts.
• Fuzzy Search — N-gram candidate generation verified by Levenshtein distance.
• Hybrid Search — Combines FTS and vector scores for blended retrieval.

Installation

Add the module to your Nim project:

import barabadb/search/engine

No additional dependencies are required; the search module is part of the core barabadb package.

Basic Usage

import barabadb/search/engine

let config = defaultSearchConfig()
var search = newUnifiedSearchEngine(config)

# Index documents
search.indexDocument(1, "The quick brown fox", {"title": "Animals"}.toTable)
search.indexDocument(2, "Lazy dog sleeps all day", {"title": "Pets"}.toTable)

# BM25 search
let results = search.search("quick fox", limit = 10)

# Phrase search
let phrases = search.searchPhrase(@["quick", "brown"], slop = 0)

# Boolean query
let boolResults = search.searchBoolean("quick AND (fox OR dog)")

# Fuzzy search
let fuzzy = search.searchFuzzy("quik", maxDistance = 2)

# Prefix search
let prefix = search.searchPrefix("quic*")

# Vector search
search.indexVector(1, @[0.1'f32, 0.2, 0.3], {"category": "A"}.toTable)
let vecResults = search.searchVector(@[0.15'f32, 0.25, 0.35], k = 10)

# Hybrid search (combines FTS + vector)
let hybrid = search.hybridSearch("fox", @[0.1'f32, 0.2, 0.3], k = 10)

Advanced Features

Faceted Search

Faceted search lets you filter results by categorical metadata and retrieve aggregated counts per facet value.

# Index with facets
search.indexDocument(1, "Nim programming book",
  fields = {"author": "John"}.toTable,
  facets = {"category": @["programming", "books"], "language": @["nim"]}.toTable)

# Filter by facets
let filters = @[FacetFilter(field: "category", values: @["programming"])]
let filteredDocs = search.filterByFacets(filters)

# Get facet counts
let counts = search.getFacetCounts("category")

Field Boosting

Field boosting adjusts the relative importance of matches in different fields. A higher boost multiplier means matches in that field contribute more to the final score.

search.setFieldBoost("title", 3.0)  # Title matches 3x more important
search.setFieldBoost("author", 2.0)

Multi-Language Support

The search engine ships with Porter2 stemmers for several languages. Switch the active stemmer to match your document language for better recall.

search.setLanguage(langBulgarian)  # Switch to Bulgarian stemmer

Supported stemmers: English (langEnglish), Bulgarian (langBulgarian), German (langGerman), French (langFrench), Russian (langRussian).

Segment Management

The index is organized into segments that are merged periodically. Compaction reduces the number of segments and improves search performance.

# Compact segments for better performance
search.compact()

# Get statistics
echo "Documents: ", search.documentCount()
echo "Terms: ", search.termCount()

Boolean Query Syntax

The boolean query parser supports a rich syntax for composing complex search expressions.

Operator	Example	Description
AND (default)	quick brown	Both terms required
AND (explicit)	quick AND brown	Both terms required
OR	quick OR brown	Either term
NOT	quick NOT brown	Exclude brown
Phrase	"quick brown fox"	Exact phrase
Proximity	"quick fox"~3	Within 3 words
Wildcard	quic*	Prefix match
Fuzzy	quik~2	Max 2 edits
Grouping	(quick OR slow) AND fox	Boolean groups
Range	price:[10 TO 100]	Numeric range

Examples

# Simple conjunction — both terms must appear
let r1 = search.searchBoolean("database indexing")

# Disjunction with exclusion
let r2 = search.searchBoolean("search OR retrieval NOT deprecated")

# Phrase with proximity
let r3 = search.searchBoolean("\"quick fox\"~5")

# Grouped boolean with field range
let r4 = search.searchBoolean("(nim OR rust) AND performance score:[80 TO 100]")

Performance Characteristics

HNSW Vector Search

The vector index uses a Hierarchical Navigable Small World graph with heap-based searchLayer:

• Speed: 2.4x faster than linear scan on the heap-optimized path.
• Recall@10: 92–99% depending on dataset size and dimensionality.
• Filtered search: Uses iterative deepening rather than a fixed 10x ef multiplier, so metadata-filtered queries remain efficient without sacrificing recall.

Segment-Based Indexing

Documents are indexed into immutable segments that are merged during compaction:

• Auto-segmentation: A new segment is created every 50,000 documents.
• Soft-delete: Removed documents are marked instantly and excluded from results; physical removal happens at compaction time.
• Periodic compaction: search.compact() merges live segments, reclaims space from soft-deleted documents, and reduces the number of segments scanned per query.

N-gram Fuzzy Search

Fuzzy matching is a two-phase process:

1. Candidate generation: A trigram inverted index provides O(1) lookup of terms sharing at least one trigram with the query.
2. Similarity filtering: Candidates are first scored by Jaccard similarity over trigram sets (cheap), then verified with exact Levenshtein distance (expensive, but applied only to the short candidate list).

Architecture

UnifiedSearchEngine
├── SegmentIndex (FTS with BM25)
│   └── Multiple segments (auto-merge)
├── NGramIndex (fuzzy/prefix/wildcard)
│   └── Trigram inverted index
├── FacetIndex (categorical filtering)
│   └── Per-field value → docId mapping
├── HNSWIndex (vector search)
│   └── Heap-optimized searchLayer
└── Porter2 Stemmers (EN/BG/DE/FR/RU)

Each sub-index is independently testable and can be used in isolation if only a subset of search capabilities is needed.

Migration from FTS Engine

If you are upgrading from the standalone FTS engine, the migration is straightforward.

Old code:

import barabadb/fts/engine
var idx = newInvertedIndex()
idx.addDocument(1, "text")
let results = idx.search("query")

New code:

import barabadb/search/engine
var search = newUnifiedSearchEngine()
search.indexDocument(1, "text")
let results = search.search("query")

Key changes:

Old API	New API	Notes
newInvertedIndex()	newUnifiedSearchEngine()	Includes all sub-indexes
addDocument(id, text)	indexDocument(id, text, fields, facets)	Fields and facets are optional
search(query)	search(query, limit)	Limit parameter added

The old barabadb/fts/engine module is deprecated and will be removed in a future release.

Benchmark Results

Benchmarks run on a single thread, 128-dimensional vectors, HNSW parameters M=16, efConstruction=200, efSearch=50.

N=1K:   insert=0.24s  search=0.30ms  recall@10=99.6%
N=5K:   insert=2.64s  search=0.94ms  recall@10=97.8%
N=10K:  insert=6.94s  search=1.09ms  recall@10=92.6%
N=50K:  insert=70.67s search=2.26ms  recall@10=75.5%

• insert — total wall-clock time to index N documents (including vector insertion).
• search — mean latency per hybrid search query.
• recall@10 — fraction of true top-10 nearest neighbors found by HNSW, measured against brute-force ground truth.