Baradb/docs/en/fts.md

Full-Text Search Engine

Inverted index with BM25 and TF-IDF ranking for text search.

Usage

import barabadb/fts/engine

var idx = newInvertedIndex()
idx.addDocument(1, "Nim is a fast programming language")
idx.addDocument(2, "Python is popular for data science")

# BM25 search
let results = idx.search("programming language")

# TF-IDF search
let tfidf = idx.searchTfidf("programming language")

# Fuzzy search (typo tolerance)
let fuzzy = idx.fuzzySearch("programing", maxDistance = 2)

# Wildcard search
let wild = idx.regexSearch("prog*")

Ranking Methods

BM25

Best matching ranking algorithm:

let bm25 = idx.searchBM25("query terms")

TF-IDF

Term Frequency-Inverse Document Frequency:

let tfidf = idx.searchTfidf("query terms")

Search Features

Feature	Description
Fuzzy search	Levenshtein distance tolerance
Wildcard	Prefix, suffix, and infix wildcards
Regex	Regular expression patterns
Phrase search	Exact phrase matching with slop support
Proximity search	Terms within a configurable distance window
Boolean	AND, OR, NOT operators with nested expressions
Faceted search	Category filtering, counts, and aggregation
Hybrid search	Combined full-text + vector (HNSW) with RRF fusion
Segment indexing	Incremental indexing with automatic compaction
Field boosting	Per-field relevance weights

SQL Interface

Full-text search is also available directly in BaraQL:

-- Create a table with text column
CREATE TABLE articles (id INT PRIMARY KEY, title TEXT, body TEXT);

-- Create an FTS index
CREATE INDEX idx_fts ON articles(body) USING FTS;

-- Search with the @@ operator (BM25 ranking)
SELECT * FROM articles WHERE body @@ 'machine learning';

-- Search with multiple terms
SELECT * FROM articles WHERE body @@ 'quick brown fox';

Multi-Language Support

import barabadb/fts/multilang

# Supported languages: EN, BG, DE, FR, RU
var tokenizer = newTokenizer("bg")  # Bulgarian
let tokens = tokenizer.tokenize("Търсене в пълен текст")

Features per language:

• Tokenization
• Stop words
• Stemming
• Language detection

Advanced Search

The new src/barabadb/search/ module provides a unified search engine with segment-based indexing for high-performance search operations.

UnifiedSearchEngine

import barabadb/search/engine

# Create search engine with default configuration
var engine = newUnifiedSearchEngine()

# Index documents with fields and facets
engine.indexDocument(
  docId = 1,
  text = "Nim is a fast programming language",
  fields = {"title": "Nim Overview"}.toTable,
  facets = {"category": @["programming"], "level": @["beginner"]}.toTable
)

# Basic search
let results = engine.search("programming language", limit = 10)

# Phrase search (exact phrase matching)
let phrase = engine.searchPhrase(@["fast", "programming"], slop = 0)

# Proximity search (terms within distance)
let proximity = engine.searchProximity(@["fast", "language"], maxDistance = 5)

# Boolean queries
let boolResults = engine.searchBoolean("programming AND (fast OR efficient)")
let boolResults2 = engine.searchBoolean("Nim AND NOT Python")
let boolResults3 = engine.searchBoolean("\"exact phrase\" OR wildcard*")

# Fuzzy search with typo tolerance
let fuzzy = engine.searchFuzzy("programing", maxDistance = 2)

# Prefix and wildcard search
let prefix = engine.searchPrefix("prog", limit = 10)
let wildcard = engine.searchWildcard("prog*", limit = 10)

Faceted Search

import barabadb/search/engine
import std/sets

# Index documents with facets
engine.indexDocument(
  docId = 1,
  text = "Nim tutorial",
  facets = {"category": @["programming", "tutorial"], "difficulty": @["beginner"]}.toTable
)

# Get facet counts
let counts = engine.getFacetCounts("category", limit = 10)
for count in counts:
  echo count.value, ": ", count.count

# Filter by facets
var filters = @[
  FacetFilter(field: "category", values: @["programming"], exclude: false),
  FacetFilter(field: "difficulty", values: @["advanced"], exclude: true)
]
let matchingDocs = engine.filterByFacets(filters)

# Aggregate multiple facets
let agg = engine.facets.aggregate(@["category", "difficulty"], matchingDocs)

Hybrid Search (Text + Vector)

import barabadb/search/engine
import barabadb/vector/engine

# Index vectors
engine.indexVector(1, @[0.1, 0.2, 0.3], {"title": "Doc 1"}.toTable)

# Hybrid search combining text and vector similarity
let hybrid = engine.hybridSearch(
  queryText = "programming",
  queryVec = @[0.1, 0.2, 0.3],
  k = 10,
  textWeight = 1.0,
  vecWeight = 1.0
)

# Filtered vector search
proc filterMeta(meta: Table[string, string]): bool =
  meta.getOrDefault("category") == "programming"

let filtered = engine.searchVectorFiltered(@[0.1, 0.2, 0.3], k = 10, filterMeta)

Configuration and Management

# Custom configuration
var config = defaultSearchConfig()
config.language = langBulgarian
config.maxSegmentSize = 100_000
config.ngramSize = 3
config.enableFacets = true

var engine = newUnifiedSearchEngine(config)

# Set field boosts for relevance tuning
engine.setFieldBoost("title", 2.0)
engine.setFieldBoost("body", 1.0)

# Change language
engine.setLanguage(langBulgarian)

# Compact segments for better performance
engine.compact()

# Get statistics
echo "Documents: ", engine.documentCount()
echo "Terms: ", engine.termCount()

# Remove documents
engine.removeDocument(1)