dimgigov
965ed2f675
perf: optimize FTS and HNSW engines + real PostgreSQL benchmarks
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
FTS Engine (src/barabadb/fts/engine.nim):
- Fix bm25Score doing O(n) linear scan per document
- Cache IDF per token instead of recomputing for each doc
- Use entry.termFreq directly instead of searching postings again
- Result: FTS search +438% (249 -> 1360 queries/s)
HNSW Vector Engine (src/barabadb/vector/engine.nim):
- Optimize distance functions with float32 + 4x loop unrolling
- Rewrite searchLayer: swap+pop instead of O(n) del, track worst-nearest
instead of sorting nearest on every iteration
- Result: HNSW insert +117% (245 -> 543 ops/s), search 2.2x faster
Benchmarks:
- Add real PostgreSQL comparison script (benchmarks/pg_bench.py)
- Add report generator (benchmarks/generate_report.py)
- Fix compare.nim cpuTime() bug (was dividing by 1M incorrectly)
- Add nimble tasks: bench_pg, bench_report
Docs:
- Update README.md and docs/en/performance.md with real measured numbers
- Add benchmarks/REAL_COMPARISON.md
Version bump: 1.1.7 -> 1.1.8
2026-05-29 17:11:22 +03:00
dimgigov
967c0855a5
Bug fixes: composite PK, nl_to_sql sandbox, FK check, SQL injection, storage correctness
...
Critical fixes:
- Composite PK: execInsert + validateConstraints use all PK columns
- nl_to_sql: non-SELECT SQL no longer executed directly during validation
- FK check: removed O(N) scanMemTable fallback, uses db.get() with SSTables
- exprToSql: nkIdent wrapped in quotes to prevent SQL injection
- restoreSchema: try/except around tokenize/parse for crash resilience
- recovery.nim: lastTxnId tracking + putUnsafe/deleteUnsafe without WAL
- SCRAM: verifyClientProof length check + DefaultIterationCount restored
Storage fixes:
- lsm.nim: SSTable sort order fixed (ascending), close() flushes all memtables
- compaction.nim: tombstones preserved during compaction
- wal.nim: header written for empty existing files, readEntries checks magic
- btree.nim: B+ tree leaf split keeps boundary key
- bloom.nim: deserialize raises on short data
- mmap.nim: bounds checks for adviseWillNeed/DontNeed + posix.close()
Protocol fixes:
- zerocopy.nim: readString bounds check
- wire.nim: deserializeValue 32-bit underflow check
- auth.nim: JWT JSON escaping for claims
- server.nim: readUint32BE bounds check + specific exception handling
- raft.nim: readData checks + specific exception handling
Tests:
- Added Composite Primary Key test suite (4 tests)
Build: 0 warnings, 0 errors
2026-05-18 11:33:11 +03:00
dimgigov
19fa760604
Week 4: Production Hardening — Thread Safety, Fuzz Tests, Property-Based Tests, NimForum Smoke Test
...
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
Thread Safety:
- Add lock to BTreeIndex (insert/get/scan/remove/contains)
- Add lock to InvertedIndex (addDocument/removeDocument/search/bm25/tfidf/fuzzy/regex)
- Add ctxLock to ExecutionContext for DDL/schema metadata, autoIncCounters, sequences
- Deep-copy sessionVars in cloneForConnection() to isolate per-connection state
- Fix activeConnections check-then-act race with Lock
Wire Protocol Hardening:
- Fix RangeDefect in deserializeValue for invalid FieldKind bytes
- Fix IndexDefect in fkFloat32/fkFloat64 by reusing readUint32/readUint64
- Fix int32/int64 cast from uint32/uint64 to use cast[] instead of constructor
- Fix parseHeader to validate MsgKind instead of holey enum cast
Property-Based Tests (tests/prop_test.nim):
- 8 invariants for evalExprValue: literal types, commutativity, identity,
double negation, NULL propagation
Wire Protocol Fuzz (tests/fuzz_test.nim):
- 7 fuzz tests: random bytes deserialization, truncated buffers,
MsgKind casts, parseHeader logic, message roundtrips
NimForum Adapter Smoke Test (tests/nimforum_smoke_test.nim):
- Full TCP server lifecycle with adapter CRUD and parameterized queries
Test Results: 423 tests pass, 0 failures
2026-05-17 11:57:55 +03:00
dimgigov
6aaabb518d
fix: високо и средно приоритетни бъгове в query, protocol, storage, vector, fts
...
Поправени проблеми:
- Wire protocol: bounds checking, max length limits, recursion depth limit
- SQL injection: escape-ване на quotes в exprToSql string literals
- ReDoS: escape-ване на regex metachars в LIKE/ILIKE
- Stale BTree indexes: добавен BTree.remove() + изтриване при UPDATE/DELETE
- SSL: quoteShell за всички shell команди с пътища
- Boolean literals: parser вече разпознава tkTrue/tkFalse
- Unary minus: lowerExpr map-ва ukNeg към irNeg (вместо irNot)
- Non-aggregate UDFs: lowerExpr създава irekFuncCall вместо NULL literal
- UTF-8 FTS: tokenize използва runes вместо байтове
- HNSW: добавен Lock за thread-safe insert/search
292 теста, 0 failure-а.
2026-05-12 23:02:54 +03:00
dimgigov
096c8347cf
feat: complete Phase 3 + new production roadmap for Web/ERP
...
- Thread-safety: locks in LSMTree and Graph engines
- Raft network transport: async TCP, serialization, heartbeat, 3-node election test
- CI/CD: GitHub Actions workflow
- Cleanup: remove dead code, unused imports, build artifacts
- New PLAN.md targeting production Web/ERP readiness
- 216 tests passing
2026-05-06 10:40:34 +03:00
dimgigov
805dbc8ad6
fix: comprehensive bug audit — 15 critical/high/medium fixes
...
Critical fixes:
- B-Tree splitChild: fixed out-of-bounds on child.children.len (mid+1..len → mid+1..<len)
- Wire protocol: all deserializeValue cases now properly assign to result
- Wire protocol: fkBool/fkInt8 now increment pos after reading
- MVCC: write-write conflict now detected for completed transactions
- MVCC: commit only marks visible versions as deleted (not concurrent uncommitted)
- Lexer: buffer overrun on backslash escape at end of input — added bounds check
- Bulgarian/Russian stemmers: fixed UTF-8 byte offsets for Cyrillic suffixes (2 bytes per char)
High fixes:
- WebSocket: added missing std/tables import
- Raft: added randomize() call for election timeouts
- MemTable.get: binary search O(log n) instead of linear scan O(n)
Medium fixes:
- Zerocopy: string length now written as 4-byte int32 (was single byte)
- Raft appendEntries: guard against nextIdx underflow
- Columnar: null tracking with nulls seq on ColumnPtr
- Removed dead code in lexer readIdent (redundant true/false checks)
- Updated Bulgarian stemming test to match corrected byte offsets
All 214 tests pass across 48 suites with 0 failures.
2026-05-06 02:46:37 +03:00
dimgigov
3ed3036b11
feat: cross-modal engine, 2PC, gossip, client lib, import/export, multi-lang FTS — 196 tests
...
Cross-Modal Engine:
- Unified query interface across document, vector, graph, FTS
- Hybrid search with weighted scoring across all engines
- 2PC (two-phase commit) for cross-modal transactions
Gossip Protocol:
- Membership management with alive/suspect/dead states
- Incarnation-based conflict resolution
- Fanout-based gossip message dissemination
Client Library:
- Async and sync client with connection string parser
- Fluent query builder (select/from/where/join/groupBy/having/orderBy/limit)
Import/Export:
- JSON, CSV, NDJSON formats
- Parse and generate with proper quoting/escaping
Multi-Language FTS:
- Tokenizers for EN, BG, DE, FR, RU
- Language-specific stemmers and stop word lists
- Automatic language detection
TLS/SSL configuration module.
37 new tests (196 total, all passing)
2026-05-06 01:46:15 +03:00
dimgigov
67213826a8
feat: compaction, page cache, WebSocket, rate limiter, TF-IDF, fuzzy search, regex, metadata filter — 73 tests
...
- SSTable compaction: size-tiered strategy, level-based scheduling
- Page cache: LRU eviction, hit rate tracking, capacity management
- WebSocket: full duplex streaming, frame encoding, ping/pong
- Rate limiter: token bucket + sliding window algorithms
- FTS: TF-IDF ranking, Levenshtein fuzzy matching, wildcard regex
- Vector: metadata filtering on HNSW search
- 16 new tests (73 total, all passing)
2026-05-06 01:03:58 +03:00
dimgigov
5c84aeccf8
feat: MVCC, wire protocol, HTTP API, schema system, CLI shell — 39 tests
...
- MVCC: multi-version concurrency control with snapshot isolation
- Deadlock detection: wait-for graph with cycle detection
- Wire Protocol: binary message format, 16 types, big-endian serialization
- HTTP/REST API: router with CORS, path params, JSON
- Schema system: types, links, properties, migration diffing
- CLI: interactive query shell (bara shell)
- 18 new tests (39 total, all passing)
2026-05-06 00:32:02 +03:00
dimgigov
6935889877
feat: initial BaraDB — multimodal database engine in Nim
...
- LSM-Tree storage engine with WAL, bloom filter, MemTable
- BaraQL query language: lexer (80+ tokens), recursive descent parser, AST
- Vector engine: HNSW + IVF-PQ indexes, 4 distance metrics
- Graph engine: adjacency list, BFS/DFS, Dijkstra, PageRank
- Full-Text Search: inverted index, BM25 ranking, stemming, stop words
- Type system: 17 types (int/float/string/uuid/json/vector/...)
- Async TCP server
- 21 passing tests
2026-05-06 00:22:12 +03:00
dimgigov