Benchmarks
Methodology, hardware, metrics, reproducibility, and interpretation of Wolbarg v0.2.1 dual-backend performance results.
What is it?
How Wolbarg v0.2.1 measures startup, insert, search, retrieval, hybrid/filters/MMR, ingest, forget, compression, memory, database size, and push-to-failure concurrency on SQLite and PostgreSQL.
Two suites — do not mix them:
| Suite | Embeddings | What it measures |
|---|---|---|
| Storage (mock) | Local mock OpenAI-compatible server | SDK + database ceiling (I/O, indexes, concurrency) |
| LIVE | Real providers (OpenAI, etc.) | End-to-end latency including network + provider time |
Interactive charts: /benchmarks. Raw downloads: /benchmarks/benchmark.json · /benchmarks/benchmark.md. Public suite: wolbarg-benchmarks.
The published site artifact is scale=quick (corpora 100 and 1k). Larger scales (10k / 100k) are not in this publish — run the suite locally rather than inventing a curve.
Why does it exist?
Vendor graphs without methodology mislead. These docs explain what was measured so you can compare fairly and reproduce locally — especially the difference between mock stress and live API spots.
Methodology
Mock vs LIVE (read this)
Primary stress and push-to-failure concurrency use a local mock OpenAI-compatible embedding/LLM server. Live OpenAI is not used for failure ramps because API rate limits and quota errors would dominate long before SQLite or PostgreSQL contention, masking true Wolbarg/storage breaking points.
A separate LIVE spot suite (npm run benchmark:live) reports real-network latency for representative insert / search / hybrid / ingest / compress / vision paths. It does not ramp concurrency to failure.
Failure criteria (breaking ramps)
A concurrency level fails when:
errorRate > 1%, orp95 latency > 5s, or- a hard integrity/exception failure (duplicate IDs, crash, etc.)
Reports record lastHealthyLevel and breakingLevel with reason (error_rate | p95_sla | exception | integrity | cap).
Storage matrix
Every workload runs on:
| Backend | Notes |
|---|---|
| SQLite | Local file + WAL |
| PostgreSQL | Local Docker only (pgvector/pg17 via benchmark/docker-compose.yml). Hosted Neon/Supabase/Railway URLs are refused — they measure network RTT, not Wolbarg. |
Start Postgres before dual-backend runs:
cd benchmark
npm run postgres:up
npm run benchmark:brutalPrefer fresh local Docker numbers over older hosted or misconfigured Postgres runs when comparing backends.
Modes & scales
| Command | Mode | Scale | Failure ramps |
|---|---|---|---|
npm run benchmark | mock | full | yes |
npm run benchmark:quick | mock | quick | no |
npm run benchmark:brutal | mock | brutal | yes (up to 4096) |
npm run benchmark:live | live | spot | no |
npm run benchmark:sqlite | mock brutal | sqlite only | yes |
npm run benchmark:postgres | mock brutal | postgres only | yes |
Metrics
| Benchmark | What it means |
|---|---|
| Startup cold/warm | Time to ready() + probe providers |
| Insert ops/sec | Sustained remember throughput |
| Search / Retrieval | Semantic recall latency at scale |
| Hybrid / Filters / MMR | v0.2 retrieval feature overhead |
| Ingest / Chunking | Document → chunks → embed → store |
| Forget / Clear | Delete latency + integrity |
| Concurrency baseline | Fixed writer counts on one client |
| Breaking concurrency | Write / read / mixed ramps until SLA or hard fail |
| Compression % | Active-set reduction after compress |
| DB size | On-disk / relation growth |
| Memory usage | Heap / RSS snapshots |
Hardware (example)
Always read the environment block in the latest Benchmarks.md artifact for the run you cite (Node, CPU, RAM, mode, backends, dims).
Dataset
Synthetic memories with fixed text templates. Mock dims default 384; live typically 1536 (text-embedding-3-small). Labels (100, 1000, 10000, 100000) are memory counts.
Reproducibility
cd benchmark
npm install
cp .env.example .env # set DATABASE_URL for postgres; API keys for --live
npm run benchmark:brutal
# or
npm run benchmark:livePrefer recording: date, SDK version (wolbarg@0.2.1), Node version, CPU/RAM, mode (mock/live), backends, and git SHA.
Interpretation
- Startup ms — agents open memory without multi-second cold starts
- Search vs scale — know when Postgres/pgvector or sharding strategies matter
- Breaking levels — last healthy concurrent readers/writers before SLA breach
- Mock ≠ hosted SaaS — do not compare mock embed timings to managed GPU indexes
- Storage (mock) ≠ LIVE — never mix the two suites in one comparison cell
Related pages
- Performance
- Architecture
- SQLite · PostgreSQL
- Live page: /benchmarks
- Raw: /benchmarks/benchmark.json · /benchmarks/benchmark.md