Opt-in write-time exact and near-duplicate detection in Wolbarg 0.4 — update existing facts instead of inserting duplicates, with RememberResult.action and update().

What is it?

Wolbarg 0.4 adds opt-in write-time deduplication. When enabled, remember() can update an existing active memory instead of always inserting a new UUID.

Without dedupe, every remember() inserts a new row. Agents that restate facts create duplicates, inflate compress() cost, and noise recall — even when MMR hides near-duplicates at search time.

Use dedupe for facts, preferences, and durable state. Keep append-only (default) for episodic logs and time-series observations.

Compatibility

Behavior	Default
Dedupe	Off (0.3-compatible append-only)
`remember()` return	`RememberResult` = `MemoryRecord` + `action` (additive field)
History	New event type `"updated"` when an upsert replaces content/metadata
Schema	`content_hash` column + unique active hash index (auto-migrated)

Upgrading from 0.3.x requires no code changes unless you opt into dedupe or read action.

Enabling

Constructor

import { wolbarg, openaiEmbedding } from "wolbarg";

const ctx = wolbarg({
  organization: "my-org",
  database: { provider: "sqlite", url: "./memory.db" },
  embedding: openaiEmbedding({
    apiKey: process.env.OPENAI_API_KEY!,
    model: "text-embedding-3-small",
  }),
  memory: {
    dedupe: {
      enabled: true,
      strategy: "exact-or-near", // default when enabled
      nearThreshold: 0.92,       // cosine similarity
      nearCandidateLimit: 8,
    },
  },
});

const result = await ctx.remember({
  agent: "assistant",
  content: { text: "User prefers dark mode" },
  metadata: { source: "chat" },
});
// result.action === "created" | "updated"

Per-call override

// Force on for one write
await ctx.remember({
  agent: "assistant",
  content: { text: "User prefers dark mode" },
  dedupe: true,
});

// Exact-only for one write
await ctx.remember({
  agent: "assistant",
  content: { text: "Timezone is IST" },
  dedupe: { strategy: "exact" },
});

// Force off even if constructor enabled
await ctx.remember({
  agent: "logger",
  content: { text: "Heartbeat at T+60s" },
  dedupe: false,
});

Config reference

Field	Default	Meaning
`enabled`	`false`	Master switch
`strategy`	`"exact-or-near"` when enabled	`"exact"` \| `"near"` \| `"exact-or-near"`
`nearThreshold`	`0.92`	Minimum cosine similarity for near match
`nearCandidateLimit`	`8`	Max ANN candidates considered for near-dup

Detection pipeline

Exact — normalize content (NFC, trim, collapse whitespace; case preserved), hash, look up unique active hash for the same organization + agent.
Near (when strategy includes near) — embed the new text, compare cosine similarity against up to nearCandidateLimit active memories for that agent; accept if ≥ nearThreshold.
No match — insert a new row (action: "created").

Match scope

Included	Excluded
Same `organization`	Other orgs
Same `agent`	Other agents
Non-archived active rows	Archived rows after `compress()`

Archived memories are never match targets — compress must stay free to summarize history without resurrecting dead hashes as “live” facts.

Update behavior

Case	Behavior
Exact match	Merge metadata (shallow), bump `updated_at`, keep existing embedding
Near match	Replace content, re-embed, merge metadata
No match	Insert new row

Additional side effects:

History records an "updated" event on upsert.
subscribe() emits "update" (not "remember").
Concurrent exact-dedupe races stay unique (v4 suite: 1 id, 11 updates under contention).

RememberResult

interface RememberResult extends MemoryRecord {
  action: "created" | "updated";
}

rememberBatch() returns RememberResult[] — each item carries its own action.

Explicit update()

When you already know the memory id:

const result = await ctx.update({
  id: memoryId,
  content: { text: "Revised fact" },
  metadata: { edited: true },
});
// result.action === "updated"

Use update() for user edits and admin tools; use dedupe for automatic fact convergence on write.

When to leave dedupe off

Audit / event logs where each utterance is a distinct fact
Time-series observations (“temp was 72°F at 10:01”)
Workflows that must never merge paraphrases
Debug sessions where you want append-only archaeology

Worked example — preference convergence

const a = await ctx.remember({
  agent: "assistant",
  content: { text: "User prefers dark mode" },
  metadata: { source: "onboarding" },
  dedupe: true,
});
// a.action === "created"

const b = await ctx.remember({
  agent: "assistant",
  content: { text: "User prefers dark mode" },
  metadata: { source: "settings", confirmed: true },
  dedupe: true,
});
// b.action === "updated"
// b.id === a.id
// metadata merged: { source: "settings", confirmed: true }

Memory upsert & deduplication