Two-stage contradiction detection: why one embedding threshold isn't enough for agent memory

The problem

skull's long-term memory stores persona facts about the user across sessions, searchable by embedding similarity. The obvious next feature is: when a new fact contradicts an old one — the user said they prefer terse answers, and later says they prefer detailed explanations — the old fact should stop being surfaced, without literally deleting it (it stays visible in history, just excluded from future recall).

The naive implementation is one cosine-similarity threshold: if the new fact is similar enough to an old one, treat it as an update. That doesn't work, and the reason is measurable, not theoretical.

What the real numbers actually looked like

Testing against realistic phrasing — not hand-matched sentence pairs designed to make the threshold look good — genuinely contradicting fact pairs scored embedding similarity anywhere from 0.37 to 0.87, depending on how much vocabulary the two sentences happened to share. Unrelated pairs scored 0.02 to 0.1. That gap looks clean until you notice the contradicting range is enormous — a threshold set low enough to catch a 0.37 contradiction will also catch plenty of merely-related-but-compatible facts sitting in that same range.

It gets worse for deletion specifically. A separate feature, forget_fuzzy, needs to match a fact the model paraphrased instead of quoting exactly. Testing that path surfaced a genuinely uncomfortable result: "User loves dark chocolate" vs. "User loves milk chocolate" scored a higher similarity (0.817) than several genuine paraphrases of unrelated facts (0.69–0.87). A wrong-preference fact and a real paraphrase of something else entirely both land in overlapping territory. No single threshold, at any value, is safe to act on alone — especially for something destructive like a delete.

Stage 1: a deliberately loose embedding pre-filter

Given that, the embedding check's job changes. Instead of making the actual contradiction decision, it exists only to cheaply skip facts that are obviously unrelated:

SUPERSEDE_CANDIDATE_MIN_SCORE = 0.35

candidates = store.search(fact, k=3, min_score=SUPERSEDE_CANDIDATE_MIN_SCORE)

At 0.35, this catches essentially every real contradiction pair from testing (which started around 0.37) while still filtering out the bulk of genuinely unrelated facts sitting near 0.02–0.1. It's intentionally loose — false positives here just mean one extra cheap LLM call in stage 2, which is a much cheaper mistake than a false negative that silently makes the whole feature not fire on real phrasing (the actual failure mode hit during development, before the threshold was loosened).

Stage 2: the LLM makes the real call

Every candidate that passes the loose filter goes to one small, non-streaming LLM call that answers a single yes/no question:

def _confirm_supersedes(old_fact: str, new_fact: str) -> bool:
    prompt = (
        "Two facts about the same user:\n"
        f'A (existing): "{old_fact}"\n'
        f'B (new): "{new_fact}"\n\n'
        "Does B supersede or contradict A - i.e. is B an update/correction to "
        "the same specific thing A states, making A no longer current? "
        "Answer with exactly one word: yes or no."
    )
    try:
        resp = requests.post(..., json={"max_tokens": 5, "stream": False, ...})
        answer = resp.json()["choices"][0]["message"]["content"].strip().lower()
        return answer.startswith("yes")
    except Exception:
        return False

This is where "prefers terse answers" vs. "prefers detailed explanations" actually gets distinguished from "likes terse answers" vs. "likes concise code" — a distinction cosine similarity alone genuinely cannot make, since both pairs are about the same general topic and score similarly. Note the fallback: any exception — a timeout, a malformed response — defaults to False, not supersede. A missed supersede just leaves one extra fact in memory a little longer; a wrongly-triggered one would bury a fact that's still true.

The fuzzy-delete path needed its own question, not a reused one

forget_fuzzy looked at first like it could reuse _confirm_supersedes — both are "is this the same thing, essentially" checks. It couldn't. Supersede asks does B update/contradict A; a safe delete needs to ask is this the SAME fact, just reworded — a materially different question, especially once "dark chocolate" vs. "milk chocolate" scoring 0.817 made clear that two different-but-similar-shaped facts can look more alike than two real paraphrases of the same fact:

def _confirm_same_fact(candidate_fact: str, requested_fact: str) -> bool:
    prompt = (
        ...
        "Is \"Stored\" the SAME underlying fact as \"Requested\", just worded "
        "differently - not merely a related fact on a similar topic? Answer "
        "with exactly one word: yes or no."
    )

Same two-stage shape, same conservative default-to-False on failure — but a distinct confirmation prompt, because "did this change" and "is this the same thing" are not the same question, and conflating them is exactly how a fuzzy delete ends up removing the wrong fact.

The general shape of the fix

Neither stage alone is sufficient, and that's the actual lesson: a cheap signal (embedding similarity) narrows the field so an expensive signal (an LLM call) only has to run on plausible candidates, and the expensive signal is the one actually trusted to make an consequential decision. The cheap stage's threshold doesn't need to be precise — it needs to be loose enough to never miss a real case, because the expensive stage is what corrects for its false positives. Tuning a single threshold to try to do both jobs at once is where this kind of system tends to quietly fail on exactly the phrasing that wasn't in the test set.