PII scrubbing - xysq documentation

Say you’re pushing support tickets into a vault, and those tickets have names, emails, order numbers, all the stuff a real conversation carries. You want the vault to remember the problem (a firmware bug, a refund, a login issue) without remembering who reported it. That’s what PII scrubbing does: it rewrites identifying details out of the content before the first byte is stored.

Why ingest-time

Every layer downstream of a push inherits whatever the push contained: the wiki pages distilled from it, the facts pulled out of it, the embeddings computed over it. And the raw log itself is immutable (it’s the audit trail) so nothing written to it can be edited out later. That leaves one place to catch PII: the door. Scrub once, before the first durable write, and every derived layer is clean by construction.

The vault toggle

pii_scrub is a vault setting, off by default. Turn it on at create:

from xysq import Xysq

with Xysq() as client:
    vault = client.vaults.create("customers", project_id=project_id, pii_scrub=True)

or later in the app, under the vault’s Settings. Flipping it on only affects pushes from that point forward: existing content isn’t rewritten, because (as above) the log is immutable. The Settings panel says this too, so nobody expects a flip to launder history.

Pushing with `pii` and `known_pii`

A push can override the vault default with pii=True, and it can hand over known_pii: the identifiers you already know because they came from your own system of record (a helpdesk, a CRM). This is close to free for most integrations and it makes the scrub deterministic instead of best-effort.

result = client.vaults.push(
    vault.vault_id,
    "user: Alice Chen (alice@example.com) reported that ticket #A-1042, "
    "her Widget Pro keeps flickering on startup.\n"
    "agent: escalated to firmware, will follow up.",
    known_pii=["Alice Chen", "alice@example.com", "#A-1042"],
)

What gets stored is neutral:

user: The customer reported that ticket <redacted>, their Widget Pro
keeps flickering on startup.
agent: escalated to firmware, will follow up.

Sent, never stored: "Alice Chen", "alice@example.com", "#A-1042". known_pii is transient input to the scrub, not a field on the source; it never lands in the log. Every sensitive match, deterministic or known_pii, collapses to the same literal token: <redacted>. There’s no per-kind phrasing (no [email] vs [phone]) because a typed label is itself a hint about what was there. Prose identity is different: the model pass rewrites a name into a gender-neutral noun like “the customer” or “they”, not into <redacted>, because the sentence still needs to read. If the scrub can’t produce a clean result, the push fails closed: a 422 with the message "content could not be scrubbed of personal data; nothing was stored", and nothing is stored (not even a pending row). A partially-scrubbed push is worse than a rejected one, so xysq doesn’t do partial.

How it works

Three stages run inside the push, before the first durable write:

Deterministic scrubbers. Pattern-based detectors catch the list below and every known_pii string (case, whitespace, and punctuation don’t matter). Replaced with <redacted>.
Model neutralization. One model call rewrites remaining prose identity into neutral voice: “the customer”, “they”, never a gendered pronoun (gender is an inference, not a fact, and we don’t ink inferences into the log).
Deterministic gate. The same detectors from step one re-run against the model’s output. If anything survives, the model pass retries once; if it still fails, the push is rejected. The model rewrites, the gate guarantees.

Titles go through the same deterministic pass and gate (no model call - a title is a handful of words, not worth the round trip). A title the deterministic pass can’t fully clean rejects the push, same as content.

The deterministic detector list

This is the actual guarantee. It catches exactly this, no more:

Emails
Phone numbers with 10 or more digits
Card numbers that pass a Luhn check
IPv4 and IPv6 addresses
IBANs, mod-97 validated (not just IBAN-shaped)
Every known_pii string you pass, matched with case, whitespace, and punctuation normalized so "Alice Chen" also catches "alice chen"

Anything on this list, or anything you named in known_pii, is deterministically absent from what’s stored - the gate checks every push, so a survivor fails the push rather than shipping quietly. Everything else (a name you never passed in, a physical address, a nickname) goes through the model pass only: best-effort, not gate-verified.

What’s actually guaranteed

Be precise about what “scrubbed” means here, because not every kind of identity is removable the same way:

Category	Guarantee
The deterministic list above, and everything in `known_pii`	Deterministically absent. The gate verifies this on every push; a survivor fails the push, not the guarantee.
Unknown prose identity (a name you never passed in `known_pii`)	Best-effort. The model pass catches most of it, but it’s probabilistic, not gate-verified, since the gate only checks for what it can detect.
Quasi-identifiers (“the only customer in Iceland with two units”)	Not removable. No scrub can strip context that’s identifying by elimination rather than by naming. We’re not claiming otherwise.

That middle row is why known_pii matters: anything you can name, the gate can guarantee. Anything you don’t name is left to the model’s best judgment.

Things worth knowing

Thread turns are raw until flush. client.threads.append writes to a working buffer verbatim; scrubbing runs when a flush promotes those turns to long-term memory, not when each turn lands in the buffer. See Thread-level memory.
A writer key can opt out per push. pii=False on a single push bypasses scrubbing even on a pii_scrub vault. Read the promise as “masked unless a writer explicitly opts out”, not “impossible to store raw”.
IPv4-mapped IPv6 is a known edge. Forms like ::ffff:1.2.3.4 may be only partially masked. Don’t rely on this shape being fully caught.
Re-pushing identical content doesn’t dedupe on a pii_scrub vault. Dedup keys off the stored (scrubbed) hash, and the model rewrite isn’t deterministic, so two pushes of the same raw text can scrub to two different strings and land as two sources.
Binary uploads are rejected on a pii_scrub vault. There’s no text to scrub before storage, so a file push comes back 422. Push text content instead.

Best practices

Pass known_pii from your system of record. If your helpdesk knows the customer’s name and email, hand them over. It’s the difference between “guaranteed absent” and “best-effort”.
Use opaque customer ids in metadata, not emails. A customer_id metadata value that’s an internal id, not alice@example.com, keeps the metadata layer as identity-free as the content layer.
Keep identity in metadata, semantics in content. The content should read like it’s about anyone; anything that has to tie sources back to a specific customer belongs in a declared metadata key, not in prose.

Works with filters

A pii_scrub vault still supports metadata filters and tags as usual. Scrubbing changes what the content says, not how you search it.

​Why ingest-time

​The vault toggle

​Pushing with pii and known_pii

​How it works

​The deterministic detector list

​What’s actually guaranteed

​Things worth knowing

​Best practices

​Works with filters