boromat/polanon
boromat/polanon has 29 GitHub stars and 9 forks. GitHot measured +3 stars in 24 hours. Last checked 2026-11-15 20:15 UTC. Last push 2026-11-14.
Measured star growth
- in 24h
- +3
- in 7 days
- Update delayed
- in 15 days
- Update delayed
- in 1 month
- Update delayed
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
Reversible PII masking for LLM calls and RAG, Polish-first: checksum-validated PESEL, NIP and IBAN, Polish names in every case, stable tokens, streaming restore and Haystack components.
AIREADME.md
Originalpolanon
English | Polski
Reversible PII masking for LLM calls, Polish-first. Mask personal data before a prompt leaves your network, send only tokens to the model, and put the originals back into its answer — even while it streams. Pure Python: the standard library is the whole core.

pip install polanon
from polanon import Anonymizer, default_guards
anonymizer = Anonymizer(default_guards(with_ner=False))
result = anonymizer.anonymize_sync(
"Proszę o korektę faktury dla Jana Kowalskiego, PESEL 50810112354, "
"konto PL61109010140000071219812874, mail [email protected]."
)
print(result.anonymized)
# Proszę o korektę faktury dla <OSOBA_1>, PESEL <PESEL_1>, konto <IBAN_1>, mail <EMAIL_1>.
answer = "Korekta dla <OSOBA_1> (PESEL <PESEL_1>) jest gotowa." # what the model replied
print(Anonymizer.restore(answer, result.mapping))
# Korekta dla Jana Kowalskiego (PESEL 50810112354) jest gotowa.
The model only ever sees <OSOBA_1>. The mapping that turns it back into a name never leaves your
process. anonymize_sync works in scripts, Django and Flask views, Celery tasks and notebooks; in
async code, await anonymizer.anonymize(...) does the same work in a worker thread.
Why another PII library
General-purpose tools are built around English. Polish data needs three things they do not do out of the box:
- Checksums. A PESEL, NIP, REGON, ID card number, IBAN or payment card is validated, not just pattern-matched: the checksum rejects most random digit strings — for a PESEL, together with the birth date, about 98% of them — while a valid number always passes. KRS numbers, which have no public checksum, and passports are recognised next to their label as well. Eleven digits right after the word PESEL are masked even with a typo in them.
- Seven grammatical cases. "Jan Kowalski", "Jana Kowalskiego" and "z Janem Kowalskim" are the same person. Polish first names are recognised in every case, in lowercase and typed without Polish letters, together with the surname that agrees with them, without any language model.
- Polish model labels. Polish spaCy models tag people as
persName, notPERSON. A pipeline configured for English labels finds no names at all, and says nothing about it.
It also reads foreign invoices: EU, UK and Swiss VAT numbers, company register numbers, Bankgiro, Plusgiro and sort codes, international phone numbers, street addresses and postcodes.
How it compares
| polanon | Presidio | pii-veil | piimask | |
|---|---|---|---|---|
| Restores the model's answer | Yes, from stable tokens such as <OSOBA_1>, also while it streams |
Not built in: its deanonymizer decrypts spans at known positions; its docs include a sample that restores an LLM answer | Yes, from stable tokens and a saved mapping | Yes, from stable tokens |
| Polish identifiers with a checksum | PESEL, NIP, REGON, ID card, IBAN, payment cards | PESEL; IBAN and payment cards in general | PESEL, NIP, REGON, IBAN, payment cards; ID card and passport by pattern only | None; IBAN and payment cards in general |
| Polish names in every grammatical case | For listed first names and the surnames that agree with them, without a language model; spaCy adds the rest | With a Polish spaCy model and its persName label mapped |
Not detected | Not detected; its README suggests adding an NER model |
| What the core needs | Nothing beyond the standard library | spaCy, numpy, pydantic and a few more; a language model only for names and places | pii-core and typer | Nothing beyond the standard library |
| On polanon's sample documents | 98% fully masked with pl_core_news_md, 86% with rules alone |
66% with the same model (presidio-analyzer 2.2.366) | Not measured | Not measured |
| License | MIT | MIT | Apache-2.0 | MIT |
Compared from each project's own documentation and source on 3 November 2026. Only Presidio was
run on the sample documents, with the script in benchmarks/, and that is not neutral ground: the
eleven synthetic documents were written by polanon's author, who tuned its rules on them, and
Presidio had only the Polish model, also for the three invoices from abroad. Each has its place:
Presidio for many languages, structured data and its Azure integration; pii-veil for Polish
identifiers under Apache-2.0 when names do not matter; piimask for language-neutral identifiers
(e-mail, phone numbers, payment cards, IBAN, IP addresses, URLs) and US social security numbers,
with fake-value, hashing and partial strategies and an encrypted vault. polanon is for Polish text
on its way to a model, where the names in it
and the restored answer matter as much as the numbers.
Install
| You want | Install |
|---|---|
| Identifiers, contacts, secrets, Polish names, companies by legal form | pip install polanon |
| Also places, and names and organisations no rule can see | pip install "polanon[ner]" then python -m spacy download pl_core_news_md |
| The stage-2 judges (a local Ollama model, or a remote one) | pip install "polanon[judge]" |
| Components for Haystack 3 pipelines | pip install "polanon[haystack]" |
| A gateway for tools that use the OpenAI API | pip install "polanon[gateway]" |
Then use default_guards() with the model, or default_guards(with_ner=False) without it.
Another spaCy model is one argument away: default_guards(spacy_model="pl_core_news_lg").
Package names copied from pip work too (pl-core-news-lg). A local model directory can be passed
as a path; it must contain meta.json and config.cfg. Missing packages or model directories are
reported when the guards are built, while the model itself loads on first use.
An installed model package takes priority over a local path with the same name.
From the command line
Text comes from the arguments, from a file with --file, or from standard input. Masked text goes
to stdout and the mapping to stderr, or to a file with --mapping:
polanon --version # print the installed version
polanon "Jan Kowalski, PESEL 50810112354"
polanon --file umowa.txt --ner --mapping mapping.json > masked.txt
cat umowa.txt | polanon --ner > masked.txt 2> mapping.json
A non-zero exit status means something went wrong and stderr says what. --file reads the file's
bytes itself and tells UTF-8, UTF-16 (what Windows PowerShell 5.1 writes with >) and cp1250
apart; text it cannot decode for certain is refused rather than masked half-blind.
On Windows, prefer --file and --mapping to pipes and redirection. Windows PowerShell 5.1
re-encodes text piped into a program as ASCII, so Polish letters arrive as ? and a garbled name
is not recognised, and its 2> wraps stderr in error records. If you do pipe, run
$OutputEncoding = [System.Text.UTF8Encoding]::new($false) first; in cmd.exe, run chcp 65001.
Streaming
A provider streams arbitrary fragments, so <OSOBA_1> can arrive as <OSO and BA_1>. The
restorer holds back anything that could still grow into a token:
restorer = Anonymizer.stream_restorer(result.mapping)
for chunk in stream: # "Dzień dobry <OSO", "BA_1>, PESEL <PESEL", "_1> zweryfikowany."
print(restorer.feed(chunk), end="")
print(restorer.flush())
# Dzień dobry Jana Kowalskiego, PESEL 50810112354 zweryfikowany.
With Claude, GPT or a local model
Mask before the call, restore while the answer streams. With the Anthropic SDK:
import anthropic
from polanon import Anonymizer, default_guards
anonymizer = Anonymizer(default_guards(with_ner=False))
masked = anonymizer.anonymize_sync(question)
restorer = Anonymizer.stream_restorer(masked.mapping)
with anthropic.Anthropic().messages.stream(
model="claude-opus-5-5",
max_tokens=64000,
system="Personal data is replaced by tokens such as <OSOBA_1>. Copy a token exactly as written.",
messages=[{"role": "user", "content": masked.anonymized}],
) as stream:
for text in stream.text_stream:
print(restorer.feed(text), end="")
print(restorer.flush())
Telling the model what the tokens are keeps it from rewriting them. The
examples go further:
openai_chat.py does the same for any OpenAI-compatible endpoint (OpenAI, OpenRouter, Azure, vLLM,
a local Ollama), and anthropic_chat.py holds a conversation whose history keeps masked text only,
so an answer restored for the reader is never sent back in the clear. Both are tested in CI against
the real SDKs, offline.
A chat turn with tools
A turn of a chat with tools sends more than the question: the history, and the result of every tool
the model calls. anonymizer.turn() masks all of it with one mapping, puts the originals back into
the tool arguments and the answer, and refuses instead of sending anything it could not mask:
import json
import anthropic
from polanon import Anonymizer, MaskingUnavailableError, default_guards, token_instructions
anonymizer = Anonymizer(default_guards(with_ner=False))
client = anthropic.Anthropic()
def chat(history: list[dict], clients: list[str]) -> None:
"""One turn. `history` ends with the question; SYSTEM, TOOLS and run_tool are your own."""
try:
turn = anonymizer.turn(known=clients, unknown_blocks="drop")
messages = turn.mask_messages(history)
while True:
restorer = turn.stream_restorer()
with client.messages.stream(
model="claude-opus-5-5",
max_tokens=64000,
system=SYSTEM + "\n\n" + token_instructions("en"),
tools=TOOLS,
messages=messages,
) as stream:
for text in stream.text_stream:
print(restorer.feed(text), end="")
response = stream.get_final_message()
print(restorer.flush())
messages.append({"role": "assistant", "content": response.content})
if response.stop_reason != "tool_use":
return
results = []
for block in response.content:
if block.type == "tool_use":
result = run_tool(block.name, turn.restore_data(block.input))
masked = json.dumps(turn.mask_data(result, default=str), ensure_ascii=False)
results.append(
{"type": "tool_result", "tool_use_id": block.id, "content": masked}
)
messages.append({"role": "user", "content": results})
except MaskingUnavailableError:
print("Masking is on but not working, so nothing more was sent to the model.")
turn.stream_restorer() puts the originals back as each response streams in, also a token cut
between two events, as in Streaming. The tests run this example as written here
against the real anthropic SDK, offline, so it cannot drift from the library.
A turn or the lower API. Take a turn whenever a model gets a list of messages, with tools or
without: it keeps one mapping for all of them, masks a value found in a later message in the
earlier ones too, and refuses on any failure, all of which you would otherwise write yourself. Call
anonymize_sync(), Anonymizer.restore() and StreamRestorer directly for a single text, such as
a document or a prompt you build yourself, when you need the spans of the result, or when the
stage-2 judge must run, which a turn does not take.
One mapping for the turn. A client in the history, in the question and in a tool result is the
same <FIRMA_1>. To keep the tokens from one turn to the next, start the next one with
anonymizer.turn(mapping=turn.mapping).
What mask_messages masks. Messages in the Anthropic or the OpenAI format: their text, the
input of tool_use, tool results and the arguments of tool_calls, with JSON kept valid and ids
and tool names left alone. A value recognised only in the question is masked where it stood earlier
in the history, unrecognised. A block nothing can mask, such as an image or a model's signed
reasoning, is refused, or left out with unknown_blocks="drop". mask_data masks the strings in a
value; keys and numbers stay as they are, so keep personal data in string values, and an object
such as a date needs default=str.
Values you already know. known= takes what you know is sensitive, such as your clients' names
from the database. They are masked wherever they appear, also when they are on the allowlist.
Refusing instead of leaking. Whatever fails while masking raises MaskingUnavailableError, and
nothing half masked comes back. default_guards() raises NerModelUnavailableError, which is one,
when spaCy or the model is missing, and ner_available() tells a settings page whether they are
installed. Once the answer is in, turn.unknown_tokens(answer) lists the tokens the model made up,
and turn.sweep(restored) also looks for personal data the answer added.
Masking for RAG and search indexes
Tokens numbered per call mean nothing outside their conversation: a contract masked for a search
index and a question masked a week later can each call a different person <OSOBA_1>, and the
search finds nothing. Give the anonymizer a secret key and every token is computed instead, from
HMAC-SHA256 of the value under the key: the same in every document, every process and every
question, with no shared state. "Jan Kowalski", "Jana Kowalskiego" and "z Janem Kowalskim" get one
token, and so does a NIP with its dashes or without them.
import os
from polanon import Anonymizer, InMemoryMappingStore, Pseudonyms, default_guards, stored_mapping
key = bytes.fromhex(os.environ["POLANON_KEY"]) # 32 random bytes per company, from a secret store
store = InMemoryMappingStore() # in production, an encrypted table of yours, apart from the index
pseudonyms = Pseudonyms(key, collisions=store.collisions)
anonymizer = Anonymizer(default_guards(with_ner=False), pseudonyms=pseudonyms)
document = anonymizer.index_chunks(
[
"Umowę z Firmą Kot Sp. z o.o. podpisał Jan Kowalski.",
"Pełnomocnikiem Jana Kowalskiego jest Anna Nowak, tel. 600 100 200.",
],
store,
)
for text in document.texts: # tokens only: these go to the embedder and the index
index.add(text, embed(text)) # index and embed are your own
question = anonymizer.mask_query("Kto jest pełnomocnikiem Jana Kowalskiego?", store)
passages = index.search(embed(question.anonymized)) # the question has the passages' tokens
mapping = question.mapping | stored_mapping(store, question.anonymized, *passages)
restorer = Anonymizer.stream_restorer(mapping) # restores the answer, as in Streaming
The tests run this example as written here, with a stand-in index and embedder. Each part:
Nothing in the index names a value. document.texts holds tokens where the values were; what
each token stands for goes to the store. Never copy a mapping into a passage's metadata, which is
stored and searched with it. document.mappings[i] restores passage i exactly as it was written,
"Jana Kowalskiego" in the genitive; keep it with the passage, encrypted, outside the index.
A store of your own. MappingStore has two methods: get(tokens) and add(entries), which
stores only the tokens it does not hold yet, atomically per token, as a unique key on the token
does. It returns what it already holds for the others. Encrypt StoredValue.value, which is the
personal data; its fingerprint names nothing without the key. Keep store.collisions next to it
and pass it to every Pseudonyms: it holds tokens and fingerprints, nothing else.
Which form a name comes back in. One token stands for "Jan Kowalski" and "Jana Kowalskiego". The store keeps the nominative, because the model writes the token in a sentence of its own, and "Umowę podpisał Jana Kowalskiego" would be wrong whichever document came first. A passage restored with its own mapping keeps the case it was written in.
Names the store already holds. A surname on its own, as in "Co podpisał Kowalczyk?", is a name
no rule reads. index_chunks stores the surname of every person whose first name the rules know,
next to the full name, and mask_query and index_chunks look up every run of up to four words in
the store and mask the ones it holds a token for. A question then gets the surname's token, and a
surname masked in one document stays masked in the next. Pass lookup=False for a store too slow
for that.
The key. At least 32 random bytes, one key per company and a scope= per index that must not be
linked to another. Keep it in a secret manager, out of the index, the code and the logs;
Pseudonyms never shows, pickles or copies it. A new key, scope or number of digits changes every
token, so the index has to be built again: store pseudonyms.settings with the index and compare
them at startup. To erase a person, delete the passages holding pseudonyms.token(name, "PERSON"),
and those holding the surname's token where nobody else with that surname is indexed. Deleting the
store's entries alone does not unlink them, since the key turns a guess back into the token.
Collisions. Twelve digits make two values of one type share a token with a chance of about 1 in 20,000 at 10,000 values, and 1 in 200 at 100,000. Wherever two such values meet, in the store or in one text, the one that came second gets four digits more.
Limits. A name the rules cannot decline, such as a rare first name or "Krawca" from "Krawiec", and a company name in another case, "Firmą Kot" beside "Firma Kot", keep a token per form, which costs a missed search but leaks nothing. A word alone in lowercase is never looked up in the store. And one token everywhere is the point and the price: whoever reads many masked passages can tell which of them are about the same person. The threat model says what that means.
With Haystack
pip install "polanon[haystack]" adds components that put the masking into a
Haystack 3 pipeline. The key is read from POLANON_KEY, 32
random bytes in hexadecimal:
from haystack import Pipeline
from haystack.components.preprocessors import DocumentSplitter
from haystack.components.writers import DocumentWriter
from polanon import InMemoryMappingStore
from polanon.integrations.haystack import (
PolanonDocumentMasker,
PolanonQueryMasker,
PolanonRestorer,
PolanonStreamRestorer,
)
mappings = InMemoryMappingStore() # in production, an encrypted table of yours
indexing = Pipeline()
indexing.add_component("mask", PolanonDocumentMasker(store=mappings))
indexing.add_component("split", DocumentSplitter(split_by="period"))
indexing.add_component("embed", document_embedder) # any embedder: it gets masked text
indexing.add_component("write", DocumentWriter(document_store))
indexing.connect("mask", "split")
indexing.connect("split", "embed")
indexing.connect("embed", "write")
asking = Pipeline() # with an embedder, a retriever, a prompt builder and a generator of yours
asking.add_component("mask", PolanonQueryMasker(store=mappings)) # first
asking.add_component("restore", PolanonRestorer(store=mappings)) # after the generator
asking.connect("mask.query", "embed.text")
asking.connect("mask.query", "prompt.query")
asking.connect("retrieve.documents", "restore.documents")
asking.connect("llm.replies", "restore.replies")
asking.connect("mask.mapping", "restore.mapping")
stream = PolanonStreamRestorer(mappings, on_text=lambda text: print(text, end=""))
asking.run({"mask": {"query": question, "stream": stream}, "llm": {"streaming_callback": stream}})
examples/haystack_rag.py
is the whole of it, offline, with Haystack's stand-in embedder and generator, and the tests run it.
examples/haystack_openai.py
is the same with OpenAI's embedder and model: it is paid, so the tests only build it.
Mask first. The document masker comes before the splitter, so a whole document is read and a
company or an address is masked where it stands: cut at full stops first, "Firmą Kot Sp. z o.o."
fell apart and went out unmasked. Passages that come split already are masked together when they
name one source in source_id, parent_id or __parent_id. Split by word, sentence, period or
passage: a splitter that counts characters cuts tokens in two, and a half matches nothing.
Nothing in the document store names a value. The masked documents are new ones: tokens in the
content and in every string of meta, nested keys and the members of sets included. A document's
id is a plain SHA-256 of its text, so every string of meta shaped like one, which is what
splitters write there, points to the masked twin of its document or becomes a keyed fingerprint.
An id Haystack computed is computed again from the masked document; one you chose is masked like
text and kept, so upserts and deleting by id still work. What the tokens stand for goes to the
mapping store only, never into meta. A document with
a blob or an embedding is refused, since both were made from the original. The tests check every
document in the store, and the file it saves, for each original value.
The answer, for the reader. PolanonQueryMasker sends the masked question to the embedder and
the prompt builder, and its mapping to PolanonRestorer, which restores the replies and the
documents found. Its documents are for display only: never write them to a store or send them to a
model. PolanonStreamRestorer restores a streamed answer as the generator's streaming_callback;
with Pipeline.stream(), feed it the chunks instead: stream.feed(chunk.content). Values you know
are sensitive, such as client names, go in at run time, {"mask": {"known": clients}}, so they
never end up in a saved pipeline.
Saving a pipeline. The key is a Haystack Secret, and a saved pipeline holds the name of its
variable only. A mapping store is saved through its own to_dict, which should say where it
connects and never what it holds; InMemoryMappingStore has none, so a pipeline with it is built
in code. Haystack 3 loads classes from its own modules only, so allow this one and your store's:
Pipeline.loads(text, allowed_modules=["polanon.integrations.haystack", "myapp.vault"]).
Keep content tracing off. With HAYSTACK_CONTENT_TRACING_ENABLED, Haystack records what every
component receives, and the masker receives the originals. A run that fails keeps its inputs in
the error's snapshot, written to disk with HAYSTACK_PIPELINE_SNAPSHOT_SAVE_ENABLED. Both are off
by default.
A gateway for any OpenAI-compatible tool
Tools such as Hermes Agent, n8n, Open WebUI or LibreChat talk to a model through the OpenAI API.
polanon-gateway stands between such a tool and the provider, on the machine of whoever runs it:
every message is masked before it leaves, and the answer comes back with the original values,
streamed or not, tool calls included.
pip install "polanon[gateway,ner]"
python -m spacy download pl_core_news_md
polanon-gateway --upstream https://api.openai.com/v1
Then point the tool's base URL at http://127.0.0.1:8787/v1 and keep its API key: the gateway
passes it on to the provider. A local model works the same way, --upstream http://localhost:11434/v1 for Ollama.
| Tool | Where the base URL goes |
|---|---|
| Hermes Agent | base_url of the model or of a named provider in config.yaml |
| n8n | the Base URL of the OpenAI credential |
| Open WebUI | the URL of an OpenAI API connection |
| LibreChat | the baseURL of a custom endpoint in librechat.yaml |
What it does with a call:
- It serves
/v1/chat/completionsand the Responses API,/v1/responses, streamed or not,/v1/embeddingsand/v1/models, and refuses every other endpoint, so nothing it cannot mask leaves as it came. - With the Responses API the whole conversation comes with every call, as Hermes Agent sends it
with
store: false. A call that leaves it at the provider, withprevious_response_idorconversation, or asks for abackgroundresponse to fetch later, would bring back tokens no one can restore, and is refused. Encrypted reasoning passes as it is: the provider sealed it from masked text. - Every message, tool result and earlier tool call is masked, and so is the prose of the tools and
of an answer format: descriptions, titles and example values. Tool names, types and parameter
names stay, so the tools keep working. A participant's
nameis left out. - An image, a file or audio cannot be masked, so the call is refused;
--drop-unsupportedleaves such content out and sends the rest. - Before anything leaves, the payload is searched once more for every original value, and the call is refused if one is still there.
- A note telling the model what the tokens are joins the system message:
--instructions en, ornoneto leave it out. - It keeps nothing on disk. A tool sends the whole conversation with every call, and each call is
masked and restored on its own. With
POLANON_KEYset to 64 hexadecimal characters, a person keeps one token in every call and in every grammatical case.
Keys stay with whoever runs the gateway, and come from the environment only. By default the
tool's own key is passed on; POLANON_UPSTREAM_KEY sends another one instead. The gateway listens
on 127.0.0.1. To reach it from another machine, set --host and POLANON_GATEWAY_KEY, which
every client must then send: without the key it refuses to start. The log names the path, the
status, the time and how many values of each type were masked, never a value.
There is no hosted service: the gateway runs where it is installed, and whoever runs it answers for that machine, its keys and the provider the calls go to. Not supported yet: Anthropic's Messages API. The threat model says what the gateway protects and what it does not.
What it detects
Checksum-validated Polish identifiers (PESEL, NIP, REGON, ID card, IBAN, payment cards), KRS numbers
next to their label, e-mail addresses and phone numbers, street addresses and postcodes, links to
a person's profile (LinkedIn, GitHub, GitHub Pages, GitLab, social media) up to the handle, IPv4
and IPv6, MAC addresses, vehicle registrations after their label ("nr rej."), VINs, passport
numbers, birth dates, API keys, tokens and passwords, Polish first names with their surnames, and
company names anchored on a legal form ("Sp. z o.o.", "S.A.", "Ltd", "GmbH"). Once a person or a
profile link is masked, the surname or the handle alone is masked wherever it appears in the same
text: "Kowalski" after "Jan Kowalski", "jkowalski" in a JSON field after "github.com/jkowalski".
With the ner extra, a spaCy model adds places and the names nothing else marks. An optional
second stage lets a small local model review the masked text for what still identifies someone
by context. Other dates are recognised too, but left readable unless you ask for them.
Text copied from PDFs, Word or macOS is read as a person would read it: letters written as a base letter plus an accent, no-break spaces and en dashes inside numbers, and invisible zero-width characters do not hide a value. The mapping keeps each value as it was first written.
On the annotated sample documents in the repository, the rules alone fully mask 86% of the
values and the rules with pl_core_news_md mask 98%, with nothing left half-masked. Twelve short
documents are a regression check, not a research benchmark: see the
benchmark notes for what is
missed and why, and for Presidio on the first eleven of them with the same model: 66% fully
masked, 76% at its most lenient.
Worth knowing
Tokens are stable per conversation, not global, unless you give a key, as for a search index. Pass the mapping from the latest result to the next call: the same value keeps the same token, and a token is never reused for another value. The mapping you pass in is read, not changed, so keep the one each result returns:
mapping = result.mapping
second = anonymizer.anonymize_sync("Ten sam PESEL 50810112354", mapping)
second.anonymized # "Ten sam PESEL <PESEL_1>"
mapping = second.mapping # everything so far, for the next turn
A token restores the form it replaced, unless the model names a case. Masking happens on the
text as written, so a name masked in the genitive comes back in the genitive. The model may write
the case its own sentence needs after a colon, and a person's name comes back declined:
Szanowny Panie <OSOBA_1:wołacz> gives "Szanowny Panie Janie Kowalski" where the text said "Jana
Kowalskiego". token_instructions("pl") asks for it, and every restore understands it, streams and
the gateway included. Declined are names of one or two words the rules read for sure: a listed
first name and a surname such as Kowalski, Nowak, Bednarek, Mickiewicz or Duda. A surname alone
that may be a man's or a woman's, a foreign name, a company and a place come back as written, and
the case is dropped. Spellings that differ only in letter
case, in the kind of whitespace (a no-break space, a line break, a run of spaces) or in invisible
characters are the same value, and so are an identifier's groups written apart or together: they
share its token, and restore writes the spelling seen first. Passwords and custom types are the
exception, since two passwords that differ in case are two passwords, and a password never shares
a token with a name that reads alike. A one-word name found in all lowercase keeps a token of its
own too, for the reason below.
Every occurrence of a masked value is masked, also where no detector would recognise it on its own: as written, in capitals, across a line break, with a zero-width character in it, with its groups spaced differently, and around whatever a detector found inside it. Two exceptions keep ordinary words readable. A capitalised one-word name is not matched in all lowercase, because Zima, Kot and Wilk are surnames and "zima" is winter; identifiers such as a VIN or an e-mail address are matched in any case. A number shorter than six digits, such as a PIN, is not matched inside a longer number, a date, a time or an amount: "15.04.2025", "2024–2025", "2025,50" and "20250" stay readable when the PIN is 2025, while "w 2025 roku" and "PIN/PUK 2025/36912874" do not.
A token you write stays yours. A placeholder such as <PESEL_1> in your own text is never handed
to real data. It joins the mapping as standing for itself, so the reservation holds in later turns
too.
The mapping is the sensitive half. result.anonymized holds a token for everything the
detectors found, and only what they missed in the clear; result.mapping is the key that undoes
it. Store the mapping the way you would store the original document.
Replaying a conversation? mask_known_values(text, mapping) masks every value a mapping already
holds in text that was masked earlier, such as history sent again after a new value got its token.
The same call checks a payload before it leaves: mask_known_values(payload, mapping) == payload
means no value the mapping holds is left where the library would mask it.
Words of your own. In-house names no rule can know, such as a project codename or a client, go on a denylist and are masked wherever they appear, in any case and across line breaks:
guards = default_guards(denylist=["Projekt Wichura", "Zakład Nowak"], with_ner=False)
Checking the answer. After restoring, anonymizer.sweep(restored, mapping) reports tokens the
model invented and personal data its answer added: report.is_clean, or report.unknown_tokens
and report.new_spans to look at.
Choosing what to mask
Anonymizer(guards) masks DEFAULT_ENABLED: every type in ENTITY_TYPES except DATE, because a
date rarely identifies anyone and masking every deadline leaves the model unable to answer. A birth
date does identify someone, so a date after "ur.", "data urodzenia", "DOB" or "geb." is its own
type, BIRTH_DATE, and stays masked. Pass enabled to choose for yourself:
from polanon import DEFAULT_ENABLED, ENTITY_TYPES, Anonymizer, default_guards
guards = default_guards(with_ner=False)
Anonymizer(guards, enabled=DEFAULT_ENABLED | {"DATE"}) # dates too
Anonymizer(guards, enabled={"PESEL", "NIP", "IBAN"}) # identifiers only
list(ENTITY_TYPES) # every name you can use
A name that is neither built in nor declared in a guard's entity_types raises ValueError, so a
typo such as "PERSONS" cannot quietly switch masking off.
A second opinion: the judge
Rules and a language model miss what identifies someone only by context: "the finance director of the only steelworks in town". A judge is a small model that reads the masked text after stage 1 and flags such fragments. Run it locally with Ollama, so the review never leaves your network:
import asyncio
from polanon import Anonymizer, JudgeError, default_guards
from polanon.judge import OllamaJudge
anonymizer = Anonymizer(
default_guards(),
judge=OllamaJudge(model="qwen2.5:1.5b"),
judge_fallback="block", # refuse to go on if the judge is down
)
async def main(text: str) -> None:
try:
result = await anonymizer.anonymize(text)
except JudgeError:
... # the judge was down, and "block" means nothing goes out
asyncio.run(main("..."))
It needs pip install "polanon[ner,judge]", the spaCy model, a running Ollama and
ollama pull qwen2.5:1.5b. With judge_fallback="send_stage1" the call goes on when the judge
is down, and result.judge_failed says that stage 1 went out alone.
Judges are asynchronous, so anonymize_sync refuses an anonymizer that has one. RemoteJudge asks an
OpenAI-compatible endpoint instead; the masked text then leaves your network, which makes it a
testing aid rather than a privacy control.
Names that should stay readable
A language model flags "Facebook" or "Google Ads" as an organisation, which masks the very thing a question is about. Pass an allowlist and those detections are dropped before tokens are assigned:
from polanon import WELL_KNOWN_NAMES, Allowlist, Anonymizer, default_guards
anonymizer = Anonymizer(default_guards(), allowlist=Allowlist(WELL_KNOWN_NAMES))
WELL_KNOWN_NAMES holds platforms and office tools, and public institutions and filing programs
such as ZUS, Płatnik, NFZ and KSeF. A one-word name also matches its Polish case forms
("Facebooku", "Canvie", "ZUS-ie"). Only detections of names are dropped, and only when every word in them is on the list; identifiers, the denylist and
values a conversation has already masked stay masked.
The pieces
| Module | What it is for |
|---|---|
guards/ |
deterministic detectors. Each returns Spans in original-text coordinates |
pipeline.py |
runs the guards in a thread, resolves overlaps, assigns stable tokens |
policy.py |
the entity taxonomy and the token naming (<OSOBA_1>, <PESEL_1>) |
judge/ |
the optional second stage: a model reviews the masked text for what still identifies |
stream_restore.py |
puts originals back into a streamed reply without splitting a token |
turn.py |
masks everything one call to a model sends, with one mapping, and restores what it gets back |
pseudonyms.py |
computes stable tokens from a secret key, the same in every document and process |
index.py |
masks documents and questions for a search index, with the store of what tokens hide |
integrations/haystack.py |
Haystack components that mask documents and questions and restore answers |
gateway/ |
a gateway compatible with the OpenAI API, run on the operator's own machine |
output_sweep.py |
flags invented tokens and personal data the answer introduced |
Adding a detector
Subclass Guard, return spans in original-text coordinates and respect the enabled set. A type of
your own goes in entity_types, which turns it on by default; it becomes the label of its tokens,
so it is written in uppercase letters and underscores:
import re
from polanon import Anonymizer, Guard, Span, default_guards
class EmployeeIdGuard(Guard):
name = "employee_id"
entity_types = frozenset({"EMPLOYEE_ID"})
def detect(self, text: str, enabled: set[str]) -> list[Span]:
if "EMPLOYEE_ID" not in enabled:
return []
return [
Span(m.start(), m.end(), "EMPLOYEE_ID", 1.0, self.name)
for m in re.finditer(r"\bEMP-\d{5}\b", text)
]
anonymizer = Anonymizer([*default_guards(with_ner=False), EmployeeIdGuard()])
anonymizer.anonymize_sync("Wniosek EMP-11111").anonymized # "Wniosek <EMPLOYEE_ID_1>"
The score decides who wins where detections overlap: 1.0 for a value a checksum, a label or an
official format vouches for, 0.85 for a statistical guess; the judge's findings score 0.5. A Polish
name with its surname scores 0.9 and a first name alone 0.8, so a guess of a place wins over it. A
secret is masked whole, with whatever was detected inside it. A span outside the text, a type
spelled any other way, or one spelled like a built-in token label (OSOBA, DATA) raises
ValueError naming your guard.
Before you rely on it
Masking is pseudonymisation, not anonymisation in the legal sense, and no detector catches everything. The threat model says what is covered, what is not, and what stays your application's job, starting with keeping the mapping safe.
polanon was built as the masking engine of PolGate, a self-hosted hub that lets a team use Claude or GPT on real documents without sending personal data to the provider. The library knows nothing about it — no HTTP, no database, no settings — and a test fails if it ever starts to.
Used by
PolGate is a self-hosted company chat for Claude and GPT built on polanon: every message and uploaded document is masked on the server before it reaches the model, and the answer streams back with the originals in place. It adds what a team needs around the library: an encrypted vault for the mapping, retention that deletes, an audit log of types and counts, and an admin panel for the policy, including the allowlist and denylist.
PolCV writes CVs for Polish job postings and can mask the candidate's name, contact details and company names with polanon before any text reaches the model, then put them back into the answer. It adds a guard of its own for what the profile already knows, such as a LinkedIn link with the name in it, as Adding a detector describes. polanon stays optional there: the app installs it on request, talks to it through a small JSON-lines bridge, and sends nothing at all while masking is on but polanon is not working.
Contributing
Bug reports, missed values and new detectors are welcome: see CONTRIBUTING.md. A way to get an original value past the masking is a security report, made privately as SECURITY.md describes.
MIT licensed.