codelion/pts

---

What PTS is

When a model works through a problem, a few points along the way decide the answer. Some of those points are hidden in the residual stream as a concept the model has not said yet. Some are a single emitted token. Some are a whole sentence of reasoning.

PTS treats all three as the same thing at different scales. It searches for them, scores each one by how much it changes the model's chance of getting the answer right, labels them with one shared vocabulary, and links them into a single graph.

latent meta-token / workspace event        Latent PTS
        |
emitted pivotal token                       Token PTS
        |
sentence-level thought anchor               Sentence PTS
        |
success / failure probability shift

The score is the same at every scale:

event_importance = outcome_with_event - outcome_without_or_altered_event

The three scales

Each scale builds on existing work. PTS is the frame that holds the three together.

Scale	What it finds	How it's scored	Builds on
Latent PTS	Concepts active in the mid-layer workspace, not yet emitted	J-lens readout score	Anthropic's workspace / J-lens [1]
Token PTS	Emitted tokens that flip success probability	P(success | prefix+token) - P(success | prefix)	Phi-4 Pivotal Token Search [2]
Sentence PTS	Reasoning sentences that flip success probability	P(success | prefix+sentence) - P(success | +alternative)	Thought Anchors [3]

The question PTS is built to test: do latent meta-tokens in the workspace tend to show up just before the emitted tokens and sentences that matter?

Install

git clone https://github.com/codelion/pts.git && cd pts
pip install -e .
pip install -e '.[mlx]'   # optional: Apple-silicon steering, serving and discovery
pip install -e '.[optiq]' # optional: also mlx-optiq's mixed-precision quants (mlx-community/*-OptiQ-*)

import pts does not load torch or transformers. The schema, storage, classification, and linking layers are plain Python. Model code loads only when you use it.

Quick start

# Token PTS: emitted pivotal tokens (the original idea)
pts run --granularity token --model Qwen/Qwen3-0.6B --output-path events.jsonl

# Sentence PTS: thought anchors
pts run --granularity sentence --model Qwen/Qwen3-0.6B --output-path events.jsonl

# Latent PTS: add workspace meta-tokens to a dataset you already have.
pts fit-jlens --model Qwen/Qwen3-0.6B --output-path ./jlens         # calibrate once
pts enrich --input-path events.jsonl --output-path events_latent.jsonl \
           --model Qwen/Qwen3-0.6B --jlens-path ./jlens \
           --readout-method jlens --with-latent --shuffle-control

# All three scales, linked into one graph
pts run --granularity all --model Qwen/Qwen3-0.6B \
        --readout-method jlens --jlens-path ./jlens --output-path events.jsonl

No J-lens yet? --jlens-path also takes a reference lens straight from the Hub, e.g. hf://neuronpedia/jacobian-lens/qwen3-1.7b/jlens/Salesforce-wikitext/Qwen3-1.7B_jacobian_lens.pt. Or use --readout-method logit_lens, which needs no calibration. It is the same readout with J = I, and it is a weaker signal. See docs/latent_pts.md.

Explore any result in the hosted visualizer, or run it locally with cd visualizer && python app.py.

Results

We enriched two reasoning models and checked whether the J-lens (what an activation is pushing the model to say later) beats a logit-lens control (what it would say now). That comparison is what tells a real workspace apart from plain next-token structure.

	Qwen3-0.6B	DeepSeek-R1-1.5B
Meta-token category matches the event it precedes, vs chance	2.6x	3.6x
J-lens lift	2.76x	3.64x
logit-lens lift (control)	2.41x	3.29x
J-lens beats the control?	trends ahead, overlapping (n≈100)	yes (n=239)

The gap is clearer on the bigger model, which is the direction the workspace idea predicts. These are observational results, not causal ones. The dataset cards have the full per-model write-ups: Qwen, DeepSeek.

Commands

Command	Does
pts run --granularity token|sentence|latent|all	Search for pivotal events
pts enrich --with-latent	Add latent meta-token events to a dataset
pts fit-jlens	Calibrate a Jacobian lens for a model
pts link	Link latent, token, and sentence events into chains
pts migrate	Read older pivotal-token / thought-anchor files into the schema
pts export --format …	causal_events, metatokens, pivotal_tokens, thought_anchors, dpo, reasoning_patterns
pts push	Upload to Hugging Face
pts build-vector	Build a steering vector from positive and negative prompts
pts steer	Generate with one or more steering vectors applied
pts serve	OpenAI-compatible server with per-request steering (MLX or PyTorch)
pts discover	Token PTS on a labelled behavior eval, turned into validated steering vectors (MLX)

DPO pairs and reasoning patterns are export formats. reasoning_patterns is each pivotal token's activation, clustered and labelled with one of five reasoning patterns, for OptiLLM's autothink. It was called steering before; the steering vectors PTS builds, validates and runs are the format in Steering below. See docs/compatibility.md.

Steering

A steering vector is one format on every backend: steering_config.json plus steering_vectors.safetensors, loaded from a directory, a file, or hf://org/repo. MLX (pip install -e '.[mlx]', Apple silicon) and PyTorch apply it at the same point, the residual stream after decoder block L, and agree to 1e-5 on the same weights.

pts build-vector --model M --positive pos.txt --negative neg.txt --layers 16 --output-path vec/
pts steer --model M --vector vec/:-8 --prompt "..." --chat --compare       # MLX
pts steer --backend torch --model M --vector vec/:-8 --prompt "..."        # PyTorch
pts serve --model M --vector syco=vec/ --preset honest=syco:-8 --port 8080                  # MLX
pts serve --backend torch --model M --vector syco=vec/ --preset honest=syco:-8 --port 8080  # PyTorch

pts serve takes a steering field on each request (or a named preset), and never batches or shares a KV cache between differently-steered requests.

pts discover runs token PTS on a behavior eval (for example Anthropic's model-written sycophancy evals), builds vectors from the pivotal tokens, and checks them against shuffled-sign and random-position controls, CAA, a random vector, and a prompt baseline on held-out questions. It reports when the PTS vector does not beat them. It needs a model whose reasoning comes before its answer and finishes within --max-tokens; on an answer-first model the pivotal tokens are the answer letters and the vector reduces to CAA.

An eval is JSONL. Rows with answer_matching_behavior / answer_not_matching_behavior are A/B or Yes/No questions (Anthropic's format). Rows with a pattern are open-ended prompts: the behavior is a final answer matching that regex, which is how a topic like the Golden Gate Bridge is searched for (CAA needs labelled answers, so it is skipped for these). examples/evals/golden_gate_prompts.jsonl has 598 prompts where the bridge is a natural but optional mention (160 hand-written, the rest filled in from templates by build_golden_gate_prompts.py); keep the ones your model mentions it on some of the time.

pts discover --model M --eval hf://Anthropic/model-written-evals/sycophancy/sycophancy_on_nlp_survey.jsonl --output-dir out/
pts push --input-path out/ --hf-repo-id you/M-sycophancy   # vectors + report + a model card

Held-out questions the model always (or never) answers with the behavior leave a vector no room to move them. --held-out-pool N screens N extra questions and keeps, for selection and test, only those in the same [--min-prob, --max-prob] band the search uses; the discovery questions do not change. --methods caa validates only the PTS vector and CAA, a quicker check of whether the behavior is steerable at all (with --reuse-search, no new search); a card made from such a run never calls a vector passing, because the skipped controls were not beaten.

Each vector's layer and strength are picked on the selection questions: settings that make the model answer less often or disturb its predictions on neutral text (--min-answered-ratio, --max-kl) are dropped, and of the rest the weakest one that moves the behavior about as far as the best (within one standard error) is kept, so a lucky draw at full strength does not win.

pts push on a discover directory publishes the PTS vectors (increase/, decrease/), the full report.json, and a model card generated from it. The card's verdict is computed: it says a vector beats its controls only when every paired interval excludes zero and the random-vector null agrees. --eval-id gives a local eval's public name (local paths are never published), and --limitation adds a caveat you measured to the card. To read steered answers in the mode a vector was validated in, pass pts steer the same --chat-template-args and sampling flags; --compare runs both answers from one --seed.

Related work

PTS pulls three lines of work into one framework. The framework is the contribution; each scale rests on prior work.

• Token PTS is the Pivotal Token Search idea from the Phi-4 technical report [2], turned from a standalone token method into one scale of a larger object.
• Sentence PTS is Thought Anchors [3], the reasoning steps that matter, recast as the sentence scale of the same event.
• Latent PTS reads the hidden workspace with a Jacobian lens, following Anthropic's workspace work [1]. It is an independent reimplementation from the paper's published equations (no code was released), checked against a brute-force autograd Jacobian but not validated against the authors' own results. "Meta-token" is our term, not the paper's.

Put simply: what Phi-4 found in tokens and Thought Anchors found in sentences is the same thing at different scales, and the workspace work describes where it lives before it is emitted. PTS is the frame around all three. It is inspired by and compatible with that work, not the same as it.

Documentation

• docs/latent_pts.md: the J-lens and how latent search works
• docs/dataset_schema.md: the event schema
• docs/compatibility.md: reading older datasets

Datasets

• codelion/Qwen3-0.6B-pts
• codelion/DeepSeek-R1-Distill-Qwen-1.5B-pts

References

1. Anthropic, Verbalizable Representations Form a Global Workspace in Language Models (2026). transformer-circuits.pub/2026/workspace
2. Microsoft, Phi-4 Technical Report (2024), which introduces Pivotal Token Search. arXiv:2412.08905
3. P. C. Bogdan, U. Macar, N. Nanda, A. Conmy, Thought Anchors: Which LLM Reasoning Steps Matter? (2025). arXiv:2506.19143

Citation

@software{pts,
  title = {PTS: Pivotal Token Search},
  author = {Asankhaya Sharma},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/codelion/pts}
}