Skip to content
Projects

cimeister/tokenizer-intrinsic-evals

View on GitHub

cimeister/tokenizer-intrinsic-evals has 58 GitHub stars and 13 forks. GitHot measured 0 in 7 days and 0 in 15 days. Last checked 2026-11-15 22:14 UTC. Last push 2026-08-31.

How these counts are measured

Measured star growth

in 24h
Update delayed
in 7 days
0
in 15 days
0
in 1 month
Update delayed

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

TokEval: intrinsic quality metrics for tokenizers across natural language, code, and math

AI

README.md

Original

TokEval

A toolkit for computing intrinsic quality metrics for tokenizers across natural language, code and math.

The metrics, and the pretraining experiments that check which of them predict downstream behaviour, are described in TokEval: A Tokenizer Evaluation Suite (COLM 2026).

Installing the package puts three commands on the path: tokenizer-analysis, tokenizer-visualize and tokenizer-sanity-check.

This supersedes tokenizer-analysis-suite. The install name changed to tokenizer-intrinsic-evals; the import name is still tokenizer_analysis. The prior suite is preserved on the legacy-suite branch and the legacy-suite-final tag. Several metrics changed what they compute, so numbers from the old suite are not comparable with numbers from this one: CHANGELOG.md lists every such change with its measured effect.

Contents

Reference material is in docs/: METRICS.md defines every metric, CONFIGURATION.md is the flag and config reference, OUTPUT.md describes the results file, SANITY_CHECKS.md documents the 16 tokenizer health checks, VISUALIZATION.md covers the plotting and LaTeX tools, and EXTENDING.md covers the Python API and adding your own tokenizers and metrics.

Install

Python 3.10 or newer, tested on Linux. The commands below use uv; install that first if it is not already on your path.

git clone https://github.com/cimeister/tokenizer-intrinsic-evals.git
cd tokenizer-intrinsic-evals
uv sync

uv run puts the console scripts on the path; activate the venv instead if you prefer, and drop the prefix.

Not published to PyPI. To use it as a dependency rather than working in the checkout, pip install git+https://github.com/cimeister/tokenizer-intrinsic-evals.git. The demo data and the example configs are not in the installable package. Use a checkout for those.

MorphScore, parquet input and the FLORES+ fetcher are optional extras with an install order that matters: see Optional extras.

Quick Start

Compare two tokenizers on one corpus. gpt2 and xlm-roberta-base download from the Hugging Face Hub on first use; both are public.

cat > corpus.txt <<'CORPUS'
The quick brown fox jumps over the lazy dog.
नई दिल्ली भारत की राजधानी है।
def compute_total(items): return sum(i.price for i in items)
The invoice totalled 1234567 euros on 2024-03-15.
CORPUS

cat > my_tokenizers.json <<'TOKENIZERS'
{
  "gpt2":  {"class": "huggingface", "path": "gpt2"},
  "xlm-r": {"class": "huggingface", "path": "xlm-roberta-base"}
}
TOKENIZERS

uv run tokenizer-analysis --tokenizer-config my_tokenizers.json --input corpus.txt

Results are written to results/analysis_results.json, with plots beside it.

The metrics

Eighteen metrics. One line each below; the definition, worked example and caveats for each are in docs/METRICS.md, and the exact key path for each headline value is in Metric names and results keys.

Metric Measures Better
compression_rate text units per token higher
fertility tokens per word or character lower
token_length mean token size in bytes and characters neither
vocabulary_utilization share of the declared vocabulary the corpus used higher
renyi_efficiency how evenly token probability mass is spread higher
bigram_entropy how predictable each token's successor is higher
trigram_entropy the same over a two-token context higher
morphscore whether token boundaries fall on morpheme boundaries higher
three_digit_boundary_alignment whether numbers split on place-value boundaries higher
numeric_magnitude_consistency whether tokens per digit stay stable across magnitudes lower
operator_isolation_rate whether operators are their own tokens higher
reconstruction_fidelity whether decoding restores the input exactly higher
utf8_token_integrity whether tokens are complete UTF-8 sequences higher
ast_boundary_alignment whether token boundaries fall on syntax-tree leaves higher
identifier_fragmentation how many tokens an identifier costs lower
indentation_consistency whether whitespace tokens scale with indent depth higher
tokenizer_fairness_gini how unevenly encoding cost falls across languages lower
encoding_speed wall-clock cost of the run, not a quality measure lower

"Better" gives the direction for the quantity as defined. "neither" marks a metric that describes a tokenizer without ranking it.

Running on your own data

--input takes one file (.txt, .json, .jsonl, .parquet) or a directory of them. A .txt file is one document per line. For several corpora at once, and for the cross-lingual metrics, use --language-config instead. Both are described in Data Configuration.

A tokenizer path is either a Hub model id or a local tokenizer.json. Ten tokenizer classes are supported; see Tokenizer Configuration.

Settings that change what is measured

The Quick Start command uses the default code corpus, the default math corpus and the default character error rate budget. Each of the three changes what the metrics are computed on, so numbers from that command are not comparable with numbers from the command below.

uv run tokenizer-analysis \
    --tokenizer-config my_tokenizers.json \
    --language-config my_languages.json \
    --code-ast-config my_code.json \
    --use-builtin-math-data \
    --cer-time-budget 0
  • --code-ast-config supplies real source code. Without it the code metrics run on built-in synthetic samples and the run warns. The two are not comparable.
  • --use-builtin-math-data, or --math-data FILE, points the three digit metrics at a corpus with long numbers in it. Without either they run on whatever numbers the prose happens to contain, which on the bundled corpus is 1797 digit spans of which 6 exceed 4 digits, so the place-value boundaries at L-6 and L-9 are almost never exercised.
  • --cer-time-budget 0 removes the cap on the character error rate, so mean_cer and whitespace_fidelity are measured rather than reported as null for slow tokenizers.

The evaluation corpus

The configs in configs/ and the --use-sample-data demo read parallel/<iso639-3>_<Script>.txt. Those files are not in this repository. They come from FLORES+, which is CC-BY-SA 4.0 and is fetched rather than redistributed here. Cite NLLB Team et al., Nature 630 (2024) if you use it.

uv pip install datasets            # or: uv sync --extra flores
hf auth login                      # FLORES+ is gated, approval is automatic
uv run python scripts/fetch_flores.py                              # 13 languages
uv run python scripts/fetch_flores.py --all                        # every language

A run that names a missing file aborts with an error that prints the fetch command above. It never proceeds on a smaller corpus than the config declares. Nothing in the library requires FLORES+: --input and --language-config take your own corpus.

Reading the output

results/
├── analysis_results.json       # every metric, per tokenizer and per language
├── analysis_results_full.json  # with --save-full-results
└── *.svg                       # one chart per metric, unless --no-plots

Every metric is written as <metric>.per_tokenizer.<tok>.global for the headline value and .per_language.<lang> for the breakdown, with metadata.aggregation naming which average the global is. A value that could not be computed is null, never 0.0.

docs/OUTPUT.md describes the schema, the aggregation labels, the null convention and the run_metadata provenance block.

tokenizer-visualize and tokenizer-sanity-check

tokenizer-visualize renders token boundaries on source text, for inspecting how tokenizers split code, math and multilingual content.

uv run tokenizer-visualize --tokenizer-config configs/sample_tokenizers.json

tokenizer-sanity-check runs 16 checks against one tokenizer: byte coverage, whitespace and digit handling, special tokens, determinism, Unicode normalization, vocabulary integrity and reachability. A failing check sets a non-zero exit code, so it can gate a tokenizer before a full run.

uv run tokenizer-sanity-check huggingface:tokenizers/bpe.json

Both bundled demo tokenizers fail this check on purpose: bpe.json and unigramlm.json each cover 94 of 256 byte values. unigramlm.json also logs a warning that it cannot report its declared special tokens, so the check falls back to the 13 generic ones. That follows from the same file and is expected.

docs/SANITY_CHECKS.md documents the 16 checks, the five severities and the exit codes. docs/VISUALIZATION.md covers tokenizer-visualize, the metric plots and the LaTeX table generator.

Results for nine open-source tokenizers

benchmarks/open_source/REPORT.md has the full metric set for nine open-source tokenizers on 13 FLORES+ languages of translated news, 1500 source files across 15 programming languages, and the 285 bundled math expressions. Three of its results:

Pooled over the 13 languages, bytes per token ranges from 2.036 for GPT-2 to 5.067 for XLM-RoBERTa base. On English alone the order reverses: GPT-2 is highest at 4.805 and XLM-RoBERTa base lowest at 4.263. Vocabulary size is not held constant across the nine: it correlates with bytes per token at Spearman 0.883.

The Gini coefficient over per-byte costs and the Gini coefficient over per-line costs correlate at Spearman 0.650 over the nine, and the two orderings differ at the top. Llama 3 has the lowest per-byte coefficient, 0.0772, and a per-line coefficient of 0.0926. XLM-RoBERTa base has the lowest per-line coefficient, 0.0494, and a per-byte coefficient of 0.0976. The per-line coefficient is published only when every language has the same line count, which is the case for FLORES+.

An operator isolation rate of 1.000 does not mean compound operators are kept whole. BERT base uncased has an isolation rate of 1.000 and a compound preservation rate of 0.000 over 88,398 compound operators such as == and ->, where the other eight range from 0.844 to 0.997.

bash benchmarks/open_source/run.sh

That regenerates the report and the results file beside it. The results file records the package version, the commit, and a hash of the configs and the FLORES+ files.

Troubleshooting

No module named 'morphscore': initialize submodules, then install MorphScore into the project environment: git submodule update --init --recursive && uv pip install -e ./morphscore. If it worked before and stopped, a later uv sync removed it; see Optional extras.

Unknown tokenizer class: the available classes are listed under Tokenizer Configuration, plus any you register with register_tokenizer_class().

FileNotFoundError: a relative data_path in a language config resolves against the package root, while --input and a relative tokenizer path resolve against your working directory. See How data_path is resolved.

Cannot read the parquet file: uv sync --extra parquet, naming any other extras in the same command.

_tkinter.TclError: no display name: export MPLBACKEND=Agg before running on a headless server.

A code language is missing from the results: three of the 19 configured languages are excluded by design, and a grammar that crashes or exceeds TOKEVAL_PARSE_TIMEOUT_S is reported as unmeasured and named in the log. See docs/METRICS.md.

Contributing

  1. Fork the repository.
  2. Create a feature branch.
  3. Ensure the existing tests pass: uv run pytest tokenizer_analysis/tests -q.
  4. Submit a pull request.

Adding a tokenizer class, adding a metric, and the repository layout are covered in docs/EXTENDING.md.

Documents

File Contents
docs/METRICS.md The definition, worked example and caveats for every metric
docs/CONFIGURATION.md Every CLI flag and every config file format
docs/OUTPUT.md The results file: schema, aggregation, null convention, provenance
docs/SANITY_CHECKS.md The 16 tokenizer health checks, the five severities, the exit codes
docs/VISUALIZATION.md Metric plots, tokenizer-visualize, the LaTeX table generator
docs/EXTENDING.md Python API, repository layout, adding tokenizers and metrics
benchmarks/open_source/REPORT.md Nine open-source tokenizers measured on the full metric set
CHANGELOG.md Changes from tokenizer-analysis-suite, breaking ones first
CITATION.cff Machine-readable citation metadata

License

MIT. See LICENSE. The evaluation corpora and the tokenizers named in the example configs are downloaded at run time under their own licenses and are not redistributed here.

Citation

@inproceedings{meister2026tokeval,
  title         = {TokEval: A Tokenizer Evaluation Suite},
  author        = {Meister, Clara},
  booktitle     = {Conference on Language Modeling},
  year          = {2026},
  eprint        = {2608.18062},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2608.18062}
}