Skip to content
项目

ashvardanian/StringWars has 160 GitHub stars and 7 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-19 23:41 UTC. Last push 2026-11-14.

这些数字如何统计

实测星标增长

24小时
0
7天
正在收集历史
15天
正在收集历史
1个月
正在收集历史

各周期内的星标净变化,周期相互重叠,并非每日新增。缺失的测量保持空白。星标代表关注度,不代表软件质量。

关于

Comparing performance-oriented string-processing libraries for substring search, multi-pattern matching, hashing, edit-distances, sketching, and sorting across CPUs and GPUs in Rust 🦀 and Python 🐍

数据

README.md

原文

StringWars

Text Processing on CPUs & GPUs, in Python & Rust

StringWars banner

There are many great libraries for string processing! Mostly, of course, written in Assembly, C, and C++, but some in Rust as well.

Where Rust decimates C and C++, is the simplicity of dependency management, making it great for benchmarking "Systems Software" and lining up apples-to-apples across native crates and their Python bindings. So, to accelerate the development of the StringZilla C, C++, and CUDA libraries (with Rust and Python bindings), I've created this repository to compare it against some of my & communities most beloved Rust projects, like:

Of course, the functionality of the projects is different, as are the APIs and the usage patterns. So, I focus on the workloads for which StringZilla was designed and compare the throughput of the core operations. Notably, I also favor modern hardware with support for a wider range SIMD instructions, like mask-equipped AVX-512 on x86 starting from the 2015 Intel Skylake-X CPUs or more recent predicated variable-length SVE and SVE2 on Arm, that aren't often supported by existing libraries and tooling.

[!IMPORTANT]
The numbers in the tables below are provided for reference only and may vary depending on the CPU, compiler, dataset, and tokenization method. Most of them were obtained on Intel Sapphire Rapids (SPR) and Granite Rapids (GNR) CPUs and Nvidia Hopper-based H100 and Blackwell-based RTX 6000 Pro GPUs, using Rust with -C target-cpu=native optimization flag. To replicate the results, please refer to the Replicating the Results section below.

Benchmarks at a Glance

Hash

Many hashing libraries exist, but they often lack reproducible outputs, streaming support, or cross-language availability. Throughput on short words and long lines:

                    Short Words                  Long Lines
Rust:
stringzilla::hash   ████████████████████ 1.71    ████████████████████ 10.60 GB/s
aHash::hash_one     █████████████▍       1.15    ███████████████▏      8.02 GB/s
xxh3::xxh3_64       ███████████▊         1.01    ████████████████▋     8.83 GB/s
std::hash           ████▋                0.40    ██████▌               3.48 GB/s

Python:
stringzilla.hash    ████████████████████ 0.13    ████████████████████  8.56 GB/s
hash                ██████████████████▌  0.12    █████████▎            3.98 GB/s
xxhash.xxh3_64      █████▋               0.04    █████████████▉        5.94 GB/s

See hash/README.md for details

Multi-Way Hashing

Bloom and cuckoo filters need many independent hashes of the same key; StringZilla's hash_multiseed prepares the key once and replays cheap per-seed rounds, while the alternatives re-prepare it every 64–128 bits. Digest throughput at a 1024-bit digest (16 independent hashes per word):

Rust:
stringzilla::hash_multiseed ████████████████████ 71.85 G bits/s
xxh3::xxh3_128              ████████▏            29.30 G bits/s
stringzilla::hash           ██████               21.77 G bits/s

Python:
stringzilla.hash_multiseed  ████████████████████ 6.48 G bits/s
stringzilla.hash            █▌                   0.51 G bits/s
xxhash.xxh3_128             ▉                    0.31 G bits/s

See containers/README.md for details

Case-Insensitive UTF-8 Search

Unicode-aware case-insensitive search with full case folding (ß↔SS, σ↔ς). Throughput searching across ~100MB multilingual corpora:

Rust:
                      English                      German
stringzilla           ████████████████████ 11.91   ████████████████████ 9.937 GB/s
icu                   ▏                     0.07   ▏                     0.07 GB/s

                      Russian                      Korean
stringzilla           ████████████████████  6.63   ████████████████████ 32.69 GB/s
icu                   ▏                     0.13   ▏                     0.21 GB/s

Python:
                      English                      German
stringzilla           ████████████████████  5.22   ████████████████████  5.66 GB/s
regex                 ██▋                   0.72   ███                   0.84 GB/s

                      Russian                      Korean
stringzilla           ████████████████████  5.31   ████████████████████ 18.67 GB/s
regex                 ████████              2.14   ████▋                 4.27 GB/s

See normalization/README.md for details

Exact Substring Search

Substring search is offloaded to C's memmem or strstr in most languages, but SIMD-optimized implementations can do better. Throughput on long lines:

                    Left to right                Reverse order
Rust:
memmem::Finder      ████████████████████ 10.24
stringzilla         ███████████████████▋ 10.08   ████████████████████ 9.928 GB/s
std::str            ███████████████████▊ 10.13   ███████████▏          5.53 GB/s

Python:
stringzilla         ████████████████████ 10.98   ████████████████████ 10.77 GB/s
str                 ██                    1.15   ██████▋               3.58 GB/s

See find/README.md for details

Byte-Set Search

Searching for character sets (tabs, HTML markup, digits) commonly uses regex or Aho-Corasick automata. Throughput counting all matches on long lines:

Rust:
stringzilla         ████████████████████   7.61 GB/s
regex::find_iter    ████████████▊          4.86 GB/s
aho_corasick        █▏                     0.47 GB/s

Python:
stringzilla         ████████████████████   8.19 GB/s
re.finditer         ▍                      0.18 GB/s

See find/README.md for details

Multi-Pattern Search

Matching a whole dictionary of needles against a whole corpus in one pass is what Aho-Corasick automata are for, and how the automaton is laid out in memory decides the throughput far more than the search loop does. Counting, resolving a leftmost cover, rewriting, and BM25 scoring all ride the same compiled dictionary. Numbers await a quiet machine; the sweep and its framing are in substrings/README.md.

Two things that table has to say out loud: every rival is single-threaded and CPU-only, so the honest head-to-head is against stringzilla::Substrings<1cpu> rather than its multi-core or GPU rows; and BM25 here is a scan against a fixed query, not an inverted index, so bm25 and bm25s are compared on index build and on query separately.

See substrings/README.md for details

UTF-8 Processing

Different scripts stress UTF-8 differently: Korean has 3-byte Hangul with single-byte whitespace (representative for tokenization), Arabic uses 2-byte characters, English is mostly 1-byte ASCII. Throughput on AMD Zen5 Turin:

Newline splitting:
                      English                     Arabic
stringzilla           ████████████████ 14.39      ████████████████████ 17.08 GB/s
stdlib                ██                1.77      ██                    1.70 GB/s

Whitespace splitting:
                      English                     Korean
stringzilla           ████████████████████ 0.76   ████████████████████ 1.75 GB/s
stdlib                ██████████████████▊  0.72   ██████████▍          0.91 GB/s
icu::WhiteSpace       ██▋                  0.10   █▌                   0.14 GB/s

Case folding on bicameral scripts (Latin, Cyrillic, Greek, Armenian) plus Chinese for reference:

Case folding:
                      English 16x                 German 6x
stringzilla           ████████████████████ 7.53   ████████████████████ 2.59 GB/s
stdlib                ██▌                  0.48   ███▎                 0.43 GB/s

                      Russian 10x                 French 5x
stringzilla           ████████████████████ 2.20   ████████████████████ 1.84 GB/s
stdlib                ██                   0.22   ███▊                 0.35 GB/s

                      Greek 5x                    Armenian 4x
stringzilla           ████████████████████ 1.00   ████████████████████  908 MB/s
stdlib                ████▍                0.22   ████▉                 223 MB/s

                      Vietnamese 1.3x             Chinese 4x
stringzilla           ████████████████████  352   ████████████████████ 1.21 GB/s
stdlib                █████████████▏        265   █████▍                325 MB/s

Codepoint indexing — the byte offset of the Nth codepoint — is where SIMD pays off most: the standard library rescans the buffer byte-by-byte, while StringZilla jumps to it with a single SIMD scan.

Byte offset of the Nth codepoint, full corpora:
                      English                      Korean
stringzilla::find_nth ████████████████████  6.65   ████████████████████ 11.31 GB/s
std::char_indices     ██▎                   0.77   █▏                   0.67 GB/s

See tokenization/README.md and normalization/README.md for details

Sequence Operations

Dataframe libraries and search engines rely heavily on string sorting. SIMD-accelerated comparisons and specialized radix sorts can outperform generic algorithms. Every competitor is configured for a stable sort to match StringZilla, whose argsort writes the permutation into a caller-owned buffer — a NumPy out= array in Python — for a zero-allocation index sort. Throughput on short words:

Rust:
stringzilla.argsort ████████████████████  209.32 M cmp/s
polars::sort        ███████████████████▋  205.21 M cmp/s
arrow::lexsort      ███████████▊          122.58 M cmp/s
std::sort_by_key    ███▌                   37.46 M cmp/s

Python:
polars.sort         ████████████████████  229.90 M cmp/s
stringzilla.argsort ███████████████████▏  219.92 M cmp/s
pyarrow.sort        ██████▎                72.99 M cmp/s
list.sort           ████▍                  51.10 M cmp/s

GPU: cudf on H100 reaches 9,463 M cmp/s on short words.

See sequence/README.md for details

Random Generation

Random byte generation and lookup tables are common in image processing and bioinformatics. Throughput on long lines:

Rust:
stringzilla         ████████████████████  9.844 GB/s
zeroize             ████████▉              4.41 GB/s
rand_xoshiro        ███████▎               3.59 GB/s

Python:
stringzilla         ████████████████████  18.97 GB/s
pycryptodome        ████████████▉         12.26 GB/s
numpy.Philox        █▌                     1.48 GB/s

See memory/README.md for details

Similarity Scoring

Edit distance is essential for search engines, data cleaning, NLP, and bioinformatics. It's computationally expensive with O(n·m) complexity, but GPUs and multi-core parallelism help. Levenshtein distance on ~1,000 byte lines (MCUPS = Million Cell Updates Per Second):

Rust:
                        1 Core                       1 Socket
bio::levenshtein        █▏                      823
rapidfuzz               ████████████████████ 14,316
stringzilla<384x GNR>   ██████████████████▎  13,084  ████████████████████ 3,084,270 MCUPS
stringzilla<B200>                                    ██████▍                998,620 MCUPS
stringzilla<H100>                                    ██████                 925,890 MCUPS

See similarities/README.md for details

Window Overlap

Scoring the exact share of byte windows two variable-length strings have in common enables near-duplicate matching in large-scale retrieval.

See fingerprints/README.md for details

Encryption

ChaCha20 and AES256 encryption throughput comparison on long lines:

Rust:
ring::aes256        ████████████████████   2.69 GB/s
ring::chacha20      ████████▏              1.11 GB/s
libsodium::chacha20 █████                  0.66 GB/s

See encryption/README.md for details

Replicating the Results

Replicating the Results in Rust

To pull and compile all the dependencies, you can call:

RUSTFLAGS="-C target-cpu=native" cargo build --benches --all-features                  # to compile everything
RUSTFLAGS="-C target-cpu=native" cargo check --benches --all-features --all-targets    # to fail on warnings

By default StringWars links stringzilla in CPU mode. If the machine has an NVIDIA GPU with CUDA installed, enable the CUDA kernels explicitly when running benches, for example:

RUSTFLAGS="-C target-cpu=native" \
    STRINGWARS_DATASET=README.md \
    STRINGWARS_TOKENS=lines \
    STRINGWARS_FILTER=GPU \
    cargo bench --features "cuda bench_similarities" --bench bench_similarities --jobs 1

Wars always take long, and so do these benchmarks. Every one of them includes a warm-up phase to ensure that the CPU caches are filled and the results are not affected by cold start or SIMD-related frequency scaling. Each of them accepts a few environment variables to control the dataset, the tokenization, and the time limits. Suite-specific ones are listed in each benchmark's file-level documentation, which you can print with:

awk '/^"#\]/ { exit } { print }' find/bench.rs

The shared harness variables are below, and the Python scripts read the same ones, taking no flags beyond --help. Defaults come from stringwars.toml, and every run prints the values it uses. An empty variable means the same as an unset one, and a value that does not parse stops the run before it starts.

Variable Default Meaning
STRINGWARS_SEED 42 Seed of the drawn inputs, like normalization's needles, or random
STRINGWARS_FILTER none Regex over benchmark names, falling back to a substring match
STRINGWARS_WARMUP 1s Warm-up cap per benchmark, like 1s or 200ms
STRINGWARS_TIME_LIMIT 10s Measurement cap per benchmark, like 10s or 500ms
STRINGWARS_BYTES per suite, 256MB Bytes read from the dataset, like 256MB, rounded down to a power of two
STRINGWARS_BATCH_PER_CORE per suite Items per core, or per GPU streaming multiprocessor, in batched suites
STRINGWARS_DEVICES cuda:0 CUDA devices in GPU suites, like cuda:0,cuda:1, one row per device
STRINGWARS_THREADS all cores Cores for multi-core rows, 0 for all of them
STRINGWARS_DATASET per suite Path to the textual dataset file
STRINGWARS_TOKENS per suite Tokenization mode: file, lines, or words
STRINGWARS_UNIQUE false 1 or true deduplicates the tokens read
STRINGWARS_MIN_SAMPLES 10 Fewest samples a row needs before it may converge
STRINGWARS_TARGET_SPREAD 0.025 Relative half-width at which a row counts as converged
STRINGWARS_RESULTS_DIR none Directory for per-row NDJSON records
STRINGWARS_COUNTERS false 1 or true adds cycles-per-byte and IPC columns, Rust on Linux only
STRINGWARS_COLLISIONS false 1 or true adds collision rates to hash, Rust only

"Per suite" defaults are in stringwars.toml. Sizes are whole bytes or KB, MB, GB and TB in any case, and booleans are 0, 1, true or false. STRINGWARS_DEVICES is read by the suites that put rows on a GPU, which are similarities, fingerprints and sequence, and any other suite refuses it rather than ignoring it. Each selected device gets its own row, named after its ordinal as in <cuda:1>, and a device named in the variable that cannot be opened stops the run.

Here is an example of a common benchmark run on a Unix-like system:

RUSTFLAGS="-C target-cpu=native" \
    STRINGWARS_DATASET=README.md \
    STRINGWARS_TOKENS=lines \
    cargo bench --features bench_hash --bench bench_hash --jobs $(nproc)

On Windows using PowerShell you'd need to set the environment variable differently:

$env:STRINGWARS_DATASET="README.md"
cargo bench --features bench_hash --bench bench_hash --jobs $(nproc)

Replicating the Results in Python

Each suite has a dependency group of the same name in pyproject.toml, with exact versions in uv.lock. From the repository root, uv run --group <suite> installs that group together with the shared stringwars.py harness and runs the script:

STRINGWARS_DATASET=README.md STRINGWARS_TOKENS=lines uv run --group find find/bench.py
uv run --group hash hash/bench.py --help
uv run --group memory memory/bench.py --help
uv run --group sequence sequence/bench.py --help
uv run --group similarities similarities/bench.py --help
uv run --group fingerprints fingerprints/bench.py --help
uv run --group tokenization tokenization/bench.py --help
uv run --group normalization normalization/bench.py --help
uv run --group containers containers/bench.py --help
uv run --group encryption encryption/bench.py --help

The cuDF rows need RAPIDS from NVIDIA's index, which cannot share uv.lock with the CPU pins, so layer it on top:

uv run --group sequence --with-requirements requirements-cuda.txt sequence/bench.py

Datasets

UTF8 Corpus

For mixed UTF data, I've used the XL Sum dataset for multilingual extractive summarization. It's 4.7 GB in size (1.7 GB compressed), 1'004'598 lines long, and contains 268'435'456 tokens of mean length 8. To download, unpack, and run the benchmarks, execute the following bash script in your terminal:

mkdir -p data/xlsum && curl -fL -o data/xlsum/xlsum.csv.gz https://github.com/ashvardanian/xl-sum/releases/download/v1.0.0/xlsum.csv.gz
gzip -d data/xlsum/xlsum.csv.gz
STRINGWARS_DATASET=data/xlsum/xlsum.csv cargo bench --features bench_hash --bench bench_hash --jobs $(nproc)

Alternatively, for a much smaller and faster run, check out the Big List of Naughty Strings (BLNS):

mkdir -p data/blns && curl -fL -o data/blns/blns.txt https://raw.githubusercontent.com/minimaxir/big-list-of-naughty-strings/master/blns.txt
STRINGWARS_DATASET=data/blns/blns.txt cargo bench --features bench_hash --bench bench_hash --jobs $(nproc)

Multilingual Wikipedia Corpus

The Cohere Wikipedia dataset provides pre-processed JSONL files for different languages. This may be the optimal dataset for relative comparison of UTF-8 decoding and matching enginges in each individual environment. Not all Wikipedia languages are available, but the following have been selected specifically:

  • Chinese (zh): 3-byte CJK characters, rare 1-byte punctuation
  • Korean (ko): 3-byte Hangul syllables, frequent 1-byte punctuation
  • Arabic (ar): 2-byte Arabic script, with regular 1-byte punctuation
  • French (fr): Mixed 1-2 byte Latin with high diacritic density
  • English (en): Mostly 1-byte ASCII baseline

To download and decompress one file from each language:

mkdir -p data/wikipedia-22-12
curl -fL -o data/wikipedia-22-12/wiki_en.jsonl.gz https://huggingface.co/datasets/Cohere/wikipedia-22-12/resolve/main/en/000.jsonl.gz && gunzip data/wikipedia-22-12/wiki_en.jsonl.gz
curl -fL -o data/wikipedia-22-12/wiki_zh.jsonl.gz https://huggingface.co/datasets/Cohere/wikipedia-22-12/resolve/main/zh/000.jsonl.gz && gunzip data/wikipedia-22-12/wiki_zh.jsonl.gz
curl -fL -o data/wikipedia-22-12/wiki_ko.jsonl.gz https://huggingface.co/datasets/Cohere/wikipedia-22-12/resolve/main/ko/000.jsonl.gz && gunzip data/wikipedia-22-12/wiki_ko.jsonl.gz
curl -fL -o data/wikipedia-22-12/wiki_ar.jsonl.gz https://huggingface.co/datasets/Cohere/wikipedia-22-12/resolve/main/ar/000.jsonl.gz && gunzip data/wikipedia-22-12/wiki_ar.jsonl.gz
curl -fL -o data/wikipedia-22-12/wiki_fr.jsonl.gz https://huggingface.co/datasets/Cohere/wikipedia-22-12/resolve/main/fr/000.jsonl.gz && gunzip data/wikipedia-22-12/wiki_fr.jsonl.gz
curl -fL -o data/wikipedia-22-12/wiki_de.jsonl.gz https://huggingface.co/datasets/Cohere/wikipedia-22-12/resolve/main/de/000.jsonl.gz && gunzip data/wikipedia-22-12/wiki_de.jsonl.gz
curl -fL -o data/wikipedia-22-12/wiki_es.jsonl.gz https://huggingface.co/datasets/Cohere/wikipedia-22-12/resolve/main/es/000.jsonl.gz && gunzip data/wikipedia-22-12/wiki_es.jsonl.gz
curl -fL -o data/wikipedia-22-12/wiki_it.jsonl.gz https://huggingface.co/datasets/Cohere/wikipedia-22-12/resolve/main/it/000.jsonl.gz && gunzip data/wikipedia-22-12/wiki_it.jsonl.gz

Each JSONL file contains one JSON object per line with fields: id, title, text (paragraph content), url, wiki_id, and paragraph_id.

CC-100 Corpus

The CC-100 corpus provides large monolingual text files (1-80 GB) for 100+ languages, extracted from Common Crawl. Files are XZ-compressed plain text with documents separated by double-newlines.

Workload Relevant Scripts Best Test Languages
Case Folding Latin, Cyrillic, Greek, Armenian Turkish (I/i), German (ss → SS), Greek, Russian
Normalization Indic, Arabic, Vietnamese, Korean Vietnamese, Hindi, Korean, Arabic
Whitespace Tokenization Most scripts except CJK/Thai English, Russian, Arabic vs. Chinese, Japanese, Thai
Grapheme Clusters Indic, Thai, Khmer, Myanmar Thai, Tamil, Myanmar, Khmer
RTL Handling Arabic, Hebrew Arabic, Hebrew, Persian

Bicameral scripts with various case folding rules:

mkdir -p data/cc-100
curl -fL https://data.statmt.org/cc-100/en.txt.xz | xz -d > data/cc-100/cc100_en.txt      # 82 GB - English
curl -fL https://data.statmt.org/cc-100/de.txt.xz | xz -d > data/cc-100/cc100_de.txt      # 18 GB - German
curl -fL https://data.statmt.org/cc-100/tr.txt.xz | xz -d > data/cc-100/cc100_tr.txt      # 5.4 GB - Turkish
curl -fL https://data.statmt.org/cc-100/ru.txt.xz | xz -d > data/cc-100/cc100_ru.txt      # 46 GB - Russian
curl -fL https://data.statmt.org/cc-100/uk.txt.xz | xz -d > data/cc-100/cc100_uk.txt      # 14 GB - Ukrainian
curl -fL https://data.statmt.org/cc-100/el.txt.xz | xz -d > data/cc-100/cc100_el.txt      # 7.4 GB - Greek
curl -fL https://data.statmt.org/cc-100/hy.txt.xz | xz -d > data/cc-100/cc100_hy.txt      # 776 MB - Armenian
curl -fL https://data.statmt.org/cc-100/ka.txt.xz | xz -d > data/cc-100/cc100_ka.txt      # 1.1 GB - Georgian
curl -fL https://data.statmt.org/cc-100/pl.txt.xz | xz -d > data/cc-100/cc100_pl.txt      # 12 GB - Polish
curl -fL https://data.statmt.org/cc-100/cs.txt.xz | xz -d > data/cc-100/cc100_cs.txt      # 4.4 GB - Czech
curl -fL https://data.statmt.org/cc-100/nl.txt.xz | xz -d > data/cc-100/cc100_nl.txt      # 7.9 GB - Dutch
curl -fL https://data.statmt.org/cc-100/fr.txt.xz | xz -d > data/cc-100/cc100_fr.txt      # 14 GB - French
curl -fL https://data.statmt.org/cc-100/es.txt.xz | xz -d > data/cc-100/cc100_es.txt      # 14 GB - Spanish
curl -fL https://data.statmt.org/cc-100/pt.txt.xz | xz -d > data/cc-100/cc100_pt.txt      # 13 GB - Portuguese
curl -fL https://data.statmt.org/cc-100/it.txt.xz | xz -d > data/cc-100/cc100_it.txt      # 7.8 GB - Italian

Unicameral scripts without case folding, but with other normalization/segmentation challenges:

mkdir -p data/cc-100
curl -fL https://data.statmt.org/cc-100/ar.txt.xz | xz -d > data/cc-100/cc100_ar.txt      # 5.4 GB - Arabic (RTL)
curl -fL https://data.statmt.org/cc-100/he.txt.xz | xz -d > data/cc-100/cc100_he.txt      # 6.1 GB - Hebrew (RTL)
curl -fL https://data.statmt.org/cc-100/fa.txt.xz | xz -d > data/cc-100/cc100_fa.txt      # 20 GB - Persian (RTL)
curl -fL https://data.statmt.org/cc-100/hi.txt.xz | xz -d > data/cc-100/cc100_hi.txt      # 2.5 GB - Hindi (Devanagari)
curl -fL https://data.statmt.org/cc-100/bn.txt.xz | xz -d > data/cc-100/cc100_bn.txt      # 860 MB - Bengali
curl -fL https://data.statmt.org/cc-100/ta.txt.xz | xz -d > data/cc-100/cc100_ta.txt      # 1.3 GB - Tamil
curl -fL https://data.statmt.org/cc-100/te.txt.xz | xz -d > data/cc-100/cc100_te.txt      # 536 MB - Telugu
curl -fL https://data.statmt.org/cc-100/th.txt.xz | xz -d > data/cc-100/cc100_th.txt      # 8.7 GB - Thai (no spaces)
curl -fL https://data.statmt.org/cc-100/vi.txt.xz | xz -d > data/cc-100/cc100_vi.txt      # 28 GB - Vietnamese
curl -fL https://data.statmt.org/cc-100/zh-Hans.txt.xz | xz -d > data/cc-100/cc100_zh.txt # 14 GB - Chinese
curl -fL https://data.statmt.org/cc-100/ja.txt.xz | xz -d > data/cc-100/cc100_ja.txt      # 15 GB - Japanese
curl -fL https://data.statmt.org/cc-100/ko.txt.xz | xz -d > data/cc-100/cc100_ko.txt      # 14 GB - Korean (Jamo)
curl -fL https://data.statmt.org/cc-100/my.txt.xz | xz -d > data/cc-100/cc100_my.txt      # 46 MB - Myanmar
curl -fL https://data.statmt.org/cc-100/km.txt.xz | xz -d > data/cc-100/cc100_km.txt      # 153 MB - Khmer
curl -fL https://data.statmt.org/cc-100/am.txt.xz | xz -d > data/cc-100/cc100_am.txt      # 133 MB - Amharic (Ethiopic)
curl -fL https://data.statmt.org/cc-100/si.txt.xz | xz -d > data/cc-100/cc100_si.txt      # 452 MB - Sinhala

Leipzig Corpora Collection

The Leipzig Corpora Collection provides pre-segmented sentences in 200+ languages. Each tar.gz contains *-sentences.txt (tab-separated id\tsentence), *-words.txt (frequencies), and co-occurrence files. Standard sizes: 10K, 30K, 100K, 300K, 1M sentences. Check for newer years at the download page.

Bicameral scripts with various case folding rules:

mkdir -p data/leipzig-1m data/leipzig-300k
curl -fL https://downloads.wortschatz-leipzig.de/corpora/eng_wikipedia_2016_1M.tar.gz | tar -xzf - -O 'eng_wikipedia_2016_1M/eng_wikipedia_2016_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_en.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/deu_wikipedia_2021_1M.tar.gz | tar -xzf - -O 'deu_wikipedia_2021_1M/deu_wikipedia_2021_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_de.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/tur_wikipedia_2021_1M.tar.gz | tar -xzf - -O 'tur_wikipedia_2021_1M/tur_wikipedia_2021_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_tr.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/rus_wikipedia_2021_1M.tar.gz | tar -xzf - -O 'rus_wikipedia_2021_1M/rus_wikipedia_2021_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_ru.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/ukr_wikipedia_2021_1M.tar.gz | tar -xzf - -O 'ukr_wikipedia_2021_1M/ukr_wikipedia_2021_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_uk.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/ell_wikipedia_2021_1M.tar.gz | tar -xzf - -O 'ell_wikipedia_2021_1M/ell_wikipedia_2021_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_el.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/hye_wikipedia_2021_1M.tar.gz | tar -xzf - -O 'hye_wikipedia_2021_1M/hye_wikipedia_2021_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_hy.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/kat_wikipedia_2021_300K.tar.gz | tar -xzf - -O 'kat_wikipedia_2021_300K/kat_wikipedia_2021_300K-sentences.txt' | cut -f2 > data/leipzig-300k/leipzig300K_ka.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/pol_wikipedia_2021_1M.tar.gz | tar -xzf - -O 'pol_wikipedia_2021_1M/pol_wikipedia_2021_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_pl.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/ces_wikipedia_2021_1M.tar.gz | tar -xzf - -O 'ces_wikipedia_2021_1M/ces_wikipedia_2021_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_cs.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/nld_wikipedia_2021_1M.tar.gz | tar -xzf - -O 'nld_wikipedia_2021_1M/nld_wikipedia_2021_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_nl.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/fra_wikipedia_2021_1M.tar.gz | tar -xzf - -O 'fra_wikipedia_2021_1M/fra_wikipedia_2021_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_fr.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/spa_wikipedia_2021_1M.tar.gz | tar -xzf - -O 'spa_wikipedia_2021_1M/spa_wikipedia_2021_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_es.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/por_wikipedia_2021_1M.tar.gz | tar -xzf - -O 'por_wikipedia_2021_1M/por_wikipedia_2021_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_pt.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/ita_wikipedia_2021_1M.tar.gz | tar -xzf - -O 'ita_wikipedia_2021_1M/ita_wikipedia_2021_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_it.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/lit_wikipedia_2021_300K.tar.gz | tar -xzf - -O 'lit_wikipedia_2021_300K/lit_wikipedia_2021_300K-sentences.txt' | cut -f2 > data/leipzig-300k/leipzig300K_lt.txt

Unicameral scripts without case folding, but with other normalization/segmentation challenges:

mkdir -p data/leipzig-1m data/leipzig-300k data/leipzig-30k data/leipzig-10k
curl -fL https://downloads.wortschatz-leipzig.de/corpora/ara_wikipedia_2021_1M.tar.gz | tar -xzf - -O 'ara_wikipedia_2021_1M/ara_wikipedia_2021_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_ar.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/heb_wikipedia_2021_1M.tar.gz | tar -xzf - -O 'heb_wikipedia_2021_1M/heb_wikipedia_2021_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_he.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/fas_wikipedia_2014_1M.tar.gz | tar -xzf - -O 'fas_wikipedia_2014_1M/fas_wikipedia_2014_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_fa.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/hin_wikipedia_2021_1M.tar.gz | tar -xzf - -O 'hin_wikipedia_2021_1M/hin_wikipedia_2021_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_hi.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/ben_wikipedia_2021_1M.tar.gz | tar -xzf - -O 'ben_wikipedia_2021_1M/ben_wikipedia_2021_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_bn.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/tam_wikipedia_2021_1M.tar.gz | tar -xzf - -O 'tam_wikipedia_2021_1M/tam_wikipedia_2021_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_ta.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/tel_wikipedia_2021_300K.tar.gz | tar -xzf - -O 'tel_wikipedia_2021_300K/tel_wikipedia_2021_300K-sentences.txt' | cut -f2 > data/leipzig-300k/leipzig300K_te.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/tha_wikipedia_2021_10K.tar.gz | tar -xzf - -O 'tha_wikipedia_2021_10K/tha_wikipedia_2021_10K-sentences.txt' | cut -f2 > data/leipzig-10k/leipzig10K_th.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/vie_wikipedia_2021_1M.tar.gz | tar -xzf - -O 'vie_wikipedia_2021_1M/vie_wikipedia_2021_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_vi.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/zho_wikipedia_2018_1M.tar.gz | tar -xzf - -O 'zho_wikipedia_2018_1M/zho_wikipedia_2018_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_zh.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/jpn_wikipedia_2018_1M.tar.gz | tar -xzf - -O 'jpn_wikipedia_2018_1M/jpn_wikipedia_2018_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_ja.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/kor_wikipedia_2021_1M.tar.gz | tar -xzf - -O 'kor_wikipedia_2021_1M/kor_wikipedia_2021_1M-sentences.txt' | cut -f2 > data/leipzig-1m/leipzig1M_ko.txt
curl -fL https://downloads.wortschatz-leipzig.de/corpora/amh_wikipedia_2021_30K.tar.gz | tar -xzf - -O 'amh_wikipedia_2021_30K/amh_wikipedia_2021_30K-sentences.txt' | cut -f2 > data/leipzig-30k/leipzig30K_am.txt

To produce a mixed dataset with rows in all languages:

mkdir -p data/leipzig-1gb
cat data/leipzig-*/*.txt | shuf | head -c 1G > data/leipzig-1gb/leipzig1GB.txt

DNA Corpus

For bioinformatics workloads, I use the following datasets with increasing string lengths:

mkdir -p data/acgt
curl -fL -o data/acgt/acgt_100.txt 'https://huggingface.co/datasets/ashvardanian/StringWars/resolve/main/acgt_100.txt?download=true'
curl -fL -o data/acgt/acgt_1k.txt 'https://huggingface.co/datasets/ashvardanian/StringWars/resolve/main/acgt_1k.txt?download=true'
curl -fL -o data/acgt/acgt_10k.txt 'https://huggingface.co/datasets/ashvardanian/StringWars/resolve/main/acgt_10k.txt?download=true'
curl -fL -o data/acgt/acgt_100k.txt 'https://huggingface.co/datasets/ashvardanian/StringWars/resolve/main/acgt_100k.txt?download=true'
curl -fL -o data/acgt/acgt_1m.txt 'https://huggingface.co/datasets/ashvardanian/StringWars/resolve/main/acgt_1m.txt?download=true'
curl -fL -o data/acgt/acgt_10m.txt 'https://huggingface.co/datasets/ashvardanian/StringWars/resolve/main/acgt_10m.txt?download=true'

Unicode Testing Data

The Unicode Project comes with many adversarial string examples for case folding, grapheme segmentation, word, and sentence breaks. The following script, downloads all of them and composes them ino a single randomly shuffled document:

source_urls=(
    https://www.unicode.org/Public/UCD/latest/ucd/NormalizationTest.txt
    https://www.unicode.org/Public/UCD/latest/ucd/CaseFolding.txt
    https://www.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt
    https://www.unicode.org/Public/UCD/latest/ucd/auxiliary/GraphemeBreakTest.txt
    https://www.unicode.org/Public/UCD/latest/ucd/auxiliary/WordBreakTest.txt
    https://www.unicode.org/Public/UCD/latest/ucd/auxiliary/SentenceBreakTest.txt
    https://www.unicode.org/Public/UCD/latest/ucd/auxiliary/LineBreakTest.txt
    https://www.unicode.org/Public/emoji/latest/emoji-test.txt
    https://www.unicode.org/Public/security/latest/confusables.txt
)

for url in "${source_urls[@]}"; do
    curl -fL --create-dirs -o "unicode/${url##*/}" "$url"
done

# Concatenate every file and shuffle all lines into one adversarial document. The shuffle is
# reproducible: a seeded AES-CTR keystream feeds `shuf --random-source`, so the same seed
# always yields the same ordering.
seed=42
seeded_random() { openssl enc -aes-256-ctr -pass "pass:$1" -nosalt </dev/zero 2>/dev/null; }
cat unicode/*.txt | shuf --random-source=<(seeded_random "$seed") > unicode_tests.txt

Deep Profiling

In case you are profiling the some of the internal kernels of mentioned libraries, here are a few example commands to get around. Such as using ncu for NVIDIA GPUs to evaluate the register usage and occupancy of the CUDA kernels used in StringZilla's Levenshtein distance calculation:

/usr/local/cuda/bin/ncu \
  --metrics launch__registers_per_thread,launch__occupancy_per_block_size,sm__warps_active.avg.pct_of_peak_sustained_active,sm__throughput.avg.pct_of_peak_sustained_elapsed,dram__throughput.avg.pct_of_peak_sustained_elapsed,dram__bytes.sum \
  --target-processes all \
  --kernel-name "levenshtein_on_each_cuda_thread" \
  --launch-skip 5 \
  --launch-count 1 \
  bash -c 'STRINGWARS_DATASET=data/acgt/acgt_100.txt STRINGWARS_BATCH_PER_CORE=65536 STRINGWARS_TOKENS=lines STRINGWARS_FILTER="uniform/stringzilla::LevenshteinDistances\(1xGPU\)" cargo bench --features "cuda bench_similarities" --bench bench_similarities --jobs 1'

Using perf on Linux to analyze the CPU-side performance of SIMD-accelerated substring search:

perf record -e cpu-clock -g graph,0x400000 -o perf.data -- cargo bench --features "bench_similarities" --bench bench_similarities --jobs 1
perf report -i perf.data

Contributing

After cloning, enable the repository's git hooks once:

git config core.hooksPath .githooks

The pre-commit hook rejects banner marks (three or more consecutive - or =) in Python and Rust sources. Use plain text for section titles, and # region: / # endregion: at module scope for foldable sections.

It also runs ruff check and ruff format --check on staged Python (the project requires Python 3.13). Fix issues with ruff check --fix . && ruff format . rather than silencing them with # noqa.