Skip to content
Projects

clipperhouse/uax29 has 125 GitHub stars and 8 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-16 19:12 UTC. Last push 2026-02-16.

How these counts are measured

Measured star growth

in 24h
0
in 7 days
Update delayed
in 15 days
Update delayed
in 1 month
Update delayed

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.

Developer Tools

README.md

Original

This package tokenizes (splits) words, sentences and graphemes, based on Unicode text segmentation (UAX #29), for Unicode 17. Details and usage are in the respective packages:

uax29/graphemes

uax29/words

uax29/phrases

uax29/sentences

Why tokenize?

Any time our code operates on individual words, we are tokenizing. Often, we do it ad hoc, such as splitting on spaces, which gives inconsistent results. The Unicode standard is better: it is multi-lingual, and handles punctuation, special characters, etc.

Uses

The uax29 module has 4 tokenizers. In decreasing granularity: sentences → phrases → words → graphemes. Words and graphemes are the most common uses.

You might use words for inverted indexes, full-text search, TF-IDF, BM25, embeddings, etc.

If you're doing embeddings, the definition of “meaningful unit” will depend on your application. You might choose sentences, phrases, words, or a combination.

Conformance

We use the official Unicode test suites. Status:

Go

Quick start

go get "github.com/clipperhouse/uax29/v2/words"
import "github.com/clipperhouse/uax29/v2/words"

text := "Hello, 世界. Nice dog! 👍🐶"

tokens := words.FromString(text)

for tokens.Next() {                     // Next() returns true until end of data
	fmt.Println(tokens.Value())         // Do something with the current token
}

See also

jargon, a text pipelines package for CLI and Go, which consumes this package.

Prior art

blevesearch/segment

rivo/uniseg

Other language implementations

C# (also by me)

JavaScript

Rust

Java

Python