clipperhouse/uax29
clipperhouse/uax29 has 125 GitHub stars and 8 forks. GitHot measured 0 stars in 24 hours and 0 in 7 days. Last checked 2026-11-22 10:30 UTC. Last push 2026-02-16.
实测星标增长
- 24小时
- 0
- 7天
- 0
- 15天
- 正在收集历史
- 1个月
- 正在收集历史
各周期内的星标净变化,周期相互重叠,并非每日新增。缺失的测量保持空白。星标代表关注度,不代表软件质量。
关于
A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.
开发工具README.md
原文This package tokenizes (splits) words, sentences and graphemes, based on Unicode text segmentation (UAX #29), for Unicode 17. Details and usage are in the respective packages:
Why tokenize?
Any time our code operates on individual words, we are tokenizing. Often, we do it ad hoc, such as splitting on spaces, which gives inconsistent results. The Unicode standard is better: it is multi-lingual, and handles punctuation, special characters, etc.
Uses
The uax29 module has 4 tokenizers. In decreasing granularity: sentences → phrases → words → graphemes. Words and graphemes are the most common uses.
You might use words for inverted indexes, full-text search, TF-IDF, BM25, embeddings, etc.
If you're doing embeddings, the definition of “meaningful unit” will depend on your application. You might choose sentences, phrases, words, or a combination.
Conformance
We use the official Unicode test suites. Status:
Quick start
go get "github.com/clipperhouse/uax29/v2/words"
import "github.com/clipperhouse/uax29/v2/words"
text := "Hello, 世界. Nice dog! 👍🐶"
tokens := words.FromString(text)
for tokens.Next() { // Next() returns true until end of data
fmt.Println(tokens.Value()) // Do something with the current token
}
See also
jargon, a text pipelines package for CLI and Go, which consumes this package.
Prior art
Other language implementations
C# (also by me)
探索相关项目
加载中…