A language processing tool for Sinhalese (සිංහල).

Update 2020.11.01: Fixed pypi package. Use 'pip install sinling' to install sinling directly from repository.

Update 2020.08.16: Add pypi package @ https://pypi.org/project/sinling/.

Update 2020.08.16: Integrated Part of speech tagger and stemmer tool.

Update 2019.07.21: This tool no longer requires java to run sinhala tokenizer. All java code is ported to Python implementation for convenience.

Versions

The current release is 0.3.6, and the usage examples below describe its API.

Sinling 0.4 will be a redesign with a new API, and it will not be backward compatible with 0.3. The 0.3 API (SinhalaTokenizer, SinhalaStemmer, POSTagger, word_joiner, word_splitter and preprocess) will be removed in 0.4 without a deprecation period.

If your code uses the 0.3 API, pin the version before 0.4 is released:

pip install "sinling<0.4"

or, in requirements.txt:

sinling<0.4

Installation

Run the following command in your virtualenv to install this package.

pip install sinling

How to use

Sinhala Tokenizer

from sinling import SinhalaTokenizer

tokenizer = SinhalaTokenizer()

sentence = '...'  # your sentence

tokenizer.tokenize(sentence)

Sinhala Stemmer (Experimental)

from sinling import SinhalaStemmer

stemmer = SinhalaStemmer()

word = '...'  # your sentence

stemmer.stem(word)  # returns (base, suffix)

The stemmer looks words up in a stem dictionary first and falls back to stripping the longest known suffix. Pass SinhalaStemmer(use_dictionary=False) to use suffix stripping only.

Please cite sinhala-stemmer if you are using this implementation.

Part-of-Speech Tagger

from sinling import SinhalaTokenizer, POSTagger

tokenizer = SinhalaTokenizer()

document = '...'  # may contain multiple sentences

tokenized_sentences = [tokenizer.tokenize(f'{ss}.') for ss in tokenizer.split_sentences(document)]

tagger = POSTagger()

pos_tags = tagger.predict(tokenized_sentences)

Word Joiner (Morphological Joiner)

from sinling import preprocess, word_joiner

w1 = preprocess('මුනි')
w2 = preprocess('උතුමා')
results = word_joiner.join(w1, w2)
# Returns a list of possible results after applying join rules ['මුනිතුමා', ...]

Word Splitter (Morphological Splitter) / corpus based - experimental

from sinling import word_splitter

word = '...'
results = word_splitter.split(word)
# Returns a dict containing debug information, base word and affix

Visit here to see some sample splits.

Contributions

• Contact [email protected] if you would like to contribute to this project.

License

Apache License Version 2.0, January 2004 http://www.apache.org/licenses/