UBC-NLP/afrolid

AfroLID, a neural LID toolkit for 517 African languages and varieties. AfroLID exploits a multi-domain web dataset manually curated from across 14 language families utilizing five orthographic systems. AfroLID is described in this paper: AfroLID: A Neural Language Identification Tool for African Languages.

What's New in AfroLID v1.5?

• Fine-tuned on SERENGETI, a massively multilingual language model covering 517 African languages and language varieties.
• Enhanced model performance, improving macro-F1 from 95.95 to 97.41.
• Built on Hugging Face Transformers for seamless integration.
• Optimized for easy use with the Hugging Face pipeline.
• Better efficiency and accuracy, making it more robust for African langauges identification.

How to use AfroLID v1.5?

from transformers import pipeline

afrolid = pipeline("text-classification", model='UBC-NLP/afrolid_1.5', use_fast=False)

input_text = "6Acï looi aya në wuöt dït kɔ̈k yiic ku lɔ wuöt tɔ̈u tëmec piny de Manatha ku Eparaim ku Thimion , ku ɣään mec tɔ̈u të lɔ rut cï Naptali"

result = afrolid(input_text)

# Extract the label and score from the first result
language = result[0]['label']
score = result[0]['score']

print(f"detected language: {language}\tscore: {round(score*100, 2)}")

Output:

detected langauge: dip	score: 99.99

Requirements

• Download AfroLID model:

    wget https://demos.dlnlp.ai/afrolid/afrolid_model.tar.gz
    tar -xf afrolid_model.tar.gz

Installation

• To install AfroLID and develop directly using pip:

    pip install -U afrolid

• To install AfroLID and develop directly GitHub repo using pip:

    pip install -U git+https://github.com/UBC-NLP/afrolid.git

• To install AfroLID and develop locally:

    git clone https://github.com/UBC-NLP/afrolid.git
    cd afrolid
    pip install .

Getting Started

The full documentation contains instructions for getting started, translation using diffrent methods, intergrate AfroLID with your code, and provides more examples.

Colab Examples

(1) Integrate AfroLID with your python code

Content	Colab link
Install AfroLID Download AfroLID's model Initial AfroLID object Get language prediction(s) Integrate with Pandas

(2) Command Line Interface

Command	Content	Colab link
afrolid_cli	Usage and Arguments Examples

Supported languages

Please refer to suported-languages

License

afrolid(-py) is Apache-2.0 licensed. The license applies to the pre-trained models as well.

Citation

If you use AfroLID toolkit or the pre-trained models for your scientific publication, or if you find the resources in this repository useful, please cite our paper as follows (to be updated):

@article{adebara2022afrolid,
  title={AfroLID: A Neural Language Identification Tool for African Languages},
  author={Adebara, Ife and Elmadany, AbdelRahim and Abdul-Mageed, Muhammad and Inciarte, Alcides Alcoba},
  booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
  month = December,
  year = "2022",
}

Acknowledgments

We gratefully acknowledge support from Canada Research Chairs (CRC), the Natural Sciences and Engineering Research Council of Canada (NSERC; RGPIN-2018-04267), the Social Sciences and Humanities Research Council of Canada (SSHRC; 435-2018-0576; 895-2020-1004; 895-2021-1008), Canadian Foundation for Innovation (CFI; 37771), Digital Research Alliance of Canada, UBC ARC-Sockeye, Advanced Micro Devices, Inc. (AMD), and Google. Any opinions, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of CRC, NSERC, SSHRC, CFI, CC, AMD, Google, or UBC ARC-Sockeye.