nghuyong/ERNIE-Pytorch
nghuyong/ERNIE-Pytorch has 930 GitHub stars and 122 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-20 22:23 UTC. Last push 2023-07-26.
Crecimiento de estrellas medido
- in 24h
- 0
- in 7 days
- Recopilando historial
- in 15 days
- Recopilando historial
- in 1 month
- Recopilando historial
Cambios netos en períodos que se solapan, no totales diarios. Los datos no disponibles se dejan en blanco. Las estrellas indican interés, no calidad.
Acerca de
ERNIE Pytorch Version
Aprendizaje automáticoREADME.md
OriginalERNIE-Pytorch
This project is to convert ERNIE from paddlepaddle to huggingface's format (in Pytorch).
News: ERNIE has been merged into huggingface/[email protected] !!
Get Started
pip install --upgrade transformers
Take ernie-1.0-base-zh as an example:
from transformers import BertTokenizer, ErnieModel
tokenizer = BertTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh")
model = ErnieModel.from_pretrained("nghuyong/ernie-1.0-base-zh")
Supported Models
| Model Name | Language | Description |
|---|---|---|
| ernie-1.0-base-zh | Chinese | Layer:12, Heads:12, Hidden:768 |
| ernie-2.0-base-en | English | Layer:12, Heads:12, Hidden:768 |
| ernie-2.0-large-en | English | Layer:24, Heads:16, Hidden:1024 |
| ernie-3.0-xbase-zh | Chinese | Layer:20, Heads:16, Hidden:1024 |
| ernie-3.0-base-zh | Chinese | Layer:12, Heads:12, Hidden:768 |
| ernie-3.0-medium-zh | Chinese | Layer:6, Heads:12, Hidden:768 |
| ernie-3.0-mini-zh | Chinese | Layer:6, Heads:12, Hidden:384 |
| ernie-3.0-micro-zh | Chinese | Layer:4, Heads:12, Hidden:384 |
| ernie-3.0-nano-zh | Chinese | Layer:4, Heads:12, Hidden:312 |
| ernie-health-zh | Chinese | Layer:12, Heads:12, Hidden:768 |
| ernie-gram-zh | Chinese | Layer:12, Heads:12, Hidden:768 |
You can find all the supported models from huggingface's model hub: huggingface.co/nghuyong, and model details from paddle's official repo: PaddleNLP and ERNIE.
Details
I want to convert the model from paddle version by myself 😉
The following will take ernie-1.0-base-zh as an example to show how to convert.
- Download the paddle-paddle version ERNIE model. Execute the following code
import paddlenlp
tokenizer = paddlenlp.transformers.ErnieTokenizer.from_pretrained("ernie-1.0-base-zh")
model = paddlenlp.transformers.ErnieForMaskedLM.from_pretrained("ernie-1.0-base-zh")
And then you will get the model in ~/.paddlenlp/models/ernie-1.0-base-zh/, move to this project path.
pip install -r requirements.txtpython convert.py- Now, a folder named
convertwill be in the project path, and there will be three files in this folder:config.json,pytorch_model.binandvocab.txt.
I want to check the calculation results before and after model conversion 😁
python test.py --task logit_check
You will get the output:
huggingface result
pool output: [-1. -1. 0.9981035 -0.9996652 -0.78173476 -1. -0.9994901 0.97012603 0.85954666 0.9854131 ]
paddle result
pool output: [-0.99999976 -0.99999976 0.9981028 -0.9996651 -0.7815545 -0.99999976 -0.9994898 0.97014064 0.8594844 0.985419 ]
It can be seen that the result of our convert version is the same with the official paddlepaddle's version.
I want to reproduce the cloze test in ERNIE1.0's paper 😆
python test.py --task cloze_check
You will get the output:
huggingface result
prediction shape: torch.Size([47, 18000])
predict result: ['西', '游', '记', '是', '中', '国', '神', '魔', '小', '说', '的', '经', '典', '之', '作', ',', '与', '《', '三', '国', '演', '义', '》', '《', '水', '浒', '传', '》', '《', '红', '楼', '梦', '》', '并', '称', '为', '中', '国', '古', '典', '四', '大', '名', '著', '。']
[CLS] logit: [-15.693626 -19.522263 -10.429456 ... -11.800728 -12.253127 -14.375117]
paddle result
prediction shape: [47, 18000]
predict result: ['西', '游', '记', '是', '中', '国', '神', '魔', '小', '说', '的', '经', '典', '之', '作', ',', '与', '《', '三', '国', '演', '义', '》', '《', '水', '浒', '传', '》', '《', '红', '楼', '梦', '》', '并', '称', '为', '中', '国', '古', '典', '四', '大', '名', '著', '。']
[CLS] logit: [-15.693538 -19.521954 -10.429307 ... -11.800765 -12.253114 -14.375412]
Citation
If you use this work in a scientific publication, I would appreciate that you can also cite the following BibTex entry:
@misc{nghuyong2019@ERNIE-Pytorch,
title={ERNIEPytorch},
author={Yong Hu},
howpublished={\url{https://github.com/nghuyong/ERNIE-Pytorch}},
year={2019}
}