Skip to content
Projects

doctolib-lab/doctobert has 60 GitHub stars and 7 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-16 22:53 UTC. Last push 2026-07-21.

How these counts are measured

Measured star growth

in 24h
0
in 7 days
Collecting history
in 15 days
Collecting history
in 1 month
Collecting history

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

A medical encoder, pretrained from scratch on curated and LLM-rephrased medical web data

AI

README.md

Original

DoctoBERT

DoctoBERT

🤗 Blog | 📄 Paper | 💻 Code | 🌐 FineMed | 🩺 DoctoBERT

DoctoBERT is a family of medical encoders, pretrained from scratch on medical text curated from the open web, which provides the scale, source, and stylistic diversity that hand-curated corpora lack.

This repository holds the reproducible data-curation pipeline, which combines two complementary levers: multi-axis annotation that scores each document along three axes (subdomain, educational quality, and medical-term density) to select the documents most valuable for pretraining, and signal-amplifying rephrasing that uses an LLM to rewrite documents into denser variants with broader entity contexts to improve their training utility. Model pretraining builds on the ModernBERT codebase, included here as the ModernBERT/ submodule.

Installation

Clone with the ModernBERT submodule:

git clone --recurse-submodules https://github.com/doctolib-lab/doctobert.git
cd doctobert

Install dependencies:

conda env create -f environment.yaml   # creates the `data-proc` env
conda activate data-proc

Data curation

The corpus is built in three stages, listed in data-flow order. Each script has a matching .slurm runner.

data_curation/
├── data_processing/              # acquire, mix, chunk, stats
│   ├── download_hf.slurm              # pull source corpora (FineWeb-2, FinePDFs, FineWiki)
│   ├── split_document.py              # split long docs into token-bounded chunks
│   ├── mix_and_sample.py              # build + sample the filtered/rephrased training mixture
│   └── stats_dataset.py               # token / document counts
├── data_classification/          # distill the 3 annotators, then apply + filter at scale
│   ├── llm_annotate.py                # distill labels from an LLM teacher (quality / topic / entities)
│   ├── postprocess_llm_annotation.py  # parse + normalize raw LLM labels into training data
│   ├── train_classifier.py            # train a text classifier (edu-quality scorer / subdomain)
│   ├── train_gliner2.py               # train the GLiNER medical-entity extractor (term density)
│   ├── gliner_annotate.py             # apply the entity extractor across the corpus
│   ├── run_dataset_classifier.py      # apply the quality + subdomain classifiers at scale
│   ├── filter_by_domain.py            # prefilter to medical-domain documents
│   ├── eval_classifier.py             # evaluate the trained classifiers
│   └── eval_ner.py                    # evaluate the entity extractor
└── data_synthesis/               # synthetic generation + MGA rephrasing + postprocess
    ├── llm_generate.py                # generate synthetic docs (clinical case / dialogue / EHR / ICD / vocab)
    ├── llm_rewrite.py                 # two-stage MGA rephrasing
    ├── postprocess_extract.py         # parse LLM JSON, language + repetition filter
    └── postprocess_datatrove.py       # deserialize entities to struct columns, reshard

Model building

MDS conversion, tokenizer training, and checkpoint conversion. Pretraining itself lives in the ModernBERT/ submodule, with the DoctoBERT configs under ModernBERT/yamls/doctobert_fr/.

model_building/
├── mds_conversion/               # corpus to training shards
│   └── convert_to_mds.py              # parquet to MosaicML Streaming (MDS) shards
├── tokenizer/                    # build + evaluate the SentencePiece tokenizer
│   ├── prep_spm_input.py              # assemble the tokenizer training corpus
│   ├── minhash_deduplication.py       # MinHash-deduplicate the corpus
│   ├── train_spm.py                   # train the SentencePiece BPE tokenizer (32k / 50k vocab)
│   ├── convert_spm_to_hf.py           # convert the SPM model to a HF tokenizer
│   ├── add_tokens.py                  # add special tokens
│   ├── trim_tokens.py                 # trim unused vocab
│   └── eval_tokenizer.py              # fertility / coverage checks
└── model/                        # checkpoint merging + format conversion
    ├── merge_sharded_checkpoints.py   # merge FSDP-sharded training checkpoints
    ├── convert_flexbert_to_hf.py      # FlexBERT/Composer checkpoint to Hugging Face format
    └── convert_hf_to_flexbert.py      # HF model to FlexBERT format