superdoc/docx-corpus
superdoc/docx-corpus has 71 GitHub stars and 9 forks. GitHot measured 0 stars in 24 hours and 0 in 7 days. Last checked 2026-11-17 16:09 UTC. Last push 2026-11-03.
Measured star growth
- in 24h
- 0
- in 7 days
- 0
- in 15 days
- Update delayed
- in 1 month
- Update delayed
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
The largest open corpus of classified docx documents
Machine LearningREADME.md
Original
docx-corpus
736K+ real Word documents from the public web, classified and open.
Built by SuperDoc · docxcorp.us · Hugging Face · API
[!IMPORTANT] Documents scraped before
477d1b9can end with 4 extra bytes after the ZIP data, which makes Word for Mac report unreadable content (#8). New scrapes are fixed. To check or repair older local copies, runpython3 scripts/fix_trailing_warc_bytes.py.
Get the data
# Corpus stats
curl https://api.docxcorp.us/stats
# Find documents by type, topic, language, and minimum confidence
curl "https://api.docxcorp.us/documents?type=legal&lang=en&min_confidence=0.8"
# Download a filtered set
curl "https://api.docxcorp.us/manifest?type=legal&lang=en" -o manifest.txt
wget -i manifest.txt -P ./corpus/
Pipeline
Common Crawl → cdx-filter → scrape → extract → classify → export
- cdx-filter: an AWS Lambda finds
.docxURLs in Common Crawl's indexes. - scrape: downloads each file, validates it, deduplicates it, and stores it in R2.
- extract: extracts text with Docling and detects the language with lingua.
- classify: labels type and topic with fine-tuned XLM-RoBERTa models.
- export: publishes the metadata to Hugging Face as Parquet.
Quick start
Requires Bun and Docker.
bun install
cp .env.example .env
docker compose up -d # Local Postgres
bun run corpus status # Pipeline stats
Commands
bun run corpus cdx-filter --latest 1 # Filter the newest crawl not yet filtered
bun run corpus crawls # List filtered crawls in R2
bun run corpus scrape --crawl 3 # Scrape the latest 3 crawls
bun run corpus scrape --retry-failed # Retry failed URLs in the latest crawl
bun run corpus extract # Extract text from pending documents
bun run corpus classify # Classify pending documents
bun run corpus classify --modal # Classify on Modal GPUs
bun run corpus export --push # Push to Hugging Face
bun run corpus status # Show pipeline stats
Run bun run corpus <command> --help for every option.
Configuration
Set these in .env. See .env.example for the rest.
| Variable | Needed for |
|---|---|
DATABASE_URL |
Every command |
CLOUDFLARE_ACCOUNT_ID, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, R2_BUCKET_NAME |
scrape, crawls, and cdx-filter without --crawl. Without them, extract uses STORAGE_PATH (default ./corpus). |
AWS_PROFILE or AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY |
cdx-filter. Needs lambda:InvokeFunction on the cdx-filter function. |
ANTHROPIC_API_KEY |
Labeling training data for classification |
Development
bun run lint
bun run typecheck
bun test
The scraper's database tests run only when TEST_DATABASE_URL is set. They delete every row in documents, so point it at a separate database:
docker exec docx-corpus-postgres-1 createdb -U postgres docx_corpus_test 2>/dev/null || true
docker exec -i docx-corpus-postgres-1 psql -U postgres -d docx_corpus_test < db/schema.sql
TEST_DATABASE_URL=postgres://postgres:postgres@localhost:5432/docx_corpus_test bun test
| Path | What |
|---|---|
apps/cli |
The corpus command |
apps/cdx-filter |
Lambda that filters Common Crawl indexes (deploy) |
apps/api |
Cloudflare Worker for api.docxcorp.us |
apps/site |
Static Astro site for docxcorp.us |
packages/* |
Scraper, extractor, and shared database and storage code |
scripts/classification |
Classifier training and inference (details) |
scripts/export-hf.py |
Hugging Face export |
db/schema.sql |
Postgres schema; db/migrations/ for existing databases |
Docker
The image idles so you can docker exec into it for long runs. Its .env needs a DATABASE_URL the container can reach (localhost is the container itself). Without R2 credentials, extract uses /app/corpus, so mount a volume there.
docker build -t docx-corpus .
docker run -d --name docx-corpus --env-file .env -v "$PWD/corpus:/app/corpus" docx-corpus
docker exec -it docx-corpus bun run corpus scrape --crawl 3
Takedown requests
If you own a document and want it removed, email [email protected] with its hash or URL and proof of ownership. Requests are processed within 7 days.
License
MIT