Skip to content
Projects

superdoc/docx-corpus has 71 GitHub stars and 9 forks. GitHot measured 0 stars in 24 hours and 0 in 7 days. Last checked 2026-11-17 16:09 UTC. Last push 2026-11-03.

How these counts are measured

Measured star growth

in 24h
0
in 7 days
0
in 15 days
Update delayed
in 1 month
Update delayed

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

The largest open corpus of classified docx documents

Machine Learning

README.md

Original

docx-corpus

docx-corpus

736K+ real Word documents from the public web, classified and open.
Built by SuperDoc · docxcorp.us · Hugging Face · API

CLI CDX Filter License: MIT

[!IMPORTANT] Documents scraped before 477d1b9 can end with 4 extra bytes after the ZIP data, which makes Word for Mac report unreadable content (#8). New scrapes are fixed. To check or repair older local copies, run python3 scripts/fix_trailing_warc_bytes.py.

Get the data

# Corpus stats
curl https://api.docxcorp.us/stats

# Find documents by type, topic, language, and minimum confidence
curl "https://api.docxcorp.us/documents?type=legal&lang=en&min_confidence=0.8"

# Download a filtered set
curl "https://api.docxcorp.us/manifest?type=legal&lang=en" -o manifest.txt
wget -i manifest.txt -P ./corpus/

Pipeline

Common Crawl → cdx-filter → scrape → extract → classify → export
  1. cdx-filter: an AWS Lambda finds .docx URLs in Common Crawl's indexes.
  2. scrape: downloads each file, validates it, deduplicates it, and stores it in R2.
  3. extract: extracts text with Docling and detects the language with lingua.
  4. classify: labels type and topic with fine-tuned XLM-RoBERTa models.
  5. export: publishes the metadata to Hugging Face as Parquet.

Quick start

Requires Bun and Docker.

bun install
cp .env.example .env
docker compose up -d       # Local Postgres
bun run corpus status      # Pipeline stats

Commands

bun run corpus cdx-filter --latest 1      # Filter the newest crawl not yet filtered
bun run corpus crawls                     # List filtered crawls in R2
bun run corpus scrape --crawl 3           # Scrape the latest 3 crawls
bun run corpus scrape --retry-failed      # Retry failed URLs in the latest crawl
bun run corpus extract                    # Extract text from pending documents
bun run corpus classify                   # Classify pending documents
bun run corpus classify --modal           # Classify on Modal GPUs
bun run corpus export --push              # Push to Hugging Face
bun run corpus status                     # Show pipeline stats

Run bun run corpus <command> --help for every option.

Configuration

Set these in .env. See .env.example for the rest.

Variable Needed for
DATABASE_URL Every command
CLOUDFLARE_ACCOUNT_ID, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, R2_BUCKET_NAME scrape, crawls, and cdx-filter without --crawl. Without them, extract uses STORAGE_PATH (default ./corpus).
AWS_PROFILE or AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY cdx-filter. Needs lambda:InvokeFunction on the cdx-filter function.
ANTHROPIC_API_KEY Labeling training data for classification

Development

bun run lint
bun run typecheck
bun test

The scraper's database tests run only when TEST_DATABASE_URL is set. They delete every row in documents, so point it at a separate database:

docker exec docx-corpus-postgres-1 createdb -U postgres docx_corpus_test 2>/dev/null || true
docker exec -i docx-corpus-postgres-1 psql -U postgres -d docx_corpus_test < db/schema.sql
TEST_DATABASE_URL=postgres://postgres:postgres@localhost:5432/docx_corpus_test bun test
Path What
apps/cli The corpus command
apps/cdx-filter Lambda that filters Common Crawl indexes (deploy)
apps/api Cloudflare Worker for api.docxcorp.us
apps/site Static Astro site for docxcorp.us
packages/* Scraper, extractor, and shared database and storage code
scripts/classification Classifier training and inference (details)
scripts/export-hf.py Hugging Face export
db/schema.sql Postgres schema; db/migrations/ for existing databases

Docker

The image idles so you can docker exec into it for long runs. Its .env needs a DATABASE_URL the container can reach (localhost is the container itself). Without R2 credentials, extract uses /app/corpus, so mount a volume there.

docker build -t docx-corpus .
docker run -d --name docx-corpus --env-file .env -v "$PWD/corpus:/app/corpus" docx-corpus
docker exec -it docx-corpus bun run corpus scrape --crawl 3

Takedown requests

If you own a document and want it removed, email [email protected] with its hash or URL and proof of ownership. Requests are processed within 7 days.

License

MIT