superdoc/docx-corpus
superdoc/docx-corpus has 71 GitHub stars and 9 forks. GitHot measured 0 stars in 24 hours and 0 in 7 days. Last checked 2026-11-18 12:13 UTC. Last push 2026-11-11.
측정된 스타 증가
- in 24h
- 0
- in 7 days
- 0
- in 15 days
- 기록 수집 중
- in 1 month
- 기록 수집 중
서로 겹치는 기간의 순증가량이며 일별 합계가 아닙니다. 측정값이 없으면 비워 둡니다. 스타는 관심도를 나타내며 품질을 보장하지 않습니다.
소개
The largest open corpus of classified docx documents
머신러닝README.md
Original
docx-corpus
736K+ real Word documents from the public web, classified and open.
Built by SuperDoc · docxcorp.us · Hugging Face · API
[!IMPORTANT] Documents scraped before
477d1b9can end with 4 extra bytes after the ZIP data, which makes Word for Mac report unreadable content (#8). New scrapes are fixed. To check or repair older local copies, runpython3 scripts/fix_trailing_warc_bytes.py.
Get the data
# Corpus stats
curl https://api.docxcorp.us/stats
# Find documents by type, topic, language, and minimum confidence
curl "https://api.docxcorp.us/documents?type=legal&lang=en&min_confidence=0.8"
# Download a filtered set
curl "https://api.docxcorp.us/manifest?type=legal&lang=en" -o manifest.txt
wget -i manifest.txt -P ./corpus/
Pipeline
Common Crawl → cdx-filter → scrape → extract → classify → export
- cdx-filter: an AWS Lambda finds
.docxURLs in Common Crawl's indexes. - scrape: downloads each file, validates it, deduplicates it, and stores it in R2.
- extract: extracts text with Docling and detects the language with lingua.
- classify: labels type and topic with fine-tuned XLM-RoBERTa models.
- export: publishes the metadata to Hugging Face as Parquet.
Quick start
Requires Bun and Docker.
bun install
cp .env.example .env
docker compose up -d # Local Postgres
bun run corpus status # Pipeline stats
Commands
bun run corpus cdx-filter --latest 1 # Filter the newest crawl not yet filtered
bun run corpus crawls # List filtered crawls in R2
bun run corpus scrape --crawl 3 # Scrape the latest 3 crawls
bun run corpus scrape --retry-failed # Retry failed URLs in the latest crawl
bun run corpus extract # Extract text from pending documents
bun run corpus classify # Classify pending documents
bun run corpus classify --modal # Classify on Modal GPUs
bun run corpus export --push # Push to Hugging Face
bun run corpus status # Show pipeline stats
Run bun run corpus <command> --help for every option.
Configuration
Set these in .env. See .env.example for the rest.
| Variable | Needed for |
|---|---|
DATABASE_URL |
Every command |
CLOUDFLARE_ACCOUNT_ID, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, R2_BUCKET_NAME |
scrape, crawls, and cdx-filter without --crawl. Without them, extract uses STORAGE_PATH (default ./corpus). |
AWS_PROFILE or AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY |
cdx-filter. Needs lambda:InvokeFunction on the cdx-filter function. |
ANTHROPIC_API_KEY |
Labeling training data for classification |
Development
bun run lint
bun run typecheck
bun test
The scraper's database tests run only when TEST_DATABASE_URL is set. They delete every row in documents, so point it at a separate database:
docker exec docx-corpus-postgres-1 createdb -U postgres docx_corpus_test 2>/dev/null || true
docker exec -i docx-corpus-postgres-1 psql -U postgres -d docx_corpus_test < db/schema.sql
TEST_DATABASE_URL=postgres://postgres:postgres@localhost:5432/docx_corpus_test bun test
| Path | What |
|---|---|
apps/cli |
The corpus command |
apps/cdx-filter |
Lambda that filters Common Crawl indexes (deploy) |
apps/api |
Cloudflare Worker for api.docxcorp.us |
apps/site |
Static Astro site for docxcorp.us |
packages/* |
Scraper, extractor, and shared database and storage code |
scripts/classification |
Classifier training and inference (details) |
scripts/export-hf.py |
Hugging Face export |
db/schema.sql |
Postgres schema; db/migrations/ for existing databases |
Docker
The image idles so you can docker exec into it for long runs. Its .env needs a DATABASE_URL the container can reach (localhost is the container itself). Without R2 credentials, extract uses /app/corpus, so mount a volume there.
docker build -t docx-corpus .
docker run -d --name docx-corpus --env-file .env -v "$PWD/corpus:/app/corpus" docx-corpus
docker exec -it docx-corpus bun run corpus scrape --crawl 3
Takedown requests
If you own a document and want it removed, email [email protected] with its hash or URL and proof of ownership. Requests are processed within 7 days.
License
MIT