scrapfly/scrapfly-scrapers
scrapfly/scrapfly-scrapers has 1,126 GitHub stars and 207 forks. GitHot measured +9 in 7 days. Last checked 2026-11-11 00:09 UTC. Last push 2026-11-02.
Scalable Python web scraping scripts for +40 popular domains
Measured star growth
- in 24h
- Update delayed
- in 7 days
- +9
- in 15 days
- Collecting history
- in 1 month
- Collecting history
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
README.md
OriginalScrapFly Scrapers 🕷️
English · 简体中文 · 日本語 · Español · Русский
This repository contains educational example scrapers for popular web scraping targets using the ScrapFly web scraping API and Python.
Most scrapers use a simple web scraping stack:
- Python version 3.10+
- Scrapfly's Python SDK for sending HTTP requests, getting past blocking and parsing the HTML with the built-in parsel selector.
- asyncio for writing concurrent code using the async/await syntax.
- JMESPath and nested-lookup for JSON parsing when needed.
- loguru for logging.
To learn more about web scraping, see our full tutorials on how to scrape these targets (and many others) in the scrapeguide directory.
For AI agents and LLM tools, llms.txt lists every scraper with its functions and sample output, and explains how to run a scraper and how to write a new one.
Quick start
Every scraper lives in its own folder and runs on its own (Python 3.10+ and Poetry):
git clone https://github.com/scrapfly/scrapfly-scrapers.git
cd scrapfly-scrapers/airbnb-scraper # any scraper folder from the list below
export SCRAPFLY_KEY="YOUR SCRAPFLY KEY" # from https://scrapfly.io/dashboard
poetry install --no-root
poetry run python run.py # example scrape, saves its output in ./results
poetry run pytest test.py # the scraper's tests
Each scraper folder's README lists what it scrapes.
List of Scrapers
Below is the list of available web scrapers for the supported domains along with their scrape guide, sample datasets, and status. 👇
The status comes from the nightly test run, which also orders the table: scrapers whose tests pass come first, failing and untested ones last.
A badge reads passing (every test passed), N/M passing (some tests passed), failing (none passed) or not run; click it to see that scraper's last results, with a link to each test's log.
Fair Use and Legal Disclaimer
This repository contains educational reference material to illustrate how accessible web scraping can be and the provided programs are not intended to be used in web scraping production. That being said, Scrapfly team is constantly updating and improving all of this code for optimal experience.
Scrapfly does not offer legal advice and as always, consult a lawyer when creating programs that interact with other people's websites though here's a good general intro of what NOT to do:
- Do not store PII (personally identifiable information) of EU citizens who are protected by GDPR.
- Do not scrape and repurpose entire public datasets which can be protected by database protection laws in some countries.
- Do not scrape at rates that could damage the website and scrape only publicly available data.
License
This repository is licensed under the Non-Profit Open Software License 3.0.
Explore related projects
n8n-io/n8n
Fair-code workflow automation platform with native AI capabilities. Combine visual building with custom code, self-host or cloud, 400+ integrations.
210,087 starslanggenius/dify
Build Agentic workflows, RAG pipelines, with rich AI model and tool support on one collaborative workspace. Deploy on cloud, VPC, or self-hosted, so teams move from prototype to production without rebuilding the stack.
160,773 starsmicrosoft/PowerToys
Microsoft PowerToys is a collection of utilities that supercharge productivity and customization on Windows
140,833 starsstablyai/orca
Orca is the place for working with a fleet of agents (e.g. Grok Bot, Claude Code). Run any coding agent with your own subscription. Available on desktop, mobile and remote runtime.
121,781 stars