ksanjeev284/reddit-universal-scraper
ksanjeev284/reddit-universal-scraper has 620 GitHub stars and 95 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-19 23:42 UTC. Last push 2026-10-22.
What you can do
- 📊 Full Scraping - Available posts, loaded comments, images, videos, galleries
- 📈 Web Dashboard - Beautiful Streamlit UI with 7 tabs
- 🚀 REST API - Connect Metabase, Grafana, DuckDB
- 🔌 Plugin System - Extensible post-processing (sentiment, dedupe, keywords)
Crecimiento de estrellas medido
- in 24h
- 0
- in 7 days
- Recopilando historial
- in 15 days
- Recopilando historial
- in 1 month
- Recopilando historial
Cambios netos en períodos que se solapan, no totales diarios. Los datos no disponibles se dejan en blanco. Las estrellas indican interés, no calidad.
Acerca de
Universal Reddit Scraper - Works on any Subreddit or User
AprendizajeREADME.md
Original🤖 Universal Reddit Scraper Suite
A full-featured Reddit scraper with analytics dashboard, REST API, scheduled scraping, plugins, and more. Two collection modes: free browser-page capture, or approved OAuth API access.
✨ Features
| Feature | Description |
|---|---|
| 📊 Full Scraping | Available posts, loaded comments, images, videos, galleries |
| 📈 Web Dashboard | Beautiful Streamlit UI with 7 tabs |
| 🚀 REST API | Connect Metabase, Grafana, DuckDB |
| 🔌 Plugin System | Extensible post-processing (sentiment, dedupe, keywords) |
| 📋 Job Tracking | Full history with status, duration, errors |
| 🧪 Dry Run Mode | Test scrape rules without saving data |
| 📦 Parquet Export | Analytics-ready format for DuckDB/warehouses |
| 😀 Sentiment Analysis | Analyze post/comment sentiment |
| 📅 Scheduled Scraping | Cron-style job scheduling |
| 📧 Notifications | Discord & Telegram alerts |
| 🗄️ SQLite Database | Structured storage with auto-backup |
🚀 Quick Start
# Install dependencies
pip install -r requirements.txt
# Configure approved API credentials (see .env.example)
# Copy .env.example to .env only if you do not already have .env
python main.py --check-access
# Scrape a subreddit
python main.py python --mode full --limit 100
# Launch dashboard
python main.py --dashboard
# Opens at http://localhost:8501
📋 Requirements
- Python 3.10+
- ffmpeg (optional, for video with audio)
# Windows (via chocolatey)
choco install ffmpeg
# macOS
brew install ffmpeg
# Ubuntu/Debian
sudo apt install ffmpeg
Free browser extension
The Browser Collector Companion works independently in Chrome/Edge while you browse Reddit signed in. It captures loaded posts, comments and media links, supports filters and session limits, and exports JSON or CSV without paid services or API keys. It can also send collections to the application for database storage, dashboard analytics and search. Collect only data you are authorized to use.
Load the extension/ folder through Developer mode → Load unpacked, reload Reddit, then use Start capture or Capture this page. It observes loaded content as you browse; it does not retrieve an entire archive or automatically expand unloaded comments.
# Connect the extension to the application
python main.py --api
# In another terminal: copy this pairing token into the extension
python main.py --bridge-token
# Or import a standalone JSON export without starting the API
python main.py --import-browser path/to/export.json
# Build the unpackable release ZIP
python scripts/package_extension.py
Pairing uses optional localhost permissions and a local token. The browser's Reddit cookies remain in the browser. Python proxy support remains available, including stable country/session targeting; the extension uses your existing browser proxy settings.
Reddit access and reliability
Reddit blocks unauthenticated API traffic. The direct Python scraper uses oauth.reddit.com with a descriptive User-Agent, cached OAuth tokens, token renewal, bounded retries and shared request pacing. Both sync and async scrapers respect Retry-After and Reddit rate-limit headers. API requests are serialized within each client; media downloads remain concurrent. Independent processes share your app quota, so schedule them accordingly.
See Reddit access setup for credentials, restrictions, troubleshooting and the Developer Platform migration timeline. Missing credentials, denied access, HTML responses and exhausted retries produce explicit failures. CLI failures exit with status 1; jobs preserve collected posts and report failed status when a later request fails.
An optional stable network proxy can be configured with PROXY_URL or --proxy. ScrapingAnt country/session targeting is available through PROXY_COUNTRY / --proxy-country and PROXY_SESSION_ID / --proxy-session. --proxy none disables it. Mirrors, browser impersonation, warm-up requests and automatic proxy rotation are no longer used. A proxy does not grant API approval.
Listings expose a limited window of posts, not a complete historical archive. Comment collection includes the initially loaded tree up to the configured depth; more placeholders are not expanded. Deleted or restricted content cannot be recovered.
📖 All Commands
🔄 Scraping
# Collect available posts + media + loaded comments
python main.py delhi --mode full --limit 100
# Fast history-only (no media/comments)
python main.py delhi --mode history --limit 500
# Live monitor (checks every 5 min)
python main.py delhi --mode monitor
# Scrape a user's posts
python main.py spez --user --mode full --limit 50
# Skip media or comments
python main.py delhi --no-media --limit 200
python main.py delhi --no-comments --limit 200
🧪 Dry Run Mode
Test scrape rules without saving any data:
python main.py python --mode full --limit 50 --dry-run
Output:
🧪 DRY RUN MODE - No data will be saved
🧪 DRY RUN COMPLETE!
📊 Would scrape: 100 posts
💬 Would scrape: 245 comments
🔌 Plugins
Enable post-processing plugins:
# List available plugins
python main.py --list-plugins
# Run with plugins enabled
python main.py python --mode full --plugins
Built-in Plugins:
| Plugin | Description |
|---|---|
sentiment_tagger |
Adds sentiment scores to posts |
deduplicator |
Removes duplicate posts |
keyword_extractor |
Extracts top keywords |
Create custom plugins in plugins/ folder.
📊 Dashboard
python main.py --dashboard
# Opens at http://localhost:8501
Dashboard Tabs:
- 📊 Overview - Stats & charts
- 📈 Analytics - Sentiment & keywords
- 🔍 Search - Query scraped data
- 💬 Comments - Comment analysis
- ⚙️ Scraper - Start new scrapes
- 📋 Job History - View all jobs
- 🔌 Integrations - API, export, plugins
🚀 REST API
python main.py --api
# API at http://localhost:8000
# Docs at http://localhost:8000/docs
Endpoints:
| Endpoint | Description |
|---|---|
GET /posts |
List posts with filters |
GET /comments |
List comments |
GET /subreddits |
All scraped subreddits |
GET /jobs |
Job history |
GET /query?sql=... |
Raw SQL queries |
GET /grafana/query |
Grafana time-series |
📦 Export & Maintenance
# Export to Parquet (for DuckDB/warehouses)
python main.py --export-parquet python
# View job history
python main.py --job-history
# Backup database
python main.py --backup
# Optimize database
python main.py --vacuum
📅 Scheduled Scraping
# Scrape every 60 minutes
python main.py --schedule delhi --every 60
# With options
python main.py --schedule delhi --every 30 --mode full --limit 50
🔍 Search & Analytics
# Search scraped data
python main.py --search "credit card" --min-score 100
# Run sentiment analysis
python main.py --analyze delhi --sentiment
# Extract keywords
python main.py --analyze delhi --keywords
🐳 Docker
Quick Start
# Build
docker build -t reddit-scraper .
# Run scrape
docker run --env-file .env -v ./data:/app/data reddit-scraper python --limit 100
# Run with plugins
docker run --env-file .env -v ./data:/app/data reddit-scraper python --plugins
Docker Compose (Full Stack)
# Start API + Dashboard
docker-compose up -d
# Access:
# Dashboard: http://localhost:8501
# API: http://localhost:8000/docs
Deploy to AWS/VPS
# SSH into your server
ssh user@your-server-ip
# Clone repo
git clone https://github.com/ksanjeev284/reddit-universal-scraper.git
cd reddit-universal-scraper
# Start services
docker-compose up -d
# Open firewall ports
sudo ufw allow 8000
sudo ufw allow 8501
Access:
http://your-server-ip:8501→ Dashboardhttp://your-server-ip:8000/docs→ API
🔗 External Integrations
Metabase
- Start API:
python main.py --api - Add HTTP datasource:
http://localhost:8000 - Query:
/posts?subreddit=python&limit=100
Grafana
- Install "JSON API" or "Infinity" plugin
- Add datasource:
http://localhost:8000 - Use
/grafana/queryfor time-series
DuckDB
import duckdb
# Export to Parquet first
# python main.py --export-parquet python
# Query directly
duckdb.query("SELECT * FROM 'data/parquet/*.parquet'").df()
📁 Project Structure
reddit-scraper/
├── main.py # CLI entry point
├── config.py # Settings
├── analytics/ # Sentiment & keywords
├── alerts/ # Discord/Telegram
├── api/ # REST API server
├── dashboard/ # Streamlit UI
├── export/ # Database & exports
├── plugins/ # Post-processing plugins
├── scheduler/ # Cron scheduling
├── search/ # Search engine
└── data/
├── r_subreddit/ # Scraped data
├── backups/ # DB backups
└── parquet/ # Parquet exports
📊 Data Output
posts.csv
| Column | Description |
|---|---|
| id | Reddit post ID |
| title | Post title |
| author | Username |
| score | Net upvotes |
| num_comments | Comment count |
| post_type | text/image/video/gallery |
| selftext | Post body |
| sentiment_score | -1.0 to 1.0 (with plugins) |
comments.csv
| Column | Description |
|---|---|
| comment_id | Comment ID |
| post_permalink | Parent post |
| author | Username |
| body | Comment text |
| score | Upvotes |
⚙️ Environment Variables
# Notifications
export DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..."
export TELEGRAM_BOT_TOKEN="123456:ABC..."
export TELEGRAM_CHAT_ID="987654321"
📜 License
MIT License - Feel free to use, modify, and distribute.
🤝 Contributing
Pull requests welcome! For major changes, please open an issue first.