Skip to content
Proyectos

ksanjeev284/reddit-universal-scraper

Ver en GitHub

ksanjeev284/reddit-universal-scraper has 620 GitHub stars and 95 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-19 23:42 UTC. Last push 2026-10-22.

Cómo se miden estas cifras

What you can do

  • 📊 Full Scraping - Available posts, loaded comments, images, videos, galleries
  • 📈 Web Dashboard - Beautiful Streamlit UI with 7 tabs
  • 🚀 REST API - Connect Metabase, Grafana, DuckDB
  • 🔌 Plugin System - Extensible post-processing (sentiment, dedupe, keywords)

Crecimiento de estrellas medido

in 24h
0
in 7 days
Recopilando historial
in 15 days
Recopilando historial
in 1 month
Recopilando historial

Cambios netos en períodos que se solapan, no totales diarios. Los datos no disponibles se dejan en blanco. Las estrellas indican interés, no calidad.

Acerca de

Universal Reddit Scraper - Works on any Subreddit or User

Aprendizaje

README.md

Original

🤖 Universal Reddit Scraper Suite

Docker Build & Publish

A full-featured Reddit scraper with analytics dashboard, REST API, scheduled scraping, plugins, and more. Two collection modes: free browser-page capture, or approved OAuth API access.

image

✨ Features

Feature Description
📊 Full Scraping Available posts, loaded comments, images, videos, galleries
📈 Web Dashboard Beautiful Streamlit UI with 7 tabs
🚀 REST API Connect Metabase, Grafana, DuckDB
🔌 Plugin System Extensible post-processing (sentiment, dedupe, keywords)
📋 Job Tracking Full history with status, duration, errors
🧪 Dry Run Mode Test scrape rules without saving data
📦 Parquet Export Analytics-ready format for DuckDB/warehouses
😀 Sentiment Analysis Analyze post/comment sentiment
📅 Scheduled Scraping Cron-style job scheduling
📧 Notifications Discord & Telegram alerts
🗄️ SQLite Database Structured storage with auto-backup

🚀 Quick Start

# Install dependencies
pip install -r requirements.txt

# Configure approved API credentials (see .env.example)
# Copy .env.example to .env only if you do not already have .env
python main.py --check-access

# Scrape a subreddit
python main.py python --mode full --limit 100

# Launch dashboard
python main.py --dashboard
# Opens at http://localhost:8501

📋 Requirements

  • Python 3.10+
  • ffmpeg (optional, for video with audio)
# Windows (via chocolatey)
choco install ffmpeg

# macOS
brew install ffmpeg

# Ubuntu/Debian
sudo apt install ffmpeg

Free browser extension

The Browser Collector Companion works independently in Chrome/Edge while you browse Reddit signed in. It captures loaded posts, comments and media links, supports filters and session limits, and exports JSON or CSV without paid services or API keys. It can also send collections to the application for database storage, dashboard analytics and search. Collect only data you are authorized to use.

Load the extension/ folder through Developer mode → Load unpacked, reload Reddit, then use Start capture or Capture this page. It observes loaded content as you browse; it does not retrieve an entire archive or automatically expand unloaded comments.

# Connect the extension to the application
python main.py --api
# In another terminal: copy this pairing token into the extension
python main.py --bridge-token

# Or import a standalone JSON export without starting the API
python main.py --import-browser path/to/export.json

# Build the unpackable release ZIP
python scripts/package_extension.py

Pairing uses optional localhost permissions and a local token. The browser's Reddit cookies remain in the browser. Python proxy support remains available, including stable country/session targeting; the extension uses your existing browser proxy settings.

Reddit access and reliability

Reddit blocks unauthenticated API traffic. The direct Python scraper uses oauth.reddit.com with a descriptive User-Agent, cached OAuth tokens, token renewal, bounded retries and shared request pacing. Both sync and async scrapers respect Retry-After and Reddit rate-limit headers. API requests are serialized within each client; media downloads remain concurrent. Independent processes share your app quota, so schedule them accordingly.

See Reddit access setup for credentials, restrictions, troubleshooting and the Developer Platform migration timeline. Missing credentials, denied access, HTML responses and exhausted retries produce explicit failures. CLI failures exit with status 1; jobs preserve collected posts and report failed status when a later request fails.

An optional stable network proxy can be configured with PROXY_URL or --proxy. ScrapingAnt country/session targeting is available through PROXY_COUNTRY / --proxy-country and PROXY_SESSION_ID / --proxy-session. --proxy none disables it. Mirrors, browser impersonation, warm-up requests and automatic proxy rotation are no longer used. A proxy does not grant API approval.

Listings expose a limited window of posts, not a complete historical archive. Comment collection includes the initially loaded tree up to the configured depth; more placeholders are not expanded. Deleted or restricted content cannot be recovered.


📖 All Commands

🔄 Scraping

# Collect available posts + media + loaded comments
python main.py delhi --mode full --limit 100

# Fast history-only (no media/comments)
python main.py delhi --mode history --limit 500

# Live monitor (checks every 5 min)
python main.py delhi --mode monitor

# Scrape a user's posts
python main.py spez --user --mode full --limit 50

# Skip media or comments
python main.py delhi --no-media --limit 200
python main.py delhi --no-comments --limit 200

🧪 Dry Run Mode

Test scrape rules without saving any data:

python main.py python --mode full --limit 50 --dry-run

Output:

🧪 DRY RUN MODE - No data will be saved
🧪 DRY RUN COMPLETE!
   📊 Would scrape: 100 posts
   💬 Would scrape: 245 comments

🔌 Plugins

Enable post-processing plugins:

# List available plugins
python main.py --list-plugins

# Run with plugins enabled
python main.py python --mode full --plugins

Built-in Plugins:

Plugin Description
sentiment_tagger Adds sentiment scores to posts
deduplicator Removes duplicate posts
keyword_extractor Extracts top keywords

Create custom plugins in plugins/ folder.

📊 Dashboard

python main.py --dashboard
# Opens at http://localhost:8501

Dashboard Tabs:

  • 📊 Overview - Stats & charts
  • 📈 Analytics - Sentiment & keywords
  • 🔍 Search - Query scraped data
  • 💬 Comments - Comment analysis
  • ⚙️ Scraper - Start new scrapes
  • 📋 Job History - View all jobs
  • 🔌 Integrations - API, export, plugins

🚀 REST API

python main.py --api
# API at http://localhost:8000
# Docs at http://localhost:8000/docs

Endpoints:

Endpoint Description
GET /posts List posts with filters
GET /comments List comments
GET /subreddits All scraped subreddits
GET /jobs Job history
GET /query?sql=... Raw SQL queries
GET /grafana/query Grafana time-series

📦 Export & Maintenance

# Export to Parquet (for DuckDB/warehouses)
python main.py --export-parquet python

# View job history
python main.py --job-history

# Backup database
python main.py --backup

# Optimize database
python main.py --vacuum

📅 Scheduled Scraping

# Scrape every 60 minutes
python main.py --schedule delhi --every 60

# With options
python main.py --schedule delhi --every 30 --mode full --limit 50

🔍 Search & Analytics

# Search scraped data
python main.py --search "credit card" --min-score 100

# Run sentiment analysis
python main.py --analyze delhi --sentiment

# Extract keywords
python main.py --analyze delhi --keywords

🐳 Docker

Quick Start

# Build
docker build -t reddit-scraper .

# Run scrape
docker run --env-file .env -v ./data:/app/data reddit-scraper python --limit 100

# Run with plugins
docker run --env-file .env -v ./data:/app/data reddit-scraper python --plugins

Docker Compose (Full Stack)

# Start API + Dashboard
docker-compose up -d

# Access:
# Dashboard: http://localhost:8501
# API: http://localhost:8000/docs

Deploy to AWS/VPS

# SSH into your server
ssh user@your-server-ip

# Clone repo
git clone https://github.com/ksanjeev284/reddit-universal-scraper.git
cd reddit-universal-scraper

# Start services
docker-compose up -d

# Open firewall ports
sudo ufw allow 8000
sudo ufw allow 8501

Access:

  • http://your-server-ip:8501 → Dashboard
  • http://your-server-ip:8000/docs → API

🔗 External Integrations

Metabase

  1. Start API: python main.py --api
  2. Add HTTP datasource: http://localhost:8000
  3. Query: /posts?subreddit=python&limit=100

Grafana

  1. Install "JSON API" or "Infinity" plugin
  2. Add datasource: http://localhost:8000
  3. Use /grafana/query for time-series

DuckDB

import duckdb

# Export to Parquet first
# python main.py --export-parquet python

# Query directly
duckdb.query("SELECT * FROM 'data/parquet/*.parquet'").df()

📁 Project Structure

reddit-scraper/
├── main.py              # CLI entry point
├── config.py            # Settings
├── analytics/           # Sentiment & keywords
├── alerts/              # Discord/Telegram
├── api/                 # REST API server
├── dashboard/           # Streamlit UI
├── export/              # Database & exports
├── plugins/             # Post-processing plugins
├── scheduler/           # Cron scheduling
├── search/              # Search engine
└── data/
    ├── r_subreddit/     # Scraped data
    ├── backups/         # DB backups
    └── parquet/         # Parquet exports

📊 Data Output

posts.csv

Column Description
id Reddit post ID
title Post title
author Username
score Net upvotes
num_comments Comment count
post_type text/image/video/gallery
selftext Post body
sentiment_score -1.0 to 1.0 (with plugins)

comments.csv

Column Description
comment_id Comment ID
post_permalink Parent post
author Username
body Comment text
score Upvotes

⚙️ Environment Variables

# Notifications
export DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..."
export TELEGRAM_BOT_TOKEN="123456:ABC..."
export TELEGRAM_CHAT_ID="987654321"

📜 License

MIT License - Feel free to use, modify, and distribute.

🤝 Contributing

Pull requests welcome! For major changes, please open an issue first.