Research · Prompt

Reddit Signal Scraper

Mines Reddit for the pains people can't stop complaining about, clustered by problem space and ranked by how often they recur and how willing people are to pay.

REDDIT SIGNAL SCRAPER

# Reddit Signal Scraper — build spec

## Context / why this exists
The best business ideas come from listening to real people describe real pain. Reddit is the largest unfiltered archive of that pain — people complaining about tools they hate, asking for products that don't exist, and saying out loud what they'd happily pay for. Buried in that noise are recurring, high-intensity problems that keep coming up across threads. If I can systematically mine Reddit for those complaints, cluster them by problem space, and rank them by how often they recur and how monetizable they look, I can turn doomscrolling into a repeatable idea pipeline.

Build me a scraper + classifier that continuously pulls pain signals from the subreddits I care about, groups them into problem spaces, and surfaces the clusters worth building for — not one-off gripes.

## Goal (one sentence)

Enter your email to unlock the full prompt (+108 more lines) and copy it.

Bonus: turn the signals into a visual report

Once the scraper has populated data/reddit_signals.db, paste this second prompt to generate a shareable HTML report of the strongest problem spaces straight from your data. No dashboard server required.

REDDIT SIGNAL — VISUAL REPORT

# Visual report from the Reddit signal data

## Goal
Read the SQLite database produced by the Reddit Signal Scraper (data/reddit_signals.db) and generate a single, self-contained visual report I can share — a static HTML file with charts, no server required.

## Input
- SQLite file: data/reddit_signals.db
- Tables: signals, problem_clusters, scrape_runs (schema already exists)
- Read-only. Do not modify the database.

## Output
- One file: reports/reddit-signal-report.html
- Fully self-contained: inline CSS + a single charting library via CDN (Chart.js). No build step; it opens in any browser.

## Sections (in order)
1. Header — date range covered, total signals captured, number of active problem clusters, and how many signals showed explicit willingness to pay. Big, scannable numbers.
2. Opportunity leaderboard — top 15 problem clusters by blended score (recurrence + average intensity + explicit-pay count) as a horizontal bar chart. Label each bar with cluster name, signal count, and average intensity.
3. Heating up vs cooling down — clusters ranked by change in signal count (last 30 days vs the prior 60). Show the top 5 rising and top 5 falling, each with a trend arrow and percent change.
4. Pain intensity map — distribution of signals by intensity (1-5) and monetization signal (none / implicit / explicit), as a stacked bar.
5. Cluster cards — for the top 8 clusters, a card with: label, one-paragraph description, signal count, average intensity, and up to 6 example quotes (excerpt, subreddit, permalink).
6. Raw appendix — a compact HTML table of every signal (subreddit, excerpt, score, intensity, monetization signal, permalink) with client-side sort and a subreddit filter.

## Style
- Dark, technical, editorial. Monospace for labels and numbers; a clean sans for body copy.
- One accent color for emphasis; keep everything else muted grayscale. The data is the visual — no clip-art or stock imagery.
- Desktop-first, but it must not break on mobile.

## Rules
- Derive every number from the database; never invent data. Render null values as "—".
- Show excerpts, never raw usernames (the DB stores only hashes anyway).
- Put the SQL used for each section in an HTML comment directly above that section so I can audit it.
- If a section has no data, render it with an empty-state note rather than omitting it.

## Start by
Printing the row counts per table and the min/max created_utc so we confirm the data range, then generate the report.