Universal Web Monitoring Agent

This repository runs scheduled GitHub Actions jobs that monitor research sources and store results as Markdown reports:

• LessWrong filter: reads recent LessWrong posts (GraphQL API, with the RSS feed as a fallback) and filters for long ML/NLP posts with likely visualizations.
• Page watchers: scrape the Anthropic Interpretability team page and the Goodfire research page, detect new publication entries, and send an alert email on hits.
• arXiv AI Top Papers (weekly): ranks the last 7 days of AI-related arXiv submissions by reference count and by citation-weighted references.
• Lab Publications filter (temporarily disabled): matched arXiv papers against a list of lab names.
• Pythia Google watch: daily Google web search for the EleutherAI Pythia models, with a 24-hour search, overlapping outage catch-up, relevance filtering and a persistent outbox. All queued hits are bundled into one Pushover-mail digest per run, with a link to the full report. Requires the additional SERPAPI_API_KEY search secret. Setup and coverage.
• Competition Tracer: extensible daily tracking of TIG algorithm submissions, Sakana Core War / Digital Red Queen, Sudoku-Bench status and related assembly/evolution papers, including Google's Paradigms of Intelligence research. Per-source depth, cumulative reports and one daily email digest use the existing SMTP/Pushover route. Sources, setup and coverage.

Conventions

• Fail loudly. A monitor that cannot read its source must produce a red run, not an empty report. An empty report is indistinguishable from a quiet day and hides outages for as long as nobody looks; a visible gap in reports/ is honest. lw_filter.py (LW_STRICT), page_watch.py (WATCH_MIN_ENTRIES) and arxiv_top_papers.py all follow this.
• Log the evidence. Every refused HTTP request logs its status, timing, the relevant response headers and a body snippet, so an incident is diagnosable from a single run's log.
• Ask, don't evade. These jobs identify themselves in the User-Agent, honour robots.txt (LessWrong asks for Crawl-Delay: 3) and keep to roughly one request a day. If a source deliberately blocks us, the response is to reduce load and contact the maintainers — never to rotate addresses, disguise the client or bypass a challenge.

How it works

• GitHub Actions runs on a daily schedule or via workflow_dispatch.
• lw_filter.py requests recent LessWrong posts over HTTP instead of browser automation, using whichever transport answers (see below).
• The script filters posts by lookback window, minimum word count, visualization heuristics, and ML/NLP topic keywords.
• Matching posts are written to reports/YYYY-MM-DD.md.
• Processed post IDs are stored in seen.json so the workflow does not report the same post twice.

Transports

LW_TRANSPORT selects how posts are fetched; the default auto tries them in order and reports the winner in the report header.

Transport	Endpoint	Coverage	Notes
graphql	POST /graphql	Full window, paginated via offset	Preferred: one request returns baseScore, wordCount, frontpageDate and the full post HTML. Uses the selector argument; the older input: { terms: … } form is deprecated server-side.
feed	GET /feed.xml?view=…	10 items per view, hard-capped server-side	Fallback. The feed's <description> is the same contents.html GraphQL returns and its <guid> is the bare post ID, so the filter heuristics and seen.json work unchanged. baseScore is unavailable and renders as n/a; a Frontpage post outside the 10 newest frontpage items is reported as Personal/All (unconfirmed). Responses are CDN-cached, so this path can stay reachable when the uncacheable GraphQL POST is not — which is why no after query parameter is sent: it is part of the edge cache key, and a daily-changing value would force a MISS to the origin on every run. The lookback window is enforced client-side for both transports instead.

Failure behaviour

• A run that cannot fetch posts over any transport writes no report, leaves seen.json untouched, exits non-zero and sends the failure alert email. Set LW_STRICT=0 to downgrade that to a green run with a degraded report.
• The day's report is cumulative. Reports are keyed by UTC date, and a second run of the same day only classifies posts absent from seen.json — so rewriting the file from that run's matches alone would silently delete the earlier ones, unrecoverably, since their IDs are already marked seen. Each day's matches are therefore also kept in reports/.matches/YYYY-MM-DD.json, and every run renders the union. A backlog catch-up run adds to the day's report instead of replacing it.
• If a report exists whose match index is missing (written before the index existed, or lost) and this run would write fewer matches than it lists, the write is refused, the matched IDs are not recorded in seen.json, and the run exits non-zero — so those posts stay eligible for the next run rather than being suppressed with nothing written anywhere.
• Total time spent waiting out retries is bounded by LW_TIME_BUDGET_MINUTES, deliberately below the workflow's timeout-minutes. GitHub enforces that timeout by cancelling the job, and cancellation would otherwise skip the alert step — so the script gives up first.
• Posts that arrive without body content are skipped without being marked seen, so a transport quirk cannot drop a post permanently.
• A refusal carrying no rate-limit budget headers, a non-JSON body and near-zero latency is treated as a standing edge block rather than a quota: the script stops retrying after LW_BLOCK_STRIKES attempts and moves to the next transport instead of burning its whole budget.
• The workflow's Probe LessWrong reachability step curls /api/agent/ping (LessWrong's documented reachability probe), /feed.xml and /graphql on every run and never fails the job. If /api/agent/ping is refused as well, the whole host is unreachable from GitHub's runners and no transport change will help — the remedy is to contact the LessWrong developers (POST /api/agent/feedback, or an issue on ForumMagnum) and describe the job's traffic, not to work around the block.

Page watchers (Anthropic Interpretability & Goodfire research)

Two separate workflows watch one concrete URL each instead of matching names on arXiv:

• anthropic-interpretability-watch.yml → https://www.anthropic.com/research/team/interpretability (3×/day at 06:00, 14:00, 22:00 UTC = 00/08/16h CEST)
• goodfire-research-watch.yml → https://www.goodfire.ai/research (3×/day at 06:15, 14:15, 22:15 UTC, staggered to avoid push races)

Both run page_watch.py, which extracts publication links from the page (rendered anchors plus URLs embedded in script/JSON payloads), compares them against a persisted baseline (seen_anthropic_interpretability.json / seen_goodfire_research.json), and writes a report to reports/ only when something changed. Design decisions:

• Fail loudly: if fewer entries than a per-site minimum can be extracted (page redesign, bot block, client-only rendering), the run fails instead of silently treating a degraded page as "no news". Override with the WATCH_MIN_ENTRIES environment variable.
• Baseline on first run: the first run records all current entries without alerting; only entries appearing later trigger the alert email.
• Reports are only written on baseline initialization or new hits, so the reports/ directory is not flooded with empty daily files.

Alert email (repository secrets)

On a hit, send_alert.py sends a plain-text email via SMTP. All personal data lives exclusively in GitHub Actions repository secrets (Settings → Secrets and variables → Actions), so nothing private appears in this public repository or its logs:

Secret	Required	Description
SMTP_HOST	yes	SMTP server of the sending account
SMTP_PORT	no	587 (STARTTLS, default) or 465 (implicit TLS)
SMTP_USERNAME	yes	Login of the sending account
SMTP_PASSWORD	yes	Password / app password of the sending account
ALERT_EMAIL_TO	yes	Recipient address(es), comma-separated — kept secret on purpose
ALERT_EMAIL_FROM	no	From address, defaults to SMTP_USERNAME

If a hit occurs while these secrets are missing, the email step fails visibly (listing only the missing secret names) so a hit can never be dropped silently. Both workflows accept two Run workflow inputs: dry_run tests scraping without committing or emailing, and test_email sends a test alert immediately so the SMTP secrets can be verified without waiting for a real hit.

arXiv AI Top Papers (weekly)

arxiv_top_papers.py runs once a week (arxiv-top-papers.yml, Sundays 22:30 UTC) and covers everything submitted to the AI/computing categories cs.AI, cs.LG, cs.CL, cs.NE (override with ARXIV_CATEGORIES) during the last 7 days (LOOKBACK_DAYS). It writes reports/arxiv_top15_YYYY-MM-DD.md with two rankings:

1. Top 15 by number of references — first-order: how many works each paper cites.
2. Top 15 by citation-weighted references — every single reference is weighted by the current citation count of the cited work: weighted score = Σ citation_count(reference).

Reference lists and citation counts come from the Semantic Scholar Graph API. Notes:

• The optional repository secret S2_API_KEY (a free Semantic Scholar API key) raises the rate limit considerably; without it the script paces itself more conservatively and the run takes longer. An invalid or not-yet-activated key (Semantic Scholar answers 403) is detected at runtime: the job logs a warning and falls back to unauthenticated requests instead of failing.
• The unauthenticated pool is shared and often rate-limited (429) for minutes at a time; the job waits that out patiently (S2_MAX_429_RETRIES, up to ~30 min per request) instead of aborting. An overall time budget (S2_TIME_BUDGET_MINUTES, default 240) caps the Semantic Scholar phase: when exhausted, remaining per-paper reference fetches are skipped and the report is written with the data collected so far (affected papers keep their reference count but get no weighted score).
• Very fresh papers may not be indexed by Semantic Scholar yet (or their bibliographies may not be parsed yet); the report header shows exactly how many papers could be ranked.
• The job is stateless: each run is a self-contained weekly snapshot, so there is no seen_*.json file.
• Offline unit tests (test_arxiv_top_papers.py, mocked APIs) run in the workflow before the ranking step.

Lab Publications filter (temporarily disabled)

lab_pubs_filter.py matched recent arXiv papers against a large list of lab names (no URLs) and produced only empty reports for weeks. Its schedule is therefore commented out in .github/workflows/lab-pubs-filter.yml; it can still be started manually via workflow_dispatch and re-enabled by uncommenting the schedule block.

Repository files

• lw_filter.py: main LessWrong filtering script
• page_watch.py: generic page watcher used by the Anthropic/Goodfire workflows
• send_alert.py: SMTP alert mailer (configured via repository secrets)
• arxiv_top_papers.py: weekly arXiv AI top-papers ranking (references & citation-weighted references)
• test_arxiv_top_papers.py: offline unit tests for the ranking script
• test_lw_filter.py: offline unit tests for the LessWrong filter (transports, retry/blocking logic, report guards)
• lab_pubs_filter.py: arXiv lab-name filter (schedule temporarily disabled)
• requirements.txt: Python dependencies for the workflows and local runs
• .github/workflows/: scheduled workflows
• reports/: generated Markdown reports
• seen.json, seen_labs.json, seen_anthropic_interpretability.json, seen_goodfire_research.json: persisted state so nothing is reported twice

Configuration (LessWrong filter)

Every value below can be set as a repository variable (Settings → Secrets and variables → Actions → Variables); lesswrong-filter.yml reads vars.<NAME> and falls back to the default. LOOKBACK_DAYS, POST_LIMIT, LW_TRANSPORT and LW_STRICT are also exposed as workflow_dispatch inputs, which take precedence — that is how you run a one-off backlog catch-up with a raised POST_LIMIT.

Variable	Description	Default
MIN_WORDS	Minimum word count for a matching post	1800
LOOKBACK_DAYS	How many days of posts to request	14
POST_LIMIT	Max posts fetched per run. Binds before LOOKBACK_DAYS: if more posts were published in the window than this, the window is truncated and the report says so. LessWrong publishes roughly 15–30 posts a day	100
POST_SCOPE	all, frontpage, or personal	all
LW_TRANSPORT	auto, graphql, or feed	auto
LW_STRICT	1 fails the run when no transport works; 0 writes a degraded report instead	1
LW_PAGE_SIZE	Posts per GraphQL request when paginating	50
LW_MAX_RETRIES	Retry budget for network and 5xx errors	5
LW_MAX_429_RETRIES	Separate retry budget for rate limiting	6
LW_MAX_BACKOFF_SECONDS	Maximum computed wait between retries	60
LW_RETRY_AFTER_CAP	Hard ceiling on an honoured Retry-After header	300
LW_BLOCK_STRIKES	Consecutive budget-less refusals treated as a standing block	2
LW_CRAWL_DELAY	Minimum seconds between requests (robots.txt asks for 3)	3
LW_TIME_BUDGET_MINUTES	Total wall clock the run may spend waiting out retries, across all transports. Keep below the workflow's timeout-minutes	10
LW_FEED_VIEWS	Comma-separated RSS views for the feed transport	Derived from POST_SCOPE
LESSWRONG_USER_AGENT	Identifiable user agent for API requests	Repository URL based default
LW_REQUEST_TIMEOUT	Request timeout in seconds	60

LW_MAX_RETRIES, LW_MAX_BACKOFF_SECONDS and LW_REQUEST_TIMEOUT also accept the older unprefixed names (MAX_RETRIES, MAX_BACKOFF_SECONDS, REQUEST_TIMEOUT); the prefixed ones win. The prefix exists because three scripts in this repository read those generic names with different intended defaults.

Tests

test_lw_filter.py and test_arxiv_top_papers.py mock all HTTP, so they run offline and are executed by their workflows before the real job:

python -m unittest test_lw_filter.py test_arxiv_top_papers.py

Running locally

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python lw_filter.py

Notes

• LessWrong requests use an identifiable user agent and stay conservatively rate-limited (LW_CRAWL_DELAY, one run a day).
• The current implementation uses a heuristic topic and visualization filter so it can run without additional services.
• LessWrong has no API key or bot registration. GraphQL authentication is a loginToken harvested from a logged-in browser session, which would mean storing a live session credential in repository secrets — deliberately not done.
• LessWrong also publishes an agent-oriented Markdown API (/api/SKILL.md, /api/latest, /api/post/[id]). It is not used here because the filter's visualization heuristic counts HTML elements (<figure>, <svg>, <canvas>, <iframe>, <table>), which the Markdown rendering discards. /api/agent/ping and /api/agent/feedback from that same API are used for the reachability probe and as the escalation path.