Skip to content

About

Participation graph of deletion discussions across ruwiki and enwiki, for Toolforge

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

talk-snapshots

Live: https://talk-snapshots.toolforge.org/ — overview in English, wikis ordered by volume, plus a coverage table of other Wikipedias (app/coverage.py); /wiki/<dbname> — per-wiki page in that wiki's language (app/i18n.py); /wiki/<dbname>/mismatches — case-by-case list where the discussion and the deletion log disagree (page deleted but discussion open, kept but deleted, delete but exists, redirect but article, no page recognised, page found nowhere), each row linking to the discussion, the page and its log (app/core/mismatch.py); /api/report.json — the same aggregates as JSON (CORS * is deliberate: the API is meant for on-wiki gadgets and user scripts); /healthz (liveness), /readyz (showcase cache built, database reachable and, on Toolforge, Toolsdb rather than SQLite).

Who argues with whom, about which pages, in deletion discussions — and what happens to those pages afterwards — across Wikipedias with one codebase.

Seven wikis are wired in: ruwiki, enwiki, dewiki, ukwiki, zhwiki, itwiki, jawiki. Adding one more is a single declarative file, see below.

Two things that are universal, and everything that is not

Universal. The participation layer (who spoke, when, replying to whom) comes from the DiscussionTools API and works on any wiki. The fate of a page (exists / redirect / deleted, when, by whom, on which grounds / moved / recreated) comes from the page and logging tables of the database replicas — the same schema on every wiki, one query for all of them.

Per wiki. Where the discussions live, how nominations are separated on a page, which pages a nomination is about, where the closing decision is written and how a comment expresses a stance. None of this is hard-coded: each axis is a small strategy in app/core/, and a wiki is a composition of strategies in app/wikis/<dbname>.py.

Axis Strategies Used by
listing DayPage, DailyLog ru/de/uk/zh · en/it/ja
pages (1:N) Subsections, HeadingLinks, TitleFromHeading, TitleAfterPrefix group nominations, headings, per-nomination pages
outcome OutcomeSection, HeadingSuffix, ClosingTemplate, CommentPattern, NoOutcome ru/uk · de · en/zh · ja · it
stance VoteWords, SectionStance, NoStance en/de/uk/zh/ja · it · —
reason ReasonClass regexes over the deletion-log comment all

Discussion outcome and page state are stored separately (discussion_outcome vs page_state): a page can be kept by the discussion and speedily deleted a month later, or deleted and recreated — those are different facts.

A nomination may concern several pages (nomination_pages): Russian Wikipedia groups articles under one heading with sub-headings per article, and a fifth of nominations there are such groups or non-article pages.

Three sources, not one

Source What it provides Module
DiscussionTools API thread structure as MediaWiki itself sees it: author, timestamp, nesting, sub-headings, transcludedfrom core/threads.py
Database replicas (Toolforge; API elsewhere) revision history of the discussion page, page state and deletion log, categories and Wikidata item snapshot core/revisions.py, core/state.py, core/topics.py
Wikidata API (wbgetentities, no maxlag) P31/P106 of the nominated page's item, taken at nomination time core/topics.py
Wikitext parsing fallback for dump processing, where an API call per page is not affordable; also the closing templates core/parse.py

Topics are snapshotted at nomination time: after deletion the categories are gone from every source (replicas, Wikidata sitelinks, even the enwiki DELSORT categories vanish when the AfD is closed). The Wikidata item itself is deleted soon after the last sitelink goes, so P31 (instance of) and P106 (occupation) are stored at the same moment as class QIDs — those survive (Q5, Q4830453 «business»…). The same pass stores the page's authors (page_authors: edits, bytes added, first/last edit per user) — after deletion the history is admin-only, and these are the people to invite to the discussion (core/authors.py).

Usage

pip install -r requirements.txt
export TS_DB=ts.sqlite            # or toolsdb:<db> on Toolforge (set by envvars; unset there = refuse to start, no SQLite fallback)
export TS_WIKIS=ruwiki,enwiki,dewiki

python -m app.cli ingest --from 2026-07-01 --to 2026-07-03   # all TS_WIKIS
python -m app.cli --wiki dewiki ingest --recent 3 --refresh-changed
python -m app.cli state                                      # page fate from logs, whole volume
python -m app.cli quality                                    # metrics vs thresholds, non-zero exit on breach
python -m app.cli logs --from 2026-07-01                     # log counters: deletions by channel, restores, moves, protections
python -m app.cli daily                                      # what the cron runs: the four above
python -m app.stats --report top                             # participation reports

--refresh-changed re-reads a day page only if its latest revision on the wiki differs from the stored one: fixing a discussion on-wiki is the trigger, there is no management UI. Page state is recomputed over the whole volume on every run — it has no window.

Adding a wiki

  1. Create app/wikis/<dbname>.py with a WIKI = WikiSpec(...): locale (months, user-namespace aliases, namespace prefixes, bots, signature format), and one strategy per axis. dewiki.py is the shortest example.
  2. Save one day of DiscussionTools output as tests/fixtures/<dbname>/dt_<day>.json and pin counts in expected_<day>.json. The registry test refuses wikis without a fixture.
  3. Add the dbname to TS_WIKIS. Nothing in app/core/ changes — a test asserts the core never mentions a wiki name.

Deployment

Procfile declares web (gunicorn serving the dashboard straight from Toolsdb with an hourly in-memory cache — no files shared between the cron and the web pod), migrate, daily, stream and smoke; toolforge.yaml schedules daily and keeps stream running.

stream is a continuous job: it listens to EventStreams recentchange, and when a deletion venue's day page (the event's day, the day before or after in UTC) is edited it re-reads the nominations there and snapshots categories, the Wikidata item with P31/P106 and the page authors for pages it has not seen in the last three days. Articles closed within hours are thus captured before deletion. Snapshots are keyed by page (page_topics, page_authors) with first/last seen, not by nomination id, which changes on every refresh. A cheap parse&prop=sections check skips passes when only replies changed; the stream position is kept in tool_state, never moves past an event whose page is still queued, and is replayed with a two-minute overlap after a restart (at most 24 hours back). migrate copies the pre-2026-10-25 topics table (keyed by nomination id) into page_topics once.

Licensed under Apache-2.0.

About

Participation graph of deletion discussions across ruwiki and enwiki, for Toolforge

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages