Wikibot

PHP CLI app for Wikipedia robot.

See https://fr.wikipedia.org/wiki/Utilisateur:CodexBot for the live bot.

• Correction and completion of bibliographic references on the french Wikipedia, using my legacy code and importing open data (GoogleBooks, OpenLibrary, Bibliothèque nationale de France, Wikidata...) based on ISBN or other book's identifiers. Lots of data cleaning and post-processing, because bibliographic data is sooo serious but inconsistent.

• Completion of "external links" from the World Wide Web : the bot acts like a web crawler and transforms the raw links (http://...) into detailed references with page's title, author, site name, date, etc. It uses metadata from OpenGraph, JSON-LD, DublinCore, TwitterCard or naive prediction from HTML. Not a lot of data postprocessing, because web data is cool but rather consistent (SEO).

• Detection of dead links (404, DNS, etc.) and replacement by web archives (provided by Wikiwix or Internet Archive).

Please do not play with this package. These programs can actually modify the live Wikipedias, and proper wiki-etiquette should be followed before running it on any wiki. See https://en.wikipedia.org/wiki/WP:Bot for rules and authorization requests.

Tech stack : PHP >=8.1 (version 1.1 on PHP7.2), MySQL, Tor, composer libraries (Symfony components, addwiki/mediawiki-api, etc), hexagonal architecture….

Installation

composer install
cp .env.dist .env   # then fill in your bot's Wikimedia credentials

A few domain-lookup data files are gitignored (manually curated or Wikidata-sourced, not meant to live in git history) and must be generated before the external-link worker can run. Two scripts fetch them from Wikidata via SPARQL:

php src/Application/CLI/wikidataFetchPresse.php      # newspaper website domain -> French name + wikilink
php src/Application/CLI/wikidataFetchScientific.php  # scientific journal website domain/name -> wikilink

These populate src/Domain/resources/data_newspapers.json, data_scientific_domain.json and data_scientific_wiki.json (used by ExternRefTransformer to decide {{article}} vs {{lien web}} and to fill in the publication's name/wikilink), plus a couple of raw Wikidata caches. Re-run them anytime to refresh the data — safe to run repeatedly, each run overwrites the previous output.

Use make on root, for the list of some available commands.

Special thanks to

• addshore (wiki API)
• biblys (ISBN formating)
• many frwiki users for quality control

Memo :

• addwiki doc http://addwiki.readthedocs.io/