Open Web Scraper

A free, local-first, no-code visual web scraper for Chromium browsers. Point it at a page, click one item in a list, and it works out the rest: the repeating structure, the fields, the pagination, and the detail pages behind each link.

Everything runs in your browser. There is no account, no server, no cloud tier, and no telemetry. Scraped data is written to your browser's own IndexedDB and never leaves the machine unless you export it yourself.

What it does

Visual picking	Hover to inspect, click to select. Every match is highlighted before you commit.
Repeating-record detection	Click one product card and it finds the other 47, plus the fields inside them.
Robust selectors	Prefers data-testid, ARIA and semantic attributes over hashed class names that change on every deploy.
Field extraction	text, innerHTML, outerHTML, href, src, alt, title, value, any attribute, plus exists/count.
Clean-up rules	trim, collapse whitespace, regex capture, find/replace, number parsing, absolute URLs, fallbacks.
Pagination	Next-button clicking, ?page={n} URL templates, query-parameter increments — or none.
Infinite scroll	Bounded by scroll count, idle rounds, page bottom, a wall-clock budget and a stop button.
Detail-page crawling	Follows each record's link, extracts more fields, merges them into the same row.
Deduplication	By whole record, by one field, or by detail URL — across every page of a run, and across re-runs, so scraping the same listing again only adds what is new.
Live results	Virtualised table that stays responsive at tens of thousands of rows. Edit, hide, reorder, delete.
Export	CSV, XLSX, JSON, JSONL. Encoded in the page; nothing is uploaded to produce a file.
Projects	Save, rename, duplicate, delete, export and import scraping configurations as JSON.
Claude / MCP ready	An optional local WebSocket bridge and a Node MCP server, both off by default.

Install it

npm install
npm run build

Then in Chrome, Chromium, Edge or Brave:

1. Open chrome://extensions
2. Turn on Developer mode
3. Click Load unpacked
4. Select the dist/ folder

Full instructions, including troubleshooting, are in SETUP.md.

Use it

1. Open the page you want to scrape in a tab.
2. Click the extension icon → Select element on this page.
3. Click one item in the list. Choose Repeating list from the chooser that appears.
4. The dashboard fills in the record selector and suggests fields. Rename them however you like.
5. Press Preview to see the first rows.
6. Configure pagination and detail pages if you need them.
7. Press Start scraping, then export from the Data section.

There is a local test site to practise on:

npm run test-site

It serves a paged product catalogue, an infinite-scroll listing, a JavaScript-rendered page, a Shadow DOM page, detail pages and some deliberately broken URLs at http://127.0.0.1:4173.

Permissions, and why each one exists

The extension asks for as little as possible, and nothing site-specific until you scrape.

Permission	Why
storage, unlimitedStorage	Projects, settings and scraped records, stored locally.
activeTab	Read the page you are looking at when you click the extension.
scripting	Inject the picker and the extraction code on demand.
tabs	Navigate between pages during a run and open detail pages in background tabs.
alarms	A heartbeat that stops the browser suspending a long run.
offscreen	Parse HTML for the optional fast detail-page mode (service workers have no DOMParser).
optional host access	Requested per site, only when you start a run. Revocable in Settings.

There are no declared content scripts, so installing the extension shows no "read your data on all websites" warning. Access to a site is granted by you, per site, when you first scrape it.

What it will not do

It scrapes publicly accessible content. It does not bypass CAPTCHAs, logins, authentication, paywalls or bot detection, and it does not disguise the browser. If a page is behind any of those, the extension will simply report that it could not read it.

Be a good citizen: leave the politeness delay on, keep concurrency low, and respect the terms of the sites you scrape.

Documentation

• SETUP.md — installing, building, browser support, troubleshooting
• ARCHITECTURE.md — how the pieces fit together and why
• MCP-INTEGRATION.md — driving the extension from Claude (npm run mcpb builds an installable bundle)
• TESTING.md — the test site, the unit suite and the end-to-end suite

Licence

MIT. See LICENSE.