jevscrape

A deterministic web scraper that pairs classical wrapper induction with a structured decision model (TypeSafe AI's Jev, via the Vercel AI Gateway) for the pipeline decisions a CSS selector cannot make on its own.

• No LLM in the loop.
• Values are DOM-node substrings by construction, so the extractor cannot emit fabricated fields.
• Jev handles typed judgments: pre-flight response classification, extraction-quality scoring, retry routing.

Install

./scripts/setup.sh
cp .env.example .env  # add AI_GATEWAY_API_KEY

Requires Node 20+, pnpm 9+, Python 3.11+.

Use it

CLI

pnpm --filter @jevscrape/pipeline scrape -- \
  --url https://example.com/product/1 \
  --schema title,price,description \
  --pretty

Returns a JSON object with the extraction, per-attribute source (learned wrapper, JSON-LD, OpenGraph, meta), Jev quality flag, cost, and latency.

MCP tool

{
  "mcpServers": {
    "jevscrape": {
      "command": "pnpm",
      "args": ["--filter", "@jevscrape/pipeline", "exec", "tsx", "src/scrape-mcp.ts"],
      "env": { "AI_GATEWAY_API_KEY": "vck_..." }
    }
  }
}

The agent then has one tool: scrape(url, schema?, useJev?).

Design

Fetch → Pre-flight (Jev) → Extract → Quality (Jev) → Route (Jev) → Store

Stages 1, 3, 6 are deterministic. Stages 2, 4, 5 are Jev calls. The extractor uses a three-tier wrapper-induction algorithm learned from K labeled example pages per site:

1. Exact match — CSS signature intersection across K pages where some DOM node's text equals the target value.
2. Contains match — same, but the value is a substring of a labeled container's text; the inducer learns the shared prefix and suffix to strip at extraction time.
3. Landmark contains — relaxes signature uniqueness and adds a text-label filter; needed for product-page lists like Amazon's.

Repo layout

• packages/pipeline/ — the TypeScript pipeline. Fetch, extract, wrapper induction, Jev client, scrape() API, MCP server, CLIs.
• eval/ — Python evaluation harness. SWDE loader, F1 metrics, rule-only / hybrid / LLM baseline runners, cross-system comparison with bootstrap CIs.
• scripts/ — induce-wrappers (TypeScript), download-swde (shell), setup.
• docs/interop.md — JSONL/subprocess contract between the TS pipeline and the Python harness.

Tests

pnpm --filter @jevscrape/pipeline test   # 14 TypeScript tests
python3 -m pytest eval/                  # 16 Python tests

License

MIT.