Independent v0.7.0 validation + fixes for the 8 bugs it found

CLAUDE.md

@@ -102,12 +102,12 @@ experiments/
 ## Testing
 
 ```bash
-pytest                    # 553 unit/integration tests (no network)
+pytest                    # 569 unit/integration tests (no network)
 pytest -m live            # 6 live tests (hits jsonplaceholder + httpbin)
 pytest -v                 # Verbose output
 ```
 
-**Total: 559 tests (553 + 6 live), all passing in <8s.**
+**Total: 575 tests (569 offline + 6 live; live tests require network access and are reported separately).**
 
 **Test files:**
 - `test_analyzer.py` — Traffic analysis & grouping (23 tests)
@@ -150,7 +150,6 @@ pytest -v                 # Verbose output
 - `test_monitor.py` — Diff computation, watcher, webhook, registry CRUD (41 tests)
 - `test_screenshot.py` — Screenshot model, CLI help, formats (8 tests)
 
-**Total: 500 tests (494 + 6 live), all passing.**
 
 ## Live Validation (8 Experiments)
 

VALIDATION-REPORT.md

@@ -0,0 +1,168 @@
+# site2cli v0.7.0 — Independent Validation Report
+
+**Date:** 2026-07-17
+**Method:** Fresh install in a clean Linux container (Python 3.11), full test suite, and a
+live end-to-end exercise of the entire CLI surface against a purpose-built local test
+website (HTML storefront backed by a JSON XHR API, multi-page site, robots.txt, mutable
+content page) using a real headless Chromium via Playwright.
+
+The sandbox's network policy blocks arbitrary external hosts, so the 8 live experiments
+(`experiments/`) and the 6 `@pytest.mark.live` tests could not reach their targets
+(jsonplaceholder, httpbin, etc.). Everything else was validated for real — including full
+browser-based CDP capture — against localhost.
+
+---
+
+## Verdict
+
+**The core claim is real.** `site2cli discover` drove a real browser, captured the site's
+XHR traffic over CDP, and produced — with zero LLM involvement (`--no-enhance`) — a valid
+OpenAPI 3.1 spec with correctly templatized path parameters (`/api/users/{id}`), inferred
+query and JSON-body parameters, a working Python client, a working zero-dep JS ES-module
+client, an HTML coverage report with gap candidates, and a replayable trace. `site2cli run`
+then executed those endpoints as direct HTTP calls (Tier 3) with no browser. The
+progressive-formalization architecture is not vaporware.
+
+**But it's an alpha with real rough edges.** One headline command (`screenshot`) is
+completely broken, generated MCP tool names violate the MCP spec, robots.txt handling
+fails open, and several of the README's quantified claims are simulations or estimates,
+not measurements.
+
+---
+
+## What was verified working (against a live local site)
+
+| Area | Result |
+|---|---|
+| Unit tests | **553 passed, 1 skipped** (claim: 553) in 16.5s (claim: <8s; machine-dependent) |
+| ruff lint | clean (`src/` + `tests/`) |
+| `discover` (real browser, CDP capture) | 4/4 XHR endpoints captured; GET query params, POST JSON body params, path-ID templatization all correct |
+| OpenAPI 3.1 output | validates with `openapi-spec-validator`; correct `servers` URL incl. port |
+| Generated Python client | all 4 methods executed successfully against the live API (after working around naming bug #1) |
+| Generated JS client (`client.mjs`) | loads as ES module in Node, exports `createClient` |
+| HTML coverage report | contains endpoint table + gap candidates |
+| Trace save/replay | `--save-trace` then `--replay` re-generates all artifacts fully offline |
+| `run` (Tier 3 direct API) | path substitution, POST bodies, `--json/--compact/--keys-only/--grep/--limit` all work; success counters increment |
+| `scrape` | correct main-content markdown extraction |
+| `crawl` | BFS crawled all 6 pages, correct per-page markdown files, `--sitemap` mode works |
+| `monitor` | baseline snapshot → content change → clean unified diff + watch registry. Works exactly as advertised |
+| `chunk` | heading-strategy chunking correct |
+| `mcp generate` + MCP protocol | spoke JSON-RPC to the generated server: `initialize`, `tools/list`, `tools/call` all execute real API calls end-to-end |
+| `init` | valid Claude MCP config JSON |
+| `community export` | bundle produced |
+| `daemon` | JSON-RPC over Unix socket: status + graceful stop work |
+| `sites list/show`, `config show`, `version` | work |
+
+Not testable in this environment: `search` (DuckDuckGo blocked), `extract` (needs
+`ANTHROPIC_API_KEY`), LLM-enhanced discovery, auth/OAuth device flow against real
+providers, experiments 8–15 (network policy).
+
+---
+
+## Bugs found (all reproduced, with locations)
+
+1. **Generated Python client is syntactically invalid for digit-leading hosts** —
+   `src/site2cli/discovery/client_generator.py:50`. `_sanitize_name()` prefixes `_` for
+   names starting with a digit, but the following `.replace("_", "")` strips it back off:
+   `127.0.0.1` → `class 127001Client:` → `SyntaxError`. Affects any digit-leading domain
+   (e.g. `9gag.com`). Tier 3 then silently skips the broken client. Ironically, CLAUDE.md
+   documents a previous naming bug fix in this same function.
+
+2. **Registry stores the user's relative `-o` path as `openapi_spec_path`** — after
+   `discover ... -o spec.json`, running `site2cli mcp generate <domain>` from any other
+   directory crashes with `FileNotFoundError: 'spec.json'`, and Tier 3 spec loading
+   breaks the same way. The spec path should be resolved to absolute (or the registry
+   copy under `~/.site2cli/specs/` should be authoritative).
+
+3. **Action-name mismatch + silent browser fallback** — actions are registered as
+   `GET_/api/products`, but `discover` prints `GET /api/products` and the spec/client use
+   `operationId: get_api_products`. Running the natural name doesn't error — the router
+   silently falls back to Tier 1 and navigates to `https://{domain}/` (dropping the
+   registered `base_url`'s scheme *and* port), surfacing as a confusing Playwright
+   `ERR_CONNECTION_REFUSED` traceback. Unknown actions should fail fast with the list of
+   valid names; the browser fallback should honor `base_url`.
+
+4. **robots.txt compliance fails open on non-standard ports** —
+   `src/site2cli/crawl/crawler.py:41` uses `parsed.hostname` (port stripped), so
+   `RobotsChecker` fetches `http://host/robots.txt` instead of `http://host:8777/robots.txt`;
+   the fetch fails and the checker allows everything (`load()` swallows all exceptions).
+   Verified: the crawler happily fetched `/private/secret.html` despite
+   `Disallow: /private/`. Standard-port sites are unaffected, but fail-open is the wrong
+   default for a compliance feature.
+
+5. **`screenshot` is completely broken in v0.7.0** — `src/site2cli/screenshot/capture.py:60`
+   calls `create_browser_context(proxy=..., profile=..., session=...)` but the factory
+   (`src/site2cli/browser/context.py:21`) accepts neither `proxy` nor `session` →
+   instant `TypeError` on every invocation. The 8 screenshot tests cover the model/help/
+   formats but never invoke the capture path.
+
+6. **Generated MCP tool names violate the MCP spec** — `tools/list` returns names like
+   `127_get_api_users_{id}`. Tool names must match `^[a-zA-Z0-9_-]+$`; real clients
+   (Claude Desktop, Claude API) will reject these tools. Path-parameter braces must be
+   sanitized. Also, parameter typing is inconsistent for agents: query params are typed
+   `string` while JSON-body params are typed `integer` (e.g. `limit` vs `max_results`),
+   producing validation errors for the obvious call.
+
+7. **`daemon start` doesn't daemonize** — `cli.py:1852` runs the server in the foreground
+   despite the help text "Start the *background* browser daemon".
+
+8. **Health checks give false negatives** — `src/site2cli/health/monitor.py:63` probes GET
+   endpoints with `HEAD` (others with `OPTIONS`). Servers that don't implement HEAD
+   (common; Python's stdlib server returns 501) are marked BROKEN even when every real
+   call succeeds — observed here: all 4 endpoints FAIL while Tier 3 `run` works perfectly.
+
+---
+
+## Claims audit
+
+| Claim | Verdict |
+|---|---|
+| "553 unit tests, all passing" | **True** (1 skipped) |
+| Discover → spec → clients → MCP pipeline works | **True**, verified end-to-end with a real browser |
+| "Four artifacts every run" | **True** (spec, .py client, .mjs client, HTML report) |
+| "<1s per action (API)" after discovery | **True** for Tier 3 (~100ms locally) |
+| "74% cheaper than browser-use" | **Simulation, not a measurement** — experiment 11 uses hardcoded cost constants ($0.05/run browser-use vs $0.0001 Tier 3, "based on industry data") |
+| "2M x faster than manual" | **Arithmetic on an assumption** — experiment 10 divides an assumed 40 hours/API of manual effort by measured runtime |
+| "Reliability ~15-35% on benchmarks vs >95%" | **Uncited** in README |
+| "100% health check accuracy" | Measured against the experiment's own scenario set; real-world HEAD-based probes false-negative (bug #8) |
+| "62% coverage of known unofficial APIs" | Plausible methodology (compares against documented endpoint lists) but not reproducible here (network) |
+| CLAUDE.md internal consistency | Contradicts itself: "559 tests" in one place, "500 tests" in another |
+
+---
+
+## Update: all 8 bugs fixed
+
+All bugs below were fixed on this branch and re-verified live against the same
+local test site (plus 16 new regression tests; suite now 569 passing):
+
+1. Digit-leading hosts now generate `class Site127001Client:` (valid Python).
+2. `discover` stores absolute spec/client paths; `mcp generate` also falls back
+   to `~/.site2cli/specs/` for legacy relative entries.
+3. Action names are now canonical operationIds (`get_api_products`) shared by
+   the spec, registry, clients, and MCP tools; `discover` prints the runnable
+   name per endpoint; unknown actions fail fast listing valid names (with
+   normalized matching for legacy `GET_/path` registries).
+4. robots.txt is fetched from the full `host:port` netloc; the crawler no
+   longer fetches `Disallow`ed paths on non-standard ports.
+5. `screenshot` works (context factory gained a `proxy` override; capture now
+   passes `session_manager`/`session_name`).
+6. MCP tool names are sanitized to `^[a-zA-Z0-9_-]+$`, and query params are
+   type-inferred (`limit=5` → integer) so tool schemas are consistent.
+7. `daemon start` detaches by default (logs to `~/.site2cli/daemon.log`);
+   `--foreground` preserves the old behavior.
+8. Health checks fall back to GET when a server answers 405/501 to
+   HEAD/OPTIONS — no more BROKEN reports for healthy endpoints.
+
+## How big a deal is it?
+
+The differentiated core — watch a site's own frontend traffic, formalize it into a typed,
+agent-ready API surface (CLI + OpenAPI + clients + MCP), then never pay browser/LLM cost
+again for those actions — **works as described and is genuinely useful**. That loop, plus
+offline trace replay, is something Firecrawl-style scrapers don't do, and the local-first,
+no-API-key posture is real.
+
+Tempering that: it's `0.7.0` alpha-quality at the edges (one command dead on arrival, MCP
+names that real clients would reject, fail-open robots), the headline benchmark numbers
+are projections rather than measurements, and discovery inherently only finds what the
+frontend happens to call while you browse. Treat the engineering as solid and promising,
+and the marketing numbers as directional.

src/site2cli/__main__.py

@@ -0,0 +1,6 @@
+"""Allow running as ``python -m site2cli``."""
+
+from site2cli.cli import app
+
+if __name__ == "__main__":
+    app(prog_name="site2cli")

src/site2cli/browser/context.py

@@ -24,6 +24,7 @@ async def create_browser_context(
     session_manager: Any | None = None,
     session_name: str | None = None,
     inject_cookies_for: str | None = None,
+    proxy: dict | None = None,
 ) -> AsyncIterator[tuple[Browser, BrowserContext, Page]]:
     """Create a browser context with cookies, profile, and session support.
 
@@ -34,6 +35,7 @@ async def create_browser_context(
         session_manager: Optional SessionManager to reuse sessions.
         session_name: Session name for session_manager lookup.
         inject_cookies_for: Domain to inject stored cookies for.
+        proxy: Playwright proxy dict; overrides the configured proxy.
 
     Yields:
         Tuple of (browser, context, page).
@@ -56,8 +58,8 @@ async def create_browser_context(
     browser: Browser | None = None
 
     try:
-        # Resolve proxy settings
-        proxy_config = config.proxy.get_playwright_proxy()
+        # Resolve proxy settings (explicit argument wins over config)
+        proxy_config = proxy if proxy is not None else config.proxy.get_playwright_proxy()
 
         if profile:
             # Profile mode: launch_persistent_context with user data dir

src/site2cli/cli.py

@@ -99,7 +99,11 @@ def discover(
         generate_js_client,
         save_js_client,
     )
-    from site2cli.discovery.spec_generator import generate_openapi_spec, save_spec
+    from site2cli.discovery.spec_generator import (
+        generate_openapi_spec,
+        make_unique_operation_ids,
+        save_spec,
+    )
     from site2cli.discovery.trace import load_trace, save_trace
     from site2cli.models import (
         DiscoveredAPI,
@@ -223,27 +227,24 @@ async def do_capture():
 
     # Step 6: Register site
     registry = _get_registry()
+    action_names = make_unique_operation_ids(endpoints)
     actions = [
         SiteAction(
-            name=(
-                ep.description.replace(" ", "_").lower()
-                if ep.description
-                else f"{ep.method}_{ep.path_pattern}"
-            ),
+            name=name,
             description=ep.description,
             tier=Tier.API,
             endpoint=ep,
         )
-        for ep in endpoints
+        for ep, name in zip(endpoints, action_names)
     ]
     site = SiteEntry(
         domain=domain,
         base_url=api.base_url,
         description=api.description,
         actions=actions,
         auth_type=auth_type,
-        openapi_spec_path=str(spec_path),
-        client_module_path=str(client_path),
+        openapi_spec_path=str(spec_path.resolve()),
+        client_module_path=str(client_path.resolve()),
     )
     registry.add_site(site)
 
@@ -253,10 +254,10 @@ async def do_capture():
         f"[bold green]Discovered {len(endpoints)}"
         f" capabilities for {domain}:[/bold green]"
     )
-    for ep in endpoints:
+    for ep, action_name in zip(endpoints, action_names):
         name = ep.description or f"{ep.method} {ep.path_pattern}"
         params = ", ".join(p.name for p in ep.parameters[:5])
-        console.print(f"  - {name} ({params})")
+        console.print(f"  - {name} ({params})  [dim]run: {action_name}[/dim]")
 
 
 @app.command()
@@ -306,6 +307,13 @@ def run(
     with console.status(f"[bold green]Executing {action} on {domain}..."):
         result = _run_async(router.execute(domain, action, param_dict))
 
+    if isinstance(result, dict) and "error" in result and "available_actions" in result:
+        console.print(f"[red]{result['error']}[/red]")
+        console.print("Available actions:")
+        for name in result["available_actions"]:
+            console.print(f"  - {name}")
+        raise typer.Exit(1)
+
     # Apply output filters
     if grep or limit is not None or keys_only:
         from site2cli.output_filter import filter_result
@@ -1495,7 +1503,22 @@ def mcp_generate(
         console.print("[red]No OpenAPI spec found for this site.[/red]")
         raise typer.Exit(1)
 
-    spec = load_spec(Path(site.openapi_spec_path))
+    spec_path = Path(site.openapi_spec_path)
+    if not spec_path.exists():
+        # Legacy registries stored relative paths; fall back to the specs dir
+        for ext in ("json", "yaml"):
+            candidate = config.specs_dir / f"{domain}.{ext}"
+            if candidate.exists():
+                spec_path = candidate
+                break
+        else:
+            console.print(
+                f"[red]Spec file not found: {site.openapi_spec_path}[/red]"
+                f" Re-run `site2cli discover {domain}` to regenerate it."
+            )
+            raise typer.Exit(1)
+
+    spec = load_spec(spec_path)
     code = generate_mcp_server_code(site, spec)
 
     output_path = (
@@ -1849,16 +1872,61 @@ def init(
 
 
 @daemon_app.command("start")
-def daemon_start() -> None:
+def daemon_start(
+    foreground: bool = typer.Option(
+        False, "--foreground", "-f", help="Run in the foreground instead of detaching"
+    ),
+) -> None:
     """Start the background browser daemon."""
-    from site2cli.daemon.server import DaemonServer
-
     config = get_config()
-    server = DaemonServer(config.daemon_socket_path)
-    console.print(
-        f"[green]Starting daemon on {config.daemon_socket_path}...[/green]"
-    )
-    _run_async(server.run())
+
+    if foreground:
+        from site2cli.daemon.server import DaemonServer
+
+        server = DaemonServer(config.daemon_socket_path)
+        console.print(
+            f"[green]Starting daemon on {config.daemon_socket_path}...[/green]"
+        )
+        _run_async(server.run())
+        return
+
+    if config.daemon_socket_path.exists():
+        from site2cli.daemon.client import DaemonClient
+
+        client = DaemonClient(config.daemon_socket_path)
+        try:
+            _run_async(client.send("list_sessions", {}))
+            console.print("[yellow]Daemon is already running.[/yellow]")
+            return
+        except Exception:
+            # Stale socket from a previous run
+            config.daemon_socket_path.unlink()
+
+    import subprocess
+    import sys
+    import time
+
+    log_path = config.data_dir / "daemon.log"
+    with open(log_path, "ab") as log:
+        subprocess.Popen(
+            [sys.executable, "-m", "site2cli", "daemon", "start", "--foreground"],
+            stdout=log,
+            stderr=log,
+            stdin=subprocess.DEVNULL,
+            start_new_session=True,
+        )
+
+    for _ in range(40):
+        if config.daemon_socket_path.exists():
+            console.print(
+                f"[green]Daemon started on {config.daemon_socket_path}[/green]"
+                f" (log: {log_path})"
+            )
+            return
+        time.sleep(0.25)
+
+    console.print(f"[red]Daemon did not start within 10s; see {log_path}[/red]")
+    raise typer.Exit(1)
 
 
 @daemon_app.command("stop")