Unify AgentScope with Habitat evidence runtime

.github/workflows/test.yml

@@ -14,6 +14,6 @@ jobs:
         with:
           python-version: "3.11"
       - name: Install package
-        run: pip install -e .
+        run: pip install -e ".[test]"
       - name: Run tests
-        run: python -m unittest discover -v
+        run: python -m pytest -q

README.md

@@ -59,7 +59,7 @@ python3 generate.py --config myconfig.json -o dashboard.html
 
 Open `dashboard.html` in any browser. Done.
 
-Requirements: Python 3.8+, stdlib only. Zero dependencies.
+Requirements: Python 3.11+. The default runtime has no third-party dependencies; proof signing is an optional extra.
 
 ### Daily digest (optional)
 
@@ -85,6 +85,34 @@ Copy `config.example.json`:
 
 Any source may be `null` to skip it. Job definitions can optionally provide `required: false` for intentionally disabled jobs and `created_at` for reliable never-run detection. Timestamp parsing accepts timezone-aware ISO-8601 values and normalizes them before comparisons. The anomaly logic lives in `anomalies()` — extend the signature list for your own stack's failure modes.
 
+## Habitat evidence runtime (optional)
+
+AgentScope can read an existing local [Habitat](https://github.com/buttercode101/Habitat) database and include its operational evidence summary in the generated dashboard. Habitat supplies the local action ledger, claims, policy gates, and portable proof workflow; AgentScope remains the read-only supervision surface.
+
+Set `sources.habitat_db` in your config to the initialized Habitat SQLite database path. Leave it `null` to keep the existing AgentScope-only mode. The bridge opens the database read-only. Missing or unreadable evidence is shown as **UNKNOWN**, and a failed action-ledger integrity check is surfaced as a failure. Stored claim verdicts are displayed as stored records, not freshly re-verified claims.
+
+See [the integration boundary and validation checklist](docs/INTEGRATION-HABITAT.md).
+
+## Habitat runtime and portable proofs
+
+The integrated Habitat runtime is available through the same installation. It records agent actions, correlates claims to runs, checks the action-ledger hash chain, applies evidence policies, and exports portable proof bundles.
+
+```bash
+pip install -e .
+
+# Initialize a local Habitat database
+habitat init --config examples/simple.yaml
+
+# Record explicitly labelled sample data
+habitat demo
+
+# Create and verify a portable proof for the sample run
+habitat prove --run-id run-42 --claim "sample run completed" -o proof.json --card
+python tools/verify_proof.py proof.json
+```
+
+The demo run is sample data, not a live verification of an external system. A valid proof establishes properties of the recorded evidence bundle; it does not establish external truth by itself. See [the integration architecture and trust boundaries](docs/INTEGRATION-HABITAT.md).
+
 ## Scope
 
 **In:** supervision of scheduled agent work, anomaly surfacing, one-page scanning.

agentscope_habitat.py

@@ -0,0 +1,136 @@
+"""Read-only bridge from AgentScope to Habitat's local evidence database."""
+from __future__ import annotations
+
+import json
+import sqlite3
+from datetime import datetime
+from pathlib import Path
+from urllib.parse import quote
+
+from habitat.schema import Action
+from habitat.store import Store
+
+
+def _read_json_object(value):
+    decoded = json.loads(value or "{}")
+    return decoded if isinstance(decoded, dict) else {}
+
+
+def _verify_ledger(conn):
+    marker = conn.execute(
+        "SELECT value FROM schema_meta WHERE key='action_integrity_initialized'"
+    ).fetchone()
+    if not marker or marker[0] != "1":
+        return "unknown"
+    rows = conn.execute(
+        "SELECT ai.seq, ai.action_id, ai.prev_hash, ai.hash, a.* "
+        "FROM action_integrity ai JOIN action a ON a.id=ai.action_id ORDER BY ai.seq"
+    ).fetchall()
+    action_count = conn.execute("SELECT COUNT(*) FROM action").fetchone()[0]
+    if len(rows) != action_count:
+        return "failed"
+    previous = None
+    for row in rows:
+        try:
+            action = Action(
+                row["id"], row["habitat_id"], datetime.fromisoformat(row["timestamp"]),
+                row["actor"], row["action"], row["status"], row["job_id"],
+                _read_json_object(row["details"]), row["run_id"],
+            )
+            if row["prev_hash"] != previous or row["hash"] != Store._action_hash(action, previous):
+                return "failed"
+            previous = row["hash"]
+        except (KeyError, TypeError, ValueError, json.JSONDecodeError):
+            return "failed"
+    return "intact"
+
+
+def load_habitat_snapshot(database_path):
+    """Return a bounded, read-only operational summary; never mutate Habitat state."""
+    if not database_path:
+        return None
+    path = Path(database_path).expanduser()
+    if not path.is_file():
+        return {"available": False, "reason": "database_missing", "path": str(path)}
+    uri = "file:" + quote(str(path.resolve()), safe="/") + "?mode=ro"
+    try:
+        conn = sqlite3.connect(uri, uri=True, timeout=2)
+        conn.row_factory = sqlite3.Row
+        try:
+            tables = {row[0] for row in conn.execute(
+                "SELECT name FROM sqlite_master WHERE type='table'"
+            )}
+            required = {"habitat", "job", "signal", "claim", "action", "action_integrity", "schema_meta"}
+            if not required.issubset(tables):
+                return {"available": False, "reason": "incomplete_schema", "path": str(path)}
+            habitat = conn.execute(
+                "SELECT id, name, status, model, updated_at FROM habitat LIMIT 1"
+            ).fetchone()
+            if habitat is None:
+                return {"available": False, "reason": "not_initialized", "path": str(path)}
+            jobs = conn.execute(
+                "SELECT COUNT(*) total, SUM(CASE WHEN enabled=1 THEN 1 ELSE 0 END) enabled "
+                "FROM job WHERE habitat_id=?", (habitat["id"],)
+            ).fetchone()
+            signals = conn.execute(
+                "SELECT severity, COUNT(*) total FROM signal WHERE habitat_id=? "
+                "AND resolved_at IS NULL GROUP BY severity", (habitat["id"],)
+            ).fetchall()
+            claims = conn.execute(
+                "SELECT status, COUNT(*) total FROM claim WHERE habitat_id=? GROUP BY status",
+                (habitat["id"],)
+            ).fetchall()
+            return {
+                "available": True,
+                "name": habitat["name"],
+                "status": habitat["status"],
+                "model": habitat["model"],
+                "updated_at": habitat["updated_at"],
+                "jobs_total": jobs["total"] or 0,
+                "jobs_enabled": jobs["enabled"] or 0,
+                "active_signals": {row["severity"]: row["total"] for row in signals},
+                "claims": {row["status"]: row["total"] for row in claims},
+                "ledger_integrity": _verify_ledger(conn),
+            }
+        finally:
+            conn.close()
+    except (OSError, sqlite3.Error, ValueError) as exc:
+        return {"available": False, "reason": "database_unreadable", "detail": str(exc)[:160], "path": str(path)}
+
+
+def render_habitat_panel(snapshot):
+    """Render an escaped, read-only Habitat summary for the generated dashboard."""
+    if snapshot is None:
+        return ""
+    import html
+
+    esc = html.escape
+    if not snapshot.get("available"):
+        reason = snapshot.get("reason", "unknown")
+        return (
+            '<div class="panel attn"><h2>Habitat evidence runtime</h2>'
+            '<div class="anom"><span class="badge" style="color:#b91c1c;background:#fee2e2">UNKNOWN</span> '
+            'Habitat state unavailable (' + esc(reason) + '). No assurance status is inferred.</div></div>'
+        )
+    claims = snapshot.get("claims", {})
+    signals = snapshot.get("active_signals", {})
+    signal_count = sum(signals.values())
+    ledger = snapshot.get("ledger_integrity", "unknown")
+    ledger_label = {"intact": "INTACT", "failed": "FAILED", "unknown": "UNKNOWN"}.get(ledger, "UNKNOWN")
+    ledger_color = "#15803d" if ledger == "intact" else "#b91c1c"
+    return (
+        '<div class="panel"><h2>Habitat evidence runtime</h2>'
+        '<div class="anom"><strong>' + esc(snapshot.get("name", "Habitat")) + '</strong> · runtime '
+        + esc(snapshot.get("status", "unknown")) + ' · '
+        + str(snapshot.get("jobs_enabled", 0)) + '/' + str(snapshot.get("jobs_total", 0)) + ' jobs enabled</div>'
+        '<div class="anom">Active signals: ' + str(signal_count)
+        + ' · Claims recorded: ' + str(sum(claims.values()))
+        + ' (verified ' + str(claims.get("verified", 0))
+        + ', rejected ' + str(claims.get("rejected", 0))
+        + ', inconclusive ' + str(claims.get("inconclusive", 0)) + ')</div>'
+        '<div class="anom">Action ledger: <span class="badge" style="color:' + ledger_color
+        + ';background:#e5e7eb">' + ledger_label + '</span>'
+        + (' · Active critical signals: ' + str(signals.get("critical", 0)) if signals.get("critical") else '')
+        + (' · Active warnings: ' + str(signals.get("warning", 0)) if signals.get("warning") else '')
+        + '</div><div class="mut" style="font-size:11px;margin-top:8px">Read-only snapshot. Recorded claim verdicts are not independently re-verified by this dashboard.</div></div>'
+    )

config.example.json

@@ -9,7 +9,8 @@
     "leads_query": [
       "SELECT count(*) FROM leads",
       "SELECT count(*) FROM leads WHERE status NOT IN ('completed','closed','lost')"
-    ]
+    ],
+    "habitat_db": null
   },
   "session_days": 7,
   "max_actions": 15,

docs/ARCHITECTURE.md

@@ -1,17 +1,70 @@
-# Full Commission architecture
+# AgentScope architecture
 
-Browser UI -> Commission API -> Evidence service -> Immutable source store -> Search index / graph -> Hermes orchestration.
+## Product boundary
 
-### Evidence object
-`id, type, statement, classification, source_ids, transcript_location, document_hash, created_at, updated_at, verification_state`
+AgentScope is the read-only supervision surface for an AI-agent crew. Habitat is the local evidence and verification runtime. The integration is intentionally one-way at the dashboard boundary: AgentScope reads Habitat state and renders a bounded summary; it does not run jobs or mutate evidence.
 
-### Source object
-`id, tier, publisher, title, canonical_url, retrieved_at, content_hash, publication_date`
+## Runtime flow
 
-### Transcript object
-`hearing_id, day, date, speaker, role, start_offset, end_offset, text, source_id`
+```text
+Hermes / scheduled agent sources
+        |
+        +----> AgentScope collectors ----> generated local HTML dashboard
+        |
+        +----> Habitat signed event intake
+                    |
+                    v
+             Habitat SQLite store
+             - jobs and runtime signals
+             - append-only action ledger
+             - correlated claims
+             - evidence policy decisions
+                    |
+                    v
+             Habitat verification + proof
+                    |
+                    v
+       AgentScope read-only SQLite bridge
+                    |
+                    v
+       Dashboard evidence/health summary
+```
 
-### Relationship object
-`subject_id, predicate, object_id, source_ids, confidence_reason`
+## Responsibilities
 
-The production version should use append-only source snapshots so later website changes cannot silently rewrite the historical record. Derived claims should point to exact source snapshots and transcript ranges.
+### AgentScope
+
+- Read local cron definitions, execution history, sessions, and optional MCP JSONL.
+- Detect operational anomalies and missing telemetry.
+- Generate a self-contained, read-only HTML dashboard.
+- Optionally read a Habitat SQLite database in SQLite `mode=ro`.
+- Display Habitat runtime state, active signals, claim verdict counts, and action-ledger integrity.
+- Treat unavailable evidence as UNKNOWN.
+
+### Habitat
+
+- Ingest structured agent events and preserve action provenance.
+- Maintain local job, signal, action, agent, and claim records.
+- Correlate claims to exact runs/actions.
+- Verify claims against trusted ledger evidence or explicit external evidence adapters.
+- Apply policy gates that can downgrade but never upgrade unsupported claims.
+- Export portable proof bundles and verify their structure, content integrity, and signatures.
+
+## Integration contract
+
+The bridge in `agentscope_habitat.py` is a read-only projection, not a second implementation of Habitat's domain model. It queries a bounded set of tables and verifies the action hash chain using Habitat's canonical hash routine. It does not re-run migrations, create claims, acknowledge signals, execute jobs, or rewrite stored verdicts.
+
+A stored claim verdict is displayed as stored state. The bridge does not label it as freshly re-verified.
+
+## Deployment model
+
+The default AgentScope product site is a static landing/demo surface. Operational dashboards are generated locally from configured sources. Habitat's local runtime can be operated locally or through its separately configured authenticated API. Do not publish a local operational database or private agent dashboard as a public demo.
+
+## Security invariants
+
+1. The default AgentScope path remains local and self-contained.
+2. Habitat database access from AgentScope is read-only.
+3. Missing telemetry is UNKNOWN, not healthy.
+4. Ledger integrity failure is a failure, not a warning hidden by acknowledgement.
+5. Proof content integrity, signature validity, publisher trust, and external truth remain distinct.
+6. The integration must not introduce an agent execution/control surface into AgentScope.

docs/DATA_MODEL.md

@@ -1,21 +1,54 @@
-# Evidence data model
+# AgentScope + Habitat data boundaries
 
-The runtime uses SQLite for the local production foundation. The same logical model can move to Postgres without changing the evidence contract.
+## AgentScope operational inputs
 
-## Core objects
+AgentScope's existing generator reads configured local sources:
 
-- **source** — canonical publisher, URL, tier and publication/retrieval metadata.
-- **snapshot** — immutable byte capture identified by SHA-256. A source can have many snapshots over time.
-- **claim** — a bounded statement with an explicit classification and verification state.
-- **claim_sources** — many-to-many provenance mapping; a claim cannot pass the gate without at least one source.
-- **hearing** — sitting-day index. `index_only` means the day is known to exist in the indexed sequence but detailed material has not been ingested into this build. `detailed` means a source-backed hearing record is present.
-- **audit** — append-only operational record of ingestion actions.
+- Job definition JSON files.
+- Execution-history SQLite database.
+- Session directory metadata.
+- Optional business counters.
+- Optional MCP/tool-call JSONL.
+- Optional acknowledgement state.
 
-## Integrity invariants
+The generated HTML contains a snapshot of the data read at generation time. It is not a live query interface.
 
-1. Claims never store source prose as if it were a source record.
-2. A claim must reference one or more source IDs before it can pass verification.
-3. A source snapshot is content-addressed by SHA-256.
-4. Re-fetching changed source content creates a new snapshot; old snapshots are retained.
-5. Missing hearing detail is represented explicitly rather than synthesized.
-6. Agent output is not persisted as factual evidence unless a human/controlled ingestion process promotes it into a sourced claim.
+## Habitat runtime records
+
+Habitat owns its local SQLite schema. AgentScope does not migrate or write this schema.
+
+| Record | Purpose |
+|---|---|
+| habitat | Runtime identity, model and overall status |
+| job | Scheduled work and latest observed execution state |
+| signal | Active/resolved anomaly signals with severity |
+| action | Trusted action ledger entries |
+| action_integrity | Hash-chain links for tamper detection |
+| claim | Bounded assertion and its stored verification verdict |
+| event | Ingested structured agent event |
+| agent | Registered agent identity and permissions |
+
+## AgentScope projection
+
+The bridge reads only:
+
+- Habitat identity and status.
+- Enabled/total job counts.
+- Counts of active signals by severity.
+- Counts of claims by stored verdict.
+- Action-ledger integrity result.
+
+It deliberately excludes agent secrets, permissions, raw event payloads, action details, claim evidence bodies, and proof contents from the dashboard summary.
+
+## Trust semantics
+
+- `available=false`: source missing, uninitialized, unreadable, or structurally incomplete; display UNKNOWN.
+- `ledger_integrity=intact`: the current action rows match the stored hash chain.
+- `ledger_integrity=failed`: the hash chain is incomplete or does not match; surface FAIL.
+- `ledger_integrity=unknown`: integrity metadata is absent or uninitialized; never treat as intact.
+- Claim counts represent stored Habitat verdicts only. They are not a fresh verification of each claim.
+- A valid proof bundle proves properties of the recorded evidence bundle, not that the external world necessarily matches the claim.
+
+## Retention and migration
+
+Habitat remains the system of record for its evidence and proof data. Preserve the original Habitat database and repository until backup/restore, schema migration, and proof-verification tests pass against the unified build. Never rewrite or deduplicate historical proof records during this integration.