Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
*
!src/
!src/**
!requirements.lock
!Dockerfile
**/__pycache__/
5 changes: 5 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
* text=auto eol=lf
# Preserve byte-identical scientific evidence on every platform.
*.json -text whitespace=cr-at-eol
*.xml -text -diff -whitespace
src/eqb/data/raw/* linguist-generated=true
47 changes: 47 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
name: Foundation checks
on:
push:
pull_request:
permissions:
contents: read
jobs:
offline:
strategy:
matrix:
os: [ubuntu-latest, windows-latest]
runs-on: ${{ matrix.os }}
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- run: python -m pip install uv==0.12.3
- run: uv sync --locked
- run: uv run --no-sync ruff check .
- run: uv run --no-sync ruff format --check .
- run: uv run --no-sync pytest
- run: uv run --no-sync eqb doctor
- run: uv run --no-sync eqb replay
- run: uv build
container:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- name: Build the local workbench image
run: docker build -t exoquery-test .
- name: Verify packaged evidence inside the image
run: docker run --rm exoquery-test python -m eqb.cli doctor
- name: Verify HTTP health and persistence directory
run: |
docker run -d --name exoquery-smoke -p 127.0.0.1:8765:8765 exoquery-test
for i in $(seq 1 30); do
if curl --fail http://127.0.0.1:8765/api/health; then exit 0; fi
sleep 1
done
docker logs exoquery-smoke
exit 1
- name: Clean up smoke test
if: always()
run: docker rm -f exoquery-smoke || true
12 changes: 12 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
.venv/
__pycache__/
.pytest_cache/
.hypothesis/
.ruff_cache/
*.egg-info/
.env
.env.*
!.env.example
.eqb/
dist/
*.sqlite3*
12 changes: 12 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
FROM python:3.12-slim
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 PYTHONPATH=/app/src EQB_DATA_DIR=/app/state
WORKDIR /app
COPY requirements.lock ./
RUN pip install --no-cache-dir --require-hashes -r requirements.lock \
&& useradd --uid 10001 --create-home researcher \
&& mkdir -p /app/state && chown researcher:researcher /app/state
COPY src ./src
USER researcher
EXPOSE 8765
HEALTHCHECK --interval=30s --timeout=5s CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8765/api/health',timeout=3)"
CMD ["python", "-m", "uvicorn", "eqb.app:create_app", "--factory", "--host", "0.0.0.0", "--port", "8765"]
140 changes: 36 additions & 104 deletions PROJECT_SCOPE.md
Original file line number Diff line number Diff line change
@@ -1,108 +1,40 @@
# ExoQuery-Bench: Project Scope
# ExoQuery-Bench: gated roadmap

Working plan for building the benchmark, the systems under test, the chat demo and the write-up. The README holds the
public summary; this file holds the build plan, decisions and done-criteria.
The long-term objective remains a reproducible comparison of models, retrieval and tool designs for astronomy archive questions, followed by a useful research assistant. The first implementation is a local evidence-and-review workbench. Its software can be tested now; scientific validity and user impact require external review.

## 1. Goal
## Acceptance gates

Give the astronomy-archive community a reproducible answer to one question: *for natural-language questions to the
NASA Exoplanet Archive, which model, retrieval setup and tool design produce correct, faithful answers, at what cost,
and where do they fail?*

Success means: a public question set, a harness anyone can rerun, a leaderboard, an error taxonomy that maps to
fixes (for example "add `default_flag` guidance to the prompt"), and a chat demo that uses the best configuration.

## 2. Verified facts the design rests on (checked 2026-10-07)

Each fact was checked in at least three places: a live TAP query, the archive documentation, and a client library or
second source.

| Fact | Live query | Documentation | Third source |
|---|---|---|---|
| `ps` has 40,194 rows for 6,375 planets; `default_flag=1` gives 6,375 | TAP `count(*)` | Column docs: `default_flag` "selected as default (1=yes, 0=no)", PS only | PyVO `TAPService` returns 6,375 |
| `pscomppars` has one row per planet (6,375) | TAP `count(*)` | Column docs list it as the composite table | astroquery examples query it per planet |
| Values are case-sensitive (`'transit'` returns 0) | TAP | TAP guide: values are case-sensitive | `'Transit'` returns 4,711 |
| `LIMIT` fails, `TOP` works | TAP error vs rows | IVOA ADQL uses `SELECT TOP n` (GAVO ADQL 2.1 notes) | astroquery docs use `select="top 10 ..."` |
| `disc_method` works but is undocumented | TAP returns codes | Absent from column docs | Absent from `TAP_SCHEMA.columns` |
| R<sub>Jup</sub> = 11.21 R<sub>Earth</sub> | n/a | Column docs define `pl_rade` and `pl_radj` separately | `astropy.units` conversion |
| AstroFetch MCP exposes 10 tools over 3 archives | `tools/list`, `list_archives` | AstroFetch agents page | Exoplanet Archive news, 2026-10-01 |
| AstroFetch's top errors are columns and units | n/a | AstroFetch FAQ | n/a (single source, quoted as the FAQ's own claim) |

## 3. Components

### 3.1 Archive layer (`src/eqb/archive.py`)
- PyVO `TAPService` for sync queries, async jobs for anything large; astroquery as a cross-check.
- On-disk cache keyed by query text and archive snapshot date; global rate limit; retries with backoff.
- `snapshot_schema()` saves `TAP_SCHEMA.tables`, `TAP_SCHEMA.columns` and the column-docs page with a date stamp.

### 3.2 Question set (`data/questions/*.jsonl`)
- v0.1: 120 hand-written items; 20 seed items first.
- Target mix: 25% lookup and filter, 20% aggregation, 15% table choice and `default_flag`, 15% units, 10% joins and
sky-position (`contains(point(...), circle(...))`), 10% TESS `toi` dispositions and spectra, 5% unanswerable.
- Difficulty easy / medium / hard; 2 paraphrases for 30 items to test wording robustness.
- Every gold query is executed and reviewed by hand before it is added; items carry tags for the trap they test.

### 3.3 Providers (`src/eqb/providers/`)
- Anthropic Claude, OpenAI, Google Gemini and an Ollama local model behind one `complete()` call, extended from the
provider layer in [JudgeGuard](https://github.com/RyanSingh0/judgeguard).
- Structured output: one JSON schema (ADQL, tables, columns, units, assumptions, refusal flag), using each provider's
native JSON or tool-calling mode, validated with Pydantic.
- Response cache so reruns and CI cost nothing.

### 3.4 Systems (`src/eqb/systems/`)
- **S0** direct prompt. **S1** curated schema in the prompt.
- **S2 RAG:** chunk column definitions, table notes and worked examples; hybrid retrieval (BM25 + embeddings) with
top-k columns injected; log retrieved chunks for retrieval metrics.
- **S3 agent:** tools `search_schema`, `validate_adql`, `run_query`, `check_units`; step budget of 6; stops on a
validated, executed answer.
- **S4 AstroFetch MCP:** an MCP client gives the model AstroFetch's own tools; throttled and cached.

### 3.5 Scoring (`src/eqb/scoring/`)
- Execution accuracy: run gold and candidate at the same time; compare as multisets with numeric tolerance; match
columns by values, not names.
- Component metrics: table accuracy, column F1, `default_flag` correctness, unit check, valid-ADQL rate, refusal
accuracy.
- LLM judge for answer faithfulness to returned rows; 60 items double-labelled by hand to report Cohen's kappa.
- Retrieval: recall@k and context precision for S2 and S3.
- Statistics: question-level bootstrap CIs; paired McNemar tests for system comparisons.

### 3.6 Error taxonomy
Automatic first-pass labels, hand-checked: wrong table, missing `default_flag`, wrong column, unit error,
case-sensitive value, SQL-not-ADQL syntax, undocumented column, wrong aggregation, timeout, hallucinated answer.

### 3.7 Leaderboard (`src/eqb/report/`)
- Static Bokeh page on GitHub Pages: accuracy by provider x system with CIs, cost-accuracy frontier, error taxonomy
bars, per-question drill-down showing gold vs candidate ADQL.

### 3.8 Chat demo (`app/`)
- FastAPI back end with the best configuration; Gradio front end on Hugging Face Spaces.
- Shows the ADQL, retrieved documentation and unit warnings with every answer.
- Thumbs up/down with a comment; reviewed feedback becomes new benchmark items.

### 3.9 Engineering
- Python 3.11+, uv, ruff, mypy, pytest with Hypothesis for the result-set comparator.
- GitHub Actions: lint, tests, offline replay of cached runs, Docker build.
- MIT license; dataset card for the question set.

## 4. Phases and done-criteria

| Phase | Work | Done when |
| Gate | Acceptance evidence | Current state |
|---|---|---|
| 0 | Repo, README, schema snapshot, 20 seed questions, comparator with tests | 20 gold queries run green in CI |
| 1 | Providers, S0 and S1, first table of results | All 4 providers scored on 20 questions with CIs |
| 2 | 120-question v0.1, S2 RAG, retrieval metrics | S2 vs S1 difference reported with a paired test |
| 3 | S3 agent, S4 MCP, error taxonomy | Every failure has a label; top-3 causes per system listed |
| 4 | Judge calibration, Bokeh leaderboard | Kappa reported; leaderboard live on GitHub Pages |
| 5 | Chat demo and feedback loop | Demo live; feedback stored and reviewable |
| 6 | IRSA and KOA extension; Research Note draft | 20 questions per new archive; RNAAS draft complete |

## 5. Risks and how they are handled

- **Archive changes weekly:** gold and candidate are executed together at scoring time; snapshot dates are logged.
- **API cost:** small question set, caching, local model as a free baseline; CI replays cached responses.
- **Load on public services:** rate limits and caching for both the TAP service and the AstroFetch MCP endpoint.
- **Judge bias:** judge used only for answer faithfulness, calibrated against hand labels, with kappa published.
- **Overfitting the benchmark:** a held-out split of 30 questions is never used while developing prompts.

## 6. Out of scope for v0.1
Fine-tuning models; image or light-curve analysis; write access to any archive.
| 0. Contracts and failure semantics | Explicit columns/units/grain/order; partial and unavailable not scored as zero; semantic regression tests | Implemented and locally tested |
| 1. Archive evidence | Raw responses, schema, checksums, complete results, independent rank controls | 17 references and 48 capture checks passed on Oct 12, 2026 |
| 2. Scientific approval | Documented review of all 20 items; critical disagreements resolved | Pending; no expert approvals claimed |
| 3. Usable review workflow | Browser evidence/compare/review flows, persistence, export, installation | Implemented; browser and local API checks performed |
| 4. Real-user pilot | 3–5 volunteers; at least one experienced archive user; concrete tasks and corrections documented | Pending recruitment through BU/LinkedIn contacts |
| 5. Model experiments | Reviewed references, frozen configurations, complete failure/cost logs, controlled comparison | Not started; no paid calls authorized |
| 6. Trustworthy reporting | Family-separated holdout; uncertainty estimates; calibrated human review of answer faithfulness | Not started |
| 7. Shared deployment | Auth, isolation, quotas, monitoring, backup/restore, retention and deployment smoke tests | Not ready; app is local-only |
| 8. Broader impact | Repeat use and documented improvements before expanding archives | Not measured |

Gates 2 and 4 do not block developing the review interface needed to complete them. They do block claims of scientific validation, benchmark rankings and public production readiness. Passing an automated test is not a guarantee of failure-proof operation.

## Next concrete work

1. Use the pilot guide to invite three to five relevant people; record permission to quote feedback separately. No outreach has been sent by the assistant.
2. Review all 20 question contracts, concentrating on reference grain, unit conventions, limit flags, nulls and catalog extrema. Preserve disagreement and sources.
3. Expand only the cases reviewers identify as useful. Keep new scientific families together when allocating future development/holdout splits.
4. Introduce a provider-neutral experiment record with prompt/schema versions, tool trace, costs, latency, attempts and failure origin. Start with imported or locally produced outputs.
5. Decide whether a small paid experiment would answer a specific unresolved question. The possible $10 cap requires a later explicit go-ahead; it has not been spent or activated.
6. Publish model results only after independent review and a frozen protocol. Deploy the shared service only after the operational gate passes.

## Decisions superseding the original plan

- Compare columns by declared meaning, not equal values under unrelated names.
- Preserve required ordering and duplicates, with explicit per-field tolerances and unit conversion.
- Treat empty/partial/unavailable results as inconclusive where correctness is unestablished.
- Do not use simultaneous archive requests as a substitute for an immutable snapshot.
- Separate public development data from a genuinely held-out future evaluation.
- Quarantine the observed direct-TAP TOP/ORDER ranking defect and verify references independently.
- Keep the first app and its review storage local; no model leaderboard or chat agent is implied by this prototype.

For implementation boundaries, see docs/ARCHITECTURE.md. For intended scientific use, see docs/DATASET_CARD.md. For recruitment and impact measures, see docs/PILOT_GUIDE.md.
Loading
Loading