Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions .claude/skills/workspace-ingest/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: workspace-ingest
version: 1.1.0
version: 1.2.0
description: Guide Codex to ingest a workspace into DataMind with source evidence, checked write receipts, and optional atomic publication.
keywords: [ingest, workspace, 入库, 导入, 文档, 表格]
---
Expand All @@ -9,7 +9,9 @@ keywords: [ingest, workspace, 入库, 导入, 文档, 表格]

Codex is the main agent: inspect sources, choose surfaces, extract relationships,
and generate SQL when needed. DataMind executes validated data operations.
MCP tools have the `datamind_` prefix; Python DataPlane tools omit it.
MCP tools have the `datamind_` prefix. The operation steps below describe MCP.
Python DataPlane tools omit the prefix and use a bound RequestContext; use
Workspace.reader for pinned Python reads.

1. Start `operation_begin(mode="write")`, then fetch this Skill with `skill_get`
using that operation_id. Reuse the ID for related calls so receipts record the
Expand All @@ -28,10 +30,12 @@ MCP tools have the `datamind_` prefix; Python DataPlane tools omit it.
from raw_file_read. Use `graph_upsert_triples` for explicit standalone facts;
use `graph_build_lineage` for file dependencies. These writes do not update
published workspace snapshots.
5. Every write returns an IngestReceipt: inspect each `results[].status`, error,
5. Every business data write returns an IngestReceipt: inspect each `results[].status`, error,
skipped sources and details. A receipt ID alone does not mean success.
Report the actual surfaces, receipt IDs, workspace revision, truncation and
failures. A failed publication keeps the previous revision readable.
failures. Build metadata tools (`build_start`, `build_freeze`, `build_export`)
return build state rather than ingestion receipts. A failed publication keeps
the previous revision readable.

KB still requires a configured embedding provider. Graph/SQL/Skills/Memory do
not require a DataMind conversation-model key. Do not switch providers silently.
Expand Down
113 changes: 25 additions & 88 deletions .env.datamind.example
Original file line number Diff line number Diff line change
@@ -1,101 +1,38 @@
# DataMind — new-style nested env (Phase 1+)
#
# Copy to `.env.datamind` (or merge into `.env`) and fill in real values.
# All new-code ENV vars use the DATAMIND__<section>__<field> prefix with
# double-underscore as the nested delimiter. The legacy .env.example keys
# (LLM_API_BASE, etc.) still drive the old main.py / server.py — nothing
# is broken by this file.

# ---- LLM (wire protocol is explicit) ----
DATAMIND__LLM__API_BASE=http://35.220.164.252:3888
DATAMIND__LLM__API_KEY=sk-your-key-here
DATAMIND__LLM__PROTOCOL=anthropic
# For an OpenAI-format gateway use:
# DATAMIND__LLM__PROTOCOL=openai_chat_completions
DATAMIND__LLM__MODEL=claude-sonnet-4-6
# Cheaper model used for subagents / background summarisation
DATAMIND__LLM__FALLBACK_MODEL=claude-haiku-4-5-20251001
# Defaults to the main protocol. Set only if the same gateway exposes the
# fallback model through a different wire protocol.
# DATAMIND__LLM__FALLBACK_PROTOCOL=anthropic
DATAMIND__LLM__TIMEOUT_S=60
DATAMIND__LLM__CONNECT_TIMEOUT_S=10
DATAMIND__LLM__MAX_RETRIES=3

# ---- Embedding ----
# provider: must match a name registered in capabilities/embedding/providers/
# Built-ins (to be added in Phase 2): openai, voyage, huggingface, siliconflow
# DataMind 2: Codex is the main agent; DataMind supplies data tools.
# Copy to .env.datamind. Keep secrets out of version control.
DATAMIND__DATA__PROFILE=default
# DATAMIND__DATA__BASE_DIR=/srv/datamind

# CLI / HTTP / UI Codex host. Omit model/provider to inherit Codex login/config.
# DATAMIND__CODEX__MODEL=your-supported-codex-model
# DATAMIND__CODEX__BASE_URL=https://your-gateway.example/v1
# DATAMIND__CODEX__API_KEY=your-private-key
# DATAMIND__CODEX__WORKSPACE=/srv/workspace
# DATAMIND__CODEX__REASONING_EFFORT=low
DATAMIND__CODEX__APPROVAL_MODE=auto_review
DATAMIND__CODEX__TIMEOUT_S=600
DATAMIND__CODEX__MAX_CONCURRENCY=4
# A trusted unattended host can use deny_all plus an explicitly authorized list.
# DATAMIND__CODEX__PREAPPROVED_TOOLS=["datamind_operation_begin"]

# KB embeddings are independent of the main Codex model.
DATAMIND__EMBEDDING__PROVIDER=openai
DATAMIND__EMBEDDING__API_BASE=https://api.openai.com/v1
DATAMIND__EMBEDDING__API_KEY=sk-your-key-here
DATAMIND__EMBEDDING__MODEL=text-embedding-3-small
# Leave blank to auto-detect dimension at first call
# DATAMIND__EMBEDDING__DIMENSION=
DATAMIND__EMBEDDING__BATCH_SIZE=32
DATAMIND__EMBEDDING__TIMEOUT_S=30
DATAMIND__EMBEDDING__CONNECT_TIMEOUT_S=10
DATAMIND__EMBEDDING__MAX_RETRIES=3
# DATAMIND__EMBEDDING__MAX_BATCH_TOKENS=8000
# DATAMIND__EMBEDDING__API_KEY=your-embedding-key
# Or install .[huggingface] and use a local model:
# DATAMIND__EMBEDDING__PROVIDER=huggingface
# DATAMIND__EMBEDDING__MODEL=BAAI/bge-small-zh-v1.5
# DATAMIND__EMBEDDING__DIMENSION=512

# ---- Retrieval ----
# strategy: simple | multi_query | hybrid (registered in capabilities/kb/providers/)
DATAMIND__RETRIEVAL__STRATEGY=simple
DATAMIND__RETRIEVAL__TOP_K=5
DATAMIND__RETRIEVAL__CHUNK_SIZE=512
DATAMIND__RETRIEVAL__CHUNK_OVERLAP=64
DATAMIND__RETRIEVAL__RERANK=false
# DATAMIND__RETRIEVAL__RERANK_MODEL=

# ---- Graph ----
# backend: networkx (built-in) | neo4j (future)
DATAMIND__GRAPH__BACKEND=networkx
# DATAMIND__GRAPH__DSN=bolt://user:pw@localhost:7687
DATAMIND__GRAPH__EMBED_ENTITIES=false

# ---- Database ----
# dialect: sqlite | mysql | postgres (registered in capabilities/db/providers/)
DATAMIND__DB__DIALECT=sqlite
# SQLite example (blank DSN = auto under storage/<profile>/demo.db)
# DATAMIND__DB__DSN=
# MySQL example:
# DATAMIND__DB__DIALECT=mysql
# DATAMIND__DB__DSN=mysql+pymysql://user:password@host:3306/dbname
# Postgres example:
# DATAMIND__DB__DIALECT=postgres
# DATAMIND__DB__DSN=postgresql+psycopg://user:password@host:5432/dbname
DATAMIND__DB__READ_ONLY=true
DATAMIND__DB__ROW_LIMIT=1000
DATAMIND__DB__QUERY_TIMEOUT_S=10.0

# ---- Memory ----
# backend: sqlite (built-in) | redis (future) | postgres (future)
DATAMIND__MEMORY__BACKEND=sqlite
# DATAMIND__MEMORY__DSN=
DATAMIND__MEMORY__SHORT_TERM_TURNS=20
DATAMIND__MEMORY__LONG_TERM_ENABLED=true

# ---- Data profile ----
# Switches data/profiles/<profile>/ and storage/<profile>/ in lockstep.
DATAMIND__DATA__PROFILE=default

# ---- Agent loop backend ----
# "native": built-in tool-use loop on the anthropic SDK — talks directly
# to whatever DATAMIND__LLM__API_BASE points at. Default. No
# extra process required.
# "sdk": uses claude-agent-sdk (which spawns the `claude` CLI) and
# routes through a local claude-code-router (CCR) instance.
# Best when your upstream gateway only speaks OpenAI format —
# CCR translates Anthropic↔OpenAI. Unlocks Hooks / Subagents /
# Compaction / Plan mode.
# Both backends use the SAME 23 DataMind tools and emit the SAME SSE
# events, so server / CLI / frontend don't care which is active.
DATAMIND__AGENT__BACKEND=native
# Only relevant when BACKEND=sdk:
# DATAMIND__AGENT__CCR_BASE_URL=http://127.0.0.1:13456
# DATAMIND__AGENT__CCR_API_KEY=dummy
DATAMIND__AGENT__MAX_TURNS=12
DATAMIND__AGENT__MAX_TOOL_CALLS=24
DATAMIND__AGENT__MAX_INPUT_TOKENS=120000
DATAMIND__AGENT__MAX_TOOL_RESULT_CHARS=12000
DATAMIND__AGENT__MAX_TOOL_RESULT_ROWS=100
DATAMIND__AGENT__WALL_CLOCK_TIMEOUT_S=300
DATAMIND__HOOKS__ENABLED=true
# DATAMIND__HOOKS__PATH_ALLOWLIST_EXTRA=["/srv/authorized-documents"]
4 changes: 4 additions & 0 deletions .github/workflows/python-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,10 @@ jobs:
python scripts/ci_required_tests.py -q
datamind/tests/test_direct_data_plane.py
datamind/tests/test_codex_host.py
datamind/tests/test_unified_host.py
datamind/tests/test_cli_codex.py
datamind/tests/test_http_api.py
datamind/tests/test_experiment_faults.py
plugins/datamind-context/tests/test_datamind_mcp.py

gate:
Expand Down
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
# Changelog

## 2.0.0 — Unified Codex host

- CLI, HTTP and browser UI now use one official Codex SDK host for reads and writes.
- Removed StoreAgent/RetrieveAgent classes, assembly, role prompts and Claude SDK/CCR adapter.
- Data services assemble independently of all agent loops and conversation credentials.
- Read requests bind profile/session, access mode and one published revision below the prompt boundary.
- Request-scoped traces preserve actual MCP evidence and receipts under Codex code mode; cancellation closes active SDK turns.
- Migrated examples/benchmarks, retained checkpoint and fault tests, and updated gateway/config migration guidance.


## 1.2.0 — Direct Codex data layer

- Local Codex plugin and optional official `openai-codex` SDK host share direct
Expand Down
Loading
Loading