diff --git a/AGENTS.md b/AGENTS.md index a389ae0..63ee628 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -3,9 +3,10 @@ **For AI agents:** Use AST-RAG to analyze this codebase. Don't guess — look up via tools. --- -After any refactoring or multi-file changes, always verify all import paths work by running the code or tests before considering the task complete +After any refactoring or multi-file changes, always verify all import paths work by running the code or tests before considering the task complete. + +Always confirm documentation language preference before creating/translating docs (default to English unless specified). -.qweAlways confirm documentation language preference before creating/translating docs (default to English unless specified) ## 🎯 Quick Start **If unsure where to start:** @@ -36,15 +37,16 @@ ast-rag callers --depth 2 | `sig` | `ast-rag sig "process(int, String)"` | Signature search | | `evaluate` | `ast-rag evaluate --all` | Check quality | | `index-folder` | `ast-rag index-folder ./ast_rag` | Index a folder | -| `workspace` | `ast-rag workspace . --apply` | Update from git diff | +| `workspace` | `ast-rag workspace . --apply` | Update from uncommitted changes | +| `stats` | `ast-rag stats` | See what is actually indexed | ### Python API ```python from ast_rag.ast_rag_api import ASTRagAPI from ast_rag.models import ProjectConfig -from ast_rag.graph_schema import create_driver -from ast_rag.embeddings import EmbeddingManager +from ast_rag.repositories import create_driver +from ast_rag.services import EmbeddingManager # Initialize cfg = ProjectConfig() @@ -146,11 +148,8 @@ curl http://localhost:6333/collections **Check index:** ```bash -# How many nodes in graph -cypher-shell "MATCH (n) RETURN count(n)" - -# How many files indexed -grep "COMPLETE" /tmp/index_*.log | wc -l +# Node/edge counts, language distribution, indexed file count +ast-rag stats ``` --- @@ -191,10 +190,10 @@ ast-rag evaluate --all ```bash # Check what's indexed -grep "COMPLETE" /tmp/index_*.log | wc -l +ast-rag stats -# Index remaining -./scripts/index-remaining.sh +# Index what is missing +ast-rag index-folder ./path/to/folder --no-schema # Run again ast-rag evaluate --all diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e739f2c..20897d9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -145,7 +145,7 @@ pytest tests/ -v ## 🎯 Areas Needing Contribution -- [ ] More language support (Go, Kotlin, Swift) +- [ ] More language support (C#, Kotlin, Swift) - [ ] Performance optimizations - [ ] Additional MCP tools - [ ] Better error handling diff --git a/README.md b/README.md index 0a9b3ca..df7d992 100644 --- a/README.md +++ b/README.md @@ -59,6 +59,20 @@ export AST_RAG_QDRANT_URL=http://myhost:6333 Or copy `ast_rag_config.example.json` to `ast_rag_config.json` and edit it (it is gitignored). +### Optional extras + +`pip install -e .` gives you the `ast-rag` CLI. The other two console scripts +need extras: + +```bash +pip install -e ".[mcp]" # ast-rag-mcp (MCP server), ast-rag-watch (file watcher) +pip install -e ".[server]" # embedding_server.py, for the GPU host +pip install -e ".[dev]" # pytest, mypy, ruff +``` + +Without `[mcp]`, `ast-rag-mcp` and `ast-rag-watch` are on `PATH` but fail with +`ModuleNotFoundError: No module named 'mcp'` / `'watchdog'`. + ## ⚙️ Configuration Create `ast_rag_config.json` in project root: @@ -75,12 +89,24 @@ Create `ast_rag_config.json` in project root: "collection_name": "ast_rag_nodes" }, "embedding": { - "model_name": "bge-m3", - "remote_url": "http://localhost:1113/v1/embeddings" + "model_name": "BAAI/bge-m3" } } ``` +Embeddings are computed locally by default. To offload them to a separate +embedding server, add `remote_url` — and `dimension` with it, which is required +whenever `remote_url` is set (`docker compose up -d` does not start such a +server): + +```json +"embedding": { + "model_name": "bge-m3", + "remote_url": "http://localhost:1113/v1/embeddings", + "dimension": 1024 +} +``` + ## 🎯 Quick Start ```bash @@ -116,30 +142,51 @@ ast-rag evaluate --all ## 🛠️ CLI Commands ``` -ast-rag init # Full indexing -ast-rag update # Update from git diff -ast-rag query "" # Semantic search -ast-rag goto # Find definition -ast-rag callers # Find callers -ast-rag refs # Find references -ast-rag sig # Signature search -ast-rag evaluate # Quality evaluation -ast-rag index-folder # Index a folder -ast-rag workspace # Show workspace changes -ast-rag sandbox # Run in Docker sandbox +ast-rag init # Full indexing +ast-rag index-folder # Index a folder +ast-rag create-database # Create a Neo4j database +ast-rag update --from-commit --to-commit + # Update from git diff a..b +ast-rag workspace # Show workspace changes +ast-rag query "" # Semantic search +ast-rag goto # Find definition +ast-rag callers # Find callers +ast-rag refs # Find references +ast-rag call-graph # Callers/callees graph around a function +ast-rag symbol-impact # Definition + refs + callers + callees +ast-rag sig # Signature search +ast-rag blocks # if/for/while/try/lambda/with blocks +ast-rag lambdas # List lambda/closure blocks +ast-rag summarize # LLM summary of a function/method/class +ast-rag analyze-stacktrace [file] # Map a stack trace onto the graph +ast-rag stats # Statistics about the indexed codebase +ast-rag cache-stats # Parse cache statistics +ast-rag evaluate # Quality evaluation +ast-rag sandbox [workdir] --cmd "" + # Run in Docker sandbox ``` +`ast-rag --help` lists the flags for each. + ## 📊 Quality -Current metrics (Phase 2): +Phase 2 benchmark run, recorded 2026-02-26 and not re-measured since: -| Metric | Target | Actual | +| Metric | Target | Measured | |--------|--------|--------| | **Pass Rate** | >80% | **100%** ✅ | | **F1 Score** | >0.85 | **0.98** ✅ | | **Precision** | >0.85 | **0.98** ✅ | | **Recall** | >0.85 | **0.97** ✅ | +The benchmarks in `benchmarks/queries/` are ground-truthed against AST-RAG's own +source, so they measure this repo rather than yours. To reproduce, index this +repo with Neo4j and Qdrant running, then from the repo root: + +```bash +ast-rag evaluate --all +``` + ## 🏗️ Architecture ``` @@ -201,9 +248,9 @@ Planned features and improvements: | Feature | Status | Description | |---------|--------|-------------| -| **Code Summaries** | 🔜 Planned | Generate AI-powered summaries for functions/classes | +| **Code Summaries** | ✅ Done | `ast-rag summarize` — LLM summaries for functions/classes ([docs](docs/SUMMARIZATION.md)) | | **Refactoring Hints** | 🔜 Planned | Detect code smells and suggest improvements | -| **More Languages** | 🔜 Planned | Go, C#, Kotlin with full AST support | +| **More Languages** | 🔜 Planned | C#, Kotlin with full AST support | | **IDE Integration** | 🔄 In Progress | MCP, skills, CLI for OpenCode, Kilocode, Claude Code, Cursor | | **Incremental Indexing** | ✅ Done | Git-based and filesystem watcher updates (improving for large codebases) | | **Rust Rewrite** | 🔜 Planned | Full rewrite in Rust for performance, type safety, and easier integration | diff --git a/docs/QUICKSTART.md b/docs/QUICKSTART.md index da1ab9a..b23588a 100644 --- a/docs/QUICKSTART.md +++ b/docs/QUICKSTART.md @@ -69,12 +69,18 @@ cat > ast_rag_config.json < --snippet # 3. Find what it calls (call graph) -ast-rag callers --depth 2 +ast-rag call-graph --direction callees --depth 2 # 4. Find key dependencies ast-rag query "database connection manager" --limit 5 diff --git a/docs/STACK_TRACE_MAPPING.md b/docs/STACK_TRACE_MAPPING.md index f97a42e..c65c698 100644 --- a/docs/STACK_TRACE_MAPPING.md +++ b/docs/STACK_TRACE_MAPPING.md @@ -1,17 +1,17 @@ # Smart Stack Trace Mapping -Сервис для автоматического анализа стектрейсов с маппингом на AST-ноды, извлечением кода и рекомендациями по исправлению. +A service that analyzes stack traces: it maps each frame to an AST node, pulls the code, and suggests fixes. -## Возможности +## Features -- **Парсинг стектрейсов** для 4 языков: Python, C++, Java, Rust -- **Маппинг на AST** — каждый фрейм сопоставляется с узлом в графе кода -- **Извлечение кода** — получение snippet'ов кода для каждого фрейма -- **Анализ root cause** — определение категории ошибки,severity, причин -- **Рекомендации по фиксу** — конкретные шаги для исправления -- **Поиск похожих проблем** — семантический поиск аналогичных ошибок в кодебазе +- **Stack trace parsing** for 4 languages: Python, C++, Java, Rust +- **AST mapping**: each frame is matched to a node in the code graph +- **Code extraction**: a code snippet for each frame +- **Root cause analysis**: error category, severity and likely causes +- **Fix suggestions**: concrete steps to fix the error +- **Similar issue search**: semantic search for similar errors in the codebase -## Поддерживаемые форматы стектрейсов +## Supported stack trace formats ### Python ``` @@ -52,27 +52,27 @@ stack backtrace: at src/main.rs:10:1 ``` -## Использование +## Usage ### CLI ```bash -# Анализ из файла +# Analyze a file ast-rag analyze-stacktrace error.log -# Анализ из stdin +# Analyze stdin echo "$STACKTRACE" | ast-rag analyze-stacktrace -# Вывод в JSON +# JSON output ast-rag analyze-stacktrace error.log -o json -# Вывод в текстовом формате +# Plain text output ast-rag analyze-stacktrace error.log -o text -# Пропустить AST маппинг для скорости +# Skip AST mapping for speed ast-rag analyze-stacktrace error.log --no-ast-mapping -# Подробный вывод +# Verbose output ast-rag analyze-stacktrace error.log -v ``` @@ -80,18 +80,18 @@ ast-rag analyze-stacktrace error.log -v ```python from ast_rag.stack_trace import StackTraceService -from ast_rag.graph_schema import create_driver -from ast_rag.embeddings import EmbeddingManager +from ast_rag.repositories import create_driver +from ast_rag.services import EmbeddingManager from ast_rag.models import ProjectConfig -# Инициализация +# Initialize config = ProjectConfig.model_validate_json(open("ast_rag_config.json").read()) driver = create_driver(config.neo4j) embed = EmbeddingManager(config.qdrant, config.embedding, neo4j_driver=driver) service = StackTraceService(driver, embed) -# Анализ стектрейса +# Analyze a stack trace trace = """ Traceback (most recent call last): File "main.py", line 42, in @@ -101,21 +101,21 @@ ValueError: Invalid input report = service.analyze(trace) -# Вывод результатов -print(report.to_markdown()) # Markdown для человека -print(report.to_json()) # JSON для машины +# Print the results +print(report.to_markdown()) # Markdown for people +print(report.to_json()) # JSON for machines -# Доступ к деталям +# Details print(f"Error: {report.error_type}") print(f"Root cause: {report.root_cause.likely_cause}") print(f"Suggested fix: {report.root_cause.suggested_fix}") print(f"Mapped frames: {report.mapped_frames}/{report.total_frames}") -# Анализ из файла +# Analyze a file report = service.analyze_from_file("error.log") ``` -## Архитектура +## Architecture ``` ┌─────────────────────────────────────────────────────────┐ @@ -179,7 +179,7 @@ report = service.analyze_from_file("error.log") └─────────────────────────────────────────────────────────┘ ``` -## Модель данных +## Data model ### StackFrame ```json @@ -226,10 +226,10 @@ report = service.analyze_from_file("error.log") } ``` -## Категории ошибок +## Error categories -| Категория | Примеры | Severity | -|-----------|---------|----------| +| Category | Examples | Severity | +|----------|----------|----------| | `null_pointer` | NullPointerException, NoneType | high | | `out_of_bounds` | IndexError, out_of_range | high | | `type_error` | TypeError, ClassCastException | medium | @@ -241,43 +241,43 @@ report = service.analyze_from_file("error.log") | `concurrency` | ConcurrentModification, Deadlock | critical | | `panic` | panic, assertion failed | critical | -## Интеграция с analyze_text +## Integration with analyze_text -Сервис использует существующий API `analyze_text` для дополнительного контекста: +The service uses the existing `analyze_text` API for extra context: ```python -# В StackTraceService.analyze() +# In StackTraceService.analyze() text_results = self._analyze_with_text_api(stacktrace) if text_results and not report.similar_issues: report.similar_issues = self._convert_text_results_to_issues(text_results) ``` -Это позволяет находить релевантный код даже когда точный маппинг на AST не удался. +This finds relevant code even when exact AST mapping fails. -## Тесты +## Tests ```bash -# Запустить тесты +# Run the tests pytest tests/test_stack_trace.py -v -# Тесты парсеров +# Parser tests pytest tests/test_stack_trace.py::TestPythonParser -v pytest tests/test_stack_trace.py::TestJavaParser -v pytest tests/test_stack_trace.py::TestCppParser -v pytest tests/test_stack_trace.py::TestRustParser -v -# Тесты моделей +# Model tests pytest tests/test_stack_trace.py::TestStackFrame -v pytest tests/test_stack_trace.py::TestStackTraceReport -v ``` -## Примеры +## Examples -См. `ast_rag/stack_trace/examples.py` для примеров стектрейсов и использования. +See `ast_rag/stack_trace/examples.py` for sample stack traces and usage. -## Расширение +## Extending -### Добавление нового парсера +### Adding a new parser ```python from .models import StackFrame, Language @@ -285,33 +285,33 @@ from .parsers import StackTraceParser class GoParser(StackTraceParser): def detect_language(self, stacktrace: str) -> Language: - # Логика детектирования - return Language.UNKNOWN # или новый enum + # Detection logic + return Language.UNKNOWN # or a new enum value def extract_error_info(self, stacktrace: str) -> tuple[str, str]: - # Извлечение типа и сообщения ошибки + # Extract the error type and message return "Error", "" def parse(self, stacktrace: str) -> list[StackFrame]: - # Парсинг фреймов + # Parse the frames return [] -# Регистрация в фабрике +# Register it in the factory StackTraceParserFactory._parsers[Language.GO] = GoParser ``` -## Ограничения +## Limitations -- Требуется запущенный Neo4j для AST маппинга -- Требуется запущенный Qdrant для семантического поиска -- Точность маппинга зависит от полноты индексации кодебазы -- Некоторые форматы стектрейсов могут требовать доработки парсеров +- AST mapping needs a running Neo4j +- Semantic search needs a running Qdrant +- Mapping accuracy depends on how completely the codebase is indexed +- Some stack trace formats may need parser work -## Будущие улучшения +## Future improvements -- [ ] Поддержка Go, TypeScript, C# -- [ ] Парсинг JSON/XML логов ошибок -- [ ] Интеграция с GitHub Issues для поиска похожих проблем -- [ ] ML-модель для классификации ошибок -- [ ] Автоматическое создание PR с фиксом -- [ ] Статистика частых ошибок по проекту +- [ ] Go, TypeScript and C# support +- [ ] Parse JSON/XML error logs +- [ ] GitHub Issues integration to find similar problems +- [ ] ML model for error classification +- [ ] Open a PR with the fix automatically +- [ ] Per-project stats on frequent errors diff --git a/docs/SUMMARIZATION.md b/docs/SUMMARIZATION.md index d9f61a2..b1d27ac 100644 --- a/docs/SUMMARIZATION.md +++ b/docs/SUMMARIZATION.md @@ -51,13 +51,13 @@ ast-rag summarize MyFunction --max-callers 10 --max-callees 10 ```python from ast_rag.ast_rag_api import ASTRagAPI -from ast_rag.graph_schema import create_driver -from ast_rag.embeddings import EmbeddingManager +from ast_rag.repositories import create_driver +from ast_rag.services import EmbeddingManager from ast_rag.summarizer import SummarizerService, NodeSummary from ast_rag.models import ProjectConfig -# Load configuration -cfg = ProjectConfig.model_validate_json("ast_rag_config.json") +# Load configuration (model_validate_json takes JSON text, not a path) +cfg = ProjectConfig.model_validate_json(open("ast_rag_config.json").read()) # Initialize API driver = create_driver(cfg.neo4j) @@ -187,7 +187,7 @@ Processes HTTP requests by validating input, calling external services, and retu |-----------|---------|-------------| | `base_url` | `http://localhost:11434/v1` | OpenAI-compatible API endpoint | | `model` | `qwen2.5-coder:14b` | Model name for summarization | -| `api_key` | `ollama` | API key (not needed for Ollama) | +| `api_key` | `None` | API key (not needed for Ollama) | | `timeout` | `120` | Request timeout in seconds | ### Supported LLM Backends diff --git a/docs/agent-scenarios.md b/docs/agent-scenarios.md index 04c1914..e42b775 100644 --- a/docs/agent-scenarios.md +++ b/docs/agent-scenarios.md @@ -154,8 +154,8 @@ ast-rag evaluate --all # Index specific folder ast-rag index-folder ./src/modified_module --no-schema -# Or all remaining -./scripts/index-remaining.sh +# Or re-index everything +ast-rag init . ``` --- @@ -172,8 +172,8 @@ ast-rag sig "process(int, String)" ast-rag sig "get*" --lang java ast-rag sig "*Handler" --lang python -# With filter -ast-rag sig "build*" --lang rust --kind Function +# With a result cap +ast-rag sig "build*" --lang rust --limit 50 ``` **Python API:** @@ -203,11 +203,11 @@ ast-rag evaluate --query benchmarks/queries/def_001.json **If quality is low:** ```bash -# Check how many indexed -grep "COMPLETE" /tmp/index_*.log | wc -l +# Check what is in the graph +ast-rag stats -# Index remaining -./scripts/index-remaining.sh +# Index what is missing +ast-rag index-folder ./path/to/folder --no-schema # Check again ast-rag evaluate --all @@ -227,7 +227,7 @@ cypher-shell "MATCH (n) RETURN count(n)" cypher-shell "MATCH (n:Method) RETURN count(n)" # 3. Check indexing -grep "COMPLETE" /tmp/index_*.log | wc -l +ast-rag stats # 4. Re-index ast-rag init /path/to/codebase diff --git a/docs/configuration.md b/docs/configuration.md index 9718d79..8d86d51 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -28,14 +28,20 @@ ### Environment Variables (optional) +These are applied last, on top of anything loaded from `ast_rag_config.json`: + ```bash -export NEO4J_URI="bolt://localhost:7687" -export NEO4J_USER="neo4j" -export NEO4J_PASSWORD="password" -export QDRANT_URL="http://localhost:6333" -export EMBEDDING_URL="http://localhost:1113/v1/embeddings" +export AST_RAG_NEO4J_URI="bolt://localhost:7687" +export AST_RAG_NEO4J_USER="neo4j" +export AST_RAG_NEO4J_PASSWORD="password" +export AST_RAG_NEO4J_DATABASE="neo4j" +export AST_RAG_QDRANT_URL="http://localhost:6333" +export AST_RAG_QDRANT_COLLECTION="ast_rag_nodes" ``` +That is the complete set. There is no environment override for the embedding +URL — set `embedding.remote_url` in the config file. + --- ## 🔧 Troubleshooting @@ -55,7 +61,7 @@ export EMBEDDING_URL="http://localhost:1113/v1/embeddings" | Connection refused | Neo4j not running | `docker run -d --name neo4j -p 7687:7687 neo4j:latest` | | Auth failed | Wrong password | Check `NEO4J_PASSWORD` in config | | Empty results | Graph not indexed | `ast-rag init /path/to/codebase` | -| Deadlock error | Parallel indexing | Use `./scripts/index-sequential.sh` | +| Deadlock error | Parallel indexing | Index serially: `ast-rag index-folder --workers 1` | ### Qdrant @@ -78,8 +84,8 @@ export EMBEDDING_URL="http://localhost:1113/v1/embeddings" | Issue | Cause | Fix | |-------|-------|-----| | Empty results | Graph not indexed | `ast-rag init .` | -| Low recall (<70%) | Not all files indexed | `./scripts/index-remaining.sh` | -| Low precision | Noise in embeddings | Tune thresholds in `embeddings.py` | +| Low recall (<70%) | Not all files indexed | Check with `ast-rag stats`, then `ast-rag index-folder --no-schema` | +| Low precision | Noise in embeddings | Retune the hybrid weights: `ast-rag query ... --vector-weight 0.8 --keyword-weight 0.2` (defaults 0.7/0.3, or set `embedding.vector_weight` / `embedding.keyword_weight`) | | Stale results | Index outdated | `ast-rag workspace . --apply` | --- @@ -120,14 +126,14 @@ cypher-shell "MATCH ()-[r]->() RETURN type(r) as type, count(r) ORDER BY count(r ### Check Indexing ```bash -# How many folders indexed -grep "COMPLETE" /tmp/index_*.log | wc -l +# Node/edge counts, language distribution, indexed file count +ast-rag stats -# Which folders completed -grep "COMPLETE" /tmp/index_*.log | cut -d: -f2 +# Same, as JSON +ast-rag stats --json -# Errors -grep "ERROR" /tmp/index_*.log +# Parse cache hit rate and configuration +ast-rag cache-stats ``` ### Check Quality @@ -191,8 +197,8 @@ performance: ### Enable Debug Logs ```bash -# For CLI -ast-rag query "test" --verbose +# For CLI (init, update, sig, evaluate, index-folder, workspace, summarize, ...) +ast-rag index-folder ./src --verbose # For Python import logging @@ -201,15 +207,12 @@ logging.basicConfig(level=logging.DEBUG) ### Indexing Logs -```bash -# Main log -tail -f index_sequential.log +Indexing logs to stderr; redirect it if you want a file to tail: -# Per-folder logs -tail -f /tmp/index_*.log - -# Errors -grep "ERROR" /tmp/index_*.log | tail -20 +```bash +ast-rag index-folder ./src --verbose 2>index.log +tail -f index.log +grep "ERROR" index.log | tail -20 ``` --- diff --git a/docs/graph-schema.md b/docs/graph-schema.md index 35cc1ce..a71e597 100644 --- a/docs/graph-schema.md +++ b/docs/graph-schema.md @@ -4,19 +4,26 @@ See also: [schema/graph_schema.cql](../schema/graph_schema.cql) for the raw Cyph ## Node Labels -`Project`, `Package`, `Module`, `Namespace`, `File`, `Class`, `Interface`, `Struct`, `Enum`, `Trait`, `Function`, `Method`, `Constructor`, `Destructor`, `Field`, `Variable`, `Parameter` +`Project`, `Package`, `Module`, `Namespace`, `File`, `Class`, `Interface`, `Struct`, `Enum`, `Trait`, `Function`, `Method`, `Constructor`, `Destructor`, `Field`, `Variable`, `Parameter`, `Block` ## Edge Types | Edge | Meaning | |---|---| -| `CONTAINS_*` | Structural nesting: Project → Package → File → Class → Method | +| `CONTAINS_*` | Structural nesting: Project → Package → File → Class → Method → Block | | `IMPORTS` / `INCLUDES` | File-level dependencies | | `CALLS` | Function/method invocation | -| `INHERITS` / `EXTENDS` | Class inheritance | +| `CROSS_FILE_CALL` | Call resolved to a definition in another file | +| `INHERITS` / `EXTENDS` | Class inheritance (`EXTENDS` for Java, `INHERITS` elsewhere) | | `IMPLEMENTS` | Interface/trait implementation | | `INJECTS` | Dependency injection — heuristic | | `OVERRIDES` | Method override | +| `TYPES` | Symbol is used as a type annotation | +| `DEPENDS_ON` | Field/variable declaration depends on a type | + +`EdgeKind` also declares `HAS_PARAMETER`, `VIRTUAL_CALL` and `LAMBDA_CALL`. +Queries treat `VIRTUAL_CALL` and `LAMBDA_CALL` as call edges and the schema +indexes them, but nothing emits any of the three yet. ## MVCC Properties @@ -31,5 +38,9 @@ Special singleton: `CurrentVersion { hash }` — tracks the active graph version ## Stable IDs -- **Node ID**: `SHA256(file_path : node_type : qualified_name)[:24]` -- **Edge ID**: `SHA256(from_id : edge_type : to_id)[:24]` +- **Node ID**: `SHA256(project_id : file_path : kind : qualified_name)[:24]` +- **Edge ID**: `SHA256(from_id : kind : to_id : dep_kind : raw_type_string : confidence)[:24]` +- **Block ID**: `SHA256(file_path : block_type : parent_function_id : start_line)[:24]` + +`project_id` is part of the node hash so several projects can share one graph. +The trailing edge fields are empty strings / `0.0` when unset. diff --git a/docs/python-api.md b/docs/python-api.md index 9356ec4..18d1f16 100644 --- a/docs/python-api.md +++ b/docs/python-api.md @@ -9,8 +9,8 @@ Programmatic access to AST-RAG from scripts and applications. ```python from ast_rag.ast_rag_api import ASTRagAPI from ast_rag.models import ProjectConfig -from ast_rag.graph_schema import create_driver -from ast_rag.embeddings import EmbeddingManager +from ast_rag.repositories import create_driver +from ast_rag.services import EmbeddingManager # Initialize cfg = ProjectConfig() @@ -63,12 +63,16 @@ for n in nodes: ### Find References +`find_references` returns a paginated dict, and each reference is a plain dict: + ```python # All usages -refs = api.find_references("processRequest", kind="Method") +result = api.find_references("processRequest", kind="Method", limit=50, offset=0) +# {"references": [...], "total": int, "limit": int, "offset": int, "has_more": bool} -for ref in refs: - print(f"{ref.node.file_path}:{ref.node.start_line} {ref.reference_type}") +for ref in result["references"]: + node = ref["node"] + print(f"{node['file_path']}:{node['start_line']} {ref['reference_type']}") ``` --- @@ -131,25 +135,37 @@ print(code) ```python diff = api.get_diff( + repo_path="/path/to/codebase", from_commit="abc123", - to_commit="def456", - file_path="src/main.py" + to_commit="def456", # defaults to HEAD ) -print(f"Added: {diff.added}, Removed: {diff.removed}") +print(f"Added: {diff['added_count']}, Deleted: {diff['deleted_count']}, " + f"Updated: {diff['updated_count']}") ``` +`get_diff` covers the whole repository — there is no per-file filter — and +returns a dict with `added` / `deleted` / `updated` node lists alongside those +counts, plus `limit`, `offset` and `has_more`. + --- ## 📊 Quality Evaluation -```python -from ast_rag.benchmarks.evaluator import BenchmarkEvaluator +There is no Python entry point for the benchmarks — they run through the CLI, +from the repo root: -evaluator = BenchmarkEvaluator() +```bash +ast-rag evaluate --all --output results.json +``` -# Run all benchmarks -results = evaluator.run_all() +The JSON it writes (`benchmarks/results/evaluation.json` by default) can be read +back directly: + +```python +import json + +results = json.load(open("benchmarks/results/evaluation.json")) print(f"Pass Rate: {results['pass_rate']*100:.1f}%") print(f"F1 Score: {results['average_metrics']['f1_score']:.2f}") @@ -159,31 +175,35 @@ print(f"F1 Score: {results['average_metrics']['f1_score']:.2f}") ## 🔄 Indexing -### Full Indexing +There is no single "index this directory" function — walking files, parsing and +writing to Neo4j are separate steps. For a whole codebase, use the CLI +(`ast-rag init ` / `ast-rag index-folder `); the pieces below are for +building on top of it. + +### Parse a File ```python -from ast_rag.graph_updater import full_index -from ast_rag.graph_schema import apply_schema +from ast_rag.services.parsing import ParserManager -# Apply schema -apply_schema(driver) +pm = ParserManager() -# Index -stats = full_index(driver, "/path/to/codebase", commit="v1.0") +path = "/path/to/file.py" +lang = pm.detect_language(path) # "python", "go", "java", ... or None +source = open(path, "rb").read() -print(f"Nodes: {stats.nodes}, Edges: {stats.edges}") +tree = pm.parse_file(path, source=source) +nodes = pm.extract_nodes(tree, path, lang, source, commit_hash="v1.0") +edges = pm.extract_edges(tree, nodes, path, lang, source, commit_hash="v1.0") ``` -### Index Folder +### Write to the Graph ```python -from ast_rag.graph_updater import index_directory +from ast_rag.repositories import apply_schema +from ast_rag.graph_updater import full_index -stats = index_directory( - driver, - "/path/to/folder", - exclude_patterns=[".git", "venv", "__pycache__"] -) +apply_schema(driver) +full_index(driver, nodes, edges, commit_hash="v1.0") # returns None ``` ### Update from Git Diff @@ -191,14 +211,14 @@ stats = index_directory( ```python from ast_rag.graph_updater import update_from_git -diff_stats = update_from_git( +diff = update_from_git( driver, - root="/path/to/codebase", - from_commit="HEAD~1", - to_commit="HEAD" + repo_path="/path/to/codebase", + old_commit="HEAD~1", + new_commit="HEAD", ) -print(f"Changed: +{diff_stats.added}, -{diff_stats.deleted}") +print(f"+{len(diff.added_nodes)} nodes, -{len(diff.deleted_node_ids)} nodes") ``` ### Update Workspace @@ -206,16 +226,20 @@ print(f"Changed: +{diff_stats.added}, -{diff_stats.deleted}") ```python from ast_rag.graph_updater import get_workspace_diff, apply_workspace_diff -# Get changes -diff = get_workspace_diff(driver, root=".") +# Get changes (read-only) +diff = get_workspace_diff(driver, repo_path=".") if not diff.is_empty: print(f"+{len(diff.added_nodes)} nodes, +{len(diff.added_edges)} edges") - + # Apply - apply_workspace_diff(driver, root=".") + apply_workspace_diff(driver, repo_path=".") ``` +All three return a `DiffResult` with `added_nodes`, `deleted_node_ids`, +`updated_nodes`, `added_edges`, `deleted_edge_ids`, `updated_edges` and the +`is_empty` property. + --- ## 🔧 Utilities @@ -223,7 +247,7 @@ if not diff.is_empty: ### Check Connection ```python -from ast_rag.graph_schema import create_driver +from ast_rag.repositories import create_driver from ast_rag.models import ProjectConfig cfg = ProjectConfig() @@ -320,7 +344,7 @@ for cls in classes: print(f"File: {cls.file_path}\n") for method in subgraph.nodes: - if method.node_type == "Method": + if method.kind == "Method": print(f"- `{method.name}`") ```