From 09855fa83097930d542252228cf7eeee3c5d23db Mon Sep 17 00:00:00 2001 From: Santiago Garces Escobar Date: Thu, 24 Sep 2026 23:35:29 -0400 Subject: [PATCH] =?UTF-8?q?docs:=20bring=20ArcGIS=20and=20security=20docs?= =?UTF-8?q?=20in=20line=20with=20#33=E2=80=93#38?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - ARCHITECTURE.md: the ArcGIS tool table listed four tools with their old arguments; it now lists get_schema and aggregate_data, the layer, paging and format arguments, and links to BUILT_IN_PLUGINS.md. - BUILT_IN_PLUGINS.md: search/aggregation tools take `query`, not `q`; replace the "appends /0" note with first-layer resolution and `layer`; schema fallback path uses the resolved layer; WHERE validation covers having and order_by; get_schema row mentions lengths and domains. - CUSTOM_PLUGINS.md: document render_rows. - SECURITY.md: ArcGIS order_by / aggregate_data / layer input checks, html_to_text, and render_rows in the structure-forgery row. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/ARCHITECTURE.md | 12 ++++++++---- docs/BUILT_IN_PLUGINS.md | 14 +++++++------- docs/CUSTOM_PLUGINS.md | 1 + docs/SECURITY.md | 11 +++++++++-- 4 files changed, 25 insertions(+), 13 deletions(-) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index cecbdfd..2a08039 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -186,10 +186,14 @@ plugins: | Tool | Description | |------|-------------| -| `arcgis__search_datasets(q, limit)` | Search the Hub catalog | -| `arcgis__get_dataset(dataset_id)` | Get metadata for a Hub item (32-char hex ID) | -| `arcgis__get_aggregations(field, q)` | Facet counts for type, tags, categories, or access | -| `arcgis__query_data(dataset_id, where, out_fields, limit)` | Query a Feature Service | +| `arcgis__search_datasets(query, limit)` | Search the Hub catalog | +| `arcgis__get_dataset(dataset_id)` | Get metadata for a Hub item: full description, data edit dates, and the service's layers and tables | +| `arcgis__get_aggregations(field, query)` | Facet counts for type, tags, categories, or access | +| `arcgis__get_schema(dataset_id, layer)` | Field names, types, aliases, lengths and domains for a layer or table | +| `arcgis__query_data(dataset_id, layer, where, out_fields, limit, offset, order_by, format)` | Query a Feature Service; total match count, next-page offset, text/json/csv output | +| `arcgis__aggregate_data(dataset_id, statistics, group_by, where, having, order_by, layer, limit, format)` | Server-side count/sum/avg/min/max/stddev, grouped by fields | + +See [BUILT_IN_PLUGINS.md](BUILT_IN_PLUGINS.md#arcgis-hub-plugin) for argument details. ### Built-in: Socrata diff --git a/docs/BUILT_IN_PLUGINS.md b/docs/BUILT_IN_PLUGINS.md index bae7b13..437673a 100644 --- a/docs/BUILT_IN_PLUGINS.md +++ b/docs/BUILT_IN_PLUGINS.md @@ -83,16 +83,16 @@ plugins: | Tool | Description | | ------------------------------------------------------------------------------------- | ------------------------------------------------------------- | -| `arcgis__search_datasets(q, limit)` | Search the Hub catalog | +| `arcgis__search_datasets(query, limit)` | Search the Hub catalog | | `arcgis__get_dataset(dataset_id)` | Get metadata for a Hub item, with its service's layers/tables | -| `arcgis__get_aggregations(field, q)` | Facet counts for type, tags, categories, or access | -| `arcgis__get_schema(dataset_id, layer)` | Field names, types and aliases for a layer or table | +| `arcgis__get_aggregations(field, query)` | Facet counts for type, tags, categories, or access | +| `arcgis__get_schema(dataset_id, layer)` | Field names, types, aliases, lengths and domains for a layer or table | | `arcgis__query_data(dataset_id, layer, where, out_fields, limit, offset, order_by, format)` | Query a Feature Service, with paging and a total match count | | `arcgis__aggregate_data(dataset_id, statistics, group_by, where, having, order_by, layer, limit, format)` | Server-side count/sum/avg/min/max/stddev, grouped by fields | ### Usage Notes -- `get_dataset` returns the Hub item metadata. Check that the item has a queryable `serviceUrl` before calling `query_data`. +- `get_dataset` returns the Hub item metadata. Check that the item has a `Service URL` before calling `query_data`, and use the listed layer ids with `layer`. - `get_aggregations` accepts `field` values: `"type"`, `"tags"`, `"categories"`, `"access"`. This is a catalog-level tool, not a DataPlugin method — it has no equivalent in other plugins. - `query_data` uses the ArcGIS Feature Service query interface. The `where` parameter is a SQL WHERE clause (e.g., `"population > 10000"`). Only Feature Layer, Feature Service, Map Service, and Table types are queryable. - Metadata: `get_dataset` shows the full description as plain text (HTML converted, capped at 12,000 characters with a truncation notice; search results keep a 300-character excerpt), the licence and access information as text, and the default layer's `Data last edited` / `Schema last edited` dates from its `editingInfo`. `get_schema` adds the length of string fields and coded-value or range domains when the service defines them. @@ -106,13 +106,13 @@ plugins: **Two-hop resolution.** `query_data` first fetches the dataset metadata via `get_dataset` to resolve the Feature Service URL, then queries the Feature Service directly. Always call `get_dataset` first and check the `service_url` field is non-empty before calling `query_data`. -**WHERE clause validation.** The `where` parameter is validated by `WhereValidator` before being sent to the Feature Service. Malformed SQL WHERE clauses are rejected before the network call. +**WHERE clause validation.** The `where` parameter (and `aggregate_data`'s `having`) is validated by `WhereValidator`, and `order_by` by `WhereValidator.validate_order_by`, before anything is sent to the Feature Service. Clauses with forbidden keywords are rejected before the network call. **Feature Service host restriction.** A dataset record could point `query_data`/`get_schema` at an arbitrary host (SSRF), so the plugin validates the Feature Service URL before querying it. Always trusted: `*.arcgis.com`, the `portal_url` host, and anything in `trusted_service_hosts`. Hub catalogs routinely reference services self-hosted on city GIS domains (`gis.charlottenc.gov`, `maps2.dcgis.dc.gov`, `gis.indy.gov`), so by default (`auto_trust_hub_services: true`) a Hub-referenced URL is also accepted when it is https, on a public DNS name (never an IP literal, single label, or `.internal`/`.local` name), and has an ArcGIS REST path (`/rest/services/.../FeatureServer|MapServer[/layer]`). The bearer `token` is never sent to auto-trusted hosts. A refused URL raises an error beginning `untrusted_service_host: ''` that names the host to add to `trusted_service_hosts`. Set `auto_trust_hub_services: false` to require an explicit allow-list. -**Schema fallback.** `get_schema` reads the layer metadata endpoint (`.../FeatureServer/0?f=json`). Some self-hosted services return HTML or an ArcGIS error envelope there while `/query` works; in that case the plugin derives the field list from a one-row query instead of failing. +**Schema fallback.** `get_schema` reads the layer metadata endpoint (`.../FeatureServer/?f=json`). Some self-hosted services return HTML or an ArcGIS error envelope there while `/query` works; in that case the plugin derives the field list from a one-row query instead of failing. -**Auto layer index.** If the dataset's service URL points at a `FeatureServer` or `MapServer` root without a layer index (e.g. `.../FeatureServer`), the plugin automatically appends `/0` to target the default layer. +**Layer resolution.** If the dataset's service URL points at a `FeatureServer` or `MapServer` root without a layer index (e.g. `.../FeatureServer`), the plugin reads the service description and uses its first layer, then its first table, falling back to `/0` only when the description cannot be read (HUD services whose only layer is id 4 or 13 are reachable this way). Pass `layer` to choose another one. Service and layer descriptions are cached per plugin instance. **Queryable item types.** `query_data` only works on the following ArcGIS item types: diff --git a/docs/CUSTOM_PLUGINS.md b/docs/CUSTOM_PLUGINS.md index ee8ec7f..4a84148 100644 --- a/docs/CUSTOM_PLUGINS.md +++ b/docs/CUSTOM_PLUGINS.md @@ -67,6 +67,7 @@ are all written this way. | `HTTP_RETRY` | Decorator adding exponential-backoff retries (3 attempts) for transient HTTP errors | | `_raise_http_error(exc, context)` | Translates `httpx.HTTPStatusError` into a user-readable `RuntimeError`, extracting portal error messages when present | | `format_records(records, max_display=None, header=None, skip_keys=..., max_chars=RECORDS_BUDGET)` | Renders query results in the standard `Record N:` style. Shows every record unless `max_display` asks for fewer (`... and X more record(s)`) or the output would pass the response size budget, in which case whole records are dropped and a `Showing N of M record(s)` notice is added | +| `render_rows(records, fmt="text", header=None, skip_keys=..., max_chars=RECORDS_BUDGET, total=None)` | Renders records as `text` (the `format_records` layout), a `json` array (one object per line) or `csv` (header plus one line per record), within the same size budget; returns `(text, shown)` so a caller can compute the next page offset | | `build_where_clause(filters)` | Builds a SQL `WHERE` body from a filter dict; escapes string values and **validates field names as plain identifiers** so SQL cannot be smuggled in through keys | ### Minimal example diff --git a/docs/SECURITY.md b/docs/SECURITY.md index ddb702e..0cd22a7 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -13,7 +13,13 @@ IDs that the connector forwards to the portal. Defenses: reject multi-statement, write, and dangerous-function queries and cap query length. - **Identifier whitelists** restrict field names, metric expressions, `ORDER - BY`, and `HAVING` values assembled by `aggregate_data`. + BY`, and `HAVING` values assembled by `aggregate_data`. For ArcGIS, + `order_by` accepts only field names with an optional `ASC`/`DESC` + (`WhereValidator.validate_order_by`); `aggregate_data` statistic fields, + output names and `group_by` fields must be plain identifiers that exist in + the layer's schema, and `having` goes through the same `WhereValidator` as + `where`. `layer` must be an integer id the service lists. All of these are + checked before any request is sent. - **`build_where_clause`** escapes values and rejects non-identifier field names. - **Redirect credential scoping** (`_create_http_client(protect_headers=…)`) @@ -55,8 +61,9 @@ All of this lives in `core/portal_content.py` and is applied centrally by | Defense | Where | Effect | | --- | --- | --- | | **Untrusted-data boundary** | `execute_tool` → `_finalize_result` → `frame_portal_content` | Every successful text result is wrapped: a one-line preamble names the source and states that the content is data, not instructions; the body sits between `<<>>` / `<<>>`; the connector's own next-step hint (`ToolHandler(guidance=…)`) is emitted **after** the closing marker so instruction-shaped text never sits inside the data region. | +| **HTML to text** | `html_to_text` | Catalog descriptions, licences and access notes stored as HTML are converted to plain text (scripts and styles dropped, entities decoded) before normalization, so markup cannot hide text from the injection scan or reach the model as raw tags. | | **Normalization** | `clean_text`, `portal_text`, `portal_line` | Strips C0/C1 controls, zero-width and bidi-override code points, Unicode tag characters (“ASCII smuggling”), private-use and unassigned code points; collapses newlines in single-line fields (titles, IDs, tags, field names); truncates with an explicit `…[truncated, N more chars]` marker; defangs any literal boundary marker inside a value. | -| **Structure forgery prevention** | `format_records`, `indent_continuation` | Record keys are single-line; multi-line values have every continuation line indented, so a value cannot start a fake `Record 2:` header or a fake connector instruction at column 0. | +| **Structure forgery prevention** | `format_records`, `render_rows`, `indent_continuation` | Record keys are single-line; multi-line values have every continuation line indented, so a value cannot start a fake `Record 2:` header or a fake connector instruction at column 0. JSON output escapes newlines; CSV values are kept on one line. | | **Size caps** | `DEFAULT_MAX_TEXT` (4 000 chars/value), `DEFAULT_MAX_LINE` (300), `DEFAULT_MAX_RESPONSE` (60 000/body), `DEFAULT_MAX_ERROR` (500) | Limits context stuffing. | | **ID validation** | `safe_id` with a per-plugin `id_pattern` | An ID is only interpolated into a `Portal:` URL or a hint if it matches the provider's ID shape (Socrata 4x4, CKAN slug/UUID, Hub hex, ODS slug); otherwise it renders as `unknown` and no link is built. Links are always built from config + validated ID, never echoed from the portal. | | **URL gating** | `BaseOpenDataPlugin.display_portal_url` (ArcGIS `_display_url` wraps it with `trusted_service_hosts`) | Portal-supplied URLs (resource downloads, license/attribution links, service endpoints) are echoed only when their host is the portal/API host or a subdomain of it (or an explicitly trusted host); otherwise only `(external: hostname)` is shown, never the URL itself. |