feat(arcgis): full plain-text descriptions, data dates, source block (#32)

[sgarcese]

Closes #32 (fork-side tracking for thealphacubicle#84).

Problem

• Descriptions were cut at 300 characters, with raw HTML. HUD's Fair Market Rents description (3,538 characters of HTML) lost the definitions and suppression rules analysts need.
• There was no data date. Only the Hub item's "modified" date was shown, so a fiscal-year dataset's vintage had to be guessed. The layer's editingInfo.dataLastEditDate has the real one.
• There was no source block, so answers carried nothing to cite.

Change

core/portal_content.html_to_text (stdlib html.parser; no new dependency)

• Block tags become lines, <li> becomes - bullets, entities are decoded, <script>/<style> content is dropped, and blank lines are collapsed.
• Text without markup passes through unchanged.
• The output still goes through clean_text and the untrusted-data frame. Converting first also means markup can't split an injection phrase so the marker scan misses it (there's a test for this).

get_dataset

• Shows the whole description as plain text, capped at 12,000 characters with the standard …[truncated, N more chars] notice. Search results keep a 300-character excerpt.
• The licence, access information and snippet are converted to text.
• Adds Data last edited and Schema last edited from the default layer's editingInfo. This costs one more request per get_dataset call. It is best-effort, and the layer description is cached.

get_schema adds the length of string fields and coded-value or range domains when the service defines them. Long domains are capped at 30 values plus "… and N more".

Source block at the end of query_data and aggregate_data:

Source:
  Dataset: Fair Market Rents (https://hudgis-hud.opendata.arcgis.com/datasets/12d25169…_0)
  Layer queried: https://services.arcgis.com/…/Fair_Market_Rents/FeatureServer/0
  Data last edited: 2025-09-30
  License: HUD and the dataset and metadata authors assume no responsibility…
  Retrieved: 2026-09-25 03:09 UTC

• query_data only uses a layer description that is already cached (from get_dataset, get_schema or an earlier call), so the block never costs an extra request. Without one, the date line is left out.
• aggregate_data reads the description anyway, so it always has the date.
• The item link is built from the configured portal_url and a validated item id.

Other

• Internal: _layer_url_for now returns (layer_url, dataset); _layer_metadata holds the per-layer cache and _layer_fields uses it.
• docs/BUILT_IN_PLUGINS.md is updated.

Deferred: reading the item's ArcGIS metadata XML for field labels (optional in thealphacubicle#84) is left for a follow-up.

Tests

• tests/unit/core/test_portal_content.py: 7 html_to_text tests (blocks, lists, entities; script/style; table cells; pass-through; markup can't hide an injection phrase).
• New tests/unit/plugins/arcgis/test_metadata.py (14 tests):
  • full description as plain text; truncation notice past the budget; short search excerpt; licence and access HTML
  • edit dates in get_dataset, and omitted when the layer can't be read
  • the source block after get_dataset (and the request count); no extra request without the cache; aggregate_data includes the date and licence; unsafe ids give no link
  • string length; coded and range domains; long domains capped; other fields unchanged
• The 300-character truncation test is replaced by tests that the description is kept whole and converted from HTML.
• pytest -n auto: 1241 passed, 95% coverage. ruff check and ruff format are clean.

Live against hudgis-hud.opendata.arcgis.com (Fair Market Rents, 12d25169…)

• The description comes through whole: 3,538 characters of HTML become 1,967 characters of plain text, with no tags and no truncation. It includes the full list of what FMRs are used for.
• The dates line reads Created: 2017-11-22 | Modified: 2026-04-20 | Data last edited: 2025-09-30 | Schema last edited: 2025-09-30. The data date matches the issue.
• The query_data footer is as shown above.

🤖 Generated with Claude Code