feat(datasource-pylon): PylonIssue read-only collection with cursor pagination

[christophebrun-forest]

Story 2 of the Pylon datasource — EXT-6. Stacked on #341 (Story 1), so it targets feature/ext-5-story-1-foundation-gem-config-resilient-client and not main — review only the last commit, or wait for #341 to merge and this base will retarget to feat/datasource-pylon (EXT-4).

What this adds

The first Pylon collection: PylonIssue in list + record-detail mode, backed by POST /issues/search and GET /issues/{id}.

• BaseCollection — the Zendesk-style template: shared STRING_OPS / NUMBER_OPS / DATE_OPS operator sets, define_schema / define_relations hooks, custom-field registration with collision reporting, and the primary-key short-circuit.
• Collections::Issue — schema, serializer and list. Split into SchemaDefinition and Serializer mixins so the field list stays readable next to the flattening logic.
• Pagination::CursorWalker — bridges Forest's offset/limit window onto Pylon's cursor.
• Datasource — registers PylonIssue.
• Client#search_issues / #fetch_issue — a SearchPage struct that normalises the three ways Pylon spells "no next page" (absent pagination block, has_next_page: false, empty cursor) down to a nil cursor.

Cursor pagination, and why the walk is capped

Forest asks for an offset/limit window; Pylon only hands out the next page of a cursor. CursorWalker walks pages until the window is covered, then slices — so a deep offset costs one request per page. /issues/search allows 20 requests per minute, so an uncapped deep-offset walk would spend an agent's entire budget on a single list view. The walk therefore stops at 20 pages / 5000 records and logs a warning naming the offset, limit and how much was collected, rather than truncating silently.

It also stops defensively on an empty page or a cursor that does not advance. Pylon does neither today, but a loop driven by a remote value should terminate on its own terms, not only on the caps.

Schema follows the live API, not the ticket

Three deliberate divergences from EXT-6's field list, all verified against the live API:

• No priority — Pylon has no such field on issues.
• first_response_time / resolution_time are Date, not durations — they are RFC3339 timestamps, so the ticket's first_response_seconds / resolution_seconds do not exist. The two genuine duration fields (time_in_status_seconds, business_hours_time_in_status_seconds) are per-status maps, mapped as Json.
• Nothing is sortable — /issues/search exposes no sort parameter at all; results always come back created_at descending. Advertising a sortable column would let the UI ask for an order the API cannot honour, so translate_sort is not ported from Zendesk.

Nested account / requester / assignee / team objects are flattened to *_id string columns until the related collections exist. Every column is read-only in this story.

Unimplemented filtering is loud, not silent

enable_search and enable_count default to off in BaseCollection — the inverse of the Zendesk template — because both land with the condition-tree translator in a later story. Until then, a condition tree or search term the collection cannot honour is dropped with a warning naming exactly what was discarded, so an unfiltered result set never reads as a filtered one.

The one filter that is honoured is the primary-key lookup: id equal / id in short-circuits to GET /issues/{id}, since /issues/search has no id operator. A 404 there yields no record rather than an error — a deleted issue, or one outside the token's scope, should read as "no record", not as a failed page.

Monorepo wiring

One .rubocop.yml exclude (Metrics/AbcSize on base_collection.rb). .releaserc.js stays untouched for the same reason as in #341 — the release wiring is owned by EXT-13 (Story 9), which carries the exact 3-spot patch as a comment.

Test plan

• BUNDLE_GEMFILE=Gemfile-test bundle exec rspec → 119 examples, 0 failures, coverage 100% (303/303 LOC, threshold 90)
• bundle exec rubocop over the whole repo → 798 files, no offenses
• Specs cover the walker's window slicing, both caps, both defensive stops, the id short-circuit incl. the 404 path, the ignored-filter warnings, and the custom-field collision path

🤖 Generated with Claude Code

Note

Add forest_admin_datasource_pylon gem with read-only PylonIssue collection and cursor pagination

• Introduces a new Ruby gem that exposes Pylon issues as a read-only Forest Admin collection, supporting list and single-record fetch via the Pylon REST API.
• Client wraps Faraday with Bearer auth, retry logic (including 429 with Retry-After), and maps HTTP errors to typed APIError.
• CursorWalker bridges Forest's offset/limit pagination to Pylon's cursor-based API, with protective caps and truncation warnings.
• The PylonIssue schema is fixed and read-only; filter operators are limited to equality/in on id; condition tree filters and free-text search are accepted but ignored with a warning.
• CI lint/test jobs and coverage aggregation are extended to include the new package.

^{Macroscope summarized 9c1bb8b.}

[linear-code]

EXT-6

[qltysh]

5 new issues

Tool	Category	Rule	Count
qlty	Structure	Function with many parameters (count = 4): search_issues	3
qlty	Structure	Function with high complexity (count = 5): fetch_by_ids	2

[qltysh]

Function with many parameters (count = 4): search_issues [qlty:function-parameters]

[qltysh]

Function with many parameters (count = 6): initialize [qlty:function-parameters]

[qltysh]

Function with high complexity (count = 5): fetch_by_ids [qlty:function-complexity]

[qltysh]

Function with high complexity (count = 9): walk [qlty:function-complexity]

[qltysh]

Function with many parameters (count = 4): log_truncation [qlty:function-parameters]

[macroscopeapp]

🟠 High issue/serializer.rb:18

serialize drops every custom_fields entry: it only copies NATIVE_FIELDS and flattened party IDs into the returned record, so any field registered via Issue.new(..., custom_fields: [...]) comes back as nil in list and detail responses even though the collection schema advertises those columns. The method never reads attrs['custom_fields'], so the custom-field support added by BaseCollection is nonfunctional. Consider merging attrs.fetch('custom_fields', {}) into record before returning.

          record = NATIVE_FIELDS.to_h { |field| [field, attrs[field]] }
          PARTY_FIELDS.each { |column, source| record[column] = nested_id(attrs[source]) }
+         record.merge!(attrs.fetch('custom_fields', {}))
          record

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @packages/forest_admin_datasource_pylon/lib/forest_admin_datasource_pylon/collections/issue/serializer.rb around lines 18-21:

`serialize` drops every `custom_fields` entry: it only copies `NATIVE_FIELDS` and flattened party IDs into the returned record, so any field registered via `Issue.new(..., custom_fields: [...])` comes back as `nil` in list and detail responses even though the collection schema advertises those columns. The method never reads `attrs['custom_fields']`, so the custom-field support added by `BaseCollection` is nonfunctional. Consider merging `attrs.fetch('custom_fields', {})` into `record` before returning.

[macroscopeapp]

🟠 High collections/issue.rb:19

fetch_records returns all issues matching the id IN lookup without applying filter.page, so an id IN query with a limit or offset returns every matching record instead of the requested page. For example, ten IDs with limit: 2 returns all ten issues, and repeated calls produce overlapping windows. fetch_by_ids(ids) is returned directly; consider applying the page offset and limit to its result before returning.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @packages/forest_admin_datasource_pylon/lib/forest_admin_datasource_pylon/collections/issue.rb around line 19:

`fetch_records` returns all issues matching the `id IN` lookup without applying `filter.page`, so an `id IN` query with a limit or offset returns every matching record instead of the requested page. For example, ten IDs with `limit: 2` returns all ten issues, and repeated calls produce overlapping windows. `fetch_by_ids(ids)` is returned directly; consider applying the page offset and limit to its result before returning.

[macroscopeapp]

🟡 Medium pagination/cursor_walker.rb:28

CursorWalker#walk can accumulate more than @max_records records, so the advertised cap is not enforced. For example, if 4,500 records have been collected and the next page returns 1,000, line 29 concatenates all of them (5,500 total). When that page also satisfies the requested window or is the last page, line 32 breaks before capped? runs, so the overrun is never logged and records[offset, limit] can return rows past the configured cap. The check at line 34 only runs after the unbounded concatenation, so the budget is enforced too late. Consider clamping each batch by the remaining record budget before fetching, and/or truncating records immediately after concatenation.

          page = yield(batch_size(needed - records.size), cursor)
          records.concat(page.records)
          pages += 1
+
+          if records.size > @max_records
+            records.replace(records.take(@max_records))
+            log_truncation(offset: offset, limit: limit, pages: pages, collected: records.size)
+            break
+          end
+
          break if stop?(page, cursor) || records.size >= needed
-
          if capped?(pages, records.size)
            log_truncation(offset: offset, limit: limit, pages: pages, collected: records.size)
            break
          end

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @packages/forest_admin_datasource_pylon/lib/forest_admin_datasource_pylon/pagination/cursor_walker.rb around lines 28-37:

`CursorWalker#walk` can accumulate more than `@max_records` records, so the advertised cap is not enforced. For example, if 4,500 records have been collected and the next page returns 1,000, line 29 concatenates all of them (5,500 total). When that page also satisfies the requested window or is the last page, line 32 breaks before `capped?` runs, so the overrun is never logged and `records[offset, limit]` can return rows past the configured cap. The check at line 34 only runs after the unbounded concatenation, so the budget is enforced too late. Consider clamping each batch by the remaining record budget before fetching, and/or truncating `records` immediately after concatenation.

[macroscopeapp]

🟠 High forest_admin_datasource_pylon/forest_admin_datasource_pylon.gemspec:34

The gemspec omits forest_admin_datasource_toolkit from its runtime dependencies, so installing the published gem in an application that does not already have the toolkit fails with LoadError when require 'forest_admin_datasource_pylon' runs, because the library unconditionally executes require 'forest_admin_datasource_toolkit'. The toolkit currently appears only in development/test Gemfiles, so Bundler does not install it for end users. Add spec.add_dependency 'forest_admin_datasource_toolkit' (with the appropriate version constraint) to the gemspec.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @packages/forest_admin_datasource_pylon/forest_admin_datasource_pylon.gemspec around line 34:

The gemspec omits `forest_admin_datasource_toolkit` from its runtime dependencies, so installing the published gem in an application that does not already have the toolkit fails with `LoadError` when `require 'forest_admin_datasource_pylon'` runs, because the library unconditionally executes `require 'forest_admin_datasource_toolkit'`. The toolkit currently appears only in development/test Gemfiles, so Bundler does not install it for end users. Add `spec.add_dependency 'forest_admin_datasource_toolkit'` (with the appropriate version constraint) to the gemspec.