RFC: IND AS XBRL support via shared xbrl engine + ind_as adapter (upload → graph v1)

[MHa55anY]

Summary

This RFC proposes adding support for Indian IND AS XBRL filings in RoboSystems so they can be ingested and queried similarly to SEC XBRL, without extending the SEC EDGAR adapter into a multi-jurisdiction catch-all. The approach is a shared XBRL engine (adapters/xbrl/) plus a separate IND AS adapter (adapters/ind_as/), with a minimal v1: upload filings (S3/local) → Arelle-based processing → queryable graph. Automated India portal crawling, a shared India repository, MCP, and full SEC-feature parity are explicitly out of scope for v1.

Motivation: Problem

Today RoboSystems’ XBRL path is centered on SEC EDGAR + US filing/taxonomy assumptions (adapters/sec/). Indian companies file under the IND AS taxonomy with different packaging and distribution constraints. Those filings cannot currently enter the same parse → graph → query path.

Extending adapters/sec/ to also mean “India” would couple two jurisdictions, data sources, and operational models (EDGAR rate limits, EFM, SEC shared-repo ops, etc.) and works against clear module boundaries.

An external spike (Arelle successfully loading IND AS taxonomy-based reports) suggests the parse side is feasible; what we need is an agreed in-repo architecture before implementation PRs.

Motivation: Why Now?

• Demand for querying non-US XBRL (IND AS) is a natural extension of the platform’s financial graph story.
• A working Arelle + IND AS spike exists outside the repo, so design discussion can be grounded in something real.
• Upstream SEC has become more SEC-specific (FactSet/report lifecycle, shared-master ops, etc.), which makes “bolt India onto sec/” a worse default than extracting a thin shared XBRL layer.
• Aligning on boundaries now avoids a large, hard-to-review contribution and rework later.

Proposed Design: Overview

Principles

1. Separation of concerns: SEC stays the SEC product; IND AS is its own adapter; shared XBRL mechanics live in a neutral module.
2. Arelle for instance/DTS truth; a thin curated (or later generated) semantic profile for product-level canonical concepts — not hand-maintaining the full taxonomy as Python classes.
3. Minimal vertical slice for v1 that is queryable and extensible; defer product/ops surface area.

High-level layout

adapters/xbrl/          # shared engine: Arelle load, normalize facts/structures
adapters/ind_as/        # India-specific: taxonomy packages, upload ingest, wiring
adapters/sec/           # remains EDGAR/SEC; may gradually call into xbrl/ later

v1 flow

1. Operator/user places an IND AS filing package in S3 or local path
2. ind_as adapter invokes xbrl (Arelle + taxonomy cache/packages)
3. Emit a stable intermediate representation (subset OK) aligned where practical with existing SEC/reporting graph concepts (facts, elements, contexts, report linkage)
4. Materialize into a normal user/demo graph
5. Query via existing API/Cypher (success = basic facts / statement structure retrievable)

Explicit non-goals for v1

• India government/exchange bot crawling (sites often restrict automated download; APIs unclear)
• Shared repository for India (manifest, billing, platform-wide corpus like SEC)
• MCP / agent tooling (easy to add later once the graph exists)
• Full parity with SEC (narratives, iXBRL item extraction, corpus knowledge artifacts, incremental shared-master parking, ticker auto-resolve, full enrichment)

Proposed Design: Detailed Design

Architecture

• adapters/xbrl/

  • Taxonomy-agnostic Arelle session/load helpers (instance + DTS / taxonomy packages; offline cache)
  • Normalization: facts, contexts, units, elements, basic presentation/structure → intermediate tables/parquet (or documented subset of current SEC processor outputs)
  • No EDGAR, no EFM-only assumptions in the core path

• adapters/ind_as/

  • IND AS taxonomy package/bundle management
  • Ingest entrypoint: S3 URI or local path → process one filing (batch later)
  • Thin taxonomy semantic profile (optional in earliest PR): map a small set of concepts to canonical tags if needed for v1 queries; prefer deriving structure from Arelle first

• adapters/sec/

  • Unchanged behavior initially
  • Optional follow-up: migrate shared pieces into xbrl/ behind SEC-compatible wrappers (incremental, no big-bang rewrite in the first IND AS PR)

Data model

• Prefer reusing existing reporting/graph shapes (e.g. Report / Fact / Element / related edges, FactSet only if low-cost) so query patterns don’t permanently fork.
• v1 may populate a subset of fields; document gaps vs SEC.
• Do not require corpus-level knowledge artifacts for v1.

Ingest & runtime

• Source: S3/local upload only in v1.
• Target: private/user or demo graph (not a new shared ind repository).
• Orchestration: smallest useful surface — CLI/just recipe and/or a single Dagster job later; full nightly pipeline not required for the RFC approval bar.

APIs / product surface

• v1: existing graph query APIs / Cypher.
• MCP: Phase 1.5+ (optional), after data is queryable.

Extension points (foundation)

Hook	v1	Later
Filing source	S3/local	Approved bulk/API source if available
Taxonomy profile	IND AS packages + thin optional map	Generated profiles, richer canonical concepts
Shared repository	—	Manifest + billing + hosted corpus (SEC-like)
Enrichment / MCP	—	Semantic enricher parity, MCP tools

Implementation approach for first PRs

1. RFC approval / direction from maintainers
2. Small foundational PR(s): introduce xbrl skeleton and/or extract one SEC-safe shared helper with no behavior change
3. Spike PR or follow-up: one real IND AS package → intermediate shape → graph → documented example queries
4. Grow enrichment, automation, shared-repo only after the slice works

Alternatives Considered

Alternative	Pros	Cons
A. Extend adapters/sec/ for IND AS	Faster short-term if we only copy processors	Couples jurisdictions; fights SOLID; SEC ops/assumptions leak into India
B. Shared adapters/xbrl/ + adapters/ind_as/ (this RFC)	Clear boundaries; reusable for future taxonomies; incremental extraction	Slightly more upfront structure; needs care not to over-abstract
C. IND AS-only adapter with duplicated Arelle/processing	Isolated; no SEC touch	Duplication; harder to keep graph contracts aligned
D. v1 as shared India repository like SEC	Product parity narrative	Needs hosting, legal/source strategy, billing/manifest, crawler or bulk feed — too large for first contribution

Why this approach: B gives a solid, extensible foundation with the smallest honest v1 (upload → graph → query), while leaving shared-repo and MCP as deliberate later phases.

Implementation: Phases

1. Phase 0 — Alignment: This RFC; confirm package layout and non-goals.
2. Phase 1 — Foundation: xbrl module boundaries; optional no-behavior-change extraction from SEC; IND AS taxonomy cache/packages loadable via Arelle.
3. Phase 2 — Vertical slice: S3/local ingest → process → materialize → example Cypher/API queries on a demo/user graph.
4. Phase 3 — Harden: Better error handling, tests with redistributable fixtures, docs/just recipe.
5. Phase 4+ (optional): Thin semantic profile; MCP; shared repository if product decides to host an India corpus; any legitimate automated source.

Implementation: Dependencies

• Arelle (already used by SEC)
• IND AS taxonomy packages (versioning/cache strategy)
• Sample IND AS filings suitable for tests (licensing/redistribution must be respected; synthetic or clearly allowed fixtures preferred in-repo)
• Existing graph materialization / query stack
• Maintainer guidance on how aggressively to share code with sec/ in Phase 1 vs duplicate temporarily

Risks & Mitigations

Risk	Mitigation
Over-abstraction in xbrl/ before second taxonomy is real	Keep xbrl/ thin; only share what IND AS + SEC both need; extract incrementally
IND AS instance/DTS differences break SEC-shaped processors	Document schema deltas; allow subset output in v1; spike early with real filings
Taxonomy / filing redistribution constraints	Don’t commit proprietary filings; use S3 for real data; fixtures only if permitted
Scope creep toward SEC parity or shared repo	Hard non-goals in RFC; success = queryable vertical slice
India source automation expectations	State upload-only; crawler blocked until lawful API/bulk channel

Open Questions

• Are maintainers aligned on adapters/xbrl/ + adapters/ind_as/, vs a thinner “IND AS-only first, extract xbrl/ later”?
• For v1 graph shape: strict subset of current SEC reporting nodes/edges, vs allow a slightly India-specific schema with a mapping document?
• Preferred v1 entrypoint: just/CLI only, or also a single Dagster job?
• Any existing plans for non-US XBRL / IFRS that this should align with?
• Guidance on taxonomy package bundling (similar to SEC’s Arelle schema bundles) vs download-at-runtime with cache?

References

• robosystems/adapters/README.md — shared repository vs private adapter pattern
• robosystems/adapters/sec/README.md — current XBRL → graph pipeline
• .github/CONTRIBUTING.md — Feature → RFC → Spec workflow
• External spike: local Arelle loading of IND AS taxonomy-based reports (outside this repo)

[MHa55anY]

Note for maintainers: Please set the issue type to RFC if possible — gh could not set issue type with read-only permission on this repo (UpdateIssueIssueType). Labeling also requires write access; feel free to add area:adapters.

Also closed accidental duplicate: RoboFinSystems/robosystems#1133.

[jfrench9]

Thanks for this — it's a genuinely well-researched RFC, and the fact that you read the code before proposing shows. Setting the type to RFC and adding area:adapters.

Where you're right

Your core diagnosis holds up. adapters/sec/processors/xbrl_graph.py is ~2,200 lines that import SECClient/SEC_BASE_URL, take sec_filer/sec_report kwargs, normalize CIK, and mint http://www.sec.gov/CIK#{cik} as the canonical entity URI. Alternative A (extend sec/) would drag EDGAR assumptions into a second jurisdiction, and we agree it's the wrong default. Non-goals are scoped the way we'd want them — no crawling, no shared repository, no MCP in v1.

Two corrections and one question that we think changes the shape of the work.

1. The taxonomy-neutral Arelle layer already exists — one level up from adapters/

robosystems/arelle/ is already there and already deliberately taxonomy-neutral. Its entry points are extract_taxonomy(model_xbrl) -> rdflib.Graph and serialize_jsonld(graph, standard, version), and it feeds robosystems/taxonomy/ (discovery.py, loader.py, pins.py, writer.py) which seeds the frameworks/ library.

It lives outside adapters/ on purpose. Per robosystems/adapters/README.md, an adapter is an external service integration — a client plus transformation, optionally with a shared-repository manifest. A taxonomy-neutral XBRL engine isn't that; it's platform-scoped machinery. So we'd rather not create adapters/xbrl/: it would put neutral code in the layer reserved for service integrations, and split the neutral Arelle surface across two homes.

The extraction candidate you're looking for is real, though — it's just a different file. adapters/sec/client/arelle.py (ArelleClient, ~700 lines) is cache directory selection, bundled-schema population, offline mode, and controller lifecycle. Almost all of that is jurisdiction-neutral; only the EFM validation configuration is SEC-specific. Hoisting that into robosystems/arelle/ behind a SEC-compatible wrapper is exactly the kind of no-behavior-change foundational PR your Phase 1 describes, and we'd take it on its own merits.

So: robosystems/arelle/ grows an instance-side companion to its existing taxonomy-side extractor; adapters/ind_as/ (or equivalent) stays thin. Same separation of concerns you argued for, one layer up.

2. There are two destinations for a non-US taxonomy here, and the RFC only addresses one

This is the substantive question, and it's your open question RoboFinSystems/robosystems#2 and RoboFinSystems/robosystems#4 combined.

The corpus path — what SEC does. Filings → Arelle → intermediate tables → a graph of raw facts in the source vocabulary, served as a platform-managed public repository. This is the path your v1 flow describes.

The framework path — how a reporting taxonomy becomes something tenants actually report under. A framework is a versioned bundle in frameworks/ pinning (package, version) and (bridge, version) tuples, loaded via robosystems/taxonomy/, with per-tenant machinery in robosystems/operations/taxonomy_block/ (reporting_standard.py, reporting_extension.py, chart_of_accounts.py). The data model is already built for jurisdictional variants: Taxonomy.extension_type accepts 'jurisdiction' (robosystems/models/extensions/taxonomy.py), and Element.source already accepts 'ifrs' (robosystems/models/extensions/element.py). Bridges map an external vocabulary into our canonical reporting taxonomy so a report can render in both.

Your v1 target — "materialize into a normal user/demo graph" — sits between these. That's the ambiguity we'd want resolved before any code, because the two have almost nothing in common downstream:

• If you mean Indian filings as a queryable corpus (the SEC analogue: many filers, raw facts, query the aggregate), the corpus path is right and your architecture is roughly correct modulo the layering fix above.
• If you mean IND AS as a reporting framework an Indian entity reports under (one entity, its own chart of accounts, statements rendered in IND AS), that's the framework path — and it's mostly content: a frameworks/ind-as/ bundle plus a bridge, with far less new Python than the RFC anticipates. It would also be a materially smaller and more reviewable first contribution.

Which one are you after? The answer changes Phase 2 entirely.

To your question about existing non-US plans: yes, IFRS-adjacent frameworks are a direction we expect to go, and the mechanism we expect to use is the framework/bridge library rather than a per-jurisdiction ingestion adapter. That's why we'd like to pin down which path you mean before locking a package layout.

Answers to your other open questions

Taxonomy bundling (#5). Follow the SEC precedent: bundle, don't download at runtime. adapters/sec/arelle/bundles/ holds versioned tarballs (arelle-schemas-*.tar.gz, edgar-plugin-*.tar.gz) that ArelleClient unpacks into Arelle's cache, with offline mode on. Runtime taxonomy fetches are the thing that makes ingestion flaky and slow, and we don't want a second source of that.

v1 entrypoint (#3). A just recipe over a CLI entrypoint is enough to start. Dagster only once there's a repeating pipeline worth scheduling — a one-shot ingest doesn't need it.

Graph shape (#2). Deferred to the corpus-vs-framework answer above. If it's the corpus path, prefer a documented subset of the existing reporting nodes/edges over a parallel India-specific schema — a fork in query shape is the expensive kind of divergence.

Suggested next step

Rather than the xbrl skeleton PR, we'd start with:

1. Answer the corpus-vs-framework question above.
2. If corpus path: a no-behavior-change PR hoisting ArelleClient from adapters/sec/client/arelle.py into robosystems/arelle/, SEC calling through a wrapper, existing tests green. That stands alone, is easy to review, and is useful regardless of what happens with IND AS.
3. Then the vertical slice against one real filing.

On fixtures — agreed and appreciated that you raised it. Don't commit filings whose redistribution isn't clearly permitted; synthetic instances against the real taxonomy are fine for tests.

Thanks again for taking the time to write this up properly rather than opening a large PR cold. Happy to keep iterating here.

[MHa55anY]

Thanks for the thoughtful reply — this cleared up a lot for me.

On corpus vs framework: my long-term goal is the corpus path (SEC-style). I want Indian XBRL filings ingested and queryable so people can explore them easily, and eventually use MCP/AI to pull insights from the data — comparing companies, spotting trends, asking questions over filed facts. That's "understand what was filed," not "companies report through RoboSystems" as a first step.

I get now that the framework path is the other side of the coin — IND AS as a reporting standard inside the product (frameworks/ind-as/, bridges, tenant reporting). That's complementary and lines up with the IFRS direction you mentioned; I'd treat it as a follow-on rather than what I'm aiming for in v1.

On layering: fully on board with skipping adapters/xbrl/. Putting shared machinery in robosystems/arelle/ and keeping something like adapters/ind_as/ thin makes sense. Hoisting ArelleClient into robosystems/arelle/ with SEC calling through a wrapper sounds like the right first move.

For v1, I'm still aligned with what we scoped before, with your notes folded in: S3/local upload only, bundled taxonomies (no runtime fetch), just/CLI to start, and reusing existing graph shapes where we can instead of inventing a parallel India schema. Still out of scope for v1: shared repo, MCP, crawling, full SEC parity.

Sequencing — if this works for you:

1. Lock in corpus as the direction (happy to tweak the RFC text if that helps)
2. First PR: hoist ArelleClient, no behavior change, SEC wrapper, tests green
3. Then a vertical slice on one real filing (upload → process → a few example queries)
4. Framework/bridge and MCP once there's actually something worth querying

On fixtures: won't commit filings unless redistribution is clearly OK; synthetic instances against the real taxonomy are fine for tests.

If that ordering looks good, I'll start on the Arelle hoist — unless you'd rather framework content land first for sequencing reasons.

[jfrench9]

Moving this to xbrlkit — and with an update that makes the RFC a good deal smaller.

What changed since August

The shared XBRL engine we discussed got extracted, but it went further than either of us planned in this thread: not into robosystems/arelle/, but out of the platform repo entirely, into xbrlkit (also on PyPI). RoboSystems now depends on it like any other library.

Concretely, on the SEC side of the platform:

• adapters/sec/client/arelle.py — the ~700-line ArelleClient you were going to hoist — is now 109 lines wrapping xbrlkit.parse.load_model.
• adapters/sec/processors/xbrl_graph.py went 2,195 → 813 lines; the XBRL table schemas come from xbrlkit.schema, and the EDGAR plugin, its tarballs and the EFM validation config are gone.

So the first PR we agreed on — hoist ArelleClient, SEC through a wrapper — no longer exists as a task. Good timing: you hadn't started it, so nothing is wasted. Sorry for not flagging it here when it landed.

What that means for IND AS

The neutral layer you argued for in this RFC exists, and it's jurisdiction-neutral by construction rather than by refactor. xbrlkit loads a filing from a local path, directory, zip or URL; it resolves DTS from a persistent cache; it already reaches non-US filings — ESEF and the national regimes on filings.xbrl.org, addressed by LEI rather than CIK. There is no EDGAR assumption in the load path to work around.

That makes IND AS an xbrlkit question, not a RoboSystems one, which is why the issue is here now. It also collapses your v1:

• No adapters/xbrl/, no adapters/ind_as/, no graph materialization. Those were all scaffolding for the separation that already happened.
• v1 is: load a real IND AS filing and find out what breaks. Realistically that's DTS resolution and cache/bundle coverage for the Indian taxonomy hosts, packaging quirks, and the transform registry if MCA filings use their own ixt variants. Plus a fixture and a loader test.

That's a genuinely small, self-contained first contribution with no infrastructure attached to it.

And your actual goal gets closer than the RFC assumed

You said the point is exploring filings and asking questions over them with MCP/AI. xbrlkit ships that already:

uvx --from "xbrlkit[mcp]@latest" xbrlkit serve <path-or-url>

is a local MCP server over the parsed filing — describe_filing, resolve_element, fact_grid, statement, calculation, search_text, read_text. No graph, no index, no hosting, no key. The moment an IND AS filing loads cleanly, that works on it, and single-filing analysis needs nothing else built.

What that does not give you is the cross-filer corpus — many companies, aggregate queries, trends. That's the RoboSystems side, it's the expensive hosted part (manifest, storage, ops, billing), and it stays where you put it in the RFC: out of v1, a product decision rather than a contribution. It also gets much cheaper once the engine parses the filings, since the SEC pipeline already runs through xbrlkit.

The framework path (frameworks/ind-as/ + a bridge, for entities reporting under IND AS) is unaffected by any of this and stays in RoboSystems. You said corpus, so it's a separate conversation whenever it's wanted.

Where to pick it up

Two things would help most:

1. A filing to work against. Are MCA/exchange filings fetchable in any lawful way, or is this upload/local-path only? And can you point at one real instance — ideally one whose redistribution is clearly permitted, so it can be a fixture.
2. A first PR: a loader test against an IND AS instance, plus whatever cache or taxonomy-package work it needs to resolve offline. If the answer turns out to be "it just loads", that's a fixture, a test and a line in the README — and the interesting work moves to whatever it gets wrong in the facts.

Note the transfer dropped the RFC type and the area:adapters label (they don't exist in this repo). The design discussion above is intact and still the reference for this work.