Skip to content

Serve each published entry as Markdown and add the site index - #2571

Merged
pournasserian merged 10 commits into
v2from
2564-markdown-and-index
Oct 19, 2026
Merged

pournasserian merged 10 commits into
v2from
2564-markdown-and-index

Conversation

@pournasserian

@pournasserian pournasserian commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

Closes #2564

What

This is the next F-API (#2425) slice. Each published entry gets a Markdown representation at a stable URL, and a site index lists every published entry for agents (API-03). The read endpoints it builds on merged as #2563.

  • Core (FluentCMS.Core.Delivery):
    • EntryMarkdown.Render(item, url, entryUrl) writes front matter: the id, type and URL as quoted strings, and the version. Then come the title as the heading and every other field under its name. The fields follow the order of the definition that the published version was written against:

      • rich text as written, since it is stored as Markdown (SCH-02), inside a block quote. A quote is a container, so a fence, an HTML block or a list the rich text leaves open closes with its field;
      • other text through EntryMarkdown.Text, which escapes only what could open structure. Anywhere in the text, that is code, emphasis, links, autolinks, HTML and strikethrough, an underscore outside a word, an ampersand that starts an entity, and a trailing run of hashes. At the start of a line, it is a heading, list, quote, table or heading underline. Lines also lose their indentation. A value reads as written and can never pose as a heading, a field, a source or a link, while words and URLs keep their characters;
      • lists as bullets, each item keeping its own paragraphs and fences;
      • JSON in a fenced block;
      • whole-entry references as links to their Markdown, including references whose settings store "field": null;
      • entity-field references as the resolved published value;
      • media as the asset id;
      • on an entity, every fact, the title's included, followed by a **Source:** paragraph. Escaped text cannot form it, and quoted rich text cannot reach it. It says so when the fact is unsourced (ENT-02), and a source that is a web address is an autolink, so it stays exact.

      Unresolved references are listed with their reason.

    • EntryMarkdown.Title returns the first text field that holds text, on one line. EntryMarkdown.InlineText escapes a heading or link text.

    • IPublishedContent.IndexAsync() lists every published entry the caller may read, with its type and title, by type and then id.

    • PublishedItem carries its content type's name and the definition its version was written against (EntryVersions.ContentTypeVersion). A revision of the type waiting for review therefore changes nothing that readers see.

  • Api:
    • GET /api/v1/entries/{id}.md returns text/markdown. A malformed id is a 400 problem, as for the JSON read.
    • GET /api/v1/index returns JSON with the id, type, version, title, URL and Markdown URL of each entry.
    • GET /api/v1/index.md and GET /llms.txt return the same index in Markdown, grouped by content type. Type names and titles are escaped.
    • The JSON items gain a markdown link.
  • Docs: docs/api/delivery.md describes the Markdown form and the index.

Why

API-03 and the agent-first principle: an agent reads a site the way it reads any page, starting from /llms.txt. The Markdown URL is the entry's id, so it never changes across versions and always shows what is published. What an agent reads there must be faithful. Published text is content, never structure, so it cannot fake a field or a source.

Choices worth a look:

  • The title is the first text field that holds text. In the reference model that gives a page its headline, a post its title and a product its name. Content types have no title field to name, and adding one would be a schema change.
  • Escaping is targeted, not total, so GUIDs, URLs and identifiers stay readable in raw Markdown and links stay exact.
  • Rich text is quoted. CommonMark has no other container that ends whatever the rich text opens, and the quote marks it clearly as the value.
  • Media appears as its id until media has a delivery endpoint.
  • The index loads every published document to read titles. That is fine at the thousands of entries M1 targets (D26).

Tests

  • Unit:
    • A page renders exactly. The output includes front matter, a collapsed title, links, the resolved price, rich text, lists of text, references and rich text (multi-paragraph items and a fence), media, a boolean, an enum, a date and fenced JSON.
    • Text that tries to be a heading, a field, HTML, emphasis, an entity, an ordered or bullet list, an indented code block or a heading underline reads as text. A title ending in # keeps it.
    • On an entity, a rich text fact holding a fake **Source:** and an unclosed fence stays inside its quote. The real source of the fact and the next field follow unchanged.
    • An entity shows every fact with its source, the title fact included. A URL source is an autolink with its characters intact. A list fact's source stays out of the list.
    • URLs, words with underscores, C# and AT&T keep their characters, while emphasis, entities and a closing run of hashes are escaped.
    • The fallback heading escapes a type name holding a link, and unresolved references are listed.
    • Backslashes, brackets, autolinks and line breaks in link text stay text.
    • A backtick in JSON cannot close its fence.
  • Integration, on SQLite and PostgreSQL:
    • A published page's Markdown shows its headline, its price resolved to 99 and a link to the product. After the second version is published, it shows the new headline and version 2 at the same URL.
    • After a type revision drops headline, the published page still reads against its own definition, and the index keeps its title.
    • An agent scoped to product gets only products from the index.
    • The index lists a published post, product and page once each, with titles, and leaves out a draft and a deleted page. After a second publish it shows version 2.
  • HTTP:
    • The Markdown endpoint returns text/markdown with quoted front matter.
    • Unknown and malformed ids give problems.
    • The JSON and Markdown indexes match, with escaping, and /llms.txt equals index.md.

The full suite ran locally against a postgres:17 container as well as SQLite, and markdownlint is clean. There is no migration and no new package.

Review

A first independent review blocked this PR for three reasons:

  • plain text was written as raw Markdown, so a value could fake a field, a source, HTML or a heading;
  • rendering used the type's latest definition;
  • index link text was only partly escaped.

All three are fixed above, along with its low findings:

  • type names are escaped;
  • the front matter is quoted;
  • list items keep their blocks;
  • unresolved references are listed.

A second review confirmed those fixes, fuzzing the escaping with Markdig, and found two more Medium problems:

  • rich text went in verbatim, so an unclosed fence or HTML block could swallow later fields, and a rich text fact could show a fake source;
  • the title fact's source was never written.

Both are fixed above, along with its Low findings:

  • a list fact's source is no longer read as an extra list item;
  • "field": null references now link;
  • URLs keep their characters.

A third review gave MERGE. Its low findings are fixed: list items that start with blank lines now stay in their bullets, and the math and figure block openers of Markdown extensions are escaped at line start. Two Low findings are tracked as issues: whole-entry links to unpublished or deleted entries in #2593, and resolved references written with their own field type in #2602.

Checklist

  • Linked to an issue a maintainer has confirmed
  • Targets v2
  • Tests or docs updated where behaviour changed
  • No tool-generated co-author trailers or footers

@pournasserian pournasserian added this to the M1 Governance core milestone Oct 19, 2026
@pournasserian pournasserian added type:feature New capability v2 FluentCMS v2 (v2 branch) area:api Delivery API (API) labels Oct 19, 2026
@pournasserian pournasserian mentioned this pull request Oct 19, 2026
4 tasks done
# Conflicts:
#	docs/api/delivery.md
#	src/FluentCMS.Api/DeliveryEndpoints.cs
#	src/FluentCMS.Core/Delivery/IPublishedContent.cs
#	src/FluentCMS.Infrastructure/Delivery/PublishedContentReader.cs
#	tests/FluentCMS.IntegrationTests/Delivery/DeliveryEndpointsTests.cs
#	tests/FluentCMS.IntegrationTests/Delivery/PublishedContentTests.cs
@pournasserian
pournasserian merged commit 9c0afd7 into v2 Oct 19, 2026
2 checks passed
@pournasserian
pournasserian deleted the 2564-markdown-and-index branch October 19, 2026 08:56
pournasserian added a commit that referenced this pull request Oct 19, 2026
Closes #2572

## What

The next F-API (#2425) slice, on the read endpoints of #2563 and the
Markdown views of #2571: a preview token lets a frontend read a change
set's content resolved as it would be published, before approval
(API-04).

- **Core** (`FluentCMS.Core.Delivery`):
  - `IPreviewTokens`:
    - `IssueAsync(changeSetId)` returns the token once.
    - `RevokeAsync(tokenId)` revokes it.
- `ListAsync(changeSetId)` lists the set's tokens, newest first, without
secrets.
- Only a `UserInitiator` may issue or revoke. Agents, the system and
anonymous callers get `preview_by_people_only`.
- A set that is published, rejected or reverted gets
`change_set_not_previewable`.
  - `IPreviewContent`:
- `GetAsync(token, entryId)` returns the version the set proposes. An
entry the set unpublishes is not found, and an entry the set leaves
alone reads at its published version.
- `ChangesAsync(token)` returns the entries the set creates or updates,
in operation order.
- References to entity fields resolve to the versions the set proposes.
An entity the set unpublishes resolves as `NotPublished`.
- `PreviewTokenKeys`: 32 random bytes with an `fcmsp_` prefix. Only the
SHA-256 is stored, as for agent keys (SEC-01).
- **A token stops working** when it is revoked or expires, or when its
set is published, rejected or reverted. Unknown and malformed tokens are
refused the same way. All of these give one `preview_token_invalid`.
Reads write nothing.
- **Audit**: `preview_token.issued` and `preview_token.revoked`, with
the token as subject and the change set id, appended in the same unit of
work as the write.
- **Infrastructure**:
- `PreviewTokens` table with a unique token hash and foreign keys to the
change set and the site, with migrations for SQLite and PostgreSQL.
  - `EntryResolution` takes the entity versions a set proposes.
- `PublishedContentReader` serves both the published and the preview
reads.
- `Delivery__PreviewTokenLifetime` defaults to 7 days. A value outside
zero to 365 days stops the host at startup. It is documented in
CONTRIBUTING.
- **Api**: `GET /api/v1/preview/entries/{id}` and `GET
/api/v1/preview/changes`.
  - The token goes in `Authorization: Bearer`, never in the URL.
  - Responses carry `Cache-Control: no-store`.
  - A missing or failing token gets a 401 problem.
- **Docs**: `docs/api/delivery.md` has a Preview section.

## Why

API-04 and S1: a frontend shows reviewers the pages with the new prices
before anyone approves the price change. One resolution code path serves
both views, so a preview is what publishing would show.

Choices worth a look:

- **People only.** A token shows unapproved content to whoever holds it,
so no agent can mint one. The desk lane can add the issue and revoke
screen on these services.
- **One error for every failing token**, so a holder cannot probe which
tokens exist or why one stopped working.
- **Expiry is compared in memory** after the hash lookup, because SQLite
cannot compare stored offsets in a query.
- **Entries the set does not touch read as published**, so a preview of
one page links to the rest of the site as it is live.

## Tests

Integration, each on SQLite and PostgreSQL:

- S1: with the product at 24.99 and a new page title in an open set, the
token reads the page at version 2 with the title and 24.99, while the
published read still shows 19.99. `ChangesAsync` lists the product and
the page in operation order. After the set publishes, the token is
refused.
- The token stops working in each of these cases, with the same code:
  - after it is revoked;
  - after seven days;
  - when its set is rejected;
  - when it is well formed but unknown;
  - when it is malformed.

`ListAsync` shows the set's three tokens newest first, the revoked one
with its time. The audit has four `preview_token.issued` events and one
`preview_token.revoked`, with the token and the set.
- Refused, each with its own code:
  - an agent issuing or revoking;
  - a published set;
  - an unknown set;
  - an unknown token to revoke.

  Only the hash is stored.
- A set that unpublishes the product shows the page with a
`NotPublished` price and the product as not found. A draft outside the
set stays hidden.

HTTP, with stubs:

- A preview with the bearer token returns the entry and the changes with
`Cache-Control: no-store`.
- Without a token, or with a refused one, the response is a 401 problem.

The full suite ran locally against a `postgres:17` container as well as
SQLite. One known load-sensitive job queue test (#2536) failed once and
passed on rerun. Both migrations apply, revert and apply again.
markdownlint is clean, and no new package was added.

## Review

An independent review blocked this PR on one Medium finding. Revoking a
token that was already revoked was only refused inside the unit of work,
which broke a caller's own unit of work. It is now refused before the
unit of work with `preview_token_already_revoked`, and a test revokes
twice inside a caller's unit of work that still commits.

Its Low findings are fixed too:
- a 401 now names the `Bearer` scheme in `WWW-Authenticate`;
- previews are tested against a soft-deleted and an archived page
outside the set;
- a token lifetime that is not positive or exceeds a year is tested to
stop the host.

## Checklist

- [x] Linked to an issue a maintainer has confirmed
- [x] Targets `v2`
- [x] Tests or docs updated where behaviour changed
- [x] No tool-generated co-author trailers or footers
pournasserian added a commit that referenced this pull request Oct 19, 2026
)

Closes #2583

## What

The last F-API (#2425) slice, on the delivery endpoints of #2563, #2571
and #2577: rate limits, input validation and the read performance target
(SEC-07, PERF-03).

- **Rate limits** (`FluentCMS.Api`): `DeliveryRateLimit` is a
fixed-window policy per client address. It covers every delivery
endpoint and `/llms.txt`.
- The limit is `Delivery__RateLimit__PermitLimit` (600 by default) per
`Delivery__RateLimit__Window` (one minute by default). A limit below 1,
or a window that isn't positive, stops the host at startup.
- The policy has its own rejection: a 429 problem with `Retry-After` and
the code `rate_limited`. The sign-in forms keep their own policy and
their plain-text answer.
- `AddFluentCmsApi` now takes the configuration and registers the policy
and its options. The host passes the configuration in.
- An IPv6 client is counted by its /64 prefix, so it cannot take a fresh
address for each request.
- **Trusted proxies** (`FluentCMS.Host`):
- `ReverseProxy__KnownProxies__0` and `ReverseProxy__KnownNetworks__0`
list the proxies whose `X-Forwarded-For` names the client, before the
rate limiters run. This covers the sign-in limit too.
- With none listed, the header is ignored, so no client can choose the
address it is counted by.
  - An entry that is not an address or a network stops the host.
- **Input validation**: a content type in the path that cannot be a slug
is a 404 problem, without a database query. Every query parameter is
parsed before any read (#2563). The host's `nosniff` and content
security policy headers are now tested on delivery responses.
- **PERF-03**: a test, running alone in its own collection, loads 10,000
published entries: 50 products and pages that resolve their price. It
reads 200 random resolved pages through `IPublishedContent`, after 10
warm-up reads, and asserts the p95 is under 100 ms on SQLite and on
PostgreSQL. No new index is needed: reads use the entry primary key,
versions by entry and version, and references by entry and version.
- **Docs**: CONTRIBUTING lists the rate limit and proxy settings.
`docs/api/delivery.md` has a Limits section, and its error table has
`rate_limited`.

## Why

SEC-07 asks for rate limits and input validation on the delivery API.
The API is anonymous, so a per-address limit is what keeps one client
from exhausting the host. PERF-03 sets the bar for a single resolved
entry, and the test keeps it measured on both providers, so a regression
shows up in CI.

Choices worth a look:

- **Rows are seeded directly** for the 10,000 entries, as the directory
stores them. Loading them through the directory would take minutes, and
this test measures reads, not writes.
- **CORS and response caching are out of scope**, as #2583 says.
Server-rendered frontends need neither. A correct ETag would also have
to include the published versions of the referenced entities.

## Tests

- Over HTTP through the host:
- With a limit of 3, the fourth delivery request gets a 429 problem with
`Retry-After` and `rate_limited`. `/llms.txt` shares the limit, and
`/health` stays reachable.
- Two client addresses fall into different partitions. IPv6 addresses in
one /64 share a partition, and an IPv4-mapped address counts as its IPv4
address.
- Behind a trusted proxy, six clients named by `X-Forwarded-For` each
get their own limit. Without a trusted proxy, the header is ignored and
the fourth request is refused.
- A proxy entry that is not an address, or a network that is not one,
stops the host.
- Through the endpoints alone, a type that cannot be a slug is a 404
without the reader being asked.
- `Product-Page`, `product_page` and `a b` as a type are 404 problems
that carry `nosniff` and the content security policy.
- A permit limit of 0 stops the host from starting, and the message
names the setting.
- PERF-03 on SQLite and PostgreSQL, as described above.

The full suite ran locally against a `postgres:17` container as well as
SQLite, and markdownlint is clean. There is no migration and no new
package.

## Review

An independent review blocked this PR. Behind a reverse proxy, every
client shared the proxy's address. Honouring `X-Forwarded-For` from
anyone would have let a client pick a fresh bucket. Both are fixed with
trusted proxies from configuration. Its other findings are fixed too:
- IPv6 clients are grouped by /64;
- the slug check is tested before any read;
- the p95 test now runs in a non-parallel collection, so the rest of the
suite does not load the machine while it measures.

A second review gave MERGE. Its Low finding is fixed in the docs, which
now say that one forwarded hop is read and how to set up a proxy chain.
The sign-in limit's IPv6 grouping, which was already there before this
PR, is tracked in #2640.

## Checklist

- [x] Linked to an issue a maintainer has confirmed
- [x] Targets `v2`
- [x] Tests or docs updated where behaviour changed
- [x] No tool-generated co-author trailers or footers
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:api Delivery API (API) type:feature New capability v2 FluentCMS v2 (v2 branch)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Markdown per entry at a stable URL and a site index

1 participant