Bound stdlib file filters and define a safe symlink/file-type policy (#648)

.gitignore

@@ -21,3 +21,4 @@ __pycache__/
 .typos-oxendict-base.json
 .typos-oxendict-base.toml
 *.swo
+.hypothesis/

Cargo.toml

@@ -118,6 +118,7 @@ digest = "0.11"
 sha2 = "0.11"
 sha1 = { version = "0.11", optional = true }
 md5 = { package = "md-5", version = "0.11", optional = true }
+rustix = { version = "1.0.8", features = ["fs"] }
 itoa = "1"
 itertools = "0.12"
 indexmap = { version = "2.5", features = ["serde"] }
@@ -183,11 +184,6 @@ regex = "1.12.2"
 # the version `ortho_config` resolves, or the `FluentValue` types would differ.
 fluent-bundle = "0.16.0"
 
-# Target-specific dev-deps
-[target.'cfg(unix)'.dev-dependencies]
-# Used only to construct FIFOs and device nodes in tests.
-rustix = { version = "1.0.8", features = ["fs"] }
-
 [workspace]
 members = ["test_support"]
 

docs/security-network-command-audit.md

@@ -79,6 +79,30 @@ introduces, and concrete remediation tasks that would harden the helpers.
     budgets incrementally so long-running commands fail fast once the
     configured allowance is exceeded.
 
+## File helper findings
+
+- [x] **File-reading filters read unbounded, untrusted entries.** *(Status:
+  remediated in the bounded file-read policy.)* The `contents`, `linecount`,
+  `hash`, and `digest` filters opened caller-supplied paths and read them to
+  EOF without a byte budget or a file-type check. A contributor who replaced a
+  trusted manifest's input path with a huge regular file, a symlink to
+  `/dev/zero`, or a FIFO could exhaust Netsuke's memory, consume unbounded CPU
+  and I/O, or block a build worker indefinitely. *Remediation tasks:*
+  - Enforce a configurable byte budget while streaming, not only from metadata
+    observed before the read.
+  - Open the final entry without following symlinks and verify the opened
+    object is a regular file.
+  - Count lines incrementally instead of loading the entire file.
+  - **Remediation:** the reading filters now share one policy. The final path
+    component is opened with `O_NOFOLLOW` (a pre-open symlink check on
+    Windows), the opened handle must be a regular file, and `contents`,
+    `linecount`, `hash`, and `digest` stream against a running byte total
+    anchored to `StdlibConfig::with_file_max_read_bytes` (default 8 MiB).
+    `linecount` counts terminators incrementally instead of materializing the
+    file. Per-call `max_bytes` may narrow the ceiling and a named
+    `follow_symlinks=true` opt-in permits link following; rejections surface
+    localized diagnostics naming the path and limit without file contents.
+
 ## Next steps
 
 The tasks above can be implemented incrementally. A good first milestone is to

docs/stdlib-yaml-and-jinja-guide.md

@@ -156,6 +156,22 @@ read. Relative paths are resolved from the workspace in which Netsuke runs.
   defaults to `8` and the algorithm defaults to `sha256`. Example:
   `{{ 'fixtures/message.txt' | digest(12, 'sha512') }}`.
 
+All four filters share one safety policy: the final path component is opened
+without following symlinks, the opened object must be a regular file, and each
+read stops at a shared byte budget (8 MiB by default). A read that exceeds the
+budget, or a path that names a symlink, FIFO, or device, fails with a localized
+diagnostic quoting the path and the applicable limit. Two optional keyword
+arguments narrow a call without touching the operator ceiling:
+
+- `max_bytes` lowers the budget for one call (a value above the configured
+  budget is clamped to it). Example:
+  `{{ 'fixtures/big.bin' | contents(max_bytes=1024) }}`.
+- `follow_symlinks=true` permits the final component to be a symlink. Example:
+  `{{ 'link/version.txt' | contents(follow_symlinks=true) }}`.
+
+See the users' guide section on file reading limits for the defaults, the
+symlink policy, and the trust model these limits assume.
+
 MD5 and SHA-1 are available only in builds compiled with Cargo feature
 `legacy-digests`. Without that feature, `hash('md5')`, `hash('sha1')`, and their
 `digest` equivalents fail with a feature-specific diagnostic. New manifests

docs/users-guide.md

@@ -1600,6 +1600,41 @@ Avoid placing secrets in URLs. Netsuke logs hosts and cache keys rather than
 complete URLs, but downloaded content and commands still run within the host
 trust boundary.
 
+## Configure file reading limits
+
+The `contents`, `linecount`, `hash`, and `digest` template filters read files
+through a shared byte budget so a checkout entry cannot exhaust Netsuke's
+memory or CPU. The default budget is 8 MiB per read, matching the `fetch()`
+response limit. Hosts embedding Netsuke can raise or lower it with
+`StdlibConfig::with_file_max_read_bytes` before registering the standard
+library.
+
+The reading filters also refuse to follow a symlink as the final path component
+and reject anything that is not a regular file once opened, including FIFOs and
+device nodes. A symlinked directory used *inside* a path is unaffected; only
+the final entry is checked. Templates that deliberately read through a final
+symlink can pass `follow_symlinks=true` to accept the link:
+
+<!-- tested-example: guide-file-follow-symlinks-expression -->
+
+```jinja
+{{ 'generated/version.txt' | contents(follow_symlinks=true) }}
+```
+
+A single call may lower the budget with `max_bytes`, but never raise it above
+the configured ceiling:
+
+<!-- tested-example: guide-file-max-bytes-expression -->
+
+```jinja
+{{ 'fixtures/big.bin' | hash(max_bytes=1024) }}
+```
+
+Reads that exceed the budget fail with a diagnostic naming the path and the
+limit, never the file contents. Raise the operator budget when legitimate
+builds hash large artefacts; prefer per-call `max_bytes` narrowing when a
+manifest merely wants to bound one input.
+
 ## Interpret failures
 
 Netsuke reports failures at the earliest stage that can identify them:
@@ -1638,6 +1673,10 @@ Netsuke reduces some common quoting mistakes, but it is not a sandbox:
   On Unix, scripts use `/bin/sh -e`.
 - `shell`, `grep`, `fetch`, filesystem helpers, and ordinary recipes interact
   with the host.
+- The file-reading filters (`contents`, `linecount`, `hash`, `digest`) read at
+  most the configured byte budget, open the final path entry without following
+  symlinks, and require the opened object to be a regular file. See
+  [Configure file reading limits](#configure-file-reading-limits).
 - `glob` restricts its filesystem metadata access to a capability handle
   scoped to the pattern's literal directory prefix, so it cannot inspect
   anything outside the subtree the pattern can match; the pattern match walk

locales/ar/messages.ftl

@@ -209,6 +209,7 @@ network_policy.host.not_allowlisted = المضيف «{ $host }» ليس ضمن 
 stdlib.config.default_fetch_cache_invalid = يجب أن يكون المسار الافتراضي لذاكرة fetch المخبّأة نسبيًا.
 stdlib.config.default_which_cache_invalid = يجب أن تكون السعة الافتراضية لذاكرة which المخبّأة موجبة.
 stdlib.config.workspace_root_absolute = يجب أن يكون مسار جذر مساحة العمل مطلقًا.
+stdlib.config.file_read_limit_positive = يجب أن يكون حدّ قراءة الملفات موجبًا.
 stdlib.config.fetch_response_limit_positive = يجب أن يكون حدّ استجابة fetch موجبًا.
 stdlib.config.command_output_limit_positive = يجب أن يكون حدّ التقاط مخرجات الأوامر موجبًا.
 stdlib.config.command_stream_limit_positive = يجب أن يكون حدّ تدفّق الأوامر موجبًا.
@@ -317,6 +318,8 @@ stdlib.path.with_suffix.empty_separator = يتطلّب with_suffix فاصلًا
 stdlib.path.relative_to.mismatch = المسار { $path } ليس نسبيًا إلى { $root }.
 stdlib.path.expanduser.unsupported = توسيع ~ لمستخدم بعينه غير مدعوم.
 stdlib.path.expanduser.no_home = تعذّر توسيع ~: لم يُضبط أي متغيّر بيئة لدليل المنزل.
+stdlib.path.contents.file_too_large = تجاوز الملف '{ $path }' حدّ القراءة البالغ { $limit } بايت.
+stdlib.path.contents.not_regular_file = الملف '{ $path }' ليس ملفًا عاديًا.
 stdlib.path.contents.unsupported_encoding = ترميز غير مدعوم: «{ $encoding }».
 stdlib.path.hash.unsupported_algorithm = خوارزمية تلبيد غير مدعومة: «{ $algorithm }».
 stdlib.path.hash.unsupported_algorithm_legacy = خوارزمية تلبيد غير مدعومة: «{ $algorithm }» (فعّل الميزة «{ $feature }»).

locales/cs/messages.ftl

@@ -209,6 +209,7 @@ network_policy.host.not_allowlisted = Hostitel „{ $host }“ není na seznamu
 stdlib.config.default_fetch_cache_invalid = Výchozí cesta mezipaměti fetch musí být relativní.
 stdlib.config.default_which_cache_invalid = Výchozí kapacita mezipaměti which musí být kladná.
 stdlib.config.workspace_root_absolute = Kořenová cesta pracovního prostoru musí být absolutní.
+stdlib.config.file_read_limit_positive = File read limit must be positive.
 stdlib.config.fetch_response_limit_positive = Limit odpovědi fetch musí být kladný.
 stdlib.config.command_output_limit_positive = Limit zachyceného výstupu příkazů musí být kladný.
 stdlib.config.command_stream_limit_positive = Limit proudu příkazů musí být kladný.
@@ -317,6 +318,8 @@ stdlib.path.with_suffix.empty_separator = with_suffix vyžaduje neprázdný odd
 stdlib.path.relative_to.mismatch = { $path } není relativní vůči { $root }.
 stdlib.path.expanduser.unsupported = Rozvoj znaku ~ pro konkrétního uživatele není podporován.
 stdlib.path.expanduser.no_home = Znak ~ nelze rozvinout: není nastavena žádná proměnná prostředí domovského adresáře.
+stdlib.path.contents.file_too_large = File '{ $path }' exceeded the read limit of { $limit } bytes.
+stdlib.path.contents.not_regular_file = File '{ $path }' is not a regular file.
 stdlib.path.contents.unsupported_encoding = Nepodporované kódování „{ $encoding }“.
 stdlib.path.hash.unsupported_algorithm = Nepodporovaný hashovací algoritmus „{ $algorithm }“.
 stdlib.path.hash.unsupported_algorithm_legacy = Nepodporovaný hashovací algoritmus „{ $algorithm }“ (zapněte funkci „{ $feature }“).

locales/cy/messages.ftl

@@ -209,6 +209,7 @@ network_policy.host.not_allowlisted = Nid yw'r gwesteiwr ‘{ $host }’ ar y rh
 stdlib.config.default_fetch_cache_invalid = Rhaid i lwybr rhagosodedig storfa fetch fod yn gymharol.
 stdlib.config.default_which_cache_invalid = Rhaid i gynhwysedd rhagosodedig storfa which fod yn bositif.
 stdlib.config.workspace_root_absolute = Rhaid i lwybr gwraidd y gweithle fod yn absoliwt.
+stdlib.config.file_read_limit_positive = File read limit must be positive.
 stdlib.config.fetch_response_limit_positive = Rhaid i derfyn ymateb fetch fod yn bositif.
 stdlib.config.command_output_limit_positive = Rhaid i derfyn dal allbwn gorchmynion fod yn bositif.
 stdlib.config.command_stream_limit_positive = Rhaid i derfyn ffrwd y gorchmynion fod yn bositif.
@@ -317,6 +318,8 @@ stdlib.path.with_suffix.empty_separator = Mae with_suffix angen gwahanydd nad yw
 stdlib.path.relative_to.mismatch = Nid yw { $path } yn gymharol i { $root }.
 stdlib.path.expanduser.unsupported = Ni chefnogir ehangu ~ ar gyfer defnyddiwr penodol.
 stdlib.path.expanduser.no_home = Ni ellir ehangu ~: nid oes newidyn amgylchedd cyfeiriadur cartref wedi'i osod.
+stdlib.path.contents.file_too_large = File '{ $path }' exceeded the read limit of { $limit } bytes.
+stdlib.path.contents.not_regular_file = File '{ $path }' is not a regular file.
 stdlib.path.contents.unsupported_encoding = Amgodiad nas cefnogir: ‘{ $encoding }’.
 stdlib.path.hash.unsupported_algorithm = Algorithm stwnsio nas cefnogir: ‘{ $algorithm }’.
 stdlib.path.hash.unsupported_algorithm_legacy = Algorithm stwnsio nas cefnogir: ‘{ $algorithm }’ (galluogwch y nodwedd ‘{ $feature }’).

locales/da/messages.ftl

@@ -209,6 +209,7 @@ network_policy.host.not_allowlisted = Værten "{ $host }" står ikke på listen
 stdlib.config.default_fetch_cache_invalid = Standardstien til fetch-mellemlageret skal være relativ.
 stdlib.config.default_which_cache_invalid = Standardkapaciteten for which-mellemlageret skal være positiv.
 stdlib.config.workspace_root_absolute = Rodstien for arbejdsområdet skal være absolut.
+stdlib.config.file_read_limit_positive = File read limit must be positive.
 stdlib.config.fetch_response_limit_positive = Svargrænsen for fetch skal være positiv.
 stdlib.config.command_output_limit_positive = Grænsen for opsamlet kommandooutput skal være positiv.
 stdlib.config.command_stream_limit_positive = Strømgrænsen for kommandoer skal være positiv.
@@ -317,6 +318,8 @@ stdlib.path.with_suffix.empty_separator = with_suffix kræver en adskiller, der
 stdlib.path.relative_to.mismatch = { $path } er ikke relativ til { $root }.
 stdlib.path.expanduser.unsupported = Brugerspecifik udvidelse af ~ understøttes ikke.
 stdlib.path.expanduser.no_home = ~ kan ikke udvides: der er ingen miljøvariabler for hjemmemappen.
+stdlib.path.contents.file_too_large = File '{ $path }' exceeded the read limit of { $limit } bytes.
+stdlib.path.contents.not_regular_file = File '{ $path }' is not a regular file.
 stdlib.path.contents.unsupported_encoding = Tegnkodningen "{ $encoding }" understøttes ikke.
 stdlib.path.hash.unsupported_algorithm = Hash-algoritmen "{ $algorithm }" understøttes ikke.
 stdlib.path.hash.unsupported_algorithm_legacy = Hash-algoritmen "{ $algorithm }" understøttes ikke (slå funktionen "{ $feature }" til).

locales/de/messages.ftl

@@ -209,6 +209,7 @@ network_policy.host.not_allowlisted = Der Host „{ $host }“ steht nicht auf d
 stdlib.config.default_fetch_cache_invalid = Der voreingestellte Pfad des fetch-Caches muss relativ sein.
 stdlib.config.default_which_cache_invalid = Die voreingestellte Kapazität des which-Caches muss positiv sein.
 stdlib.config.workspace_root_absolute = Der Wurzelpfad des Arbeitsbereichs muss absolut sein.
+stdlib.config.file_read_limit_positive = File read limit must be positive.
 stdlib.config.fetch_response_limit_positive = Das Antwortlimit von fetch muss positiv sein.
 stdlib.config.command_output_limit_positive = Das Limit für erfasste Befehlsausgaben muss positiv sein.
 stdlib.config.command_stream_limit_positive = Das Stream-Limit für Befehle muss positiv sein.
@@ -317,6 +318,8 @@ stdlib.path.with_suffix.empty_separator = with_suffix benötigt ein nicht leeres
 stdlib.path.relative_to.mismatch = { $path } ist nicht relativ zu { $root }.
 stdlib.path.expanduser.unsupported = Die benutzerspezifische Erweiterung von ~ wird nicht unterstützt.
 stdlib.path.expanduser.no_home = ~ kann nicht erweitert werden: Es sind keine Umgebungsvariablen für das Heimatverzeichnis gesetzt.
+stdlib.path.contents.file_too_large = File '{ $path }' exceeded the read limit of { $limit } bytes.
+stdlib.path.contents.not_regular_file = File '{ $path }' is not a regular file.
 stdlib.path.contents.unsupported_encoding = Nicht unterstützte Kodierung „{ $encoding }“.
 stdlib.path.hash.unsupported_algorithm = Nicht unterstützter Hash-Algorithmus „{ $algorithm }“.
 stdlib.path.hash.unsupported_algorithm_legacy = Nicht unterstützter Hash-Algorithmus „{ $algorithm }“ (aktivieren Sie das Feature „{ $feature }“).

locales/el/messages.ftl

@@ -210,6 +210,7 @@ network_policy.host.not_allowlisted = Ο κόμβος «{ $host }» δεν πε
 stdlib.config.default_fetch_cache_invalid = Η προεπιλεγμένη διαδρομή της κρυφής μνήμης fetch πρέπει να είναι σχετική.
 stdlib.config.default_which_cache_invalid = Η προεπιλεγμένη χωρητικότητα της κρυφής μνήμης which πρέπει να είναι θετική.
 stdlib.config.workspace_root_absolute = Η ριζική διαδρομή του χώρου εργασίας πρέπει να είναι απόλυτη.
+stdlib.config.file_read_limit_positive = File read limit must be positive.
 stdlib.config.fetch_response_limit_positive = Το όριο απόκρισης του fetch πρέπει να είναι θετικό.
 stdlib.config.command_output_limit_positive = Το όριο καταγραφής της εξόδου εντολών πρέπει να είναι θετικό.
 stdlib.config.command_stream_limit_positive = Το όριο ροής εντολών πρέπει να είναι θετικό.
@@ -318,6 +319,8 @@ stdlib.path.with_suffix.empty_separator = Το with_suffix απαιτεί μη 
 stdlib.path.relative_to.mismatch = Το { $path } δεν είναι σχετικό ως προς το { $root }.
 stdlib.path.expanduser.unsupported = Η ανάπτυξη του ~ για συγκεκριμένο χρήστη δεν υποστηρίζεται.
 stdlib.path.expanduser.no_home = Δεν είναι δυνατή η ανάπτυξη του ~: δεν έχει οριστεί καμία μεταβλητή περιβάλλοντος για τον προσωπικό κατάλογο.
+stdlib.path.contents.file_too_large = File '{ $path }' exceeded the read limit of { $limit } bytes.
+stdlib.path.contents.not_regular_file = File '{ $path }' is not a regular file.
 stdlib.path.contents.unsupported_encoding = Μη υποστηριζόμενη κωδικοποίηση «{ $encoding }».
 stdlib.path.hash.unsupported_algorithm = Μη υποστηριζόμενος αλγόριθμος κατακερματισμού «{ $algorithm }».
 stdlib.path.hash.unsupported_algorithm_legacy = Μη υποστηριζόμενος αλγόριθμος κατακερματισμού «{ $algorithm }» (ενεργοποιήστε τη δυνατότητα «{ $feature }»).

locales/en-GB/messages.ftl

@@ -209,6 +209,7 @@ network_policy.host.not_allowlisted = Host '{ $host }' is not on the allowlist.
 stdlib.config.default_fetch_cache_invalid = Default fetch cache path must be relative.
 stdlib.config.default_which_cache_invalid = Default which cache capacity must be positive.
 stdlib.config.workspace_root_absolute = Workspace root path must be absolute.
+stdlib.config.file_read_limit_positive = File read limit must be positive.
 stdlib.config.fetch_response_limit_positive = Fetch response limit must be positive.
 stdlib.config.command_output_limit_positive = Command output capture limit must be positive.
 stdlib.config.command_stream_limit_positive = Command stream limit must be positive.
@@ -317,6 +318,8 @@ stdlib.path.with_suffix.empty_separator = with_suffix requires a non-empty separ
 stdlib.path.relative_to.mismatch = { $path } is not relative to { $root }.
 stdlib.path.expanduser.unsupported = User-specific ~ expansion is unsupported.
 stdlib.path.expanduser.no_home = Cannot expand ~: no home directory environment variables are set.
+stdlib.path.contents.file_too_large = File '{ $path }' exceeded the read limit of { $limit } bytes.
+stdlib.path.contents.not_regular_file = File '{ $path }' is not a regular file.
 stdlib.path.contents.unsupported_encoding = Unsupported encoding '{ $encoding }'.
 stdlib.path.hash.unsupported_algorithm = Unsupported hash algorithm '{ $algorithm }'.
 stdlib.path.hash.unsupported_algorithm_legacy = Unsupported hash algorithm '{ $algorithm }' (enable feature '{ $feature }').