From 23723426822c81f646f52b811602ec53a7f6cfb6 Mon Sep 17 00:00:00 2001 From: Akonwi Ngoh Date: Tue, 8 Sep 2026 21:41:59 -0400 Subject: [PATCH 1/6] docs(adr): accept compile-time embedded filesystems --- ...1-add-compile-time-embedded-filesystems.md | 587 ++++++++++++++++++ 1 file changed, 587 insertions(+) create mode 100644 docs/adrs/0071-add-compile-time-embedded-filesystems.md diff --git a/docs/adrs/0071-add-compile-time-embedded-filesystems.md b/docs/adrs/0071-add-compile-time-embedded-filesystems.md new file mode 100644 index 00000000..d616cd9b --- /dev/null +++ b/docs/adrs/0071-add-compile-time-embedded-filesystems.md @@ -0,0 +1,587 @@ +# 0071: Add Compile-Time Embedded Filesystems + +## Status + +Accepted + +## Context + +Applications commonly ship templates, static web assets, schemas, migrations, +certificates, and other files inside the executable. Ard can currently do this +through a project-local Go FFI package containing `//go:embed`, but that forces a +pure Ard application to add a Go shim solely for build-time packaging. + +Go's embedding model has two useful parts: + +- a build-time selection language for exact files, patterns, and directory + trees; and +- an immutable `embed.FS` value implementing `io/fs.FS` for runtime lookup. + +The Go directive itself is not an appropriate Ard source feature. Ard is its own +language, AIR is its target-neutral backend boundary, and future targets must be +able to implement the same behavior without parsing or emulating Go source. +Still, the Go backend should use native `//go:embed` output because it stores +large resources efficiently and interoperates with the Go filesystem ecosystem. + +Embedding also introduces compiler inputs that are not `.ard` files. Checking, +AIR lowering, generated artifacts, dependency ownership, and LSP invalidation +must all use the bytes captured for a resource during that analysis. A backend +must not reread the original source file after checking, because the resulting +binary could then contain bytes different from those that were validated. + +ADR 0049 reserves exact compiler-owned paths under `ard/*`. ADR 0069 establishes +a related precedent: build-time data should be explicit, typed, deterministic, +and carried through the checker-to-AIR pipeline rather than discovered from +ambient process state. + +## Decision + +Reserve `ard/embed` as a compiler-owned intrinsic module. It provides exact-file +embedding and a statically constructed immutable filesystem: + +```ard +use ard/embed + +let license: Str = embed::text("LICENSE") +let logo: [Byte] = embed::bytes("assets/logo.png") +let assets: embed::FS = embed::fs([ + "public", + "templates/*.html", + "all:public/.well-known", +]) +``` + +These constructors execute during checking, not at runtime. They require static +arguments, capture the selected contents in AIR, and never return `Result`. +Runtime filesystem operations on `embed::FS` do return `Result`. + +### Exact-file constructors + +`ard/embed` exposes: + +```ard +embed::text(path: Str) Str +embed::bytes(path: Str) [Byte] +``` + +The argument must be exactly one positional, non-interpolated string literal. +Computed strings, interpolation, named arguments, and references to the +constructor as a function value are rejected. Exact-file paths do not interpret +pattern metacharacters or the `all:` prefix. They must name one regular file; +files beginning with `.` or `_` may be selected explicitly when the rest of the +portable path rules permit them. + +`text` requires the selected file to contain valid UTF-8 and preserves its bytes +exactly. It does not normalize newlines, strip a byte-order mark, or otherwise +rewrite the contents. Invalid UTF-8 is a compile-time diagnostic; `bytes` is the +alternative for arbitrary data. + +Each evaluation of `bytes` produces fresh mutable list storage. Mutating one +result cannot modify another result or the compiler-owned embedded bytes. A +module-level binding is initialized once and thereafter follows ordinary Ard +list sharing and mutability rules. `Str` remains immutable. + +### Embedded filesystem constructor + +`ard/embed` also exposes: + +```ard +embed::fs(patterns: [Str]) embed::FS +``` + +The argument must be a non-empty list literal containing only +non-interpolated string literals. Computed lists, spreads, and references to the +constructor as a function value are rejected. The selected filesystem is the +union of all pattern matches. Overlapping and repeated patterns are permitted; +files are stored once by their logical path. Every pattern must independently +match at least one selectable file. + +The compiler expands every pattern and reads every selected file while checking. +The resulting file set and bytes are part of the checked program and AIR. The +constructor performs no runtime I/O and cannot fail at runtime. + +### Resource ownership and path base + +Constructor paths and patterns are relative to the root of the Ard package that +owns the module containing the constructor call. They are not relative to the +`.ard` file, compiler process working directory, generated output directory, or +consuming application. + +For example: + +```text +widgets/ +├── ard.toml +├── assets/ +│ └── icon.svg +└── ui/ + └── button.ard +``` + +A call in `widgets/ui/button.ard` uses: + +```ard +embed::text("assets/icon.svg") +``` + +When `widgets` is consumed as a dependency, that path still resolves against the +selected `widgets` package root. A dependency cannot read or be overridden by a +similarly named file in the consuming project. Git dependencies read resources +from their locked checkout; path dependencies read them from their declared +source root. + +Using package-root-relative paths follows Ard's absolute, package-oriented +module model from ADR 0013 and ensures that moving an `.ard` module does not +silently change the resource it selects. + +### Pattern language + +Ard's pattern language is based on Go embed patterns, with explicit Ard-defined +differences: paths are package-root-relative rather than Go-package-relative, +backslashes are rejected instead of acting as escapes, and nested Ard package +boundaries are enforced in addition to Go module boundaries. The compiler +implements selection itself; it does not ask the active Go toolchain to decide +which source files belong to an AIR resource set. + +Patterns have these rules: + +- `/` is the separator on every platform. +- Pattern matching otherwise follows Go `path.Match`; `*`, `?`, and character + classes do not cross `/` separators. Malformed character classes are errors. +- A path naming a directory selects its complete subtree recursively. +- Recursive directory selection excludes files and directories whose names + begin with `.` or `_`. This exclusion does not apply to an explicit wildcard + match: for example, `images/*` may select `images/.thumb`, while `images` + does not. +- The `all:` prefix includes otherwise excluded names during recursive directory + selection. +- Patterns cannot be empty or absolute and cannot contain empty, `.`, or `..` + path segments. +- Backslashes are rejected in constructor patterns rather than acting as host + separators or pattern escapes. +- Symlinks and non-regular files are not selectable. +- Selection cannot escape the owning Ard package or cross into a nested Ard + package boundary, a nested directory containing `go.mod`, or a `vendor` + directory. A `go.mod` at the owning package root does not block selection, but + a selected logical file named `go.mod` at any depth is rejected because its + generated copy would create a nested Go module boundary. +- Version-control directories `.bzr`, `.git`, `.hg`, and `.svn` are not + selectable. +- Every selected path element must be valid UTF-8. Allowed characters are + Unicode letters, ASCII digits, ASCII space, and the ASCII punctuation + `!#$%&()+,-.=@[]^_{}~`. An element cannot consist only of dots or end in a + dot. Its case-insensitive prefix before the first dot cannot be a Windows + reserved device name: `CON`, `PRN`, `AUX`, `NUL`, `COM1` through `COM9`, or + `LPT1` through `LPT9`. These fixed portable-name rules intentionally match + Go's `module.CheckFilePath` at the time of this decision; the implementation + may use that helper only alongside regression tests preserving Ard's fixed + contract. +- Empty directories are not represented. The root and every ancestor directory + of a selected file are synthesized in the embedded filesystem. + +These rules are fixed Ard semantics. Other targets must reproduce them even if +their host language has a different resource or glob implementation. + +### Runtime filesystem API + +`embed::FS` is an immutable compiler-owned nominal type. Conceptually, the +intrinsic module declares the following Ard API; the qualified names shown at +use sites come from importing the module, not from qualified declaration syntax: + +```ard +// FS is a compiler-owned opaque type with no literal syntax. + +struct DirEntry { + name: Str, + is_dir: Bool, +} + +struct FileInfo { + name: Str, + is_dir: Bool, + size: Int?, +} + +impl FS { + fn read_file(path: Str) [Byte]!Error + fn read_text(path: Str) Str!Error + fn read_dir(path: Str) [DirEntry]!Error + fn stat(path: Str) FileInfo!Error + fn sub(path: Str) FS!Error +} +``` + +`FS` has no public struct literal or empty constructor. Only `embed::fs` and +`FS.sub` construct values. It may be stored in structs, `Maybe`, `Any`, and +generic values, but it has no equality or ordering operations, is not a valid +map key, and has no automatic JSON representation. + +Runtime lookup paths follow `io/fs.ValidPath` semantics: + +- `"."` denotes the filesystem root; +- other paths are non-empty, unrooted, and slash-separated; +- empty, `.` and `..` elements are invalid; +- leading or trailing slashes are invalid; and +- backslash is an ordinary character, never a path separator. + +`read_file` requires a file and returns a fresh mutable byte list. The caller may +modify it without affecting the filesystem or later reads. `read_text` requires +a file and validates UTF-8 at runtime because its path may select any file in a +mixed binary/text filesystem. It otherwise preserves contents exactly. + +`read_dir` requires a directory, returns only its immediate children, and sorts +them lexicographically by filename. Its returned list is fresh and mutable. +`DirEntry.name` is the basename, not a full path. + +`stat` accepts files, synthesized directories, and `"."`. A regular file +reports `some(byte_count)` for `size`; a directory reports `none`. +`stat(".").name` is `"."`, including on a filesystem returned by `sub`. +Sizes are guaranteed to fit Ard `Int` by the compile-time resource limits. +Embedded filesystems do not expose permissions, ownership, timestamps, devices, +or symlinks because those values are absent or intentionally not part of +reproducible embedding. + +`sub` eagerly verifies that its path names an existing synthesized directory, +then returns an immutable view rooted there. The Go backend therefore performs +`fs.Stat` and an `IsDir` check before `fs.Sub`, whose own fallback can otherwise +validate lazily. `"."` in the returned filesystem denotes the subtree root. The +view shares immutable embedded storage with its parent. + +Copies of `embed::FS` are small handles sharing immutable content. Its methods do +not require a mutable receiver and are safe for concurrent use. Taking an Ard +mutable reference to the handle does not grant interior mutation operations. + +### Errors + +Constructor failures are compile-time diagnostics located at the invalid path or +pattern. Runtime operations return the builtin `Error` type. + +On the Go target, filesystem errors preserve Go error identity as required by +ADR 0063. In particular, callers using direct Go interop can test errors against +`io/fs.ErrInvalid`, `io/fs.ErrNotExist`, and related sentinels through +`errors.Is`. `read_text` wraps invalid UTF-8 as a path error whose cause is +`io/fs.ErrInvalid`. The operation and logical path remain available through a +Go `*fs.PathError` where applicable, but exact human-readable error messages are +not a stable language contract. + +### Go filesystem interoperability + +On the Go target, `embed::FS` lowers directly to an `io/fs.FS` interface value. +Its initial dynamic value is derived from Go `embed.FS`; `sub` may produce the +Go filesystem implementation returned by `fs.Sub`. Ard's nominal identity is a +checker and AIR guarantee and does not require a generated Go wrapper type. +Interface copies provide the specified small shared handle, and there is no nil +or zero `embed::FS` value constructible from safe Ard. + +This representation intentionally satisfies imported `io/fs.FS`. +Compiler-owned intrinsic types may declare this built-in foreign-interface +bridge without a source `impl`: + +```ard +use ard/embed +use go:net/http + +let assets = embed::fs(["public"]) +let web_filesystem = http::FS(assets) +``` + +This also permits use with APIs such as `template.ParseFS`. The language +contract promises `io/fs.FS` compatibility on the Go target. It does not promise +that direct type assertions to optional Go interfaces such as `io/fs.ReadFileFS` +or `io/fs.ReadDirFS` succeed, even when a backend value happens to implement +them. Pure Ard code uses the methods declared by `embed::FS`. + +On non-Go targets, `embed::FS` retains the same Ard API and semantics using a +target-specific immutable representation. Go interface compatibility is an +explicit target interop property, not the definition of the Ard type. + +### Resource limits + +The compiler must protect CLI and language-server processes from accidental or +malicious resource expansion. Initial limits are: + +- 16 MiB per file; +- 64 MiB per embedded filesystem; +- 128 MiB across one checked program; and +- 10,000 files across one checked program. + +The per-file limit applies to every selected logical file. The per-filesystem +limit sums each unique logical path in that set, even when different paths have +identical bytes. Identical filesystem sets are interned by owner plus their +sorted path/blob mapping. Program totals are defined entirely over checked +resource identities: each unique set contributes its entry count and the sum of +its entries' blob sizes, while exact constructors contribute each distinct +referenced blob once. A blob referenced both directly and from a set counts in +both categories, as do entries in distinct non-identical sets. This conservative +accounting is independent of backend deduplication and prevents overlapping sets +from hiding artifact growth through content hashes. + +Checking applies these ceilings to all source expressions it validates so a +test-only declaration cannot exhaust the compiler. Production AIR and artifacts +omit resources referenced only by declarations excluded under the existing +`IncludeTests` policy; test AIR retains them. Diagnostics report the observed +and allowed values. These ceilings may be raised compatibly in future releases; +configurable limits and application packaging of larger resource trees are +deferred. + +Embedded data is recoverable from the resulting artifact. Documentation must +warn users not to embed passwords, tokens, private keys, or other secrets and +must note that redistribution and license obligations apply to embedded files. + +## Checker and project loading + +The checker recognizes `ard/embed` by exact intrinsic identity, not merely by a +local import alias or function name. It constructs checked embedded-resource +expressions after validating static arguments and delegates package ownership, +path containment, and dependency-root resolution to project/module loading. +The parser does not read files. + +Resource collection is invocation-local. Process-wide embedded standard-library +caches must not retain application resources. Repeated references may share +captured immutable content internally, but their source expressions and runtime +value semantics remain distinct. + +Diagnostics cover at least: + +- non-static constructor arguments; +- an empty filesystem pattern list; +- malformed, absolute, or traversing paths; +- a pattern that matches nothing; +- unreadable files; +- symlinks and non-regular files; +- package-boundary escapes; +- invalid path-name UTF-8 or unsupported names; +- invalid file UTF-8 for `text`; +- resource limits; and +- intrinsic constructors used as function values. + +Runtime invalid paths and missing entries are values in `Result`, not compiler +diagnostics. + +## AIR representation + +AIR carries resources independently from their source filesystem and separates +content identity from logical filesystem membership: + +```text +Program.EmbeddedBlobs []EmbeddedBlob +Program.EmbeddedSets []EmbeddedSet + +EmbeddedBlob { + ID + Data []byte + Digest +} + +EmbeddedSet { + ID + OwnerPackageIdentity + Entries []EmbeddedEntry + Digest +} + +EmbeddedEntry { + Path + BlobID +} +``` + +`OwnerPackageIdentity` is the canonical root/dependency package identity +supplied by project loading and used for ownership and interning; it is not a +filesystem root or process-local checker pointer. Pattern spelling, source +module identity, and call-site attribution remain on checked expressions and AIR +source locations rather than on interned sets, so different constructors that +select identical contents can share one set deterministically. Project loading +and checking are solely responsible for proving filesystem containment before +AIR is produced. A set contains only files; its root and ancestor directories +are derived deterministically from entry paths. + +It adds a target-neutral embedded-filesystem type and explicit operations: + +```text +TypeEmbeddedFS +ExprEmbeddedText +ExprEmbeddedBytes +ExprMakeEmbeddedFS +ExprEmbeddedReadFile +ExprEmbeddedReadText +ExprEmbeddedReadDir +ExprEmbeddedStat +ExprEmbeddedSub +``` + +Exact-file expressions refer to blob IDs and filesystem expressions refer to +set IDs rather than expanding bytes into thousands of integer literal nodes. +Entries are sorted by logical path and deduplicated; sets may share immutable +blobs while retaining distinct paths. Digests make resource identity and +generated names deterministic; contents remain authoritative. + +AIR validation checks blob and set IDs, sorted unique logical paths, content +digests, derived file/directory conflicts, normalized stable package identity, +resource limits, and each operation's result type. It does not attempt to +re-prove historical filesystem containment. Serialized AIR includes captured +contents so backend generation never depends on the original source tree. + +## Go lowering and generated artifacts + +The Go backend uses native `//go:embed` over compiler-staged copies of the AIR +resources. It does not place user patterns directly in generated directives. + +Generated output contains a compiler-owned package such as +`internal/ardembed`. For an embedded filesystem it emits a shape equivalent to: + +```go +package ardembed + +import ( + "embed" + "io/fs" +) + +//go:embed all:data/a1b2c3 +var rawA1B2C3 embed.FS + +var SetA1B2C3 fs.FS = mustSub(rawA1B2C3, "data/a1b2c3") +``` + +The backend materializes the exact captured entries beneath the generated set +path, then uses `fs.Sub` so user-visible runtime paths do not contain the +compiler-owned prefix. A failed generated `fs.Sub` is an internal compiler +layout invariant and may panic during generated package initialization. + +Exact text and byte resources use unexported generated string variables +initialized by safe compiler-owned directives. Because generated Ard modules +are separate Go packages, the resource package exposes compiler-generated Go +accessors such as `TextA1B2C3() string` and `BytesA1B2C3() []byte`; these names +are not visible as Ard declarations. The byte accessor performs a conversion or +copy at each call to guarantee fresh mutable storage. + +Filesystem operations lower through `io/fs` helpers such as `fs.ReadFile`, +`fs.ReadDir`, `fs.Stat`, and `fs.Sub`, with explicit conversion into Ard's +`Result`, `DirEntry`, and `FileInfo` representations. `read_file` must preserve +the fresh-copy contract even if the underlying target helper does not. + +Generated resources should be centralized rather than duplicated into every Go +package corresponding to an Ard module. Generated Ard packages import the +compiler-owned resource package, which has no dependency back to user modules +and therefore cannot create import cycles. + +The backend artifact model expands from Go source files to a bundle containing +both generated sources and ancillary resource files. Build, run, and test +workflows write the complete bundle before invoking Go. APIs or tests that need +only rendered Go may continue to expose a source-only view, but such a view is +not a complete buildable artifact for programs using embedding. + +## Language server and incremental analysis + +Embedded files and pattern match sets are semantic compiler inputs. LSP analysis +signatures include: + +- constructor path and pattern spelling; +- sorted matched logical paths; +- each selected file's content digest; and +- missing or invalid match state. + +The language server dynamically registers `workspace/didChangeWatchedFiles` for +exact files and every directory traversed by directory or pattern selection. +Directory registrations include create/delete events so a newly matching file +invalidates analysis. Clients that do not support dynamic watched-file +registration still receive correct results when another event triggers analysis, +but cannot receive immediate diagnostics for an otherwise invisible external +resource edit. Changes to unrelated files should retain cached analysis where +practical; conservative invalidation of a traversed directory is acceptable +initially. + +Each selected file is read once into the checked resource snapshot, and backend +generation uses only those captured bytes. Ordinary filesystem APIs cannot +provide an atomic snapshot of a changing directory tree, so the compiler does +not promise one; cancelled or superseded LSP analyses must not publish their +results. + +## Implementation sequence + +1. Register `ard/embed`, its static constructors, `embed::FS`, and supporting + value types in the checker. +2. Add package-owned path and pattern resolution with containment, symlink, + UTF-8, and resource-limit diagnostics. +3. Add checked resource nodes, AIR tables/types/operations, serialization, and + strict validation. +4. Introduce a generated artifact bundle and stage captured resources. +5. Generate the compiler-owned Go resource package and native `//go:embed` + directives. +6. Lower exact-file values and filesystem methods, including fresh byte copies + and preserved errors. +7. Add the built-in `io/fs.FS` interop bridge. +8. Include resource dependencies in CLI and LSP invalidation. +9. Document patterns, package ownership, limits, generated-artifact visibility, + and the embedded-secret warning. + +## Test plan + +Checker tests cover exact files, binary and UTF-8 content, pattern syntax, +directories, `all:`, overlapping matches, package-root resolution, locked and +path dependencies, traversal, symlinks, nested package boundaries, unreadable +or unsupported files, static-argument restrictions, limits, method signatures, +and the Go filesystem bridge. + +AIR tests cover deterministic resource tables, content capture, serialization, +deduplication, embedded filesystem typing, operation payloads, malformed IDs, +bad hashes, conflicting paths, and resource limits. + +Go backend tests cover generated directives and ancillary files, hidden-file +selection, prefix removal through `fs.Sub`, exact runtime contents, fresh byte +lists, directory ordering, stat and sub behavior, invalid and missing path +errors, `errors.Is` identity, runtime UTF-8 validation, direct use with an API +accepting `io/fs.FS`, deterministic output, and run/build/test parity. Backend +generation must still succeed from captured AIR after the original resource is +changed or removed. + +LSP tests cover edits, deletion, recreation, newly matching pattern entries, +dependency resource changes, cancellation, excluded and unrelated files, and +cache reuse when resource inputs are unchanged. + +## Deferred functionality + +This decision does not add: + +- runtime filesystem access outside the captured set; +- environment-, Git-, timestamp-, or build-tag-dependent selection; +- compression, MIME inference, or content transformation; +- writable embedded filesystems; +- preservation of permissions, ownership, or timestamps; +- empty-directory entries; +- custom pattern dialects such as recursive `**`; +- manifest-named resource bundles; +- configurable resource limits; or +- guaranteed Go optional-interface conformance beyond `io/fs.FS`. + +## Consequences + +- Pure Ard applications can package individual files and directory trees without + project-local Go FFI shims. +- Embedded resources are explicit, typed, deterministic build inputs. +- Dependencies retain ownership of their resources and cannot inspect consumer + files. +- The Go target interoperates naturally with `io/fs` consumers while Ard + semantics remain target-neutral. +- AIR and generated artifacts become larger because they contain captured file + contents. +- The backend must support ancillary files in addition to rendered Go source. +- LSP invalidation must track non-Ard files and directory membership. +- Reserving `ard/embed` permanently removes that path from the separately + versioned standard-library namespace. +- Embedded contents increase executable size and remain extractable from built + artifacts. + +## Related + +- [Go `embed` package](https://pkg.go.dev/embed) +- [Go `io/fs` package](https://pkg.go.dev/io/fs) +- `docs/language-philosophy.md` +- `docs/adrs/0002-use-air-as-backend-boundary.md` +- `docs/adrs/0013-use-file-based-modules-and-absolute-imports.md` +- `docs/adrs/0031-go-backend-lowering-contract.md` +- `docs/adrs/0049-overlay-ard-intrinsics-on-an-explicit-stdlib-package.md` +- `docs/adrs/0052-adopt-structured-labeled-diagnostics.md` +- `docs/adrs/0063-preserve-imported-go-error-identity.md` +- `docs/adrs/0069-expose-manifest-build-values-through-ard-build.md` From 07b6efb748eacbc2de17f26b1fa4e0b572ec8b55 Mon Sep 17 00:00:00 2001 From: Akonwi Ngoh Date: Tue, 8 Sep 2026 22:28:35 -0400 Subject: [PATCH 2/6] feat(embed): add exact-file embedding --- compiler/air/embed_test.go | 81 +++++++++ compiler/air/expr_payload_accessors.go | 5 + compiler/air/expr_payloads.go | 8 + compiler/air/lower.go | 62 +++++-- compiler/air/nodes.go | 2 + compiler/air/serialize.go | 1 + compiler/air/types.go | 26 ++- compiler/air/validate.go | 49 ++++++ compiler/checker/checker.go | 29 +++- compiler/checker/diagnostics.go | 3 + compiler/checker/embed.go | 217 ++++++++++++++++++++++++ compiler/checker/embed_internal_test.go | 77 +++++++++ compiler/checker/embed_test.go | 162 ++++++++++++++++++ compiler/checker/module_resolver.go | 22 ++- compiler/checker/nodes.go | 18 ++ compiler/checker/std_lib.go | 7 + compiler/go/backend.go | 47 ++++- compiler/go/embed_test.go | 153 +++++++++++++++++ compiler/go/lower.go | 12 ++ compiler/main_test.go | 1 + 20 files changed, 942 insertions(+), 40 deletions(-) create mode 100644 compiler/air/embed_test.go create mode 100644 compiler/checker/embed.go create mode 100644 compiler/checker/embed_internal_test.go create mode 100644 compiler/checker/embed_test.go create mode 100644 compiler/go/embed_test.go diff --git a/compiler/air/embed_test.go b/compiler/air/embed_test.go new file mode 100644 index 00000000..8c4445d8 --- /dev/null +++ b/compiler/air/embed_test.go @@ -0,0 +1,81 @@ +package air + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/akonwi/ard/checker" + "github.com/akonwi/ard/parse" +) + +func TestLowerEmbeddedExactFilesIntoBlobReferences(t *testing.T) { + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "ard.toml"), []byte("name = \"app\"\nard = \">= 0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + content := []byte("shared embedded contents\n") + if err := os.WriteFile(filepath.Join(root, "asset.txt"), content, 0o644); err != nil { + t.Fatal(err) + } + mainPath := filepath.Join(root, "main.ard") + parsed := parse.Parse([]byte("use ard/embed\nlet page = embed::text(\"asset.txt\")\nlet raw = embed::bytes(\"asset.txt\")\n"), mainPath) + if len(parsed.Errors) > 0 { + t.Fatalf("parse errors: %v", parsed.Errors) + } + resolver, err := checker.NewModuleResolver(root) + if err != nil { + t.Fatal(err) + } + checked := checker.New(mainPath, parsed.Program, resolver) + checked.Check() + if checked.HasErrors() { + t.Fatalf("checker diagnostics: %#v", checked.Diagnostics()) + } + if err := os.Remove(filepath.Join(root, "asset.txt")); err != nil { + t.Fatal(err) + } + + program, err := Lower(checked.Module()) + if err != nil { + t.Fatalf("Lower: %v", err) + } + if len(program.EmbeddedBlobs) != 1 { + t.Fatalf("embedded blobs = %d, want 1", len(program.EmbeddedBlobs)) + } + if string(program.EmbeddedBlobs[0].Data) != string(content) { + t.Fatalf("blob data = %q", program.EmbeddedBlobs[0].Data) + } + if len(program.Globals) != 2 { + t.Fatalf("globals = %d, want 2", len(program.Globals)) + } + text := program.Globals[0].Initializer.Value + bytes := program.Globals[1].Initializer.Value + if text.Kind != ExprEmbeddedText || bytes.Kind != ExprEmbeddedBytes { + t.Fatalf("embedded kinds = %d, %d", text.Kind, bytes.Kind) + } + if text.EmbeddedBlobPayload().Blob != 0 || bytes.EmbeddedBlobPayload().Blob != 0 { + t.Fatalf("embedded blob refs = %d, %d", text.EmbeddedBlobPayload().Blob, bytes.EmbeddedBlobPayload().Blob) + } + + encoded, err := SerializeProgram(program) + if err != nil { + t.Fatalf("SerializeProgram: %v", err) + } + roundTrip, err := DeserializeProgram(encoded) + if err != nil { + t.Fatalf("DeserializeProgram: %v", err) + } + if len(roundTrip.EmbeddedBlobs) != 1 || string(roundTrip.EmbeddedBlobs[0].Data) != string(content) { + t.Fatalf("round-trip embedded blobs = %#v", roundTrip.EmbeddedBlobs) + } + + malformed := *program + duplicate := program.EmbeddedBlobs[0] + duplicate.ID = 1 + malformed.EmbeddedBlobs = append(append([]EmbeddedBlob(nil), program.EmbeddedBlobs...), duplicate) + if err := Validate(&malformed); err == nil || !strings.Contains(err.Error(), "duplicates digest") { + t.Fatalf("duplicate embedded blob validation error = %v", err) + } +} diff --git a/compiler/air/expr_payload_accessors.go b/compiler/air/expr_payload_accessors.go index ca45dc7a..1fe1ffdf 100644 --- a/compiler/air/expr_payload_accessors.go +++ b/compiler/air/expr_payload_accessors.go @@ -5,6 +5,11 @@ func (e Expr) TextPayload() *TextExprPayload { return payload } +func (e Expr) EmbeddedBlobPayload() *EmbeddedBlobExprPayload { + payload, _ := e.Payload.(*EmbeddedBlobExprPayload) + return payload +} + func (e Expr) BoolPayload() *BoolExprPayload { payload, _ := e.Payload.(*BoolExprPayload) return payload diff --git a/compiler/air/expr_payloads.go b/compiler/air/expr_payloads.go index b896fe23..05d17bad 100644 --- a/compiler/air/expr_payloads.go +++ b/compiler/air/expr_payloads.go @@ -21,6 +21,12 @@ type TextExprPayload struct { func (*TextExprPayload) exprPayload() {} +type EmbeddedBlobExprPayload struct { + Blob EmbeddedBlobID +} + +func (*EmbeddedBlobExprPayload) exprPayload() {} + type BoolExprPayload struct { Value bool } @@ -242,6 +248,8 @@ func exprPayloadIsTypedNil(payload ExprPayload) bool { switch payload := payload.(type) { case *TextExprPayload: return payload == nil + case *EmbeddedBlobExprPayload: + return payload == nil case *BoolExprPayload: return payload == nil case *EnumExprPayload: diff --git a/compiler/air/lower.go b/compiler/air/lower.go index 88378125..2fc8ea98 100644 --- a/compiler/air/lower.go +++ b/compiler/air/lower.go @@ -1,6 +1,9 @@ package air import ( + "bytes" + "crypto/sha256" + "encoding/hex" "fmt" "sort" "strconv" @@ -58,13 +61,14 @@ func LowerModulesWithOptions(modules []checker.Module, options LowerOptions) (*P type lowerer struct { program Program - moduleByPath map[string]ModuleID - moduleByName map[string]checker.Module - typeInterner *typeInterner - traits map[string]TraitID - impls map[string]ImplID - functions map[string]FunctionID - globals map[string]GlobalID + moduleByPath map[string]ModuleID + moduleByName map[string]checker.Module + typeInterner *typeInterner + traits map[string]TraitID + impls map[string]ImplID + functions map[string]FunctionID + globals map[string]GlobalID + embeddedBlobs map[string]EmbeddedBlobID cacheMethodLookups bool structMethodsByOwner map[checker.MethodOwner]map[string]*checker.FunctionDef @@ -110,12 +114,13 @@ func newLowerer(options LowerOptions, rootCount int) *lowerer { Entry: NoFunction, Script: NoFunction, }, - moduleByPath: map[string]ModuleID{}, - moduleByName: map[string]checker.Module{}, - traits: map[string]TraitID{}, - impls: map[string]ImplID{}, - functions: map[string]FunctionID{}, - globals: map[string]GlobalID{}, + moduleByPath: map[string]ModuleID{}, + moduleByName: map[string]checker.Module{}, + traits: map[string]TraitID{}, + impls: map[string]ImplID{}, + functions: map[string]FunctionID{}, + globals: map[string]GlobalID{}, + embeddedBlobs: map[string]EmbeddedBlobID{}, cacheMethodLookups: rootCount == 1, unresolvedTypeVarByType: map[checker.Type]bool{}, @@ -172,6 +177,25 @@ func (l *lowerer) functionHasUnresolvedTypeVar(def *checker.FunctionDef) bool { return l.typeHasUnresolvedTypeVar(def) } +func (l *lowerer) internEmbeddedBlob(data []byte) (EmbeddedBlobID, error) { + sum := sha256.Sum256(data) + digest := hex.EncodeToString(sum[:]) + if id, ok := l.embeddedBlobs[digest]; ok { + if !bytes.Equal(l.program.EmbeddedBlobs[id].Data, data) { + return 0, fmt.Errorf("embedded resource digest collision for %s", digest) + } + return id, nil + } + id := EmbeddedBlobID(len(l.program.EmbeddedBlobs)) + l.program.EmbeddedBlobs = append(l.program.EmbeddedBlobs, EmbeddedBlob{ + ID: id, + Data: append([]byte(nil), data...), + Digest: digest, + }) + l.embeddedBlobs[digest] = id + return id, nil +} + func (l *lowerer) mustIntern(t checker.Type) TypeID { id, err := l.internType(t) if err != nil { @@ -3929,6 +3953,18 @@ func (fl *functionLowerer) lowerExpr(expr checker.Expression) (*Expr, error) { return &Expr{Kind: ExprConstBool, Type: typeID, Payload: &BoolExprPayload{Value: e.Value}}, nil case *checker.StrLiteral: return &Expr{Kind: ExprConstStr, Type: typeID, Payload: &TextExprPayload{Value: e.Value}}, nil + case *checker.EmbeddedText: + blob, err := fl.l.internEmbeddedBlob(e.Resource.Data) + if err != nil { + return nil, err + } + return &Expr{Kind: ExprEmbeddedText, Type: typeID, Payload: &EmbeddedBlobExprPayload{Blob: blob}}, nil + case *checker.EmbeddedBytes: + blob, err := fl.l.internEmbeddedBlob(e.Resource.Data) + if err != nil { + return nil, err + } + return &Expr{Kind: ExprEmbeddedBytes, Type: typeID, Payload: &EmbeddedBlobExprPayload{Blob: blob}}, nil case *checker.RuneLiteral: return &Expr{Kind: ExprConstInt, Type: typeID, Payload: &TextExprPayload{Value: strconv.Itoa(int(e.Value))}}, nil case *checker.NeverCoercion: diff --git a/compiler/air/nodes.go b/compiler/air/nodes.go index 24e53335..e06586bc 100644 --- a/compiler/air/nodes.go +++ b/compiler/air/nodes.go @@ -62,6 +62,8 @@ const ( ExprConstFloat ExprConstBool ExprConstStr + ExprEmbeddedText + ExprEmbeddedBytes ExprPanic ExprLoadLocal ExprLoadGlobal diff --git a/compiler/air/serialize.go b/compiler/air/serialize.go index f7952774..d545faef 100644 --- a/compiler/air/serialize.go +++ b/compiler/air/serialize.go @@ -12,6 +12,7 @@ import ( func init() { gob.Register(&TextExprPayload{}) + gob.Register(&EmbeddedBlobExprPayload{}) gob.Register(&BoolExprPayload{}) gob.Register(&EnumExprPayload{}) gob.Register(&LocalExprPayload{}) diff --git a/compiler/air/types.go b/compiler/air/types.go index e0b41b69..47dc0196 100644 --- a/compiler/air/types.go +++ b/compiler/air/types.go @@ -7,6 +7,7 @@ type GlobalID int type LocalID int type TraitID int type ImplID int +type EmbeddedBlobID int const ( NoType TypeID = 0 @@ -14,16 +15,23 @@ const ( NoGlobal GlobalID = -1 ) +type EmbeddedBlob struct { + ID EmbeddedBlobID + Data []byte + Digest string +} + type Program struct { - Modules []Module - Types []TypeInfo - Traits []Trait - Impls []Impl - Globals []Global - Tests []Test - Functions []Function - Entry FunctionID - Script FunctionID + Modules []Module + EmbeddedBlobs []EmbeddedBlob + Types []TypeInfo + Traits []Trait + Impls []Impl + Globals []Global + Tests []Test + Functions []Function + Entry FunctionID + Script FunctionID } type Module struct { diff --git a/compiler/air/validate.go b/compiler/air/validate.go index 63cc05d7..91cde96e 100644 --- a/compiler/air/validate.go +++ b/compiler/air/validate.go @@ -1,6 +1,8 @@ package air import ( + "crypto/sha256" + "encoding/hex" "fmt" "reflect" @@ -49,6 +51,32 @@ func Validate(program *Program) error { if err := validateCanonicalNominalIdentities(program); err != nil { return err } + if len(program.EmbeddedBlobs) > checker.MaxEmbeddedProgramFileCount { + return fmt.Errorf("embedded blobs exceed program limit of %d files", checker.MaxEmbeddedProgramFileCount) + } + embeddedBytes := 0 + embeddedDigests := make(map[string]bool, len(program.EmbeddedBlobs)) + for i, blob := range program.EmbeddedBlobs { + if blob.ID != EmbeddedBlobID(i) { + return fmt.Errorf("embedded blob table entry %d has id %d", i, blob.ID) + } + if len(blob.Data) > checker.MaxEmbeddedFileBytes { + return fmt.Errorf("embedded blob %d exceeds file limit of %d bytes", blob.ID, checker.MaxEmbeddedFileBytes) + } + embeddedBytes += len(blob.Data) + if embeddedBytes > checker.MaxEmbeddedProgramBytes { + return fmt.Errorf("embedded blobs exceed program limit of %d bytes", checker.MaxEmbeddedProgramBytes) + } + sum := sha256.Sum256(blob.Data) + digest := hex.EncodeToString(sum[:]) + if blob.Digest != digest { + return fmt.Errorf("embedded blob %d has digest %q, want %q", blob.ID, blob.Digest, digest) + } + if embeddedDigests[digest] { + return fmt.Errorf("embedded blob %d duplicates digest %q", blob.ID, digest) + } + embeddedDigests[digest] = true + } for i, trait := range program.Traits { if trait.ID != TraitID(i) { return fmt.Errorf("trait table entry %d has id %d", i, trait.ID) @@ -930,6 +958,8 @@ func validateExprPayload(expr Expr) error { compatible = true case *TextExprPayload: compatible = expr.Kind == ExprConstInt || expr.Kind == ExprConstFloat || expr.Kind == ExprConstStr + case *EmbeddedBlobExprPayload: + compatible = expr.Kind == ExprEmbeddedText || expr.Kind == ExprEmbeddedBytes case *BoolExprPayload: compatible = expr.Kind == ExprConstBool case *EnumExprPayload: @@ -1033,6 +1063,7 @@ func exprPayloadRequired(kind ExprKind) bool { } switch kind { case ExprConstInt, ExprConstFloat, ExprConstBool, ExprConstStr, + ExprEmbeddedText, ExprEmbeddedBytes, ExprLoadLocal, ExprLoadGlobal, ExprFunctionRef, ExprCall, ExprForeignCall, ExprForeignMethodCall, ExprForeignMethodValue, ExprForeignFieldAccess, ExprForeignStructInstance, ExprForeignValue, @@ -1055,6 +1086,24 @@ func validateExpr(program *Program, fn Function, expr Expr) error { if !validTypeID(program, expr.Type) { return fmt.Errorf("expression has invalid type %d", expr.Type) } + if expr.Kind == ExprEmbeddedText || expr.Kind == ExprEmbeddedBytes { + payload := exprPayloadAs[*EmbeddedBlobExprPayload](&expr) + if payload == nil { + return fmt.Errorf("embedded expression is missing its blob payload") + } + if payload.Blob < 0 || int(payload.Blob) >= len(program.EmbeddedBlobs) { + return fmt.Errorf("embedded expression references invalid blob %d", payload.Blob) + } + typ := program.Types[expr.Type-1] + if expr.Kind == ExprEmbeddedText && typ.Kind != TypeStr { + return fmt.Errorf("embedded text expression has non-Str type %d", expr.Type) + } + if expr.Kind == ExprEmbeddedBytes { + if typ.Kind != TypeList || !validTypeID(program, typ.Elem) || program.Types[typ.Elem-1].Kind != TypeByte { + return fmt.Errorf("embedded bytes expression has non-[Byte] type %d", expr.Type) + } + } + } if err := validateTailSpread(program, expr); err != nil { return err } diff --git a/compiler/checker/checker.go b/compiler/checker/checker.go index 1bd67f51..a5d2d40a 100644 --- a/compiler/checker/checker.go +++ b/compiler/checker/checker.go @@ -9593,15 +9593,22 @@ func (c *Checker) checkExprInner(expr parse.Expression, expectedReturn Type) Exp // find the function in a module or Go package namespace modName, name := c.destructurePath(s) - if mod := c.resolveModule(modName); mod != nil && mod.Path() == "ard/unsafe" { - if c.rejectSpreadForFixedCall(s.Function.Args) { - return nil - } - switch name { - case "cast": - return c.checkUnsafeCast(s) - case "is_nil": - return c.checkUnsafeIsNil(s) + if mod := c.resolveModule(modName); mod != nil { + switch mod.Path() { + case EmbedModulePath: + if name == "text" || name == "bytes" { + return c.checkEmbedExactCall(s, name) + } + case "ard/unsafe": + if c.rejectSpreadForFixedCall(s.Function.Args) { + return nil + } + switch name { + case "cast": + return c.checkUnsafeCast(s) + case "is_nil": + return c.checkUnsafeIsNil(s) + } } } if goPkg := c.program.GoImports[modName]; goPkg != nil { @@ -10967,6 +10974,10 @@ func (c *Checker) checkExprInner(expr parse.Expression, expectedReturn Type) Exp // Check if this is accessing a module if mod := c.resolveModule(id.Name); mod != nil { + if prop, ok := s.Property.(*parse.Identifier); ok && mod.Path() == EmbedModulePath && (prop.Name == "text" || prop.Name == "bytes") { + c.addEmbedDiagnostic(DiagnosticCodeEmbedStaticArgument, "Embed constructors are not function values", "call the constructor directly with a static path", prop.GetLocation()) + return nil + } switch prop := s.Property.(type) { case *parse.StructInstance: typeArgs, ok := c.resolveStructTypeArgs(prop) diff --git a/compiler/checker/diagnostics.go b/compiler/checker/diagnostics.go index 725622a1..0681cd52 100644 --- a/compiler/checker/diagnostics.go +++ b/compiler/checker/diagnostics.go @@ -43,6 +43,9 @@ const ( DiagnosticCodeImmutableAssignment DiagnosticCode = "immutable_assignment" DiagnosticCodeIncorrectArgumentType DiagnosticCode = "incorrect_argument_type" DiagnosticCodeGoImportResolution DiagnosticCode = "go_import_resolution" + DiagnosticCodeEmbedStaticArgument DiagnosticCode = "embed_static_argument" + DiagnosticCodeEmbedResource DiagnosticCode = "embed_resource" + DiagnosticCodeEmbedTextUTF8 DiagnosticCode = "embed_text_utf8" DiagnosticCodeImportResolution DiagnosticCode = "import_resolution" DiagnosticCodeCircularImport DiagnosticCode = "circular_import" DiagnosticCodeModuleLoadFailure DiagnosticCode = "module_load_failure" diff --git a/compiler/checker/embed.go b/compiler/checker/embed.go new file mode 100644 index 00000000..486f7c4d --- /dev/null +++ b/compiler/checker/embed.go @@ -0,0 +1,217 @@ +package checker + +import ( + "fmt" + "io" + "os" + "path/filepath" + "strings" + "unicode/utf8" + + "github.com/akonwi/ard/parse" + "golang.org/x/mod/module" +) + +const ( + EmbedModulePath = "ard/embed" + MaxEmbeddedFileBytes = 16 * 1024 * 1024 + MaxEmbeddedProgramBytes = 128 * 1024 * 1024 + MaxEmbeddedProgramFileCount = 10_000 +) + +type EmbedPkg struct{} + +func (EmbedPkg) Path() string { return EmbedModulePath } +func (EmbedPkg) Program() *Program { return nil } + +func (EmbedPkg) Get(name string) Symbol { + var returnType Type + switch name { + case "text": + returnType = Str + case "bytes": + returnType = MakeList(Byte) + default: + return Symbol{} + } + return Symbol{Name: name, Type: &FunctionDef{ + Name: name, + Parameters: []Parameter{{Name: "path", Type: Str}}, + ReturnType: returnType, + }} +} + +func (c *Checker) checkEmbedExactCall(s *parse.StaticFunction, name string) Expression { + qualified := "embed::" + name + if name != "text" && name != "bytes" { + return nil + } + if len(s.Function.TypeArgs) != 0 { + c.addInvalidFunctionTypeArguments(qualified, 0, len(s.Function.TypeArgs), false, s.GetLocation(), qualified+" does not accept type arguments") + return nil + } + if c.rejectSpreadForFixedCall(s.Function.Args) { + return nil + } + if len(s.Function.Args) != 1 { + c.addArgumentCount("1", len(s.Function.Args), s.GetLocation(), "") + return nil + } + arg := s.Function.Args[0] + if arg.Name != "" { + c.addNamedArgumentsUnsupported("embed constructor", arg.GetLocation()) + return nil + } + literal, ok := arg.Value.(*parse.StrLiteral) + if !ok { + c.addEmbedDiagnostic(DiagnosticCodeEmbedStaticArgument, "Embedded resource path must be a static string", "use a non-interpolated string literal selected during checking", arg.Value.GetLocation()) + return nil + } + if c.moduleResolver == nil { + c.addEmbedDiagnostic(DiagnosticCodeEmbedResource, "Cannot resolve embedded resource", "embedding requires project package context", literal.GetLocation()) + return nil + } + resource, err := c.moduleResolver.resolveEmbeddedExactFile(c.modulePath, literal.Value) + if err != nil { + c.addEmbedDiagnostic(DiagnosticCodeEmbedResource, fmt.Sprintf("Cannot embed %q", literal.Value), err.Error(), literal.GetLocation()) + return nil + } + if name == "text" { + if !utf8.Valid(resource.Data) { + c.addEmbedDiagnostic(DiagnosticCodeEmbedTextUTF8, fmt.Sprintf("Cannot embed %q as text", literal.Value), "the file is not valid UTF-8; use embed::bytes instead", literal.GetLocation()) + return nil + } + return &EmbeddedText{Resource: resource} + } + return &EmbeddedBytes{Resource: resource} +} + +func (c *Checker) addEmbedDiagnostic(code DiagnosticCode, title string, text string, location parse.Location) { + diagnostic := newLabeledDiagnostic( + Error, + title+": "+text, + title, + text, + DiagnosticLabel{Span: c.sourceSpan(location), Message: title}, + ) + diagnostic.Code = code + c.addDiagnostic(diagnostic) +} + +func (mr *ModuleResolver) resolveEmbeddedExactFile(importerModulePath string, logicalPath string) (EmbeddedResource, error) { + if err := validateEmbeddedExactPath(logicalPath); err != nil { + return EmbeddedResource{}, err + } + packageID := mr.packageIDForModule(importerModulePath) + pkg := mr.packageInfo(packageID) + if pkg.RootPath == "" { + return EmbeddedResource{}, fmt.Errorf("owning Ard package has no source root") + } + + cacheKey := packageID + "\x00" + logicalPath + mr.embedMu.Lock() + defer mr.embedMu.Unlock() + if resource, ok := mr.embeddedResources[cacheKey]; ok { + return resource, nil + } + + root, err := os.OpenRoot(pkg.RootPath) + if err != nil { + return EmbeddedResource{}, fmt.Errorf("open owning Ard package root: %w", err) + } + defer root.Close() + + components := strings.Split(logicalPath, "/") + current := "" + for index, component := range components { + current = filepath.Join(current, component) + info, err := root.Lstat(current) + if err != nil { + return EmbeddedResource{}, fmt.Errorf("inspect embedded resource: %w", err) + } + if info.Mode()&os.ModeSymlink != 0 { + return EmbeddedResource{}, fmt.Errorf("embedded resource path must not contain a symlink: %s", logicalPath) + } + if index < len(components)-1 { + if !info.IsDir() { + return EmbeddedResource{}, fmt.Errorf("embedded resource path component is not a directory: %s", strings.Join(components[:index+1], "/")) + } + switch component { + case ".bzr", ".git", ".hg", ".svn", "vendor": + return EmbeddedResource{}, fmt.Errorf("embedded resources must not select reserved directory %q", component) + } + if _, err := root.Stat(filepath.Join(current, "ard.toml")); err == nil { + return EmbeddedResource{}, fmt.Errorf("embedded resource crosses a nested Ard package boundary: %s", strings.Join(components[:index+1], "/")) + } else if !os.IsNotExist(err) { + return EmbeddedResource{}, fmt.Errorf("inspect nested Ard package boundary: %w", err) + } + if _, err := root.Stat(filepath.Join(current, "go.mod")); err == nil { + return EmbeddedResource{}, fmt.Errorf("embedded resource crosses a nested Go module boundary: %s", strings.Join(components[:index+1], "/")) + } else if !os.IsNotExist(err) { + return EmbeddedResource{}, fmt.Errorf("inspect nested Go module boundary: %w", err) + } + continue + } + if !info.Mode().IsRegular() { + return EmbeddedResource{}, fmt.Errorf("embedded resource must be a regular file: %s", logicalPath) + } + } + + file, err := root.Open(filepath.FromSlash(logicalPath)) + if err != nil { + return EmbeddedResource{}, fmt.Errorf("open embedded resource: %w", err) + } + info, statErr := file.Stat() + if statErr != nil { + _ = file.Close() + return EmbeddedResource{}, fmt.Errorf("inspect opened embedded resource: %w", statErr) + } + if !info.Mode().IsRegular() { + _ = file.Close() + return EmbeddedResource{}, fmt.Errorf("embedded resource must be a regular file: %s", logicalPath) + } + if info.Size() > MaxEmbeddedFileBytes { + _ = file.Close() + return EmbeddedResource{}, fmt.Errorf("embedded resource is %d bytes; maximum file size is %d bytes", info.Size(), MaxEmbeddedFileBytes) + } + data, readErr := io.ReadAll(io.LimitReader(file, MaxEmbeddedFileBytes+1)) + closeErr := file.Close() + if readErr != nil { + return EmbeddedResource{}, fmt.Errorf("read embedded resource: %w", readErr) + } + if closeErr != nil { + return EmbeddedResource{}, fmt.Errorf("close embedded resource: %w", closeErr) + } + if len(data) > MaxEmbeddedFileBytes { + return EmbeddedResource{}, fmt.Errorf("embedded resource exceeds maximum file size of %d bytes", MaxEmbeddedFileBytes) + } + if len(mr.embeddedResources)+1 > MaxEmbeddedProgramFileCount { + return EmbeddedResource{}, fmt.Errorf("embedded resources exceed program limit of %d files", MaxEmbeddedProgramFileCount) + } + if mr.embeddedResourceBytes+len(data) > MaxEmbeddedProgramBytes { + return EmbeddedResource{}, fmt.Errorf("embedded resources exceed program limit of %d bytes", MaxEmbeddedProgramBytes) + } + + resource := EmbeddedResource{OwnerPackageIdentity: packageID, LogicalPath: logicalPath, Data: data} + mr.embeddedResources[cacheKey] = resource + mr.embeddedResourceBytes += len(data) + return resource, nil +} + +func validateEmbeddedExactPath(path string) error { + if path == "" { + return fmt.Errorf("embedded resource path must not be empty") + } + if strings.Contains(path, `\`) { + return fmt.Errorf("embedded resource paths use forward slashes") + } + if err := module.CheckFilePath(path); err != nil { + return fmt.Errorf("invalid embedded resource path: %w", err) + } + for _, component := range strings.Split(path, "/") { + if component == "go.mod" { + return fmt.Errorf("embedded resources must not select a file named go.mod") + } + } + return nil +} diff --git a/compiler/checker/embed_internal_test.go b/compiler/checker/embed_internal_test.go new file mode 100644 index 00000000..3532efc4 --- /dev/null +++ b/compiler/checker/embed_internal_test.go @@ -0,0 +1,77 @@ +package checker + +import ( + "os" + "path/filepath" + "testing" +) + +func TestEmbeddedExactFileSnapshotIsCachedPerResolver(t *testing.T) { + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "ard.toml"), []byte("name = \"app\"\nard = \">= 0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + path := filepath.Join(root, "asset.txt") + if err := os.WriteFile(path, []byte("first"), 0o644); err != nil { + t.Fatal(err) + } + resolver, err := NewModuleResolver(root) + if err != nil { + t.Fatal(err) + } + first, err := resolver.resolveEmbeddedExactFile("app/main", "asset.txt") + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte("second"), 0o644); err != nil { + t.Fatal(err) + } + second, err := resolver.resolveEmbeddedExactFile("app/main", "asset.txt") + if err != nil { + t.Fatal(err) + } + if string(first.Data) != "first" || string(second.Data) != "first" { + t.Fatalf("cached snapshots = %q, %q", first.Data, second.Data) + } +} + +func TestEmbeddedExactFileEnforcesProgramFileLimit(t *testing.T) { + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "ard.toml"), []byte("name = \"app\"\nard = \">= 0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "asset.txt"), []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + resolver, err := NewModuleResolver(root) + if err != nil { + t.Fatal(err) + } + for i := 0; i < MaxEmbeddedProgramFileCount; i++ { + resolver.embeddedResources[string(rune(i))+"\x00"] = EmbeddedResource{} + } + if _, err := resolver.resolveEmbeddedExactFile("app/main", "asset.txt"); err == nil { + t.Fatal("expected program file limit error") + } +} + +func TestEmbeddedExactFileAllowsReservedNameWhenItIsAFile(t *testing.T) { + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "ard.toml"), []byte("name = \"app\"\nard = \">= 0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "vendor"), []byte("file"), 0o644); err != nil { + t.Fatal(err) + } + resolver, err := NewModuleResolver(root) + if err != nil { + t.Fatal(err) + } + resource, err := resolver.resolveEmbeddedExactFile("app/main", "vendor") + if err != nil { + t.Fatal(err) + } + if string(resource.Data) != "file" { + t.Fatalf("resource data = %q", resource.Data) + } +} diff --git a/compiler/checker/embed_test.go b/compiler/checker/embed_test.go new file mode 100644 index 00000000..624fd69f --- /dev/null +++ b/compiler/checker/embed_test.go @@ -0,0 +1,162 @@ +package checker_test + +import ( + "os" + "path/filepath" + "testing" + + "github.com/akonwi/ard/checker" + "github.com/akonwi/ard/parse" +) + +func checkEmbedSource(t *testing.T, root string, source string) *checker.Checker { + t.Helper() + mainPath := filepath.Join(root, "main.ard") + result := parse.Parse([]byte(source), mainPath) + if len(result.Errors) > 0 { + t.Fatalf("parse errors: %v", result.Errors) + } + resolver, err := checker.NewModuleResolver(root) + if err != nil { + t.Fatal(err) + } + checked := checker.New(mainPath, result.Program, resolver) + checked.Check() + return checked +} + +func hasEmbedDiagnostic(checked *checker.Checker, code checker.DiagnosticCode) bool { + for _, diagnostic := range checked.Diagnostics() { + if diagnostic.Code == code { + return true + } + } + return false +} + +func TestEmbedExactFilesResolveFromOwningPackageRoot(t *testing.T) { + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "ard.toml"), []byte("name = \"app\"\nard = \">= 0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + if err := os.MkdirAll(filepath.Join(root, "assets"), 0o755); err != nil { + t.Fatal(err) + } + content := []byte("hello from an embedded file\n") + if err := os.WriteFile(filepath.Join(root, "assets", "page.txt"), content, 0o644); err != nil { + t.Fatal(err) + } + if err := os.MkdirAll(filepath.Join(root, "ui"), 0o755); err != nil { + t.Fatal(err) + } + + mainPath := filepath.Join(root, "ui", "page.ard") + source := "use ard/embed\nlet page = embed::text(\"assets/page.txt\")\nlet raw = embed::bytes(\"assets/page.txt\")\n" + result := parse.Parse([]byte(source), mainPath) + if len(result.Errors) > 0 { + t.Fatalf("parse errors: %v", result.Errors) + } + resolver, err := checker.NewModuleResolver(root) + if err != nil { + t.Fatal(err) + } + checked := checker.New(mainPath, result.Program, resolver) + checked.Check() + if checked.HasErrors() { + t.Fatalf("checker diagnostics: %#v", checked.Diagnostics()) + } + + statements := checked.Module().Program().Statements + if len(statements) != 2 { + t.Fatalf("statements = %d, want 2", len(statements)) + } + page, ok := statements[0].Stmt.(*checker.VariableDef) + if !ok { + t.Fatalf("page statement = %T", statements[0].Stmt) + } + text, ok := page.Value.(*checker.EmbeddedText) + if !ok { + t.Fatalf("page value = %T", page.Value) + } + if text.Resource.OwnerPackageIdentity != "root" || text.Resource.LogicalPath != "assets/page.txt" || string(text.Resource.Data) != string(content) { + t.Fatalf("embedded text resource = %#v", text.Resource) + } + if text.Type() != checker.Str { + t.Fatalf("embedded text type = %s", text.Type()) + } + + raw, ok := statements[1].Stmt.(*checker.VariableDef) + if !ok { + t.Fatalf("raw statement = %T", statements[1].Stmt) + } + bytes, ok := raw.Value.(*checker.EmbeddedBytes) + if !ok { + t.Fatalf("raw value = %T", raw.Value) + } + if bytes.Resource.OwnerPackageIdentity != "root" || bytes.Resource.LogicalPath != "assets/page.txt" || string(bytes.Resource.Data) != string(content) { + t.Fatalf("embedded bytes resource = %#v", bytes.Resource) + } + list, ok := bytes.Type().(*checker.List) + if !ok || list.Of() != checker.Byte { + t.Fatalf("embedded bytes type = %s", bytes.Type()) + } +} + +func TestEmbedTextRejectsInvalidUTF8(t *testing.T) { + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "ard.toml"), []byte("name = \"app\"\nard = \">= 0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "invalid.bin"), []byte{0xff}, 0o644); err != nil { + t.Fatal(err) + } + checked := checkEmbedSource(t, root, "use ard/embed\nlet value = embed::text(\"invalid.bin\")\n") + if !hasEmbedDiagnostic(checked, checker.DiagnosticCodeEmbedTextUTF8) { + t.Fatalf("missing UTF-8 diagnostic: %#v", checked.Diagnostics()) + } +} + +func TestEmbedExactFilesRequireStaticLiteralPaths(t *testing.T) { + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "ard.toml"), []byte("name = \"app\"\nard = \">= 0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + checked := checkEmbedSource(t, root, "use ard/embed\nlet path = \"asset.txt\"\nlet value = embed::bytes(path)\n") + if !hasEmbedDiagnostic(checked, checker.DiagnosticCodeEmbedStaticArgument) { + t.Fatalf("missing static argument diagnostic: %#v", checked.Diagnostics()) + } +} + +func TestEmbedExactFilesRejectMissingFilesAndSymlinks(t *testing.T) { + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "ard.toml"), []byte("name = \"app\"\nard = \">= 0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + checked := checkEmbedSource(t, root, "use ard/embed\nlet value = embed::bytes(\"missing.bin\")\n") + if !hasEmbedDiagnostic(checked, checker.DiagnosticCodeEmbedResource) { + t.Fatalf("missing resource diagnostic: %#v", checked.Diagnostics()) + } + + target := filepath.Join(root, "target.bin") + if err := os.WriteFile(target, []byte("target"), 0o644); err != nil { + t.Fatal(err) + } + if err := os.Symlink(target, filepath.Join(root, "link.bin")); err != nil { + t.Skipf("symlink unavailable: %v", err) + } + checked = checkEmbedSource(t, root, "use ard/embed\nlet value = embed::bytes(\"link.bin\")\n") + if !hasEmbedDiagnostic(checked, checker.DiagnosticCodeEmbedResource) { + t.Fatalf("missing symlink diagnostic: %#v", checked.Diagnostics()) + } +} + +func TestEmbedConstructorsCannotBeFunctionValues(t *testing.T) { + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "ard.toml"), []byte("name = \"app\"\nard = \">= 0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + checked := checkEmbedSource(t, root, "use ard/embed\nlet reader = embed::text\n") + if !hasEmbedDiagnostic(checked, checker.DiagnosticCodeEmbedStaticArgument) { + t.Fatalf("missing function-value diagnostic: %#v", checked.Diagnostics()) + } +} diff --git a/compiler/checker/module_resolver.go b/compiler/checker/module_resolver.go index cca27bd9..868790f8 100644 --- a/compiler/checker/module_resolver.go +++ b/compiler/checker/module_resolver.go @@ -15,6 +15,7 @@ import ( "regexp" "sort" "strings" + "sync" "slices" @@ -91,6 +92,10 @@ type ModuleResolver struct { modulePackages map[string]string // canonical module path -> package ID goImportAliases map[string]goImportAliasesResult buildModule Module + + embedMu sync.Mutex + embeddedResources map[string]EmbeddedResource + embeddedResourceBytes int } type goImportAliasesResult struct { @@ -1182,14 +1187,15 @@ func NewModuleResolverWithOptions(workingDir string, options BuildOptions) (*Mod } return &ModuleResolver{ - project: project, - moduleCache: make(map[string]Module), - astCache: make(map[string]*parse.Program), - overlays: make(map[string]string), - loadingChain: make([]string, 0), - modulePackages: make(map[string]string), - goImportAliases: make(map[string]goImportAliasesResult), - buildModule: newBuildModule(values), + project: project, + moduleCache: make(map[string]Module), + astCache: make(map[string]*parse.Program), + overlays: make(map[string]string), + loadingChain: make([]string, 0), + modulePackages: make(map[string]string), + goImportAliases: make(map[string]goImportAliasesResult), + buildModule: newBuildModule(values), + embeddedResources: make(map[string]EmbeddedResource), }, nil } diff --git a/compiler/checker/nodes.go b/compiler/checker/nodes.go index 902c7921..0da149b3 100644 --- a/compiler/checker/nodes.go +++ b/compiler/checker/nodes.go @@ -34,6 +34,24 @@ func (s *StrLiteral) Type() Type { return Str } +type EmbeddedResource struct { + OwnerPackageIdentity string + LogicalPath string + Data []byte +} + +type EmbeddedText struct { + Resource EmbeddedResource +} + +func (e *EmbeddedText) Type() Type { return Str } + +type EmbeddedBytes struct { + Resource EmbeddedResource +} + +func (e *EmbeddedBytes) Type() Type { return MakeList(Byte) } + type RuneLiteral struct { Value rune } diff --git a/compiler/checker/std_lib.go b/compiler/checker/std_lib.go index 5e846a42..69147246 100644 --- a/compiler/checker/std_lib.go +++ b/compiler/checker/std_lib.go @@ -19,6 +19,8 @@ func findInStdLib(path string) (Module, bool) { return AsyncPkg{}, true case "ard/unsafe": return UnsafePkg{}, true + case EmbedModulePath: + return EmbedPkg{}, true } return FindEmbeddedModule(path) @@ -193,6 +195,7 @@ var BuiltinPkgNames = map[string][]string{ "ard/result": {"ok", "err"}, "ard/async": {"start"}, "ard/unsafe": {"cast", "is_nil"}, + EmbedModulePath: {"text", "bytes"}, "builtin/Chan": {"new"}, } @@ -216,6 +219,10 @@ func (pkg UnsafePkg) Symbols() map[string]Symbol { return symbolsByName(pkg, BuiltinPkgNames[pkg.Path()]...) } +func (pkg EmbedPkg) Symbols() map[string]Symbol { + return symbolsByName(pkg, BuiltinPkgNames[pkg.Path()]...) +} + func (pkg EmptyBuiltinPkg) Symbols() map[string]Symbol { return map[string]Symbol{} } func (pkg ChannelStaticPkg) Symbols() map[string]Symbol { diff --git a/compiler/go/backend.go b/compiler/go/backend.go index 0d62e42d..0c484ec7 100644 --- a/compiler/go/backend.go +++ b/compiler/go/backend.go @@ -4,6 +4,7 @@ import ( "bytes" "encoding/json" "fmt" + "go/format" "go/token" goversion "go/version" "os" @@ -66,7 +67,7 @@ func GenerateSources(program *air.Program, options Options) (map[string][]byte, if err != nil { return nil, err } - out := make(map[string][]byte, len(generated)) + out := make(map[string][]byte, len(generated)+1) for name, file := range generated { source, err := renderFile(file) if err != nil { @@ -74,9 +75,32 @@ func GenerateSources(program *air.Program, options Options) (map[string][]byte, } out[name] = source } + if len(program.EmbeddedBlobs) > 0 { + source, err := generateEmbeddedResourceSource(program) + if err != nil { + return nil, err + } + out["internal/ardembed/embed.go"] = source + } return out, nil } +func generateEmbeddedResourceSource(program *air.Program) ([]byte, error) { + var source strings.Builder + source.WriteString("package ardembed\n\nimport _ \"embed\"\n\n") + for _, blob := range program.EmbeddedBlobs { + fmt.Fprintf(&source, "//go:embed data/%s\n", blob.Digest) + fmt.Fprintf(&source, "var blob%s string\n\n", blob.Digest) + fmt.Fprintf(&source, "func Text%s() string { return blob%s }\n\n", blob.Digest, blob.Digest) + fmt.Fprintf(&source, "func Bytes%s() []byte { return []byte(blob%s) }\n\n", blob.Digest, blob.Digest) + } + formatted, err := format.Source([]byte(source.String())) + if err != nil { + return nil, fmt.Errorf("format embedded resource package: %w", err) + } + return formatted, nil +} + func RunProgram(program *air.Program, args []string, projectInfo ...*checker.ProjectInfo) error { info := optionalProjectInfo(projectInfo) pathHint := artifactPathHint(info, inputPathFromCLIArgs(args)) @@ -407,6 +431,11 @@ func writeProgramWithStageObserver(dir string, program *air.Program, options Opt }); err != nil { return err } + if err := observeStage(observer, "go.write_embedded_resources", func() error { + return writeEmbeddedResources(dir, program) + }); err != nil { + return err + } if err := observeStage(observer, "go.copy_ffi", func() error { return copyProjectFFIDir(dir, options.ProjectInfo) }); err != nil { @@ -1179,6 +1208,22 @@ func dependencyPackageForModulePath(modulePath string, projectInfo *checker.Proj return "", "", false } +func writeEmbeddedResources(outputDir string, program *air.Program) error { + if program == nil { + return nil + } + dataDir := filepath.Join(outputDir, "internal", "ardembed", "data") + for _, blob := range program.EmbeddedBlobs { + if err := os.MkdirAll(dataDir, 0o755); err != nil { + return fmt.Errorf("create embedded resource directory: %w", err) + } + if err := os.WriteFile(filepath.Join(dataDir, blob.Digest), blob.Data, 0o644); err != nil { + return fmt.Errorf("write embedded resource %s: %w", blob.Digest, err) + } + } + return nil +} + func copyProjectFFIDir(outputDir string, projectInfo *checker.ProjectInfo) error { if projectInfo == nil || strings.TrimSpace(projectInfo.RootPath) == "" { return nil diff --git a/compiler/go/embed_test.go b/compiler/go/embed_test.go new file mode 100644 index 00000000..01cff09e --- /dev/null +++ b/compiler/go/embed_test.go @@ -0,0 +1,153 @@ +package gotarget + +import ( + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + + "github.com/akonwi/ard/air" + "github.com/akonwi/ard/checker" + "github.com/akonwi/ard/parse" +) + +func embeddedTestProgram(t *testing.T) (*air.Program, *checker.ProjectInfo, []byte) { + t.Helper() + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "ard.toml"), []byte("name = \"app\"\nard = \">= 0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + content := []byte{'h', 'e', 'l', 'l', 'o', 0, 0xff} + if err := os.WriteFile(filepath.Join(root, "asset.bin"), content, 0o644); err != nil { + t.Fatal(err) + } + mainPath := filepath.Join(root, "main.ard") + parsed := parse.Parse([]byte("use ard/embed\nlet raw = embed::bytes(\"asset.bin\")\nfn main() Int { raw.size() }\n"), mainPath) + if len(parsed.Errors) > 0 { + t.Fatalf("parse errors: %v", parsed.Errors) + } + resolver, err := checker.NewModuleResolver(root) + if err != nil { + t.Fatal(err) + } + checked := checker.New(mainPath, parsed.Program, resolver) + checked.Check() + if checked.HasErrors() { + t.Fatalf("checker diagnostics: %#v", checked.Diagnostics()) + } + program, err := air.Lower(checked.Module()) + if err != nil { + t.Fatalf("Lower: %v", err) + } + return program, resolver.GetProjectInfo(), content +} + +func TestGenerateSourcesUsesEmbeddedResourceAccessors(t *testing.T) { + program, project, _ := embeddedTestProgram(t) + sources, err := GenerateSources(program, Options{PackageName: "main", ProjectInfo: project}) + if err != nil { + t.Fatalf("GenerateSources: %v", err) + } + resourceSource := string(sources["internal/ardembed/embed.go"]) + if !strings.Contains(resourceSource, "//go:embed data/") || !strings.Contains(resourceSource, "func Bytes") { + t.Fatalf("generated resource source:\n%s", resourceSource) + } + foundAccessor := false + for name, source := range sources { + if name == "internal/ardembed/embed.go" { + continue + } + if strings.Contains(string(source), "ardembed.Bytes") { + foundAccessor = true + break + } + } + if !foundAccessor { + t.Fatalf("generated sources do not call embedded bytes accessor: %#v", sources) + } +} + +func TestEmbeddedExactFilesRunFromCapturedBytesAndReturnFreshLists(t *testing.T) { + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "ard.toml"), []byte("name = \"app\"\nard = \">= 0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + binaryContent := []byte{'h', 'e', 'l', 'l', 'o', 0, 0xff} + binaryPath := filepath.Join(root, "asset.bin") + textPath := filepath.Join(root, "page.txt") + if err := os.WriteFile(binaryPath, binaryContent, 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(textPath, []byte("hello\n"), 0o644); err != nil { + t.Fatal(err) + } + mainPath := filepath.Join(root, "main.ard") + source := `use ard/embed + +fn embedded_bytes() [Byte] { embed::bytes("asset.bin") } + +fn main() { + if embed::text("page.txt") != "hello\n" { panic("bad embedded text") } + let first = mut embedded_bytes() + first.set(0, Byte::from(0)) + let second = embedded_bytes() + if second.at(0).or(Byte::from(0)) != 104 { panic("embedded bytes shared mutable storage") } + if second.size() != 7 { panic("bad embedded byte length") } +} +` + parsed := parse.Parse([]byte(source), mainPath) + if len(parsed.Errors) > 0 { + t.Fatalf("parse errors: %v", parsed.Errors) + } + resolver, err := checker.NewModuleResolver(root) + if err != nil { + t.Fatal(err) + } + checked := checker.New(mainPath, parsed.Program, resolver) + checked.Check() + if checked.HasErrors() { + t.Fatalf("checker diagnostics: %#v", checked.Diagnostics()) + } + program, err := air.Lower(checked.Module()) + if err != nil { + t.Fatalf("Lower: %v", err) + } + if err := os.WriteFile(binaryPath, []byte("changed"), 0o644); err != nil { + t.Fatal(err) + } + if err := os.Remove(textPath); err != nil { + t.Fatal(err) + } + + output := filepath.Join(t.TempDir(), "embedded-program") + built, err := BuildProgram(program, output, resolver.GetProjectInfo()) + if err != nil { + t.Fatalf("BuildProgram: %v", err) + } + if output, err := exec.Command(built).CombinedOutput(); err != nil { + t.Fatalf("embedded program failed: %v\n%s", err, output) + } +} + +func TestWriteProgramStagesEmbeddedBlobs(t *testing.T) { + program, project, content := embeddedTestProgram(t) + out := t.TempDir() + if err := writeProgram(out, program, Options{PackageName: "main", ProjectInfo: project}); err != nil { + t.Fatalf("writeProgram: %v", err) + } + if len(program.EmbeddedBlobs) != 1 { + t.Fatalf("embedded blobs = %d", len(program.EmbeddedBlobs)) + } + path := filepath.Join(out, "internal", "ardembed", "data", program.EmbeddedBlobs[0].Digest) + got, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read staged blob: %v", err) + } + if string(got) != string(content) { + t.Fatalf("staged blob = %v, want %v", got, content) + } + if err := buildGeneratedProgram(out, filepath.Join(out, "embedded-test")); err != nil { + t.Fatalf("build generated embedded program: %v", err) + } +} diff --git a/compiler/go/lower.go b/compiler/go/lower.go index ba437516..289118ce 100644 --- a/compiler/go/lower.go +++ b/compiler/go/lower.go @@ -2427,6 +2427,18 @@ func (l *lowerer) lowerExpr(fn air.Function, expr air.Expr) (loweredExpr, error) return loweredExpr{expr: &ast.BasicLit{Kind: token.STRING, Value: fmt.Sprintf("%q", payload.Value)}}, nil } return loweredExpr{}, fmt.Errorf("string constant is missing its payload") + case air.ExprEmbeddedText, air.ExprEmbeddedBytes: + payload := expr.EmbeddedBlobPayload() + if payload == nil || payload.Blob < 0 || int(payload.Blob) >= len(l.program.EmbeddedBlobs) { + return loweredExpr{}, fmt.Errorf("embedded expression references invalid blob") + } + blob := l.program.EmbeddedBlobs[payload.Blob] + prefix := "Text" + if expr.Kind == air.ExprEmbeddedBytes { + prefix = "Bytes" + } + accessor := l.qualified("ardembed", path.Join(l.generatedModulePath, "internal", "ardembed"), prefix+blob.Digest) + return loweredExpr{expr: &ast.CallExpr{Fun: accessor}}, nil case air.ExprPanic: if expr.Target == nil { return loweredExpr{}, fmt.Errorf("panic missing target") diff --git a/compiler/main_test.go b/compiler/main_test.go index 6b5b0c8c..2161c272 100644 --- a/compiler/main_test.go +++ b/compiler/main_test.go @@ -442,6 +442,7 @@ func assertTestPipelineProfile(t *testing.T, stderr string) { "go.prepare_workspace", "go.validate_lower_render", "go.write_sources", + "go.write_embedded_resources", "go.copy_ffi", "go.write_runtime", "go.write_module", From 018a60d5b3e5455ae03983027f345f5debe0b7eb Mon Sep 17 00:00:00 2001 From: Akonwi Ngoh Date: Wed, 9 Sep 2026 00:17:19 -0400 Subject: [PATCH 3/6] feat(embed): add embedded filesystem support --- compiler/air/embed_test.go | 24 +- compiler/air/expr_payload_accessors.go | 5 + compiler/air/expr_payloads.go | 8 + compiler/air/lower.go | 72 +++- compiler/air/nodes.go | 6 + compiler/air/serialize.go | 1 + compiler/air/types.go | 16 + compiler/air/validate.go | 131 +++++- compiler/checker/checker.go | 13 +- compiler/checker/embed.go | 376 +++++++++++++++++- compiler/checker/embed_internal_test.go | 36 +- compiler/checker/embed_test.go | 131 +++++- compiler/checker/module_resolver.go | 31 +- compiler/checker/nodes.go | 35 ++ compiler/checker/std_lib.go | 2 +- compiler/go/backend.go | 67 +++- compiler/go/embed_test.go | 75 ++++ compiler/go/lower.go | 175 +++++++- compiler/lsp/analysis/engine.go | 88 +++- compiler/lsp/analysis/engine_test.go | 41 ++ compiler/lsp/server.go | 54 ++- compiler/lsp/server_test.go | 1 + ...1-add-compile-time-embedded-filesystems.md | 16 +- website/astro.config.mjs | 1 + website/src/content/docs/guide/embedding.md | 82 ++++ 25 files changed, 1422 insertions(+), 65 deletions(-) create mode 100644 website/src/content/docs/guide/embedding.md diff --git a/compiler/air/embed_test.go b/compiler/air/embed_test.go index 8c4445d8..e75f3b1f 100644 --- a/compiler/air/embed_test.go +++ b/compiler/air/embed_test.go @@ -20,7 +20,7 @@ func TestLowerEmbeddedExactFilesIntoBlobReferences(t *testing.T) { t.Fatal(err) } mainPath := filepath.Join(root, "main.ard") - parsed := parse.Parse([]byte("use ard/embed\nlet page = embed::text(\"asset.txt\")\nlet raw = embed::bytes(\"asset.txt\")\n"), mainPath) + parsed := parse.Parse([]byte("use ard/embed\nlet page = embed::text(\"asset.txt\")\nlet raw = embed::bytes(\"asset.txt\")\nlet files = embed::fs([\"asset.txt\"])\n"), mainPath) if len(parsed.Errors) > 0 { t.Fatalf("parse errors: %v", parsed.Errors) } @@ -47,8 +47,8 @@ func TestLowerEmbeddedExactFilesIntoBlobReferences(t *testing.T) { if string(program.EmbeddedBlobs[0].Data) != string(content) { t.Fatalf("blob data = %q", program.EmbeddedBlobs[0].Data) } - if len(program.Globals) != 2 { - t.Fatalf("globals = %d, want 2", len(program.Globals)) + if len(program.Globals) != 3 { + t.Fatalf("globals = %d, want 3", len(program.Globals)) } text := program.Globals[0].Initializer.Value bytes := program.Globals[1].Initializer.Value @@ -58,6 +58,13 @@ func TestLowerEmbeddedExactFilesIntoBlobReferences(t *testing.T) { if text.EmbeddedBlobPayload().Blob != 0 || bytes.EmbeddedBlobPayload().Blob != 0 { t.Fatalf("embedded blob refs = %d, %d", text.EmbeddedBlobPayload().Blob, bytes.EmbeddedBlobPayload().Blob) } + files := program.Globals[2].Initializer.Value + if files.Kind != ExprMakeEmbeddedFS || files.EmbeddedSetPayload().Set != 0 { + t.Fatalf("embedded filesystem expression = %#v", files) + } + if len(program.EmbeddedSets) != 1 || len(program.EmbeddedSets[0].Entries) != 1 || program.EmbeddedSets[0].Entries[0].Path != "asset.txt" { + t.Fatalf("embedded sets = %#v", program.EmbeddedSets) + } encoded, err := SerializeProgram(program) if err != nil { @@ -70,6 +77,9 @@ func TestLowerEmbeddedExactFilesIntoBlobReferences(t *testing.T) { if len(roundTrip.EmbeddedBlobs) != 1 || string(roundTrip.EmbeddedBlobs[0].Data) != string(content) { t.Fatalf("round-trip embedded blobs = %#v", roundTrip.EmbeddedBlobs) } + if len(roundTrip.EmbeddedSets) != 1 || roundTrip.EmbeddedSets[0].Digest != program.EmbeddedSets[0].Digest { + t.Fatalf("round-trip embedded sets = %#v", roundTrip.EmbeddedSets) + } malformed := *program duplicate := program.EmbeddedBlobs[0] @@ -78,4 +88,12 @@ func TestLowerEmbeddedExactFilesIntoBlobReferences(t *testing.T) { if err := Validate(&malformed); err == nil || !strings.Contains(err.Error(), "duplicates digest") { t.Fatalf("duplicate embedded blob validation error = %v", err) } + + invalidPath := *program + invalidPath.EmbeddedSets = append([]EmbeddedSet(nil), program.EmbeddedSets...) + invalidPath.EmbeddedSets[0].Entries = append([]EmbeddedEntry(nil), program.EmbeddedSets[0].Entries...) + invalidPath.EmbeddedSets[0].Entries[0].Path = "../escape" + if err := Validate(&invalidPath); err == nil || !strings.Contains(err.Error(), "invalid path") { + t.Fatalf("invalid embedded set path validation error = %v", err) + } } diff --git a/compiler/air/expr_payload_accessors.go b/compiler/air/expr_payload_accessors.go index 1fe1ffdf..1a3ad8b8 100644 --- a/compiler/air/expr_payload_accessors.go +++ b/compiler/air/expr_payload_accessors.go @@ -10,6 +10,11 @@ func (e Expr) EmbeddedBlobPayload() *EmbeddedBlobExprPayload { return payload } +func (e Expr) EmbeddedSetPayload() *EmbeddedSetExprPayload { + payload, _ := e.Payload.(*EmbeddedSetExprPayload) + return payload +} + func (e Expr) BoolPayload() *BoolExprPayload { payload, _ := e.Payload.(*BoolExprPayload) return payload diff --git a/compiler/air/expr_payloads.go b/compiler/air/expr_payloads.go index 05d17bad..c6be3e0b 100644 --- a/compiler/air/expr_payloads.go +++ b/compiler/air/expr_payloads.go @@ -27,6 +27,12 @@ type EmbeddedBlobExprPayload struct { func (*EmbeddedBlobExprPayload) exprPayload() {} +type EmbeddedSetExprPayload struct { + Set EmbeddedSetID +} + +func (*EmbeddedSetExprPayload) exprPayload() {} + type BoolExprPayload struct { Value bool } @@ -250,6 +256,8 @@ func exprPayloadIsTypedNil(payload ExprPayload) bool { return payload == nil case *EmbeddedBlobExprPayload: return payload == nil + case *EmbeddedSetExprPayload: + return payload == nil case *BoolExprPayload: return payload == nil case *EnumExprPayload: diff --git a/compiler/air/lower.go b/compiler/air/lower.go index 2fc8ea98..03e9c455 100644 --- a/compiler/air/lower.go +++ b/compiler/air/lower.go @@ -69,6 +69,7 @@ type lowerer struct { functions map[string]FunctionID globals map[string]GlobalID embeddedBlobs map[string]EmbeddedBlobID + embeddedSets map[string]EmbeddedSetID cacheMethodLookups bool structMethodsByOwner map[checker.MethodOwner]map[string]*checker.FunctionDef @@ -121,6 +122,7 @@ func newLowerer(options LowerOptions, rootCount int) *lowerer { functions: map[string]FunctionID{}, globals: map[string]GlobalID{}, embeddedBlobs: map[string]EmbeddedBlobID{}, + embeddedSets: map[string]EmbeddedSetID{}, cacheMethodLookups: rootCount == 1, unresolvedTypeVarByType: map[checker.Type]bool{}, @@ -177,13 +179,16 @@ func (l *lowerer) functionHasUnresolvedTypeVar(def *checker.FunctionDef) bool { return l.typeHasUnresolvedTypeVar(def) } -func (l *lowerer) internEmbeddedBlob(data []byte) (EmbeddedBlobID, error) { +func (l *lowerer) internEmbeddedBlob(data []byte, direct bool) (EmbeddedBlobID, error) { sum := sha256.Sum256(data) digest := hex.EncodeToString(sum[:]) if id, ok := l.embeddedBlobs[digest]; ok { if !bytes.Equal(l.program.EmbeddedBlobs[id].Data, data) { return 0, fmt.Errorf("embedded resource digest collision for %s", digest) } + if direct { + l.program.EmbeddedBlobs[id].Direct = true + } return id, nil } id := EmbeddedBlobID(len(l.program.EmbeddedBlobs)) @@ -191,11 +196,43 @@ func (l *lowerer) internEmbeddedBlob(data []byte) (EmbeddedBlobID, error) { ID: id, Data: append([]byte(nil), data...), Digest: digest, + Direct: direct, }) l.embeddedBlobs[digest] = id return id, nil } +func (l *lowerer) internEmbeddedSet(set checker.EmbeddedFileSet) (EmbeddedSetID, error) { + entries := make([]EmbeddedEntry, len(set.Entries)) + hash := sha256.New() + _, _ = hash.Write([]byte(set.OwnerPackageIdentity)) + _, _ = hash.Write([]byte{0}) + for index, entry := range set.Entries { + blob, err := l.internEmbeddedBlob(entry.Data, false) + if err != nil { + return 0, err + } + entries[index] = EmbeddedEntry{Path: entry.LogicalPath, Blob: blob} + _, _ = hash.Write([]byte(entry.LogicalPath)) + _, _ = hash.Write([]byte{0}) + _, _ = hash.Write([]byte(l.program.EmbeddedBlobs[blob].Digest)) + _, _ = hash.Write([]byte{0}) + } + digest := hex.EncodeToString(hash.Sum(nil)) + if id, ok := l.embeddedSets[digest]; ok { + return id, nil + } + id := EmbeddedSetID(len(l.program.EmbeddedSets)) + l.program.EmbeddedSets = append(l.program.EmbeddedSets, EmbeddedSet{ + ID: id, + OwnerPackageIdentity: set.OwnerPackageIdentity, + Entries: entries, + Digest: digest, + }) + l.embeddedSets[digest] = id + return id, nil +} + func (l *lowerer) mustIntern(t checker.Type) TypeID { id, err := l.internType(t) if err != nil { @@ -2509,6 +2546,8 @@ func (l *lowerer) internAtomicOrTraitType(t checker.Type) (TypeID, error) { info.Kind = TypeRune case checker.Str: info.Kind = TypeStr + case checker.EmbeddedFS: + info.Kind = TypeEmbeddedFS case checker.Any: info.Kind = TypeAny default: @@ -3954,17 +3993,23 @@ func (fl *functionLowerer) lowerExpr(expr checker.Expression) (*Expr, error) { case *checker.StrLiteral: return &Expr{Kind: ExprConstStr, Type: typeID, Payload: &TextExprPayload{Value: e.Value}}, nil case *checker.EmbeddedText: - blob, err := fl.l.internEmbeddedBlob(e.Resource.Data) + blob, err := fl.l.internEmbeddedBlob(e.Resource.Data, true) if err != nil { return nil, err } return &Expr{Kind: ExprEmbeddedText, Type: typeID, Payload: &EmbeddedBlobExprPayload{Blob: blob}}, nil case *checker.EmbeddedBytes: - blob, err := fl.l.internEmbeddedBlob(e.Resource.Data) + blob, err := fl.l.internEmbeddedBlob(e.Resource.Data, true) if err != nil { return nil, err } return &Expr{Kind: ExprEmbeddedBytes, Type: typeID, Payload: &EmbeddedBlobExprPayload{Blob: blob}}, nil + case *checker.EmbeddedFSValue: + set, err := fl.l.internEmbeddedSet(e.Set) + if err != nil { + return nil, err + } + return &Expr{Kind: ExprMakeEmbeddedFS, Type: typeID, Payload: &EmbeddedSetExprPayload{Set: set}}, nil case *checker.RuneLiteral: return &Expr{Kind: ExprConstInt, Type: typeID, Payload: &TextExprPayload{Value: strconv.Itoa(int(e.Value))}}, nil case *checker.NeverCoercion: @@ -4306,6 +4351,27 @@ func (fl *functionLowerer) lowerExpr(expr checker.Expression) (*Expr, error) { return fl.lowerInstanceMethod(typeID, e) case *checker.StrMethod: return fl.lowerStrMethod(typeID, e) + case *checker.EmbeddedFSMethod: + target, err := fl.lowerExpr(e.Subject) + if err != nil { + return nil, err + } + args, err := fl.lowerArgs(e.Args) + if err != nil { + return nil, err + } + kind := ExprEmbeddedFSReadFile + switch e.Kind { + case checker.EmbeddedFSReadText: + kind = ExprEmbeddedFSReadText + case checker.EmbeddedFSReadDir: + kind = ExprEmbeddedFSReadDir + case checker.EmbeddedFSStat: + kind = ExprEmbeddedFSStat + case checker.EmbeddedFSSub: + kind = ExprEmbeddedFSSub + } + return &Expr{Kind: kind, Type: typeID, Target: target, Args: args}, nil case *checker.ByteMethod: if e.Kind == checker.ByteToInt { return fl.lowerUnary(ExprToInt, typeID, e.Subject) diff --git a/compiler/air/nodes.go b/compiler/air/nodes.go index e06586bc..ce3f950b 100644 --- a/compiler/air/nodes.go +++ b/compiler/air/nodes.go @@ -64,6 +64,12 @@ const ( ExprConstStr ExprEmbeddedText ExprEmbeddedBytes + ExprMakeEmbeddedFS + ExprEmbeddedFSReadFile + ExprEmbeddedFSReadText + ExprEmbeddedFSReadDir + ExprEmbeddedFSStat + ExprEmbeddedFSSub ExprPanic ExprLoadLocal ExprLoadGlobal diff --git a/compiler/air/serialize.go b/compiler/air/serialize.go index d545faef..1d8ac470 100644 --- a/compiler/air/serialize.go +++ b/compiler/air/serialize.go @@ -13,6 +13,7 @@ import ( func init() { gob.Register(&TextExprPayload{}) gob.Register(&EmbeddedBlobExprPayload{}) + gob.Register(&EmbeddedSetExprPayload{}) gob.Register(&BoolExprPayload{}) gob.Register(&EnumExprPayload{}) gob.Register(&LocalExprPayload{}) diff --git a/compiler/air/types.go b/compiler/air/types.go index 47dc0196..61b6fbda 100644 --- a/compiler/air/types.go +++ b/compiler/air/types.go @@ -8,6 +8,7 @@ type LocalID int type TraitID int type ImplID int type EmbeddedBlobID int +type EmbeddedSetID int const ( NoType TypeID = 0 @@ -19,11 +20,25 @@ type EmbeddedBlob struct { ID EmbeddedBlobID Data []byte Digest string + Direct bool +} + +type EmbeddedEntry struct { + Path string + Blob EmbeddedBlobID +} + +type EmbeddedSet struct { + ID EmbeddedSetID + OwnerPackageIdentity string + Entries []EmbeddedEntry + Digest string } type Program struct { Modules []Module EmbeddedBlobs []EmbeddedBlob + EmbeddedSets []EmbeddedSet Types []TypeInfo Traits []Trait Impls []Impl @@ -158,6 +173,7 @@ const ( TypeByte TypeRune TypeStr + TypeEmbeddedFS TypeList TypeSlice TypeFixedArray diff --git a/compiler/air/validate.go b/compiler/air/validate.go index 91cde96e..8db8a50b 100644 --- a/compiler/air/validate.go +++ b/compiler/air/validate.go @@ -5,10 +5,23 @@ import ( "encoding/hex" "fmt" "reflect" + "strings" "github.com/akonwi/ard/checker" ) +func airTypeIsBuiltinError(program *Program, id TypeID) bool { + if !validTypeID(program, id) { + return false + } + info := program.Types[id-1] + return info.Kind == TypeTraitObject && info.Trait >= 0 && int(info.Trait) < len(program.Traits) && program.Traits[info.Trait].BuiltinError +} + +func airTypeIsEmbedStruct(info TypeInfo, name string) bool { + return info.Kind == TypeStruct && info.ModulePath == checker.EmbedModulePath && info.Name == name +} + func validGoStructTagKey(key string) bool { prefixed := false for index, char := range key { @@ -51,10 +64,8 @@ func Validate(program *Program) error { if err := validateCanonicalNominalIdentities(program); err != nil { return err } - if len(program.EmbeddedBlobs) > checker.MaxEmbeddedProgramFileCount { - return fmt.Errorf("embedded blobs exceed program limit of %d files", checker.MaxEmbeddedProgramFileCount) - } embeddedBytes := 0 + embeddedFiles := 0 embeddedDigests := make(map[string]bool, len(program.EmbeddedBlobs)) for i, blob := range program.EmbeddedBlobs { if blob.ID != EmbeddedBlobID(i) { @@ -63,9 +74,9 @@ func Validate(program *Program) error { if len(blob.Data) > checker.MaxEmbeddedFileBytes { return fmt.Errorf("embedded blob %d exceeds file limit of %d bytes", blob.ID, checker.MaxEmbeddedFileBytes) } - embeddedBytes += len(blob.Data) - if embeddedBytes > checker.MaxEmbeddedProgramBytes { - return fmt.Errorf("embedded blobs exceed program limit of %d bytes", checker.MaxEmbeddedProgramBytes) + if blob.Direct { + embeddedBytes += len(blob.Data) + embeddedFiles++ } sum := sha256.Sum256(blob.Data) digest := hex.EncodeToString(sum[:]) @@ -77,6 +88,69 @@ func Validate(program *Program) error { } embeddedDigests[digest] = true } + embeddedSetDigests := make(map[string]bool, len(program.EmbeddedSets)) + for i, set := range program.EmbeddedSets { + if set.ID != EmbeddedSetID(i) { + return fmt.Errorf("embedded set table entry %d has id %d", i, set.ID) + } + if set.OwnerPackageIdentity == "" { + return fmt.Errorf("embedded set %d has no owner package identity", set.ID) + } + hash := sha256.New() + _, _ = hash.Write([]byte(set.OwnerPackageIdentity)) + _, _ = hash.Write([]byte{0}) + setBytes := 0 + lastPath := "" + for entryIndex, entry := range set.Entries { + if err := checker.ValidateEmbeddedLogicalFilePath(entry.Path); err != nil { + return fmt.Errorf("embedded set %d has invalid path %q: %w", set.ID, entry.Path, err) + } + for _, component := range strings.Split(entry.Path, "/") { + switch component { + case ".bzr", ".git", ".hg", ".svn": + return fmt.Errorf("embedded set %d contains reserved path %q", set.ID, component) + } + } + if entryIndex > 0 && entry.Path <= lastPath { + return fmt.Errorf("embedded set %d entries are not sorted and unique", set.ID) + } + if lastPath != "" && strings.HasPrefix(entry.Path, lastPath+"/") { + return fmt.Errorf("embedded set %d path %q conflicts with file %q", set.ID, entry.Path, lastPath) + } + if entry.Blob < 0 || int(entry.Blob) >= len(program.EmbeddedBlobs) { + return fmt.Errorf("embedded set %d entry %q references invalid blob %d", set.ID, entry.Path, entry.Blob) + } + blob := program.EmbeddedBlobs[entry.Blob] + setBytes += len(blob.Data) + _, _ = hash.Write([]byte(entry.Path)) + _, _ = hash.Write([]byte{0}) + _, _ = hash.Write([]byte(blob.Digest)) + _, _ = hash.Write([]byte{0}) + lastPath = entry.Path + } + if len(set.Entries) == 0 { + return fmt.Errorf("embedded set %d is empty", set.ID) + } + if setBytes > checker.MaxEmbeddedSetBytes { + return fmt.Errorf("embedded set %d exceeds limit of %d bytes", set.ID, checker.MaxEmbeddedSetBytes) + } + embeddedBytes += setBytes + embeddedFiles += len(set.Entries) + digest := hex.EncodeToString(hash.Sum(nil)) + if set.Digest != digest { + return fmt.Errorf("embedded set %d has digest %q, want %q", set.ID, set.Digest, digest) + } + if embeddedSetDigests[digest] { + return fmt.Errorf("embedded set %d duplicates digest %q", set.ID, digest) + } + embeddedSetDigests[digest] = true + } + if embeddedFiles > checker.MaxEmbeddedProgramFileCount { + return fmt.Errorf("embedded resources exceed program limit of %d files", checker.MaxEmbeddedProgramFileCount) + } + if embeddedBytes > checker.MaxEmbeddedProgramBytes { + return fmt.Errorf("embedded resources exceed program limit of %d bytes", checker.MaxEmbeddedProgramBytes) + } for i, trait := range program.Traits { if trait.ID != TraitID(i) { return fmt.Errorf("trait table entry %d has id %d", i, trait.ID) @@ -960,6 +1034,8 @@ func validateExprPayload(expr Expr) error { compatible = expr.Kind == ExprConstInt || expr.Kind == ExprConstFloat || expr.Kind == ExprConstStr case *EmbeddedBlobExprPayload: compatible = expr.Kind == ExprEmbeddedText || expr.Kind == ExprEmbeddedBytes + case *EmbeddedSetExprPayload: + compatible = expr.Kind == ExprMakeEmbeddedFS case *BoolExprPayload: compatible = expr.Kind == ExprConstBool case *EnumExprPayload: @@ -1063,7 +1139,7 @@ func exprPayloadRequired(kind ExprKind) bool { } switch kind { case ExprConstInt, ExprConstFloat, ExprConstBool, ExprConstStr, - ExprEmbeddedText, ExprEmbeddedBytes, + ExprEmbeddedText, ExprEmbeddedBytes, ExprMakeEmbeddedFS, ExprLoadLocal, ExprLoadGlobal, ExprFunctionRef, ExprCall, ExprForeignCall, ExprForeignMethodCall, ExprForeignMethodValue, ExprForeignFieldAccess, ExprForeignStructInstance, ExprForeignValue, @@ -1094,6 +1170,9 @@ func validateExpr(program *Program, fn Function, expr Expr) error { if payload.Blob < 0 || int(payload.Blob) >= len(program.EmbeddedBlobs) { return fmt.Errorf("embedded expression references invalid blob %d", payload.Blob) } + if !program.EmbeddedBlobs[payload.Blob].Direct { + return fmt.Errorf("embedded exact-file expression references non-direct blob %d", payload.Blob) + } typ := program.Types[expr.Type-1] if expr.Kind == ExprEmbeddedText && typ.Kind != TypeStr { return fmt.Errorf("embedded text expression has non-Str type %d", expr.Type) @@ -1104,6 +1183,44 @@ func validateExpr(program *Program, fn Function, expr Expr) error { } } } + if expr.Kind == ExprMakeEmbeddedFS { + payload := exprPayloadAs[*EmbeddedSetExprPayload](&expr) + if payload == nil || payload.Set < 0 || int(payload.Set) >= len(program.EmbeddedSets) { + return fmt.Errorf("embedded filesystem expression references invalid set") + } + if program.Types[expr.Type-1].Kind != TypeEmbeddedFS { + return fmt.Errorf("embedded filesystem expression has invalid type %d", expr.Type) + } + } + if expr.Kind >= ExprEmbeddedFSReadFile && expr.Kind <= ExprEmbeddedFSSub { + if expr.Target == nil || !validTypeID(program, expr.Target.Type) || program.Types[expr.Target.Type-1].Kind != TypeEmbeddedFS { + return fmt.Errorf("embedded filesystem operation has invalid target") + } + if len(expr.Args) != 1 || !validTypeID(program, expr.Args[0].Type) || program.Types[expr.Args[0].Type-1].Kind != TypeStr { + return fmt.Errorf("embedded filesystem operation requires one Str argument") + } + result := program.Types[expr.Type-1] + if result.Kind != TypeResult || !validTypeID(program, result.Value) || !airTypeIsBuiltinError(program, result.Error) { + return fmt.Errorf("embedded filesystem operation has invalid Result type") + } + value := program.Types[result.Value-1] + validValue := false + switch expr.Kind { + case ExprEmbeddedFSReadFile: + validValue = value.Kind == TypeList && validTypeID(program, value.Elem) && program.Types[value.Elem-1].Kind == TypeByte + case ExprEmbeddedFSReadText: + validValue = value.Kind == TypeStr + case ExprEmbeddedFSReadDir: + validValue = value.Kind == TypeList && validTypeID(program, value.Elem) && airTypeIsEmbedStruct(program.Types[value.Elem-1], "DirEntry") + case ExprEmbeddedFSStat: + validValue = airTypeIsEmbedStruct(value, "FileInfo") + case ExprEmbeddedFSSub: + validValue = value.Kind == TypeEmbeddedFS + } + if !validValue { + return fmt.Errorf("embedded filesystem operation kind %d has invalid result value type", expr.Kind) + } + } if err := validateTailSpread(program, expr); err != nil { return err } diff --git a/compiler/checker/checker.go b/compiler/checker/checker.go index a5d2d40a..983f4742 100644 --- a/compiler/checker/checker.go +++ b/compiler/checker/checker.go @@ -1686,7 +1686,7 @@ func isValidMapKeyTypeSeen(t Type, context *mapKeyTypeContext) bool { return true } return ty.GoType == nil || gotypes.Comparable(ty.GoType) - case *Maybe, *List, *Slice, *Map, *Result, *Union, *FunctionDef, *Trait, *anyType: + case *Maybe, *List, *Slice, *Map, *Result, *Union, *FunctionDef, *Trait, *anyType, *embeddedFSType: return false default: return true @@ -2635,6 +2635,9 @@ func (c *Checker) areCompatible(expected Type, actual Type) bool { } } actualBase, actualIsReference := mutableRefBase(actual) + if _, ok := actualBase.(*embeddedFSType); ok && !actualIsReference && iface.Target == "go" && iface.Namespace == "io/fs" && iface.Name == "FS" { + return true + } if actualForeign, ok := actualBase.(*ForeignType); ok { return validationEqualTypes(actualForeign, iface) || foreignGoAssignableTo(actualForeign, iface) } @@ -7014,6 +7017,9 @@ func (c *Checker) createPrimitiveMethodNode(subject Expression, methodName strin if _, isResult := subjectType.(*Result); isResult { return c.createResultMethod(subject, methodName, args, fnDef) } + if _, isEmbeddedFS := subjectType.(*embeddedFSType); isEmbeddedFS { + return c.createEmbeddedFSMethod(subject, methodName, args, fnDef) + } // For user-defined types (structs, enums), use generic InstanceMethod receiverKind := ReceiverUnknown @@ -9599,6 +9605,9 @@ func (c *Checker) checkExprInner(expr parse.Expression, expectedReturn Type) Exp if name == "text" || name == "bytes" { return c.checkEmbedExactCall(s, name) } + if name == "fs" { + return c.checkEmbedFSCall(s) + } case "ard/unsafe": if c.rejectSpreadForFixedCall(s.Function.Args) { return nil @@ -10974,7 +10983,7 @@ func (c *Checker) checkExprInner(expr parse.Expression, expectedReturn Type) Exp // Check if this is accessing a module if mod := c.resolveModule(id.Name); mod != nil { - if prop, ok := s.Property.(*parse.Identifier); ok && mod.Path() == EmbedModulePath && (prop.Name == "text" || prop.Name == "bytes") { + if prop, ok := s.Property.(*parse.Identifier); ok && mod.Path() == EmbedModulePath && (prop.Name == "text" || prop.Name == "bytes" || prop.Name == "fs") { c.addEmbedDiagnostic(DiagnosticCodeEmbedStaticArgument, "Embed constructors are not function values", "call the constructor directly with a static path", prop.GetLocation()) return nil } diff --git a/compiler/checker/embed.go b/compiler/checker/embed.go index 486f7c4d..2cf6e2cc 100644 --- a/compiler/checker/embed.go +++ b/compiler/checker/embed.go @@ -1,10 +1,15 @@ package checker import ( + "crypto/sha256" + "encoding/hex" "fmt" "io" + iofs "io/fs" "os" + "path" "path/filepath" + "sort" "strings" "unicode/utf8" @@ -15,32 +20,119 @@ import ( const ( EmbedModulePath = "ard/embed" MaxEmbeddedFileBytes = 16 * 1024 * 1024 + MaxEmbeddedSetBytes = 64 * 1024 * 1024 MaxEmbeddedProgramBytes = 128 * 1024 * 1024 MaxEmbeddedProgramFileCount = 10_000 ) +type embeddedFSType struct{} + +var EmbeddedFS Type = &embeddedFSType{} + +var ( + embeddedDirEntryType = &StructDef{Name: "DirEntry", ModulePath: EmbedModulePath, Fields: map[string]Type{"name": Str, "is_dir": Bool}} + embeddedFileInfoType = &StructDef{Name: "FileInfo", ModulePath: EmbedModulePath, Fields: map[string]Type{"name": Str, "is_dir": Bool, "size": MakeMaybe(Int)}} +) + +func (*embeddedFSType) String() string { return "embed::FS" } +func (*embeddedFSType) equal(other Type) bool { _, ok := other.(*embeddedFSType); return ok } +func (*embeddedFSType) hasTrait(*Trait) bool { return false } +func (*embeddedFSType) get(name string) Type { + var returnType Type + switch name { + case "read_file": + returnType = MakeResult(MakeList(Byte), BuiltinError) + case "read_text": + returnType = MakeResult(Str, BuiltinError) + case "read_dir": + returnType = MakeResult(MakeList(embeddedDirEntryType), BuiltinError) + case "stat": + returnType = MakeResult(embeddedFileInfoType, BuiltinError) + case "sub": + returnType = MakeResult(EmbeddedFS, BuiltinError) + default: + return nil + } + return &FunctionDef{Name: name, Parameters: []Parameter{{Name: "path", Type: Str}}, ReturnType: returnType} +} + type EmbedPkg struct{} func (EmbedPkg) Path() string { return EmbedModulePath } func (EmbedPkg) Program() *Program { return nil } func (EmbedPkg) Get(name string) Symbol { + switch name { + case "FS": + return Symbol{Name: name, Type: EmbeddedFS, typeDeclaration: true} + case "DirEntry": + return Symbol{Name: name, Type: embeddedDirEntryType, typeDeclaration: true} + case "FileInfo": + return Symbol{Name: name, Type: embeddedFileInfoType, typeDeclaration: true} + } + var parameter Type = Str var returnType Type switch name { case "text": returnType = Str case "bytes": returnType = MakeList(Byte) + case "fs": + parameter = MakeList(Str) + returnType = EmbeddedFS default: return Symbol{} } return Symbol{Name: name, Type: &FunctionDef{ Name: name, - Parameters: []Parameter{{Name: "path", Type: Str}}, + Parameters: []Parameter{{Name: "path", Type: parameter}}, ReturnType: returnType, }} } +func (c *Checker) checkEmbedFSCall(s *parse.StaticFunction) Expression { + if len(s.Function.TypeArgs) != 0 { + c.addInvalidFunctionTypeArguments("embed::fs", 0, len(s.Function.TypeArgs), false, s.GetLocation(), "embed::fs does not accept type arguments") + return nil + } + if c.rejectSpreadForFixedCall(s.Function.Args) { + return nil + } + if len(s.Function.Args) != 1 { + c.addArgumentCount("1", len(s.Function.Args), s.GetLocation(), "") + return nil + } + arg := s.Function.Args[0] + if arg.Name != "" { + c.addNamedArgumentsUnsupported("embed constructor", arg.GetLocation()) + return nil + } + list, ok := arg.Value.(*parse.ListLiteral) + if !ok || len(list.Items) == 0 { + c.addEmbedDiagnostic(DiagnosticCodeEmbedStaticArgument, "Embed patterns must be a non-empty static list of strings", "files are selected while the package is checked", arg.Value.GetLocation()) + return nil + } + patterns := make([]string, len(list.Items)) + for index, item := range list.Items { + literal, ok := item.(*parse.StrLiteral) + if !ok { + c.addEmbedDiagnostic(DiagnosticCodeEmbedStaticArgument, "Embed patterns must be a static list of strings", "use non-interpolated string literals", item.GetLocation()) + return nil + } + patterns[index] = literal.Value + } + if c.moduleResolver == nil { + c.addEmbedDiagnostic(DiagnosticCodeEmbedResource, "Cannot resolve embedded filesystem", "embedding requires project package context", list.GetLocation()) + return nil + } + set, err := c.moduleResolver.resolveEmbeddedFileSet(c.modulePath, patterns) + if err != nil { + c.addEmbedDiagnostic(DiagnosticCodeEmbedResource, "Cannot construct embedded filesystem", err.Error(), list.GetLocation()) + return nil + } + return &EmbeddedFSValue{Set: set} +} + func (c *Checker) checkEmbedExactCall(s *parse.StaticFunction, name string) Expression { qualified := "embed::" + name if name != "text" && name != "bytes" { @@ -72,6 +164,9 @@ func (c *Checker) checkEmbedExactCall(s *parse.StaticFunction, name string) Expr return nil } resource, err := c.moduleResolver.resolveEmbeddedExactFile(c.modulePath, literal.Value) + if err == nil { + err = c.moduleResolver.accountEmbeddedExact(resource) + } if err != nil { c.addEmbedDiagnostic(DiagnosticCodeEmbedResource, fmt.Sprintf("Cannot embed %q", literal.Value), err.Error(), literal.GetLocation()) return nil @@ -86,6 +181,25 @@ func (c *Checker) checkEmbedExactCall(s *parse.StaticFunction, name string) Expr return &EmbeddedBytes{Resource: resource} } +func (c *Checker) createEmbeddedFSMethod(subject Expression, name string, args []Expression, declaration *FunctionDef) Expression { + var kind EmbeddedFSMethodKind + switch name { + case "read_file": + kind = EmbeddedFSReadFile + case "read_text": + kind = EmbeddedFSReadText + case "read_dir": + kind = EmbeddedFSReadDir + case "stat": + kind = EmbeddedFSStat + case "sub": + kind = EmbeddedFSSub + default: + return nil + } + return &EmbeddedFSMethod{Subject: subject, Kind: kind, Args: args, ReturnType: declaration.ReturnType} +} + func (c *Checker) addEmbedDiagnostic(code DiagnosticCode, title string, text string, location parse.Location) { diagnostic := newLabeledDiagnostic( Error, @@ -99,7 +213,7 @@ func (c *Checker) addEmbedDiagnostic(code DiagnosticCode, title string, text str } func (mr *ModuleResolver) resolveEmbeddedExactFile(importerModulePath string, logicalPath string) (EmbeddedResource, error) { - if err := validateEmbeddedExactPath(logicalPath); err != nil { + if err := ValidateEmbeddedLogicalFilePath(logicalPath); err != nil { return EmbeddedResource{}, err } packageID := mr.packageIDForModule(importerModulePath) @@ -137,7 +251,7 @@ func (mr *ModuleResolver) resolveEmbeddedExactFile(importerModulePath string, lo return EmbeddedResource{}, fmt.Errorf("embedded resource path component is not a directory: %s", strings.Join(components[:index+1], "/")) } switch component { - case ".bzr", ".git", ".hg", ".svn", "vendor": + case ".bzr", ".git", ".hg", ".svn", "ard-out", "vendor": return EmbeddedResource{}, fmt.Errorf("embedded resources must not select reserved directory %q", component) } if _, err := root.Stat(filepath.Join(current, "ard.toml")); err == nil { @@ -185,20 +299,249 @@ func (mr *ModuleResolver) resolveEmbeddedExactFile(importerModulePath string, lo if len(data) > MaxEmbeddedFileBytes { return EmbeddedResource{}, fmt.Errorf("embedded resource exceeds maximum file size of %d bytes", MaxEmbeddedFileBytes) } - if len(mr.embeddedResources)+1 > MaxEmbeddedProgramFileCount { - return EmbeddedResource{}, fmt.Errorf("embedded resources exceed program limit of %d files", MaxEmbeddedProgramFileCount) + resource := EmbeddedResource{OwnerPackageIdentity: embeddedOwnerIdentity(pkg), LogicalPath: logicalPath, Data: data} + mr.embeddedResources[cacheKey] = resource + return resource, nil +} + +func embeddedOwnerIdentity(pkg PackageInfo) string { + if pkg.Git != "" { + return pkg.ID + } + return pkg.Name +} + +func (mr *ModuleResolver) accountEmbeddedExact(resource EmbeddedResource) error { + sum := sha256.Sum256(resource.Data) + key := hex.EncodeToString(sum[:]) + mr.embedMu.Lock() + defer mr.embedMu.Unlock() + if mr.embeddedExactReferences[key] { + return nil + } + if mr.embeddedProgramFileCount+1 > MaxEmbeddedProgramFileCount { + return fmt.Errorf("embedded resources exceed program limit of %d files", MaxEmbeddedProgramFileCount) } - if mr.embeddedResourceBytes+len(data) > MaxEmbeddedProgramBytes { - return EmbeddedResource{}, fmt.Errorf("embedded resources exceed program limit of %d bytes", MaxEmbeddedProgramBytes) + if mr.embeddedProgramBytes+len(resource.Data) > MaxEmbeddedProgramBytes { + return fmt.Errorf("embedded resources exceed program limit of %d bytes", MaxEmbeddedProgramBytes) } + mr.embeddedExactReferences[key] = true + mr.embeddedProgramFileCount++ + mr.embeddedProgramBytes += len(resource.Data) + return nil +} - resource := EmbeddedResource{OwnerPackageIdentity: packageID, LogicalPath: logicalPath, Data: data} - mr.embeddedResources[cacheKey] = resource - mr.embeddedResourceBytes += len(data) - return resource, nil +func (mr *ModuleResolver) resolveEmbeddedFileSet(importerModulePath string, patterns []string) (EmbeddedFileSet, error) { + packageID := mr.packageIDForModule(importerModulePath) + pkg := mr.packageInfo(packageID) + if pkg.RootPath == "" { + return EmbeddedFileSet{}, fmt.Errorf("owning Ard package has no source root") + } + patternCacheKey := packageID + "\x00" + strings.Join(patterns, "\x00") + mr.embedMu.Lock() + if cached, ok := mr.embeddedPatternSets[patternCacheKey]; ok { + mr.embedMu.Unlock() + return cached, nil + } + mr.embedMu.Unlock() + + type patternSpec struct { + value string + includeAll bool + } + specs := make([]patternSpec, len(patterns)) + for index, original := range patterns { + value := original + includeAll := strings.HasPrefix(value, "all:") + if includeAll { + value = strings.TrimPrefix(value, "all:") + } + if value == "" || value == "." || !iofs.ValidPath(value) || strings.Contains(value, `\`) { + return EmbeddedFileSet{}, fmt.Errorf("invalid embed pattern %q", original) + } + if _, err := path.Match(value, ""); err != nil { + return EmbeddedFileSet{}, fmt.Errorf("invalid embed pattern %q: %w", original, err) + } + specs[index] = patternSpec{value: value, includeAll: includeAll} + } + + selected := map[string]bool{} + matchedPatterns := make([]bool, len(specs)) + scanRoots := map[string]bool{} + for _, spec := range specs { + scanRoot := embedPatternScanRoot(spec.value) + scanRoots[filepath.Join(pkg.RootPath, filepath.FromSlash(scanRoot))] = true + } + orderedRoots := make([]string, 0, len(scanRoots)) + for scanRoot := range scanRoots { + orderedRoots = append(orderedRoots, scanRoot) + } + sort.Strings(orderedRoots) + walk := func(filePath string, entry os.DirEntry, walkErr error) error { + if walkErr != nil { + return walkErr + } + if filePath == pkg.RootPath { + return nil + } + rel, err := filepath.Rel(pkg.RootPath, filePath) + if err != nil { + return err + } + logical := filepath.ToSlash(rel) + if entry.Type()&os.ModeSymlink != 0 { + for index, spec := range specs { + if embedPatternSelectsFile(spec.value, spec.includeAll, logical) { + return fmt.Errorf("embed pattern %q selects symbolic link %q", patterns[index], logical) + } + } + return nil + } + if entry.IsDir() { + base := entry.Name() + switch base { + case ".bzr", ".git", ".hg", ".svn", "ard-out", "vendor": + return filepath.SkipDir + } + if _, err := os.Stat(filepath.Join(filePath, "ard.toml")); err == nil { + return filepath.SkipDir + } else if !os.IsNotExist(err) { + return err + } + if _, err := os.Stat(filepath.Join(filePath, "go.mod")); err == nil { + return filepath.SkipDir + } else if !os.IsNotExist(err) { + return err + } + return nil + } + if !entry.Type().IsRegular() { + return nil + } + for index, spec := range specs { + if !embedPatternSelectsFile(spec.value, spec.includeAll, logical) { + continue + } + matchedPatterns[index] = true + selected[logical] = true + if len(selected) > MaxEmbeddedProgramFileCount { + return fmt.Errorf("embedded filesystem exceeds program limit of %d files", MaxEmbeddedProgramFileCount) + } + } + return nil + } + for _, scanRoot := range orderedRoots { + if err := filepath.WalkDir(scanRoot, walk); err != nil { + if os.IsNotExist(err) { + continue + } + return EmbeddedFileSet{}, fmt.Errorf("scan embedded resources: %w", err) + } + } + for index, matched := range matchedPatterns { + if !matched { + return EmbeddedFileSet{}, fmt.Errorf("embed pattern %q matched no files", patterns[index]) + } + } + + paths := make([]string, 0, len(selected)) + for file := range selected { + paths = append(paths, file) + } + sort.Strings(paths) + if len(paths) > MaxEmbeddedProgramFileCount { + return EmbeddedFileSet{}, fmt.Errorf("embedded filesystem exceeds program limit of %d files", MaxEmbeddedProgramFileCount) + } + set := EmbeddedFileSet{OwnerPackageIdentity: embeddedOwnerIdentity(pkg), Entries: make([]EmbeddedSetEntry, 0, len(paths))} + totalBytes := 0 + for _, logicalPath := range paths { + for _, component := range strings.Split(logicalPath, "/") { + switch component { + case ".bzr", ".git", ".hg", ".svn": + return EmbeddedFileSet{}, fmt.Errorf("embedded filesystems cannot contain reserved path %q", component) + } + } + resource, err := mr.resolveEmbeddedExactFile(importerModulePath, logicalPath) + if err != nil { + return EmbeddedFileSet{}, err + } + totalBytes += len(resource.Data) + if totalBytes > MaxEmbeddedSetBytes { + return EmbeddedFileSet{}, fmt.Errorf("embedded filesystem exceeds limit of %d bytes", MaxEmbeddedSetBytes) + } + set.Entries = append(set.Entries, EmbeddedSetEntry{LogicalPath: logicalPath, Data: resource.Data}) + } + + hash := sha256.New() + _, _ = hash.Write([]byte(set.OwnerPackageIdentity)) + _, _ = hash.Write([]byte{0}) + for _, entry := range set.Entries { + _, _ = hash.Write([]byte(entry.LogicalPath)) + _, _ = hash.Write([]byte{0}) + sum := sha256.Sum256(entry.Data) + _, _ = hash.Write(sum[:]) + } + identity := hex.EncodeToString(hash.Sum(nil)) + mr.embedMu.Lock() + defer mr.embedMu.Unlock() + if !mr.embeddedSetIdentities[identity] { + if mr.embeddedProgramFileCount+len(set.Entries) > MaxEmbeddedProgramFileCount { + return EmbeddedFileSet{}, fmt.Errorf("embedded resources exceed program limit of %d files", MaxEmbeddedProgramFileCount) + } + if mr.embeddedProgramBytes+totalBytes > MaxEmbeddedProgramBytes { + return EmbeddedFileSet{}, fmt.Errorf("embedded resources exceed program limit of %d bytes", MaxEmbeddedProgramBytes) + } + mr.embeddedSetIdentities[identity] = true + mr.embeddedProgramFileCount += len(set.Entries) + mr.embeddedProgramBytes += totalBytes + } + mr.embeddedPatternSets[patternCacheKey] = set + return set, nil } -func validateEmbeddedExactPath(path string) error { +func embedPatternScanRoot(pattern string) string { + parts := strings.Split(pattern, "/") + prefix := make([]string, 0, len(parts)) + for _, part := range parts { + if strings.ContainsAny(part, "*?[") { + break + } + prefix = append(prefix, part) + } + if len(prefix) == 0 { + return "." + } + return strings.Join(prefix, "/") +} + +func embedPatternSelectsFile(pattern string, includeAll bool, file string) bool { + if matched, _ := path.Match(pattern, file); matched { + return true + } + for directory := path.Dir(file); directory != "."; directory = path.Dir(directory) { + matched, _ := path.Match(pattern, directory) + if !matched { + continue + } + if includeAll || !embeddedRelativePathHasHiddenElement(strings.TrimPrefix(file, directory+"/")) { + return true + } + } + return false +} + +func embeddedRelativePathHasHiddenElement(relative string) bool { + for _, element := range strings.Split(relative, "/") { + if strings.HasPrefix(element, ".") || strings.HasPrefix(element, "_") { + return true + } + } + return false +} + +// ValidateEmbeddedLogicalFilePath validates the target-neutral portable path +// carried by checked resources and AIR embedded-set entries. +func ValidateEmbeddedLogicalFilePath(path string) error { if path == "" { return fmt.Errorf("embedded resource path must not be empty") } @@ -208,10 +551,17 @@ func validateEmbeddedExactPath(path string) error { if err := module.CheckFilePath(path); err != nil { return fmt.Errorf("invalid embedded resource path: %w", err) } - for _, component := range strings.Split(path, "/") { + components := strings.Split(path, "/") + for index, component := range components { if component == "go.mod" { return fmt.Errorf("embedded resources must not select a file named go.mod") } + if index < len(components)-1 { + switch component { + case ".bzr", ".git", ".hg", ".svn", "ard-out", "vendor": + return fmt.Errorf("embedded resources must not select reserved directory %q", component) + } + } } return nil } diff --git a/compiler/checker/embed_internal_test.go b/compiler/checker/embed_internal_test.go index 3532efc4..5549a1c5 100644 --- a/compiler/checker/embed_internal_test.go +++ b/compiler/checker/embed_internal_test.go @@ -47,10 +47,12 @@ func TestEmbeddedExactFileEnforcesProgramFileLimit(t *testing.T) { if err != nil { t.Fatal(err) } - for i := 0; i < MaxEmbeddedProgramFileCount; i++ { - resolver.embeddedResources[string(rune(i))+"\x00"] = EmbeddedResource{} + resolver.embeddedProgramFileCount = MaxEmbeddedProgramFileCount + resource, err := resolver.resolveEmbeddedExactFile("app/main", "asset.txt") + if err != nil { + t.Fatal(err) } - if _, err := resolver.resolveEmbeddedExactFile("app/main", "asset.txt"); err == nil { + if err := resolver.accountEmbeddedExact(resource); err == nil { t.Fatal("expected program file limit error") } } @@ -75,3 +77,31 @@ func TestEmbeddedExactFileAllowsReservedNameWhenItIsAFile(t *testing.T) { t.Fatalf("resource data = %q", resource.Data) } } + +func TestEmbeddedExactAccountingDeduplicatesIdenticalBlobs(t *testing.T) { + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "ard.toml"), []byte("name = \"app\"\nard = \">= 0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + for _, name := range []string{"first.txt", "second.txt"} { + if err := os.WriteFile(filepath.Join(root, name), []byte("same"), 0o644); err != nil { + t.Fatal(err) + } + } + resolver, err := NewModuleResolver(root) + if err != nil { + t.Fatal(err) + } + for _, name := range []string{"first.txt", "second.txt"} { + resource, err := resolver.resolveEmbeddedExactFile("app/main", name) + if err != nil { + t.Fatal(err) + } + if err := resolver.accountEmbeddedExact(resource); err != nil { + t.Fatal(err) + } + } + if resolver.embeddedProgramFileCount != 1 || resolver.embeddedProgramBytes != len("same") { + t.Fatalf("embedded accounting = %d files, %d bytes", resolver.embeddedProgramFileCount, resolver.embeddedProgramBytes) + } +} diff --git a/compiler/checker/embed_test.go b/compiler/checker/embed_test.go index 624fd69f..69ff262b 100644 --- a/compiler/checker/embed_test.go +++ b/compiler/checker/embed_test.go @@ -3,6 +3,7 @@ package checker_test import ( "os" "path/filepath" + "strings" "testing" "github.com/akonwi/ard/checker" @@ -78,7 +79,7 @@ func TestEmbedExactFilesResolveFromOwningPackageRoot(t *testing.T) { if !ok { t.Fatalf("page value = %T", page.Value) } - if text.Resource.OwnerPackageIdentity != "root" || text.Resource.LogicalPath != "assets/page.txt" || string(text.Resource.Data) != string(content) { + if text.Resource.OwnerPackageIdentity != "app" || text.Resource.LogicalPath != "assets/page.txt" || string(text.Resource.Data) != string(content) { t.Fatalf("embedded text resource = %#v", text.Resource) } if text.Type() != checker.Str { @@ -93,7 +94,7 @@ func TestEmbedExactFilesResolveFromOwningPackageRoot(t *testing.T) { if !ok { t.Fatalf("raw value = %T", raw.Value) } - if bytes.Resource.OwnerPackageIdentity != "root" || bytes.Resource.LogicalPath != "assets/page.txt" || string(bytes.Resource.Data) != string(content) { + if bytes.Resource.OwnerPackageIdentity != "app" || bytes.Resource.LogicalPath != "assets/page.txt" || string(bytes.Resource.Data) != string(content) { t.Fatalf("embedded bytes resource = %#v", bytes.Resource) } list, ok := bytes.Type().(*checker.List) @@ -160,3 +161,129 @@ func TestEmbedConstructorsCannotBeFunctionValues(t *testing.T) { t.Fatalf("missing function-value diagnostic: %#v", checked.Diagnostics()) } } + +func TestEmbedFSExpandsDirectoriesAndAllPatterns(t *testing.T) { + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "ard.toml"), []byte("name = \"app\"\nard = \">= 0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + for name, content := range map[string]string{ + "public/index.html": "index", + "public/app.js": "app", + "public/.well-known/info.txt": "hidden", + } { + path := filepath.Join(root, filepath.FromSlash(name)) + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } + + source := `use ard/embed +let normal = embed::fs(["public"]) +let complete = embed::fs(["all:public"]) +fn read(path: Str) [Byte]!Error { normal.read_file(path) } +fn read_text(path: Str) Str!Error { normal.read_text(path) } +fn subset(path: Str) embed::FS!Error { normal.sub(path) } +` + checked := checkEmbedSource(t, root, source) + if checked.HasErrors() { + t.Fatalf("checker diagnostics: %#v", checked.Diagnostics()) + } + statements := checked.Module().Program().Statements + normal := statements[0].Stmt.(*checker.VariableDef).Value.(*checker.EmbeddedFSValue) + complete := statements[1].Stmt.(*checker.VariableDef).Value.(*checker.EmbeddedFSValue) + paths := func(set checker.EmbeddedFileSet) []string { + result := make([]string, len(set.Entries)) + for index, entry := range set.Entries { + result[index] = entry.LogicalPath + } + return result + } + if got := strings.Join(paths(normal.Set), ","); got != "public/app.js,public/index.html" { + t.Fatalf("normal embedded paths = %q", got) + } + if got := strings.Join(paths(complete.Set), ","); got != "public/.well-known/info.txt,public/app.js,public/index.html" { + t.Fatalf("complete embedded paths = %q", got) + } +} + +func TestEmbedResourcesResolveFromDependencyPackageRoot(t *testing.T) { + workspace := t.TempDir() + appRoot := filepath.Join(workspace, "app") + depRoot := filepath.Join(workspace, "dep") + for _, dir := range []string{appRoot, depRoot} { + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + } + files := map[string]string{ + filepath.Join(appRoot, "ard.toml"): "name = \"app\"\nard = \">= 0.1.0\"\n\n[dependencies]\ndep = { path = \"../dep\" }\n", + filepath.Join(appRoot, "asset.txt"): "consumer", + filepath.Join(depRoot, "ard.toml"): "name = \"dep\"\nard = \">= 0.1.0\"\n", + filepath.Join(depRoot, "asset.txt"): "dependency", + filepath.Join(depRoot, "dep.ard"): "use ard/embed\nfn value() Str { embed::text(\"asset.txt\") }\n", + } + for path, content := range files { + if err := os.WriteFile(path, []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } + checked := checkEmbedSource(t, appRoot, "use dep\nfn main() Str { dep::value() }\n") + if checked.HasErrors() { + t.Fatalf("checker diagnostics: %#v", checked.Diagnostics()) + } + var dependency checker.Module + for path, imported := range checked.Module().Program().Imports { + if strings.HasSuffix(path, "/dep") { + dependency = imported + break + } + } + if dependency == nil { + t.Fatalf("dependency imports = %#v", checked.Module().Program().Imports) + } + function := dependency.Program().Statements[0].Expr.(*checker.FunctionDef) + resource := function.Body.Stmts[0].Expr.(*checker.EmbeddedText).Resource + if string(resource.Data) != "dependency" || resource.OwnerPackageIdentity != "dep" { + t.Fatalf("dependency embedded resource = %#v", resource) + } +} + +func TestEmbedFSIsNotAValidMapKey(t *testing.T) { + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "ard.toml"), []byte("name = \"app\"\nard = \">= 0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "asset.txt"), []byte("asset"), 0o644); err != nil { + t.Fatal(err) + } + checked := checkEmbedSource(t, root, "use ard/embed\nlet files = embed::fs([\"asset.txt\"])\nlet table: [embed::FS:Str] = [files: \"value\"]\n") + if !checked.HasErrors() { + t.Fatal("embed::FS map key was accepted") + } +} + +func TestEmbedFSRejectsDynamicAndUnmatchedPatterns(t *testing.T) { + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "ard.toml"), []byte("name = \"app\"\nard = \">= 0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "asset.txt"), []byte("asset"), 0o644); err != nil { + t.Fatal(err) + } + dynamic := checkEmbedSource(t, root, "use ard/embed\nlet patterns = [\"asset.txt\"]\nlet files = embed::fs(patterns)\n") + if !hasEmbedDiagnostic(dynamic, checker.DiagnosticCodeEmbedStaticArgument) { + t.Fatalf("missing dynamic-pattern diagnostic: %#v", dynamic.Diagnostics()) + } + unmatched := checkEmbedSource(t, root, "use ard/embed\nlet files = embed::fs([\"missing/*\"])\n") + if !hasEmbedDiagnostic(unmatched, checker.DiagnosticCodeEmbedResource) { + t.Fatalf("missing unmatched-pattern diagnostic: %#v", unmatched.Diagnostics()) + } + malformed := checkEmbedSource(t, root, "use ard/embed\nlet files = embed::fs([\"[bad\"])\n") + if !hasEmbedDiagnostic(malformed, checker.DiagnosticCodeEmbedResource) { + t.Fatalf("missing malformed-pattern diagnostic: %#v", malformed.Diagnostics()) + } +} diff --git a/compiler/checker/module_resolver.go b/compiler/checker/module_resolver.go index 868790f8..27e6a0e7 100644 --- a/compiler/checker/module_resolver.go +++ b/compiler/checker/module_resolver.go @@ -93,9 +93,13 @@ type ModuleResolver struct { goImportAliases map[string]goImportAliasesResult buildModule Module - embedMu sync.Mutex - embeddedResources map[string]EmbeddedResource - embeddedResourceBytes int + embedMu sync.Mutex + embeddedResources map[string]EmbeddedResource + embeddedPatternSets map[string]EmbeddedFileSet + embeddedExactReferences map[string]bool + embeddedSetIdentities map[string]bool + embeddedProgramBytes int + embeddedProgramFileCount int } type goImportAliasesResult struct { @@ -1187,15 +1191,18 @@ func NewModuleResolverWithOptions(workingDir string, options BuildOptions) (*Mod } return &ModuleResolver{ - project: project, - moduleCache: make(map[string]Module), - astCache: make(map[string]*parse.Program), - overlays: make(map[string]string), - loadingChain: make([]string, 0), - modulePackages: make(map[string]string), - goImportAliases: make(map[string]goImportAliasesResult), - buildModule: newBuildModule(values), - embeddedResources: make(map[string]EmbeddedResource), + project: project, + moduleCache: make(map[string]Module), + astCache: make(map[string]*parse.Program), + overlays: make(map[string]string), + loadingChain: make([]string, 0), + modulePackages: make(map[string]string), + goImportAliases: make(map[string]goImportAliasesResult), + buildModule: newBuildModule(values), + embeddedResources: make(map[string]EmbeddedResource), + embeddedPatternSets: make(map[string]EmbeddedFileSet), + embeddedExactReferences: make(map[string]bool), + embeddedSetIdentities: make(map[string]bool), }, nil } diff --git a/compiler/checker/nodes.go b/compiler/checker/nodes.go index 0da149b3..e09510f7 100644 --- a/compiler/checker/nodes.go +++ b/compiler/checker/nodes.go @@ -52,6 +52,41 @@ type EmbeddedBytes struct { func (e *EmbeddedBytes) Type() Type { return MakeList(Byte) } +type EmbeddedSetEntry struct { + LogicalPath string + Data []byte +} + +type EmbeddedFileSet struct { + OwnerPackageIdentity string + Entries []EmbeddedSetEntry +} + +type EmbeddedFSValue struct { + Set EmbeddedFileSet +} + +func (e *EmbeddedFSValue) Type() Type { return EmbeddedFS } + +type EmbeddedFSMethodKind uint8 + +const ( + EmbeddedFSReadFile EmbeddedFSMethodKind = iota + EmbeddedFSReadText + EmbeddedFSReadDir + EmbeddedFSStat + EmbeddedFSSub +) + +type EmbeddedFSMethod struct { + Subject Expression + Kind EmbeddedFSMethodKind + Args []Expression + ReturnType Type +} + +func (e *EmbeddedFSMethod) Type() Type { return e.ReturnType } + type RuneLiteral struct { Value rune } diff --git a/compiler/checker/std_lib.go b/compiler/checker/std_lib.go index 69147246..71b03a79 100644 --- a/compiler/checker/std_lib.go +++ b/compiler/checker/std_lib.go @@ -195,7 +195,7 @@ var BuiltinPkgNames = map[string][]string{ "ard/result": {"ok", "err"}, "ard/async": {"start"}, "ard/unsafe": {"cast", "is_nil"}, - EmbedModulePath: {"text", "bytes"}, + EmbedModulePath: {"text", "bytes", "fs", "FS", "DirEntry", "FileInfo"}, "builtin/Chan": {"new"}, } diff --git a/compiler/go/backend.go b/compiler/go/backend.go index 0c484ec7..1091f59f 100644 --- a/compiler/go/backend.go +++ b/compiler/go/backend.go @@ -87,13 +87,56 @@ func GenerateSources(program *air.Program, options Options) (map[string][]byte, func generateEmbeddedResourceSource(program *air.Program) ([]byte, error) { var source strings.Builder - source.WriteString("package ardembed\n\nimport _ \"embed\"\n\n") + source.WriteString("package ardembed\n\n") + if len(program.EmbeddedSets) > 0 { + source.WriteString("import (\n\t\"embed\"\n\t\"fmt\"\n\t\"io/fs\"\n\t\"unicode/utf8\"\n)\n\n") + } else { + source.WriteString("import _ \"embed\"\n\n") + } for _, blob := range program.EmbeddedBlobs { + if !blob.Direct { + continue + } fmt.Fprintf(&source, "//go:embed data/%s\n", blob.Digest) fmt.Fprintf(&source, "var blob%s string\n\n", blob.Digest) fmt.Fprintf(&source, "func Text%s() string { return blob%s }\n\n", blob.Digest, blob.Digest) fmt.Fprintf(&source, "func Bytes%s() []byte { return []byte(blob%s) }\n\n", blob.Digest, blob.Digest) } + for _, set := range program.EmbeddedSets { + fmt.Fprintf(&source, "//go:embed all:sets/%s\n", set.Digest) + fmt.Fprintf(&source, "var raw%s embed.FS\n\n", set.Digest) + fmt.Fprintf(&source, "var set%s fs.FS = mustSub(raw%s, \"sets/%s\")\n\n", set.Digest, set.Digest, set.Digest) + fmt.Fprintf(&source, "func FS%s() fs.FS { return set%s }\n\n", set.Digest, set.Digest) + } + if len(program.EmbeddedSets) > 0 { + source.WriteString(`func mustSub(root fs.FS, path string) fs.FS { + value, err := fs.Sub(root, path) + if err != nil { panic(err) } + return value +} + +func ReadFile(root fs.FS, path string) ([]byte, error) { + data, err := fs.ReadFile(root, path) + if err != nil { return nil, err } + return append([]byte(nil), data...), nil +} + +func ReadText(root fs.FS, path string) (string, error) { + data, err := fs.ReadFile(root, path) + if err != nil { return "", err } + if !utf8.Valid(data) { return "", &fs.PathError{Op: "read_text", Path: path, Err: fs.ErrInvalid} } + return string(data), nil +} + +func Sub(root fs.FS, path string) (fs.FS, error) { + info, err := fs.Stat(root, path) + if err != nil { return nil, err } + if !info.IsDir() { return nil, &fs.PathError{Op: "sub", Path: path, Err: fmt.Errorf("%w: not a directory", fs.ErrInvalid)} } + return fs.Sub(root, path) +} + +`) + } formatted, err := format.Source([]byte(source.String())) if err != nil { return nil, fmt.Errorf("format embedded resource package: %w", err) @@ -1214,6 +1257,9 @@ func writeEmbeddedResources(outputDir string, program *air.Program) error { } dataDir := filepath.Join(outputDir, "internal", "ardembed", "data") for _, blob := range program.EmbeddedBlobs { + if !blob.Direct { + continue + } if err := os.MkdirAll(dataDir, 0o755); err != nil { return fmt.Errorf("create embedded resource directory: %w", err) } @@ -1221,6 +1267,25 @@ func writeEmbeddedResources(outputDir string, program *air.Program) error { return fmt.Errorf("write embedded resource %s: %w", blob.Digest, err) } } + for _, set := range program.EmbeddedSets { + setRoot := filepath.Join(outputDir, "internal", "ardembed", "sets", set.Digest) + for _, entry := range set.Entries { + if entry.Blob < 0 || int(entry.Blob) >= len(program.EmbeddedBlobs) { + return fmt.Errorf("embedded set %s references invalid blob %d", set.Digest, entry.Blob) + } + resourcePath := filepath.Join(setRoot, filepath.FromSlash(entry.Path)) + rel, err := filepath.Rel(setRoot, resourcePath) + if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) || filepath.IsAbs(rel) { + return fmt.Errorf("embedded set resource escapes staging root: %q", entry.Path) + } + if err := os.MkdirAll(filepath.Dir(resourcePath), 0o755); err != nil { + return fmt.Errorf("create embedded set directory: %w", err) + } + if err := os.WriteFile(resourcePath, program.EmbeddedBlobs[entry.Blob].Data, 0o644); err != nil { + return fmt.Errorf("write embedded set resource %s: %w", entry.Path, err) + } + } + } return nil } diff --git a/compiler/go/embed_test.go b/compiler/go/embed_test.go index 01cff09e..19cce954 100644 --- a/compiler/go/embed_test.go +++ b/compiler/go/embed_test.go @@ -151,3 +151,78 @@ func TestWriteProgramStagesEmbeddedBlobs(t *testing.T) { t.Fatalf("build generated embedded program: %v", err) } } + +func TestEmbeddedFSReadsFilesAndSubdirectories(t *testing.T) { + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "ard.toml"), []byte("name = \"app\"\nard = \">= 0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + assetPath := filepath.Join(root, "public", "index.html") + if err := os.MkdirAll(filepath.Dir(assetPath), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(assetPath, []byte("

Hello

\n"), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "public", "binary.bin"), []byte{0xff, 1}, 0o644); err != nil { + t.Fatal(err) + } + mainPath := filepath.Join(root, "main.ard") + source := `use ard/embed +use go:io/fs as gofs + +let assets = embed::fs(["public"]) + +fn main() { + let text = assets.read_text("public/index.html").expect("read text") + if text != "

Hello

\n" { panic("bad text") } + let bytes = assets.read_file("public/index.html").expect("read bytes") + if bytes.size() != 15 { panic("bad bytes") } + let changed = mut assets.read_file("public/index.html").expect("mutable bytes") + changed.set(0, Byte::from(0)) + if assets.read_file("public/index.html").expect("fresh bytes").at(0).or(Byte::from(0)) != 60 { panic("read bytes shared storage") } + if gofs::ReadFile(assets, "public/index.html").expect("direct io/fs").size() != 15 { panic("bad io/fs bridge") } + if not assets.read_file("missing").is_err() { panic("missing file succeeded") } + if not assets.read_text("public/binary.bin").is_err() { panic("invalid UTF-8 succeeded") } + if not assets.sub("public/index.html").is_err() { panic("file sub succeeded") } + let entries = assets.read_dir("public").expect("read dir") + if entries.size() != 2 { panic("bad directory size") } + let entry = entries.at(1).expect("directory entry") + if entry.name != "index.html" or entry.is_dir { panic("bad directory entry") } + let info = assets.stat("public/index.html").expect("stat") + if info.name != "index.html" or info.is_dir or info.size.or(0) != 15 { panic("bad file info") } + if assets.stat(".").expect("root stat").name != "." { panic("bad root name") } + let public = assets.sub("public").expect("sub") + if public.stat(".").expect("sub root stat").name != "." { panic("bad sub root name") } + if public.read_text("index.html").expect("sub read") != text { panic("bad sub") } +} +` + parsed := parse.Parse([]byte(source), mainPath) + if len(parsed.Errors) > 0 { + t.Fatalf("parse errors: %v", parsed.Errors) + } + resolver, err := checker.NewModuleResolver(root) + if err != nil { + t.Fatal(err) + } + checked := checker.New(mainPath, parsed.Program, resolver) + checked.Check() + if checked.HasErrors() { + t.Fatalf("checker diagnostics: %#v", checked.Diagnostics()) + } + program, err := air.Lower(checked.Module()) + if err != nil { + t.Fatalf("Lower: %v", err) + } + if err := os.RemoveAll(filepath.Join(root, "public")); err != nil { + t.Fatal(err) + } + output := filepath.Join(t.TempDir(), "embedded-fs-program") + built, err := BuildProgram(program, output, resolver.GetProjectInfo()) + if err != nil { + t.Fatalf("BuildProgram: %v", err) + } + if output, err := exec.Command(built).CombinedOutput(); err != nil { + t.Fatalf("embedded FS program failed: %v\n%s", err, output) + } +} diff --git a/compiler/go/lower.go b/compiler/go/lower.go index 289118ce..f4392d91 100644 --- a/compiler/go/lower.go +++ b/compiler/go/lower.go @@ -2439,6 +2439,16 @@ func (l *lowerer) lowerExpr(fn air.Function, expr air.Expr) (loweredExpr, error) } accessor := l.qualified("ardembed", path.Join(l.generatedModulePath, "internal", "ardembed"), prefix+blob.Digest) return loweredExpr{expr: &ast.CallExpr{Fun: accessor}}, nil + case air.ExprMakeEmbeddedFS: + payload := expr.EmbeddedSetPayload() + if payload == nil || payload.Set < 0 || int(payload.Set) >= len(l.program.EmbeddedSets) { + return loweredExpr{}, fmt.Errorf("embedded filesystem expression references invalid set") + } + set := l.program.EmbeddedSets[payload.Set] + accessor := l.qualified("ardembed", path.Join(l.generatedModulePath, "internal", "ardembed"), "FS"+set.Digest) + return loweredExpr{expr: &ast.CallExpr{Fun: accessor}}, nil + case air.ExprEmbeddedFSReadFile, air.ExprEmbeddedFSReadText, air.ExprEmbeddedFSReadDir, air.ExprEmbeddedFSStat, air.ExprEmbeddedFSSub: + return l.lowerEmbeddedFSOperation(fn, expr) case air.ExprPanic: if expr.Target == nil { return loweredExpr{}, fmt.Errorf("panic missing target") @@ -4152,6 +4162,167 @@ func (l *lowerer) lowerForeignMethodCall(fn air.Function, expr air.Expr) (lowere return loweredExpr{stmts: stmts, expr: call}, nil } +func (l *lowerer) lowerEmbeddedFSOperation(fn air.Function, expr air.Expr) (loweredExpr, error) { + if expr.Target == nil || len(expr.Args) != 1 { + return loweredExpr{}, fmt.Errorf("embedded filesystem operation has invalid operands") + } + target, err := l.lowerExpr(fn, *expr.Target) + if err != nil { + return loweredExpr{}, err + } + arg, err := l.lowerExpr(fn, expr.Args[0]) + if err != nil { + return loweredExpr{}, err + } + stmts := append(append([]ast.Stmt{}, target.stmts...), arg.stmts...) + var function ast.Expr + switch expr.Kind { + case air.ExprEmbeddedFSReadFile: + function = l.qualified("ardembed", path.Join(l.generatedModulePath, "internal", "ardembed"), "ReadFile") + case air.ExprEmbeddedFSReadText: + function = l.qualified("ardembed", path.Join(l.generatedModulePath, "internal", "ardembed"), "ReadText") + case air.ExprEmbeddedFSReadDir: + return l.lowerEmbeddedFSReadDir(expr, stmts, target.expr, arg.expr) + case air.ExprEmbeddedFSStat: + stmts, pathValue := l.materializeCallOperand(stmts, arg.expr) + return l.lowerEmbeddedFSStat(expr, stmts, target.expr, pathValue) + case air.ExprEmbeddedFSSub: + function = l.qualified("ardembed", path.Join(l.generatedModulePath, "internal", "ardembed"), "Sub") + default: + return loweredExpr{}, fmt.Errorf("unsupported embedded filesystem operation %d", expr.Kind) + } + call := &ast.CallExpr{Fun: function, Args: []ast.Expr{target.expr, arg.expr}} + result, ok := l.typeInfo(expr.Type) + if !ok || result.Kind != air.TypeResult { + return loweredExpr{}, fmt.Errorf("embedded filesystem operation has non-Result type %d", expr.Type) + } + return l.lowerGoValueErrorResultCall(expr, stmts, call, result) +} + +func (l *lowerer) lowerEmbeddedFSReadDir(expr air.Expr, stmts []ast.Stmt, root ast.Expr, pathExpr ast.Expr) (loweredExpr, error) { + result, ok := l.typeInfo(expr.Type) + if !ok || result.Kind != air.TypeResult || !validTypeID(l.program, result.Value) { + return loweredExpr{}, fmt.Errorf("embedded read_dir has invalid Result type") + } + listInfo := l.program.Types[result.Value-1] + if listInfo.Kind != air.TypeList || !validTypeID(l.program, listInfo.Elem) { + return loweredExpr{}, fmt.Errorf("embedded read_dir has invalid list type") + } + resultName, rawName, errName, valueName := l.nextTemp(), l.nextTemp(), l.nextTemp(), l.nextTemp() + resultType, err := l.goType(expr.Type) + if err != nil { + return loweredExpr{}, err + } + valueType, err := l.goType(result.Value) + if err != nil { + return loweredExpr{}, err + } + entryType, err := l.goType(listInfo.Elem) + if err != nil { + return loweredExpr{}, err + } + fsDirEntry := l.qualified("fs", "io/fs", "DirEntry") + readDir := l.qualified("fs", "io/fs", "ReadDir") + stmts = append(stmts, + &ast.DeclStmt{Decl: &ast.GenDecl{Tok: token.VAR, Specs: []ast.Spec{&ast.ValueSpec{Names: []*ast.Ident{l.ident(resultName)}, Type: resultType}}}}, + &ast.DeclStmt{Decl: &ast.GenDecl{Tok: token.VAR, Specs: []ast.Spec{&ast.ValueSpec{Names: []*ast.Ident{l.ident(rawName)}, Type: &ast.ArrayType{Elt: fsDirEntry}}}}}, + &ast.DeclStmt{Decl: &ast.GenDecl{Tok: token.VAR, Specs: []ast.Spec{&ast.ValueSpec{Names: []*ast.Ident{l.ident(errName)}, Type: l.ident("error")}}}}, + &ast.AssignStmt{Lhs: []ast.Expr{l.ident(rawName), l.ident(errName)}, Tok: token.ASSIGN, Rhs: []ast.Expr{&ast.CallExpr{Fun: readDir, Args: []ast.Expr{root, pathExpr}}}}, + ) + errResult := &ast.CompositeLit{Type: resultType, Elts: []ast.Expr{&ast.KeyValueExpr{Key: l.ident("Err"), Value: l.ident(errName)}}} + successResult := &ast.CompositeLit{Type: resultType, Elts: []ast.Expr{ + &ast.KeyValueExpr{Key: l.ident("Value"), Value: l.ident(valueName)}, + &ast.KeyValueExpr{Key: l.ident("Ok"), Value: l.ident("true")}, + }} + indexName, entryName := l.nextTemp(), l.nextTemp() + entryValue := &ast.CompositeLit{Type: entryType, Elts: []ast.Expr{ + &ast.KeyValueExpr{Key: l.ident("Name"), Value: &ast.CallExpr{Fun: &ast.SelectorExpr{X: l.ident(entryName), Sel: l.ident("Name")}}}, + &ast.KeyValueExpr{Key: l.ident("IsDir"), Value: &ast.CallExpr{Fun: &ast.SelectorExpr{X: l.ident(entryName), Sel: l.ident("IsDir")}}}, + }} + successBody := []ast.Stmt{ + &ast.AssignStmt{Lhs: []ast.Expr{l.ident(valueName)}, Tok: token.DEFINE, Rhs: []ast.Expr{&ast.CallExpr{Fun: l.ident("make"), Args: []ast.Expr{valueType, &ast.CallExpr{Fun: l.ident("len"), Args: []ast.Expr{l.ident(rawName)}}}}}}, + &ast.RangeStmt{Key: l.ident(indexName), Value: l.ident(entryName), Tok: token.DEFINE, X: l.ident(rawName), Body: &ast.BlockStmt{List: []ast.Stmt{ + &ast.AssignStmt{Lhs: []ast.Expr{&ast.IndexExpr{X: l.ident(valueName), Index: l.ident(indexName)}}, Tok: token.ASSIGN, Rhs: []ast.Expr{entryValue}}, + }}}, + &ast.AssignStmt{Lhs: []ast.Expr{l.ident(resultName)}, Tok: token.ASSIGN, Rhs: []ast.Expr{successResult}}, + } + stmts = append(stmts, &ast.IfStmt{ + Cond: &ast.BinaryExpr{X: l.ident(errName), Op: token.NEQ, Y: l.ident("nil")}, + Body: &ast.BlockStmt{List: []ast.Stmt{&ast.AssignStmt{Lhs: []ast.Expr{l.ident(resultName)}, Tok: token.ASSIGN, Rhs: []ast.Expr{errResult}}}}, + Else: &ast.BlockStmt{List: successBody}, + }) + return loweredExpr{stmts: stmts, expr: l.ident(resultName)}, nil +} + +func (l *lowerer) lowerEmbeddedFSStat(expr air.Expr, stmts []ast.Stmt, root ast.Expr, pathExpr ast.Expr) (loweredExpr, error) { + result, ok := l.typeInfo(expr.Type) + if !ok || result.Kind != air.TypeResult || !validTypeID(l.program, result.Value) { + return loweredExpr{}, fmt.Errorf("embedded stat has invalid Result type") + } + resultName, rawName, errName, valueName, sizeName, nameName := l.nextTemp(), l.nextTemp(), l.nextTemp(), l.nextTemp(), l.nextTemp(), l.nextTemp() + resultType, err := l.goType(expr.Type) + if err != nil { + return loweredExpr{}, err + } + valueType, err := l.goType(result.Value) + if err != nil { + return loweredExpr{}, err + } + fileInfo := l.qualified("fs", "io/fs", "FileInfo") + stat := l.qualified("fs", "io/fs", "Stat") + valueInfo := l.program.Types[result.Value-1] + var sizeTypeID air.TypeID + for _, field := range valueInfo.Fields { + if field.Name == "size" { + sizeTypeID = field.Type + break + } + } + if !validTypeID(l.program, sizeTypeID) { + return loweredExpr{}, fmt.Errorf("embedded stat FileInfo has no size field") + } + maybeInt, err := l.goType(sizeTypeID) + if err != nil { + return loweredExpr{}, err + } + stmts = append(stmts, + &ast.DeclStmt{Decl: &ast.GenDecl{Tok: token.VAR, Specs: []ast.Spec{&ast.ValueSpec{Names: []*ast.Ident{l.ident(resultName)}, Type: resultType}}}}, + &ast.DeclStmt{Decl: &ast.GenDecl{Tok: token.VAR, Specs: []ast.Spec{&ast.ValueSpec{Names: []*ast.Ident{l.ident(rawName)}, Type: fileInfo}}}}, + &ast.DeclStmt{Decl: &ast.GenDecl{Tok: token.VAR, Specs: []ast.Spec{&ast.ValueSpec{Names: []*ast.Ident{l.ident(errName)}, Type: l.ident("error")}}}}, + &ast.AssignStmt{Lhs: []ast.Expr{l.ident(rawName), l.ident(errName)}, Tok: token.ASSIGN, Rhs: []ast.Expr{&ast.CallExpr{Fun: stat, Args: []ast.Expr{root, pathExpr}}}}, + ) + errResult := &ast.CompositeLit{Type: resultType, Elts: []ast.Expr{&ast.KeyValueExpr{Key: l.ident("Err"), Value: l.ident(errName)}}} + isDirCall := func() ast.Expr { + return &ast.CallExpr{Fun: &ast.SelectorExpr{X: l.ident(rawName), Sel: l.ident("IsDir")}} + } + successResult := &ast.CompositeLit{Type: resultType, Elts: []ast.Expr{ + &ast.KeyValueExpr{Key: l.ident("Value"), Value: l.ident(valueName)}, + &ast.KeyValueExpr{Key: l.ident("Ok"), Value: l.ident("true")}, + }} + successBody := []ast.Stmt{ + &ast.DeclStmt{Decl: &ast.GenDecl{Tok: token.VAR, Specs: []ast.Spec{&ast.ValueSpec{Names: []*ast.Ident{l.ident(sizeName)}, Type: maybeInt}}}}, + &ast.AssignStmt{Lhs: []ast.Expr{l.ident(nameName)}, Tok: token.DEFINE, Rhs: []ast.Expr{&ast.CallExpr{Fun: &ast.SelectorExpr{X: l.ident(rawName), Sel: l.ident("Name")}}}}, + &ast.IfStmt{Cond: &ast.BinaryExpr{X: pathExpr, Op: token.EQL, Y: &ast.BasicLit{Kind: token.STRING, Value: `"."`}}, Body: &ast.BlockStmt{List: []ast.Stmt{ + &ast.AssignStmt{Lhs: []ast.Expr{l.ident(nameName)}, Tok: token.ASSIGN, Rhs: []ast.Expr{&ast.BasicLit{Kind: token.STRING, Value: `"."`}}}, + }}}, + &ast.IfStmt{Cond: &ast.UnaryExpr{Op: token.NOT, X: isDirCall()}, Body: &ast.BlockStmt{List: []ast.Stmt{ + &ast.AssignStmt{Lhs: []ast.Expr{l.ident(sizeName)}, Tok: token.ASSIGN, Rhs: []ast.Expr{&ast.CallExpr{Fun: l.runtimeQualified("Some"), Args: []ast.Expr{&ast.CallExpr{Fun: l.ident("int"), Args: []ast.Expr{&ast.CallExpr{Fun: &ast.SelectorExpr{X: l.ident(rawName), Sel: l.ident("Size")}}}}}}}}, + }}}, + &ast.AssignStmt{Lhs: []ast.Expr{l.ident(valueName)}, Tok: token.DEFINE, Rhs: []ast.Expr{&ast.CompositeLit{Type: valueType, Elts: []ast.Expr{ + &ast.KeyValueExpr{Key: l.ident("Name"), Value: l.ident(nameName)}, + &ast.KeyValueExpr{Key: l.ident("IsDir"), Value: isDirCall()}, + &ast.KeyValueExpr{Key: l.ident("Size"), Value: l.ident(sizeName)}, + }}}}, + &ast.AssignStmt{Lhs: []ast.Expr{l.ident(resultName)}, Tok: token.ASSIGN, Rhs: []ast.Expr{successResult}}, + } + stmts = append(stmts, &ast.IfStmt{ + Cond: &ast.BinaryExpr{X: l.ident(errName), Op: token.NEQ, Y: l.ident("nil")}, + Body: &ast.BlockStmt{List: []ast.Stmt{&ast.AssignStmt{Lhs: []ast.Expr{l.ident(resultName)}, Tok: token.ASSIGN, Rhs: []ast.Expr{errResult}}}}, + Else: &ast.BlockStmt{List: successBody}, + }) + return loweredExpr{stmts: stmts, expr: l.ident(resultName)}, nil +} + func (l *lowerer) lowerGoValueErrorResultCall(expr air.Expr, stmts []ast.Stmt, call *ast.CallExpr, result air.TypeInfo) (loweredExpr, error) { resultTemp := l.nextTemp() valueTemp := l.nextTemp() @@ -4318,7 +4489,7 @@ func (l *lowerer) zeroValueExpr(typeID air.TypeID) (ast.Expr, error) { return l.ident("false"), nil case air.TypeStr: return &ast.BasicLit{Kind: token.STRING, Value: "\"\""}, nil - case air.TypeAny, air.TypeFunction, air.TypeTraitObject, air.TypeReference: + case air.TypeAny, air.TypeFunction, air.TypeTraitObject, air.TypeReference, air.TypeEmbeddedFS: return l.ident("nil"), nil case air.TypeParam: // A composite literal T{} is illegal for a type parameter; *new(T) @@ -4666,6 +4837,8 @@ func (l *lowerer) buildGoType(typeID air.TypeID) (ast.Expr, error) { return l.ident("bool"), nil case air.TypeStr: return l.ident("string"), nil + case air.TypeEmbeddedFS: + return l.qualified("fs", "io/fs", "FS"), nil case air.TypeMaybe: elem, err := l.goType(info.Elem) if err != nil { diff --git a/compiler/lsp/analysis/engine.go b/compiler/lsp/analysis/engine.go index f8b59239..34afca35 100644 --- a/compiler/lsp/analysis/engine.go +++ b/compiler/lsp/analysis/engine.go @@ -18,6 +18,8 @@ import ( "crypto/sha256" "encoding/hex" "fmt" + "hash" + "io" "os" "path/filepath" "runtime/debug" @@ -644,8 +646,67 @@ func (s *Snapshot) check(filePath string, relPath string, program *parse.Program // aliases and package deps participate. Check results are reusable across // snapshots that did not touch any input of this file. // -// Standard-library imports (ard/*) are skipped: the stdlib is embedded in the -// compiler binary and immutable for the lifetime of the LSP process. +// Standard-library imports (ard/*) are skipped because they are embedded in the +// compiler binary. `ard/embed` is the exception: importing it adds the owning +// package's resource tree to the signature. +func hashEmbeddedPackageTree(h hash.Hash, root string) { + if root == "" { + return + } + _ = filepath.WalkDir(root, func(filePath string, entry os.DirEntry, walkErr error) error { + if walkErr != nil { + h.Write([]byte(filePath)) + h.Write([]byte(":unreadable\x00")) + return nil + } + if filePath == root { + return nil + } + if entry.IsDir() { + switch entry.Name() { + case ".bzr", ".git", ".hg", ".svn", "ard-out", "vendor": + return filepath.SkipDir + } + if _, err := os.Stat(filepath.Join(filePath, "ard.toml")); err == nil { + return filepath.SkipDir + } + if _, err := os.Stat(filepath.Join(filePath, "go.mod")); err == nil { + return filepath.SkipDir + } + return nil + } + rel, err := filepath.Rel(root, filePath) + if err != nil { + return nil + } + h.Write([]byte(filepath.ToSlash(rel))) + h.Write([]byte{0}) + if entry.Type()&os.ModeSymlink != 0 { + if target, err := os.Readlink(filePath); err == nil { + h.Write([]byte("symlink:" + target)) + } + h.Write([]byte{0}) + return nil + } + if !entry.Type().IsRegular() { + h.Write([]byte(":irregular\x00")) + return nil + } + file, err := os.Open(filePath) + if err != nil { + h.Write([]byte(":unreadable\x00")) + return nil + } + _, copyErr := io.Copy(h, io.LimitReader(file, checker.MaxEmbeddedFileBytes+1)) + closeErr := file.Close() + if copyErr != nil || closeErr != nil { + h.Write([]byte(":read-error")) + } + h.Write([]byte{0}) + return nil + }) +} + func (s *Snapshot) signature(filePath string, content []byte, program *parse.Program, moduleResolver *checker.ModuleResolver, relPath string, trackDependencies bool) string { h := sha256.New() h.Write([]byte(filePath)) @@ -655,6 +716,7 @@ func (s *Snapshot) signature(filePath string, content []byte, program *parse.Pro seen := map[string]bool{filePath: true} seenPackageManifests := map[string]bool{} + seenEmbedPackages := map[string]bool{} projectInfo := moduleResolver.GetProjectInfo() if projectInfo != nil { seenPackageManifests[projectInfo.RootPackageID] = true @@ -678,8 +740,8 @@ func (s *Snapshot) signature(filePath string, content []byte, program *parse.Pro } h.Write([]byte{0}) } - var visit func(prog *parse.Program, importerModulePath string, importerFilePath string) - visit = func(prog *parse.Program, importerModulePath string, importerFilePath string) { + var visit func(prog *parse.Program, importerModulePath string, importerFilePath string, importerPackageID string) + visit = func(prog *parse.Program, importerModulePath string, importerFilePath string, importerPackageID string) { if prog == nil { return } @@ -690,7 +752,11 @@ func (s *Snapshot) signature(filePath string, content []byte, program *parse.Pro } deps := make([]dep, 0, len(prog.Imports)) known := true + usesEmbed := false for _, imp := range prog.Imports { + if imp.Path == checker.EmbedModulePath { + usesEmbed = true + } if imp.Kind == parse.ImportKindGo || strings.HasPrefix(imp.Path, "ard/") { continue } @@ -706,6 +772,12 @@ func (s *Snapshot) signature(filePath string, content []byte, program *parse.Pro } deps = append(deps, dep{file: resolved.FilePath, module: resolved.ModulePath, packageID: resolved.PackageID}) } + if usesEmbed && projectInfo != nil && !seenEmbedPackages[importerPackageID] { + seenEmbedPackages[importerPackageID] = true + if pkg, ok := projectInfo.Packages[importerPackageID]; ok { + hashEmbeddedPackageTree(h, pkg.RootPath) + } + } sort.Slice(deps, func(a, b int) bool { return deps[a].file < deps[b].file }) if trackDependencies { imports := make([]string, 0, len(deps)) @@ -741,10 +813,14 @@ func (s *Snapshot) signature(filePath string, content []byte, program *parse.Pro } continue } - visit(entry.program, d.module, d.file) + visit(entry.program, d.module, d.file, d.packageID) } } - visit(program, strings.TrimSuffix(relPath, ".ard"), filePath) + rootPackageID := "" + if projectInfo != nil { + rootPackageID = projectInfo.RootPackageID + } + visit(program, strings.TrimSuffix(relPath, ".ard"), filePath, rootPackageID) // Project manifest and Go module metadata participate so dependency and // FFI configuration changes invalidate checks. diff --git a/compiler/lsp/analysis/engine_test.go b/compiler/lsp/analysis/engine_test.go index 8138092a..120e4364 100644 --- a/compiler/lsp/analysis/engine_test.go +++ b/compiler/lsp/analysis/engine_test.go @@ -833,3 +833,44 @@ func TestGoSessionRepricesForNewImports(t *testing.T) { t.Error("expected a resolve diagnostic for the bogus import path") } } + +func TestEmbeddedResourceChangeInvalidatesAnalysis(t *testing.T) { + root := writeProject(t, map[string]string{ + "ard.toml": "name = \"proj\"\nard = \">= 0.1.0\"\n", + "main.ard": "use ard/embed\nlet page = embed::text(\"assets/page.txt\")\n", + "assets/page.txt": "first", + }) + engine := NewEngine(root) + workspace := NewWorkspace(engine) + mainPath := filepath.Join(root, "main.ard") + resourcePath := filepath.Join(root, "assets", "page.txt") + + first, err := workspace.Snapshot().Analyze(mainPath) + if err != nil { + t.Fatal(err) + } + if len(first.Diagnostics) != 0 { + t.Fatalf("initial diagnostics: %#v", first.Diagnostics) + } + if err := os.WriteFile(resourcePath, []byte("second"), 0o644); err != nil { + t.Fatal(err) + } + second, err := workspace.Snapshot().Analyze(mainPath) + if err != nil { + t.Fatal(err) + } + if first == second || first.Signature == second.Signature { + t.Fatal("embedded resource edit did not invalidate analysis") + } + + if err := os.Remove(resourcePath); err != nil { + t.Fatal(err) + } + missing, err := workspace.Snapshot().Analyze(mainPath) + if err != nil { + t.Fatal(err) + } + if len(missing.Diagnostics) == 0 { + t.Fatal("missing embedded resource did not produce diagnostics") + } +} diff --git a/compiler/lsp/server.go b/compiler/lsp/server.go index f59ea766..73bb4040 100644 --- a/compiler/lsp/server.go +++ b/compiler/lsp/server.go @@ -9,6 +9,7 @@ import ( "os" "path/filepath" "runtime/debug" + "sort" "strings" "sync" "time" @@ -17,6 +18,7 @@ import ( "go.lsp.dev/protocol" "go.lsp.dev/uri" + "github.com/akonwi/ard/checker" "github.com/akonwi/ard/lsp/analysis" ) @@ -43,10 +45,11 @@ type diagnosticJob struct { // Server is the Ard LSP server. type Server struct { - cache *DocumentCache - handlers map[string]jsonrpc2.Handler - conn jsonrpc2.Conn - projectRoot string + cache *DocumentCache + handlers map[string]jsonrpc2.Handler + conn jsonrpc2.Conn + projectRoot string + watchFilesDynamicSupport bool // documentStateMu makes DocumentCache metadata and analysis workspace // overlays transition as one state for concurrent snapshot capture. @@ -232,7 +235,8 @@ func handleRequestInline(method string) bool { protocol.MethodTextDocumentDidOpen, protocol.MethodTextDocumentDidChange, protocol.MethodTextDocumentDidSave, - protocol.MethodTextDocumentDidClose: + protocol.MethodTextDocumentDidClose, + protocol.MethodWorkspaceDidChangeWatchedFiles: return true default: return false @@ -287,6 +291,7 @@ func (s *Server) registerHandlers() { s.handlers[protocol.MethodTextDocumentDidChange] = s.handleDidChange s.handlers[protocol.MethodTextDocumentDidSave] = s.handleDidSave s.handlers[protocol.MethodTextDocumentDidClose] = s.handleDidClose + s.handlers[protocol.MethodWorkspaceDidChangeWatchedFiles] = s.handleDidChangeWatchedFiles // Language features s.handlers[protocol.MethodTextDocumentHover] = s.handleHover @@ -320,6 +325,7 @@ func (s *Server) handleInitialize(ctx context.Context, reply jsonrpc2.Replier, r } else if params.RootURI != "" { s.projectRoot = string(params.RootURI) } + s.watchFilesDynamicSupport = params.Capabilities.Workspace != nil && params.Capabilities.Workspace.DidChangeWatchedFiles != nil && params.Capabilities.Workspace.DidChangeWatchedFiles.DynamicRegistration s.engineMu.Unlock() result := &protocol.InitializeResult{ @@ -351,6 +357,33 @@ func (s *Server) handleInitialize(ctx context.Context, reply jsonrpc2.Replier, r } func (s *Server) handleInitialized(ctx context.Context, reply jsonrpc2.Replier, req jsonrpc2.Request) error { + if s.watchFilesDynamicSupport && s.conn != nil { + watchers := []protocol.FileSystemWatcher{{GlobPattern: "**/*"}} + if root := s.projectRootPath(); root != "" { + if project, err := checker.FindProjectRoot(root); err == nil { + dependencyRoots := make([]string, 0, len(project.Packages)) + for _, pkg := range project.Packages { + if pkg.RootPath != "" && filepath.Clean(pkg.RootPath) != filepath.Clean(project.RootPath) { + dependencyRoots = append(dependencyRoots, pkg.RootPath) + } + } + sort.Strings(dependencyRoots) + for _, dependencyRoot := range dependencyRoots { + watchers = append(watchers, protocol.FileSystemWatcher{GlobPattern: filepath.ToSlash(filepath.Join(dependencyRoot, "**", "*"))}) + } + } + } + options := protocol.DidChangeWatchedFilesRegistrationOptions{Watchers: watchers} + params := protocol.RegistrationParams{Registrations: []protocol.Registration{{ + ID: "ard-embedded-resources", + Method: protocol.MethodWorkspaceDidChangeWatchedFiles, + RegisterOptions: options, + }}} + var result any + if _, err := s.conn.Call(ctx, protocol.MethodClientRegisterCapability, params, &result); err != nil { + fmt.Fprintf(os.Stderr, "ard-lsp: could not register embedded resource watcher: %v\n", err) + } + } return reply(ctx, nil, nil) } @@ -519,6 +552,17 @@ func (s *Server) handleDidSave(ctx context.Context, reply jsonrpc2.Replier, req return reply(ctx, nil, nil) } +func (s *Server) handleDidChangeWatchedFiles(ctx context.Context, reply jsonrpc2.Replier, req jsonrpc2.Request) error { + var params protocol.DidChangeWatchedFilesParams + if err := json.Unmarshal(req.Params(), ¶ms); err != nil { + return reply(ctx, nil, fmt.Errorf("%s: %w", jsonrpc2.ErrParse, err)) + } + if len(params.Changes) > 0 { + s.scheduleDiagnosticsForOpenDocuments() + } + return reply(ctx, nil, nil) +} + func (s *Server) handleDidClose(ctx context.Context, reply jsonrpc2.Replier, req jsonrpc2.Request) error { var params protocol.DidCloseTextDocumentParams if err := json.Unmarshal(req.Params(), ¶ms); err != nil { diff --git a/compiler/lsp/server_test.go b/compiler/lsp/server_test.go index baeef34a..31d6d097 100644 --- a/compiler/lsp/server_test.go +++ b/compiler/lsp/server_test.go @@ -49,6 +49,7 @@ func TestServerInitializes(t *testing.T) { "textDocument/didChange", "textDocument/didSave", "textDocument/didClose", + "workspace/didChangeWatchedFiles", "textDocument/hover", "textDocument/definition", "textDocument/references", diff --git a/docs/adrs/0071-add-compile-time-embedded-filesystems.md b/docs/adrs/0071-add-compile-time-embedded-filesystems.md index d616cd9b..5d2f2f7a 100644 --- a/docs/adrs/0071-add-compile-time-embedded-filesystems.md +++ b/docs/adrs/0071-add-compile-time-embedded-filesystems.md @@ -165,8 +165,10 @@ Patterns have these rules: directory. A `go.mod` at the owning package root does not block selection, but a selected logical file named `go.mod` at any depth is rejected because its generated copy would create a nested Go module boundary. -- Version-control directories `.bzr`, `.git`, `.hg`, and `.svn` are not - selectable. +- Embedded filesystem paths cannot contain the version-control names `.bzr`, + `.git`, `.hg`, or `.svn`; the compiler-managed `ard-out` directory is also not + selectable. Exact `text` and `bytes` constructors stage content under encoded + names and may select a regular hidden file when the other path rules permit it. - Every selected path element must be valid UTF-8. Allowed characters are Unicode letters, ASCII digits, ASCII space, and the ASCII punctuation `!#$%&()+,-.=@[]^_{}~`. An element cannot consist only of dots or end in a @@ -386,10 +388,12 @@ EmbeddedEntry { } ``` -`OwnerPackageIdentity` is the canonical root/dependency package identity -supplied by project loading and used for ownership and interning; it is not a -filesystem root or process-local checker pointer. Pattern spelling, source -module identity, and call-site attribution remain on checked expressions and AIR +`OwnerPackageIdentity` is a location-independent identity supplied by project +loading: the manifest package name for root and path packages, and the locked +source identity for Git packages. Resource paths and content also participate in +set identity, so equal package names cannot merge different filesystems. The +identity is never a filesystem root or process-local checker pointer. Pattern +spelling, source module identity, and call-site attribution remain on checked expressions and AIR source locations rather than on interned sets, so different constructors that select identical contents can share one set deterministically. Project loading and checking are solely responsible for proving filesystem containment before diff --git a/website/astro.config.mjs b/website/astro.config.mjs index 2b59c2f0..43899e7d 100644 --- a/website/astro.config.mjs +++ b/website/astro.config.mjs @@ -203,6 +203,7 @@ export default defineConfig({ { label: "Modules", slug: "guide/modules" }, { label: "Dependencies", slug: "guide/dependencies" }, { label: "Build values", slug: "guide/build-values" }, + { label: "Embedded files", slug: "guide/embedding" }, { label: "Testing", slug: "guide/testing" }, { label: "Formatting", slug: "guide/formatting" }, ], diff --git a/website/src/content/docs/guide/embedding.md b/website/src/content/docs/guide/embedding.md new file mode 100644 index 00000000..c983d7ef --- /dev/null +++ b/website/src/content/docs/guide/embedding.md @@ -0,0 +1,82 @@ +--- +title: Embedded files +description: Package files and read-only filesystems directly into Ard programs. +--- + +Import the compiler-provided `ard/embed` module to package files into an +application. Embedded files are read while the program is checked and are +available at runtime without accessing the host filesystem. + +## Embed one file + +Use `text` for UTF-8 text and `bytes` for arbitrary data: + +```ard +use ard/embed + +let license: Str = embed::text("LICENSE") +let logo: [Byte] = embed::bytes("assets/logo.png") +``` + +Paths are relative to the Ard package root, including when the source expression +is in a nested module. They must be non-interpolated string literals. + +`text` preserves the file exactly and reports a compile-time error when its +contents are not valid UTF-8. Each evaluation of `bytes` returns a fresh mutable +list. + +## Embed a filesystem + +Use `fs` with a static list of patterns to create an immutable embedded +filesystem: + +```ard +use ard/embed + +let assets = embed::fs([ + "public", + "templates/*.html", +]) +``` + +A directory pattern includes its complete subtree. Recursive directory +selection excludes names beginning with `.` or `_`; prefix a pattern with +`all:` to include them: + +```ard +let all_assets = embed::fs(["all:public"]) +``` + +Every pattern must match at least one file. Patterns and files cannot escape the +owning package, cross nested package boundaries, or select symlinks. + +## Read embedded files + +```ard +fn homepage() Str!Error { + assets.read_text("public/index.html") +} + +fn logo() [Byte]!Error { + assets.read_file("public/logo.png") +} +``` + +An embedded filesystem provides: + +- `read_file(path)` — returns a fresh `[Byte]`; +- `read_text(path)` — reads and validates UTF-8; +- `read_dir(path)` — returns immediate entries sorted by name; +- `stat(path)` — returns the name, file kind, and optional byte size; +- `sub(path)` — returns a filesystem rooted at an embedded directory. + +Runtime paths are unrooted and slash-separated. `"."` denotes the filesystem +root. Missing and invalid runtime paths return `Error` values. + +On the Go target, `embed::FS` implements `io/fs.FS`, so it can be passed directly +to compatible Go APIs such as `net/http.FS` and `template.ParseFS`. + +:::caution +Embedded contents can be recovered from the resulting executable. Do not embed +passwords, tokens, private keys, or other secrets. +::: From 2b632f4a74fb010395ff93ac6b01b29a93c4ce3d Mon Sep 17 00:00:00 2001 From: Akonwi Ngoh Date: Wed, 9 Sep 2026 08:58:28 -0400 Subject: [PATCH 4/6] feat(examples): serve embedded chi assets --- examples/chi-server/README.md | 16 +++++++++----- examples/chi-server/main.ard | 5 +++++ examples/chi-server/public/app.css | 31 +++++++++++++++++++++++++++ examples/chi-server/public/index.html | 16 ++++++++++++++ 4 files changed, 63 insertions(+), 5 deletions(-) create mode 100644 examples/chi-server/public/app.css create mode 100644 examples/chi-server/public/index.html diff --git a/examples/chi-server/README.md b/examples/chi-server/README.md index 17219db4..9facc2db 100644 --- a/examples/chi-server/README.md +++ b/examples/chi-server/README.md @@ -1,8 +1,9 @@ # chi Graceful-Shutdown Server A pure-Ard port of chi's [graceful-shutdown example](https://github.com/go-chi/chi/blob/master/_examples/graceful/main.go): -an HTTP server built on [chi](https://github.com/go-chi/chi) that finishes in-flight -requests before exiting on `SIGINT` or `SIGTERM`. +an HTTP server built on [chi](https://github.com/go-chi/chi) that serves embedded +static assets and finishes in-flight requests before exiting on `SIGINT` or +`SIGTERM`. There is no Go shim — everything is direct `use go:` interop: @@ -18,6 +19,9 @@ There is no Go shim — everything is direct `use go:` interop: with `.sender()` for `signal::Notify` - `async::start` for the background serve goroutine - Go errors handled as identity-preserving Ard results (`Void!Error`) +- `embed::fs(["public"])` packages static assets into the executable and passes + the resulting `embed::FS` directly through Go's `io/fs.FS`-compatible + `http.FS` adapter ## Adaptations from the Go original @@ -37,9 +41,11 @@ ard run main.ard Then, in another terminal: ```sh -curl http://localhost:3333/ # "sup" -curl http://localhost:3333/slow & # takes 5 seconds -kill -INT # SIGTERM also shuts down gracefully +curl http://localhost:3333/ # "sup" +curl http://localhost:3333/static/ # embedded HTML page +curl http://localhost:3333/static/app.css # embedded stylesheet +curl http://localhost:3333/slow & # takes 5 seconds +kill -INT # SIGTERM also shuts down gracefully ``` The server logs `shutting down`, the in-flight `/slow` request finishes with diff --git a/examples/chi-server/main.ard b/examples/chi-server/main.ard index c07adc15..5b3d16dc 100644 --- a/examples/chi-server/main.ard +++ b/examples/chi-server/main.ard @@ -1,4 +1,5 @@ use ard/async +use ard/embed use go:context use go:errors @@ -24,6 +25,8 @@ use go:time fn service() mut chi::Mux { let router = chi::NewRouter() + let public = embed::fs(["public"]).sub("public").expect("embedded public directory") + let files = http::FileServer(http::FS(public)) router.Use(middleware::RequestID) router.Use(middleware::Logger) @@ -36,6 +39,8 @@ fn service() mut chi::Mux { }, ) + router.Mount("/static", http::StripPrefix("/static", files)) + router.Get( "/slow", fn(w: http::ResponseWriter, r: mut http::Request) { diff --git a/examples/chi-server/public/app.css b/examples/chi-server/public/app.css new file mode 100644 index 00000000..41b39faa --- /dev/null +++ b/examples/chi-server/public/app.css @@ -0,0 +1,31 @@ +:root { + color-scheme: light dark; + font-family: ui-sans-serif, system-ui, sans-serif; +} + +body { + display: grid; + min-height: 100vh; + margin: 0; + place-items: center; + background: #16181d; + color: #f7f7f5; +} + +main { + max-width: 42rem; + padding: 3rem; +} + +.eyebrow { + color: #e5a84b; + font-weight: 700; + letter-spacing: 0.12em; + text-transform: uppercase; +} + +h1 { + margin-block: 0.4rem 1rem; + font-size: clamp(2.5rem, 7vw, 5rem); + line-height: 0.95; +} diff --git a/examples/chi-server/public/index.html b/examples/chi-server/public/index.html new file mode 100644 index 00000000..2b3677dc --- /dev/null +++ b/examples/chi-server/public/index.html @@ -0,0 +1,16 @@ + + + + + + Ard embedded files + + + +
+

Ard + chi

+

Served from an embedded filesystem

+

This page and its stylesheet were compiled into the server binary.

+
+ + From 40daa8ebd1ee5b0abe9d31c62171dbca021fdffd Mon Sep 17 00:00:00 2001 From: Akonwi Ngoh Date: Wed, 9 Sep 2026 09:28:03 -0400 Subject: [PATCH 5/6] fix(embed): track selected resource inputs --- compiler/air/embed_test.go | 42 ++++++ compiler/checker/embed.go | 90 ++++++++++-- compiler/checker/embed_internal_test.go | 71 +++++++++ compiler/checker/embed_test.go | 43 +++++- compiler/go/embed_test.go | 33 ++++- compiler/lsp/analysis/engine.go | 155 +++++++++++--------- compiler/lsp/analysis/engine_test.go | 105 +++++++++++++ compiler/main_test.go | 62 ++++++++ website/src/content/docs/guide/embedding.md | 17 +++ 9 files changed, 532 insertions(+), 86 deletions(-) diff --git a/compiler/air/embed_test.go b/compiler/air/embed_test.go index e75f3b1f..5563558d 100644 --- a/compiler/air/embed_test.go +++ b/compiler/air/embed_test.go @@ -1,6 +1,9 @@ package air import ( + "crypto/sha256" + "encoding/hex" + "fmt" "os" "path/filepath" "strings" @@ -97,3 +100,42 @@ func TestLowerEmbeddedExactFilesIntoBlobReferences(t *testing.T) { t.Fatalf("invalid embedded set path validation error = %v", err) } } + +func embeddedSetTestDigest(set EmbeddedSet, blobs []EmbeddedBlob) string { + hash := sha256.New() + _, _ = hash.Write([]byte(set.OwnerPackageIdentity)) + _, _ = hash.Write([]byte{0}) + for _, entry := range set.Entries { + _, _ = hash.Write([]byte(entry.Path)) + _, _ = hash.Write([]byte{0}) + _, _ = hash.Write([]byte(blobs[entry.Blob].Digest)) + _, _ = hash.Write([]byte{0}) + } + return hex.EncodeToString(hash.Sum(nil)) +} + +func TestValidateEmbeddedResourceLimits(t *testing.T) { + blobData := make([]byte, checker.MaxEmbeddedFileBytes) + blobSum := sha256.Sum256(blobData) + blob := EmbeddedBlob{ID: 0, Data: blobData, Digest: hex.EncodeToString(blobSum[:])} + + oversizedSet := EmbeddedSet{ID: 0, OwnerPackageIdentity: "app"} + for index := 0; index < 5; index++ { + oversizedSet.Entries = append(oversizedSet.Entries, EmbeddedEntry{Path: fmt.Sprintf("%05d.bin", index), Blob: 0}) + } + oversizedSet.Digest = embeddedSetTestDigest(oversizedSet, []EmbeddedBlob{blob}) + if err := Validate(&Program{EmbeddedBlobs: []EmbeddedBlob{blob}, EmbeddedSets: []EmbeddedSet{oversizedSet}}); err == nil || !strings.Contains(err.Error(), "embedded set") { + t.Fatalf("oversized embedded set validation error = %v", err) + } + + zeroSum := sha256.Sum256(nil) + zeroBlob := EmbeddedBlob{ID: 0, Digest: hex.EncodeToString(zeroSum[:])} + tooMany := EmbeddedSet{ID: 0, OwnerPackageIdentity: "app"} + for index := 0; index <= checker.MaxEmbeddedProgramFileCount; index++ { + tooMany.Entries = append(tooMany.Entries, EmbeddedEntry{Path: fmt.Sprintf("%05d.txt", index), Blob: 0}) + } + tooMany.Digest = embeddedSetTestDigest(tooMany, []EmbeddedBlob{zeroBlob}) + if err := Validate(&Program{EmbeddedBlobs: []EmbeddedBlob{zeroBlob}, EmbeddedSets: []EmbeddedSet{tooMany}}); err == nil || !strings.Contains(err.Error(), "program limit") { + t.Fatalf("embedded file-count validation error = %v", err) + } +} diff --git a/compiler/checker/embed.go b/compiler/checker/embed.go index 2cf6e2cc..c627577d 100644 --- a/compiler/checker/embed.go +++ b/compiler/checker/embed.go @@ -25,6 +25,14 @@ const ( MaxEmbeddedProgramFileCount = 10_000 ) +// EmbeddedInputSpec describes statically discoverable resource constructors in +// one parsed Ard module. The LSP uses it to fingerprint only relevant resource +// inputs before deciding whether a checked result is reusable. +type EmbeddedInputSpec struct { + ExactPaths []string + PatternSets [][]string +} + type embeddedFSType struct{} var EmbeddedFS Type = &embeddedFSType{} @@ -212,6 +220,44 @@ func (c *Checker) addEmbedDiagnostic(code DiagnosticCode, title string, text str c.addDiagnostic(diagnostic) } +// EmbeddedInputSignature resolves and hashes the resource inputs selected by a +// module. Resolution errors participate so creation, removal, and newly +// matching pattern entries invalidate cached diagnostics. +func (mr *ModuleResolver) EmbeddedInputSignature(importerModulePath string, spec EmbeddedInputSpec) string { + hash := sha256.New() + exactPaths := append([]string(nil), spec.ExactPaths...) + sort.Strings(exactPaths) + for _, logicalPath := range exactPaths { + _, _ = hash.Write([]byte("exact\x00" + logicalPath + "\x00")) + resource, err := mr.resolveEmbeddedExactFile(importerModulePath, logicalPath) + if err != nil { + _, _ = hash.Write([]byte("error\x00" + err.Error() + "\x00")) + continue + } + _, _ = hash.Write([]byte(resource.OwnerPackageIdentity)) + _, _ = hash.Write([]byte{0}) + digest := sha256.Sum256(resource.Data) + _, _ = hash.Write(digest[:]) + } + for _, patterns := range spec.PatternSets { + _, _ = hash.Write([]byte("set\x00" + strings.Join(patterns, "\x00") + "\x00")) + set, err := mr.resolveEmbeddedFileSetSnapshot(importerModulePath, patterns) + if err != nil { + _, _ = hash.Write([]byte("error\x00" + err.Error() + "\x00")) + continue + } + _, _ = hash.Write([]byte(set.OwnerPackageIdentity)) + _, _ = hash.Write([]byte{0}) + for _, entry := range set.Entries { + _, _ = hash.Write([]byte(entry.LogicalPath)) + _, _ = hash.Write([]byte{0}) + digest := sha256.Sum256(entry.Data) + _, _ = hash.Write(digest[:]) + } + } + return hex.EncodeToString(hash.Sum(nil)) +} + func (mr *ModuleResolver) resolveEmbeddedExactFile(importerModulePath string, logicalPath string) (EmbeddedResource, error) { if err := ValidateEmbeddedLogicalFilePath(logicalPath); err != nil { return EmbeddedResource{}, err @@ -332,6 +378,17 @@ func (mr *ModuleResolver) accountEmbeddedExact(resource EmbeddedResource) error } func (mr *ModuleResolver) resolveEmbeddedFileSet(importerModulePath string, patterns []string) (EmbeddedFileSet, error) { + set, err := mr.resolveEmbeddedFileSetSnapshot(importerModulePath, patterns) + if err != nil { + return EmbeddedFileSet{}, err + } + if err := mr.accountEmbeddedSet(set); err != nil { + return EmbeddedFileSet{}, err + } + return set, nil +} + +func (mr *ModuleResolver) resolveEmbeddedFileSetSnapshot(importerModulePath string, patterns []string) (EmbeddedFileSet, error) { packageID := mr.packageIDForModule(importerModulePath) pkg := mr.packageInfo(packageID) if pkg.RootPath == "" { @@ -472,10 +529,19 @@ func (mr *ModuleResolver) resolveEmbeddedFileSet(importerModulePath string, patt set.Entries = append(set.Entries, EmbeddedSetEntry{LogicalPath: logicalPath, Data: resource.Data}) } + mr.embedMu.Lock() + mr.embeddedPatternSets[patternCacheKey] = set + mr.embedMu.Unlock() + return set, nil +} + +func (mr *ModuleResolver) accountEmbeddedSet(set EmbeddedFileSet) error { hash := sha256.New() _, _ = hash.Write([]byte(set.OwnerPackageIdentity)) _, _ = hash.Write([]byte{0}) + totalBytes := 0 for _, entry := range set.Entries { + totalBytes += len(entry.Data) _, _ = hash.Write([]byte(entry.LogicalPath)) _, _ = hash.Write([]byte{0}) sum := sha256.Sum256(entry.Data) @@ -484,19 +550,19 @@ func (mr *ModuleResolver) resolveEmbeddedFileSet(importerModulePath string, patt identity := hex.EncodeToString(hash.Sum(nil)) mr.embedMu.Lock() defer mr.embedMu.Unlock() - if !mr.embeddedSetIdentities[identity] { - if mr.embeddedProgramFileCount+len(set.Entries) > MaxEmbeddedProgramFileCount { - return EmbeddedFileSet{}, fmt.Errorf("embedded resources exceed program limit of %d files", MaxEmbeddedProgramFileCount) - } - if mr.embeddedProgramBytes+totalBytes > MaxEmbeddedProgramBytes { - return EmbeddedFileSet{}, fmt.Errorf("embedded resources exceed program limit of %d bytes", MaxEmbeddedProgramBytes) - } - mr.embeddedSetIdentities[identity] = true - mr.embeddedProgramFileCount += len(set.Entries) - mr.embeddedProgramBytes += totalBytes + if mr.embeddedSetIdentities[identity] { + return nil } - mr.embeddedPatternSets[patternCacheKey] = set - return set, nil + if mr.embeddedProgramFileCount+len(set.Entries) > MaxEmbeddedProgramFileCount { + return fmt.Errorf("embedded resources exceed program limit of %d files", MaxEmbeddedProgramFileCount) + } + if mr.embeddedProgramBytes+totalBytes > MaxEmbeddedProgramBytes { + return fmt.Errorf("embedded resources exceed program limit of %d bytes", MaxEmbeddedProgramBytes) + } + mr.embeddedSetIdentities[identity] = true + mr.embeddedProgramFileCount += len(set.Entries) + mr.embeddedProgramBytes += totalBytes + return nil } func embedPatternScanRoot(pattern string) string { diff --git a/compiler/checker/embed_internal_test.go b/compiler/checker/embed_internal_test.go index 5549a1c5..d2412a23 100644 --- a/compiler/checker/embed_internal_test.go +++ b/compiler/checker/embed_internal_test.go @@ -105,3 +105,74 @@ func TestEmbeddedExactAccountingDeduplicatesIdenticalBlobs(t *testing.T) { t.Fatalf("embedded accounting = %d files, %d bytes", resolver.embeddedProgramFileCount, resolver.embeddedProgramBytes) } } + +func TestEmbeddedSetAccountingDeduplicatesEquivalentSets(t *testing.T) { + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "ard.toml"), []byte("name = \"app\"\nard = \">= 0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + if err := os.MkdirAll(filepath.Join(root, "assets"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "assets", "value.txt"), []byte("value"), 0o644); err != nil { + t.Fatal(err) + } + resolver, err := NewModuleResolver(root) + if err != nil { + t.Fatal(err) + } + if _, err := resolver.resolveEmbeddedFileSet("app/main", []string{"assets"}); err != nil { + t.Fatal(err) + } + if _, err := resolver.resolveEmbeddedFileSet("app/main", []string{"assets/*.txt"}); err != nil { + t.Fatal(err) + } + if resolver.embeddedProgramFileCount != 1 || resolver.embeddedProgramBytes != len("value") { + t.Fatalf("embedded set accounting = %d files, %d bytes", resolver.embeddedProgramFileCount, resolver.embeddedProgramBytes) + } +} + +func TestEmbeddedSetEnforcesProgramByteLimit(t *testing.T) { + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "ard.toml"), []byte("name = \"app\"\nard = \">= 0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "asset.txt"), []byte("asset"), 0o644); err != nil { + t.Fatal(err) + } + resolver, err := NewModuleResolver(root) + if err != nil { + t.Fatal(err) + } + resolver.embeddedProgramBytes = MaxEmbeddedProgramBytes + if _, err := resolver.resolveEmbeddedFileSet("app/main", []string{"asset.txt"}); err == nil { + t.Fatal("expected embedded set program-byte limit error") + } +} + +func TestEmbeddedInputSignatureDoesNotConsumeResourceBudget(t *testing.T) { + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "ard.toml"), []byte("name = \"app\"\nard = \">= 0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "asset.txt"), []byte("asset"), 0o644); err != nil { + t.Fatal(err) + } + resolver, err := NewModuleResolver(root) + if err != nil { + t.Fatal(err) + } + signature := resolver.EmbeddedInputSignature("app/main", EmbeddedInputSpec{PatternSets: [][]string{{"asset.txt"}}}) + if signature == "" { + t.Fatal("empty embedded input signature") + } + if resolver.embeddedProgramFileCount != 0 || resolver.embeddedProgramBytes != 0 { + t.Fatalf("signature consumed resource budget: %d files, %d bytes", resolver.embeddedProgramFileCount, resolver.embeddedProgramBytes) + } + if _, err := resolver.resolveEmbeddedFileSet("app/main", []string{"asset.txt"}); err != nil { + t.Fatal(err) + } + if resolver.embeddedProgramFileCount != 1 || resolver.embeddedProgramBytes != len("asset") { + t.Fatalf("checked set accounting = %d files, %d bytes", resolver.embeddedProgramFileCount, resolver.embeddedProgramBytes) + } +} diff --git a/compiler/checker/embed_test.go b/compiler/checker/embed_test.go index 69ff262b..bd70752a 100644 --- a/compiler/checker/embed_test.go +++ b/compiler/checker/embed_test.go @@ -183,7 +183,7 @@ func TestEmbedFSExpandsDirectoriesAndAllPatterns(t *testing.T) { source := `use ard/embed let normal = embed::fs(["public"]) -let complete = embed::fs(["all:public"]) +let complete = embed::fs(["all:public", "public/*.js"]) fn read(path: Str) [Byte]!Error { normal.read_file(path) } fn read_text(path: Str) Str!Error { normal.read_text(path) } fn subset(path: Str) embed::FS!Error { normal.sub(path) } @@ -286,4 +286,45 @@ func TestEmbedFSRejectsDynamicAndUnmatchedPatterns(t *testing.T) { if !hasEmbedDiagnostic(malformed, checker.DiagnosticCodeEmbedResource) { t.Fatalf("missing malformed-pattern diagnostic: %#v", malformed.Diagnostics()) } + if err := os.Symlink(filepath.Join(root, "asset.txt"), filepath.Join(root, "linked.txt")); err != nil { + t.Fatal(err) + } + symlink := checkEmbedSource(t, root, "use ard/embed\nlet files = embed::fs([\"linked.txt\"])\n") + if !hasEmbedDiagnostic(symlink, checker.DiagnosticCodeEmbedResource) { + t.Fatalf("missing filesystem symlink diagnostic: %#v", symlink.Diagnostics()) + } +} + +func TestEmbedRejectsInvalidPathsAndPackageBoundaries(t *testing.T) { + root := t.TempDir() + if err := os.WriteFile(filepath.Join(root, "ard.toml"), []byte("name = \"app\"\nard = \">= 0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + if err := os.MkdirAll(filepath.Join(root, "directory"), 0o755); err != nil { + t.Fatal(err) + } + nested := filepath.Join(root, "nested") + if err := os.MkdirAll(nested, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(nested, "ard.toml"), []byte("name = \"nested\"\nard = \">= 0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(nested, "asset.txt"), []byte("nested"), 0o644); err != nil { + t.Fatal(err) + } + for name, source := range map[string]string{ + "traversal": "use ard/embed\nlet value = embed::bytes(\"../outside\")\n", + "absolute": "use ard/embed\nlet value = embed::bytes(\"/absolute\")\n", + "directory": "use ard/embed\nlet value = embed::bytes(\"directory\")\n", + "nested package": "use ard/embed\nlet value = embed::bytes(\"nested/asset.txt\")\n", + "nested pattern": "use ard/embed\nlet value = embed::fs([\"nested\"])\n", + } { + t.Run(name, func(t *testing.T) { + checked := checkEmbedSource(t, root, source) + if !hasEmbedDiagnostic(checked, checker.DiagnosticCodeEmbedResource) { + t.Fatalf("missing resource diagnostic: %#v", checked.Diagnostics()) + } + }) + } } diff --git a/compiler/go/embed_test.go b/compiler/go/embed_test.go index 19cce954..e139473a 100644 --- a/compiler/go/embed_test.go +++ b/compiler/go/embed_test.go @@ -4,6 +4,7 @@ import ( "os" "os/exec" "path/filepath" + "reflect" "strings" "testing" @@ -167,11 +168,15 @@ func TestEmbeddedFSReadsFilesAndSubdirectories(t *testing.T) { if err := os.WriteFile(filepath.Join(root, "public", "binary.bin"), []byte{0xff, 1}, 0o644); err != nil { t.Fatal(err) } + if err := os.WriteFile(filepath.Join(root, "public", ".metadata"), []byte("hidden"), 0o644); err != nil { + t.Fatal(err) + } mainPath := filepath.Join(root, "main.ard") source := `use ard/embed +use go:errors use go:io/fs as gofs -let assets = embed::fs(["public"]) +let assets = embed::fs(["all:public"]) fn main() { let text = assets.read_text("public/index.html").expect("read text") @@ -182,12 +187,18 @@ fn main() { changed.set(0, Byte::from(0)) if assets.read_file("public/index.html").expect("fresh bytes").at(0).or(Byte::from(0)) != 60 { panic("read bytes shared storage") } if gofs::ReadFile(assets, "public/index.html").expect("direct io/fs").size() != 15 { panic("bad io/fs bridge") } - if not assets.read_file("missing").is_err() { panic("missing file succeeded") } + match assets.read_file("missing") { + ok(_) => panic("missing file succeeded"), + err(error) => { + if not errors::Is(error, gofs::ErrNotExist) { panic("missing error identity") } + }, + } if not assets.read_text("public/binary.bin").is_err() { panic("invalid UTF-8 succeeded") } if not assets.sub("public/index.html").is_err() { panic("file sub succeeded") } + if assets.read_text("public/.metadata").expect("hidden file") != "hidden" { panic("bad hidden file") } let entries = assets.read_dir("public").expect("read dir") - if entries.size() != 2 { panic("bad directory size") } - let entry = entries.at(1).expect("directory entry") + if entries.size() != 3 { panic("bad directory size") } + let entry = entries.at(2).expect("directory entry") if entry.name != "index.html" or entry.is_dir { panic("bad directory entry") } let info = assets.stat("public/index.html").expect("stat") if info.name != "index.html" or info.is_dir or info.size.or(0) != 15 { panic("bad file info") } @@ -214,6 +225,20 @@ fn main() { if err != nil { t.Fatalf("Lower: %v", err) } + firstSources, err := GenerateSources(program, Options{PackageName: "main", ProjectInfo: resolver.GetProjectInfo()}) + if err != nil { + t.Fatalf("first GenerateSources: %v", err) + } + secondSources, err := GenerateSources(program, Options{PackageName: "main", ProjectInfo: resolver.GetProjectInfo()}) + if err != nil { + t.Fatalf("second GenerateSources: %v", err) + } + if !reflect.DeepEqual(firstSources, secondSources) { + t.Fatal("embedded filesystem source generation is not deterministic") + } + if resourceSource := string(firstSources["internal/ardembed/embed.go"]); !strings.Contains(resourceSource, "//go:embed all:sets/") || !strings.Contains(resourceSource, "func FS") { + t.Fatalf("generated embedded filesystem source:\n%s", resourceSource) + } if err := os.RemoveAll(filepath.Join(root, "public")); err != nil { t.Fatal(err) } diff --git a/compiler/lsp/analysis/engine.go b/compiler/lsp/analysis/engine.go index 34afca35..b47111a2 100644 --- a/compiler/lsp/analysis/engine.go +++ b/compiler/lsp/analysis/engine.go @@ -18,10 +18,9 @@ import ( "crypto/sha256" "encoding/hex" "fmt" - "hash" - "io" "os" "path/filepath" + "reflect" "runtime/debug" "sort" "strings" @@ -649,62 +648,90 @@ func (s *Snapshot) check(filePath string, relPath string, program *parse.Program // Standard-library imports (ard/*) are skipped because they are embedded in the // compiler binary. `ard/embed` is the exception: importing it adds the owning // package's resource tree to the signature. -func hashEmbeddedPackageTree(h hash.Hash, root string) { - if root == "" { - return - } - _ = filepath.WalkDir(root, func(filePath string, entry os.DirEntry, walkErr error) error { - if walkErr != nil { - h.Write([]byte(filePath)) - h.Write([]byte(":unreadable\x00")) - return nil - } - if filePath == root { - return nil +func collectEmbeddedInputSpec(program *parse.Program) checker.EmbeddedInputSpec { + spec := checker.EmbeddedInputSpec{} + if program == nil { + return spec + } + aliases := map[string]bool{} + for _, imported := range program.Imports { + if imported.Path == checker.EmbedModulePath { + aliases[imported.Alias()] = true + } + } + if len(aliases) == 0 { + return spec + } + exactSeen := map[string]bool{} + patternSetSeen := map[string]bool{} + seenPointers := map[uintptr]bool{} + var walk func(reflect.Value) + walk = func(value reflect.Value) { + if !value.IsValid() { + return } - if entry.IsDir() { - switch entry.Name() { - case ".bzr", ".git", ".hg", ".svn", "ard-out", "vendor": - return filepath.SkipDir + if value.Kind() == reflect.Interface || value.Kind() == reflect.Pointer { + if value.IsNil() { + return } - if _, err := os.Stat(filepath.Join(filePath, "ard.toml")); err == nil { - return filepath.SkipDir + if value.Kind() == reflect.Pointer { + pointer := value.Pointer() + if seenPointers[pointer] { + return + } + seenPointers[pointer] = true } - if _, err := os.Stat(filepath.Join(filePath, "go.mod")); err == nil { - return filepath.SkipDir + if value.CanInterface() { + if call, ok := value.Interface().(*parse.StaticFunction); ok { + if target, ok := call.Target.(*parse.Identifier); ok && aliases[target.Name] && len(call.Function.TypeArgs) == 0 && len(call.Function.Args) == 1 && call.Function.Args[0].Name == "" && !call.Function.Args[0].Spread { + switch call.Function.Name { + case "text", "bytes": + if literal, ok := call.Function.Args[0].Value.(*parse.StrLiteral); ok && !exactSeen[literal.Value] { + exactSeen[literal.Value] = true + spec.ExactPaths = append(spec.ExactPaths, literal.Value) + } + case "fs": + if list, ok := call.Function.Args[0].Value.(*parse.ListLiteral); ok && len(list.Items) > 0 { + patterns := make([]string, 0, len(list.Items)) + for _, item := range list.Items { + literal, ok := item.(*parse.StrLiteral) + if !ok { + patterns = nil + break + } + patterns = append(patterns, literal.Value) + } + patternKey := strings.Join(patterns, "\x00") + if patterns != nil && !patternSetSeen[patternKey] { + patternSetSeen[patternKey] = true + spec.PatternSets = append(spec.PatternSets, patterns) + } + } + } + } + } } - return nil - } - rel, err := filepath.Rel(root, filePath) - if err != nil { - return nil + walk(value.Elem()) + return } - h.Write([]byte(filepath.ToSlash(rel))) - h.Write([]byte{0}) - if entry.Type()&os.ModeSymlink != 0 { - if target, err := os.Readlink(filePath); err == nil { - h.Write([]byte("symlink:" + target)) + switch value.Kind() { + case reflect.Struct: + for index := 0; index < value.NumField(); index++ { + walk(value.Field(index)) + } + case reflect.Slice, reflect.Array: + for index := 0; index < value.Len(); index++ { + walk(value.Index(index)) + } + case reflect.Map: + iterator := value.MapRange() + for iterator.Next() { + walk(iterator.Value()) } - h.Write([]byte{0}) - return nil - } - if !entry.Type().IsRegular() { - h.Write([]byte(":irregular\x00")) - return nil - } - file, err := os.Open(filePath) - if err != nil { - h.Write([]byte(":unreadable\x00")) - return nil - } - _, copyErr := io.Copy(h, io.LimitReader(file, checker.MaxEmbeddedFileBytes+1)) - closeErr := file.Close() - if copyErr != nil || closeErr != nil { - h.Write([]byte(":read-error")) } - h.Write([]byte{0}) - return nil - }) + } + walk(reflect.ValueOf(program)) + return spec } func (s *Snapshot) signature(filePath string, content []byte, program *parse.Program, moduleResolver *checker.ModuleResolver, relPath string, trackDependencies bool) string { @@ -716,7 +743,6 @@ func (s *Snapshot) signature(filePath string, content []byte, program *parse.Pro seen := map[string]bool{filePath: true} seenPackageManifests := map[string]bool{} - seenEmbedPackages := map[string]bool{} projectInfo := moduleResolver.GetProjectInfo() if projectInfo != nil { seenPackageManifests[projectInfo.RootPackageID] = true @@ -740,8 +766,8 @@ func (s *Snapshot) signature(filePath string, content []byte, program *parse.Pro } h.Write([]byte{0}) } - var visit func(prog *parse.Program, importerModulePath string, importerFilePath string, importerPackageID string) - visit = func(prog *parse.Program, importerModulePath string, importerFilePath string, importerPackageID string) { + var visit func(prog *parse.Program, importerModulePath string, importerFilePath string) + visit = func(prog *parse.Program, importerModulePath string, importerFilePath string) { if prog == nil { return } @@ -752,11 +778,7 @@ func (s *Snapshot) signature(filePath string, content []byte, program *parse.Pro } deps := make([]dep, 0, len(prog.Imports)) known := true - usesEmbed := false for _, imp := range prog.Imports { - if imp.Path == checker.EmbedModulePath { - usesEmbed = true - } if imp.Kind == parse.ImportKindGo || strings.HasPrefix(imp.Path, "ard/") { continue } @@ -772,11 +794,10 @@ func (s *Snapshot) signature(filePath string, content []byte, program *parse.Pro } deps = append(deps, dep{file: resolved.FilePath, module: resolved.ModulePath, packageID: resolved.PackageID}) } - if usesEmbed && projectInfo != nil && !seenEmbedPackages[importerPackageID] { - seenEmbedPackages[importerPackageID] = true - if pkg, ok := projectInfo.Packages[importerPackageID]; ok { - hashEmbeddedPackageTree(h, pkg.RootPath) - } + embedSpec := collectEmbeddedInputSpec(prog) + if len(embedSpec.ExactPaths) > 0 || len(embedSpec.PatternSets) > 0 { + h.Write([]byte(moduleResolver.EmbeddedInputSignature(importerModulePath, embedSpec))) + h.Write([]byte{0}) } sort.Slice(deps, func(a, b int) bool { return deps[a].file < deps[b].file }) if trackDependencies { @@ -813,14 +834,10 @@ func (s *Snapshot) signature(filePath string, content []byte, program *parse.Pro } continue } - visit(entry.program, d.module, d.file, d.packageID) + visit(entry.program, d.module, d.file) } } - rootPackageID := "" - if projectInfo != nil { - rootPackageID = projectInfo.RootPackageID - } - visit(program, strings.TrimSuffix(relPath, ".ard"), filePath, rootPackageID) + visit(program, strings.TrimSuffix(relPath, ".ard"), filePath) // Project manifest and Go module metadata participate so dependency and // FFI configuration changes invalidate checks. diff --git a/compiler/lsp/analysis/engine_test.go b/compiler/lsp/analysis/engine_test.go index 120e4364..82f948a6 100644 --- a/compiler/lsp/analysis/engine_test.go +++ b/compiler/lsp/analysis/engine_test.go @@ -9,6 +9,8 @@ import ( "sync/atomic" "testing" "time" + + "github.com/akonwi/ard/parse" ) func writeProject(t *testing.T, files map[string]string) string { @@ -852,6 +854,16 @@ func TestEmbeddedResourceChangeInvalidatesAnalysis(t *testing.T) { if len(first.Diagnostics) != 0 { t.Fatalf("initial diagnostics: %#v", first.Diagnostics) } + if err := os.WriteFile(filepath.Join(root, "unrelated.txt"), []byte("unrelated"), 0o644); err != nil { + t.Fatal(err) + } + unchanged, err := workspace.Snapshot().Analyze(mainPath) + if err != nil { + t.Fatal(err) + } + if unchanged != first { + t.Fatal("unrelated file invalidated embedded-resource analysis") + } if err := os.WriteFile(resourcePath, []byte("second"), 0o644); err != nil { t.Fatal(err) } @@ -873,4 +885,97 @@ func TestEmbeddedResourceChangeInvalidatesAnalysis(t *testing.T) { if len(missing.Diagnostics) == 0 { t.Fatal("missing embedded resource did not produce diagnostics") } + if err := os.WriteFile(resourcePath, []byte("restored"), 0o644); err != nil { + t.Fatal(err) + } + restored, err := workspace.Snapshot().Analyze(mainPath) + if err != nil { + t.Fatal(err) + } + if len(restored.Diagnostics) != 0 || restored.Signature == missing.Signature { + t.Fatalf("recreated resource analysis = %#v", restored.Diagnostics) + } +} + +func TestNewEmbeddedPatternMatchInvalidatesAnalysis(t *testing.T) { + root := writeProject(t, map[string]string{ + "ard.toml": "name = \"proj\"\nard = \">= 0.1.0\"\n", + "main.ard": "use ard/embed\nlet files = embed::fs([\"assets/*.txt\"])\n", + "assets/first.txt": "first", + }) + workspace := NewWorkspace(NewEngine(root)) + mainPath := filepath.Join(root, "main.ard") + first, err := workspace.Snapshot().Analyze(mainPath) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "assets", "second.txt"), []byte("second"), 0o644); err != nil { + t.Fatal(err) + } + second, err := workspace.Snapshot().Analyze(mainPath) + if err != nil { + t.Fatal(err) + } + if first.Signature == second.Signature { + t.Fatal("new pattern match did not invalidate analysis") + } +} + +func TestDependencyEmbeddedResourceInvalidatesAnalysis(t *testing.T) { + workspaceRoot := t.TempDir() + appRoot := filepath.Join(workspaceRoot, "app") + depRoot := filepath.Join(workspaceRoot, "dep") + for path, content := range map[string]string{ + filepath.Join(appRoot, "ard.toml"): "name = \"app\"\nard = \">= 0.1.0\"\n\n[dependencies]\ndep = { path = \"../dep\" }\n", + filepath.Join(appRoot, "main.ard"): "use dep\nfn main() Str { dep::value() }\n", + filepath.Join(depRoot, "ard.toml"): "name = \"dep\"\nard = \">= 0.1.0\"\n", + filepath.Join(depRoot, "dep.ard"): "use ard/embed\nfn value() Str { embed::text(\"asset.txt\") }\n", + filepath.Join(depRoot, "asset.txt"): "first", + } { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } + workspace := NewWorkspace(NewEngine(appRoot)) + mainPath := filepath.Join(appRoot, "main.ard") + first, err := workspace.Snapshot().Analyze(mainPath) + if err != nil { + t.Fatal(err) + } + if len(first.Diagnostics) != 0 { + t.Fatalf("initial diagnostics: %#v", first.Diagnostics) + } + if err := os.WriteFile(filepath.Join(depRoot, "asset.txt"), []byte("second"), 0o644); err != nil { + t.Fatal(err) + } + second, err := workspace.Snapshot().Analyze(mainPath) + if err != nil { + t.Fatal(err) + } + if first.Signature == second.Signature { + t.Fatal("dependency embedded resource did not invalidate analysis") + } +} + +func TestCollectEmbeddedInputSpecHonorsAliasesAndStaticCalls(t *testing.T) { + path := filepath.Join(t.TempDir(), "main.ard") + parsed := parse.Parse([]byte(`use ard/embed as resources +let exact = resources::text("page.txt") +let files = resources::fs(["assets", "templates/*.html"]) +let dynamic_path = "ignored.txt" +let dynamic = resources::bytes(dynamic_path) +`), path) + if len(parsed.Errors) != 0 { + t.Fatalf("parse errors: %v", parsed.Errors) + } + spec := collectEmbeddedInputSpec(parsed.Program) + if !slices.Equal(spec.ExactPaths, []string{"page.txt"}) { + t.Fatalf("exact paths = %#v", spec.ExactPaths) + } + if len(spec.PatternSets) != 1 || !slices.Equal(spec.PatternSets[0], []string{"assets", "templates/*.html"}) { + t.Fatalf("pattern sets = %#v", spec.PatternSets) + } } diff --git a/compiler/main_test.go b/compiler/main_test.go index 2161c272..264011d5 100644 --- a/compiler/main_test.go +++ b/compiler/main_test.go @@ -1732,3 +1732,65 @@ func TestRemoveDependencyFromManifestMissing(t *testing.T) { t.Fatalf("manifest changed unexpectedly:\n%s", data) } } + +func TestEmbeddedFilesystemWorksAcrossRunBuildAndTest(t *testing.T) { + root := t.TempDir() + for path, content := range map[string]string{ + filepath.Join(root, "ard.toml"): "name = \"embedded_workflows\"\nard = \">= 0.1.0\"\n", + filepath.Join(root, "assets", "value.txt"): "captured", + filepath.Join(root, "main.ard"): `use ard/embed +use ard/testing + +let files = embed::fs(["assets"]) + +fn embedded_value() Str { + files.read_text("assets/value.txt").expect("embedded value") +} + +fn main() { + if embedded_value() != "captured" { panic("run/build lost embedded filesystem") } +} + +test fn embeds_in_tests() Void!Str { + try testing::assert(embedded_value() == "captured", "test lost embedded filesystem") + testing::pass() +} +`, + } { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } + mainPath := filepath.Join(root, "main.ard") + loaded, err := frontend.LoadModule(mainPath) + if err != nil { + t.Fatalf("load run module: %v", err) + } + program, err := air.Lower(loaded.Module) + if err != nil { + t.Fatalf("lower run module: %v", err) + } + if err := gotarget.RunProgram(program, []string{"ard", "run", mainPath}, loaded.ProjectInfo); err != nil { + t.Fatalf("run embedded program: %v", err) + } + + binaryPath := filepath.Join(t.TempDir(), "embedded-workflows") + built, err := buildGoBinary(mainPath, binaryPath) + if err != nil { + t.Fatalf("build embedded program: %v", err) + } + if output, err := exec.Command(built).CombinedOutput(); err != nil { + t.Fatalf("built embedded program failed: %v\n%s", err, output) + } + + var testsPassed bool + output := captureStdout(t, func() { + testsPassed = runTests(root, "embeds_in_tests", false) + }) + if !testsPassed || !strings.Contains(output, "1 passed; 0 failed; 0 panicked") { + t.Fatalf("embedded test workflow failed:\n%s", output) + } +} diff --git a/website/src/content/docs/guide/embedding.md b/website/src/content/docs/guide/embedding.md index c983d7ef..bff31736 100644 --- a/website/src/content/docs/guide/embedding.md +++ b/website/src/content/docs/guide/embedding.md @@ -76,6 +76,23 @@ root. Missing and invalid runtime paths return `Error` values. On the Go target, `embed::FS` implements `io/fs.FS`, so it can be passed directly to compatible Go APIs such as `net/http.FS` and `template.ParseFS`. +## Limits and generated files + +Embedding is bounded to keep checking and generated artifacts predictable: + +- 16 MiB per file; +- 64 MiB per embedded filesystem; and +- 128 MiB and 10,000 selected files per program. + +Repeated references to the same exact content and identical filesystem sets are +counted once. Different filesystem sets are counted independently, even when +they contain overlapping files. + +The Go target stages captured bytes under `ard-out` and generates an internal +package containing `//go:embed` directives. These are compiler-managed build +artifacts: do not import or edit them. The backend uses the bytes captured while +checking and does not reread the original resource files during code generation. + :::caution Embedded contents can be recovered from the resulting executable. Do not embed passwords, tokens, private keys, or other secrets. From 067d586a5efd791ff4db6517d20451ecc826d209 Mon Sep 17 00:00:00 2001 From: Akonwi Ngoh Date: Wed, 9 Sep 2026 10:06:14 -0400 Subject: [PATCH 6/6] docs(stdlib): document embed module --- website/astro.config.mjs | 1 + website/src/content/docs/stdlib/embed.md | 151 +++++++++++++++++++++++ 2 files changed, 152 insertions(+) create mode 100644 website/src/content/docs/stdlib/embed.md diff --git a/website/astro.config.mjs b/website/astro.config.mjs index 43899e7d..c782db1b 100644 --- a/website/astro.config.mjs +++ b/website/astro.config.mjs @@ -235,6 +235,7 @@ export default defineConfig({ label: "Modules", items: [ { label: "ard/async", slug: "stdlib/async" }, + { label: "ard/embed", slug: "stdlib/embed" }, { label: "ard/list", slug: "stdlib/list" }, { label: "ard/map", slug: "stdlib/map" }, { label: "ard/testing", slug: "stdlib/testing" }, diff --git a/website/src/content/docs/stdlib/embed.md b/website/src/content/docs/stdlib/embed.md new file mode 100644 index 00000000..af50b706 --- /dev/null +++ b/website/src/content/docs/stdlib/embed.md @@ -0,0 +1,151 @@ +--- +title: ard/embed +description: Compile files and immutable filesystems into Ard programs. +--- + +The compiler-provided `ard/embed` module captures package files while checking +and includes their contents in the generated program. It does not read the host +filesystem at runtime. + +```ard +use ard/embed + +let license = embed::text("LICENSE") +let logo = embed::bytes("assets/logo.png") +let public = embed::fs(["public"]) +``` + +Constructor arguments are static, non-interpolated string literals. Paths are +slash-separated and resolve from the root of the Ard package containing the +constructor—not from the source file's directory or the process working +directory. + +## Constructors + +### `text(path: Str) Str` + +Embed one UTF-8 text file. The returned string preserves the file contents +exactly, including line endings and a trailing newline. Checking fails if the +file is not valid UTF-8. + +```ard +use ard/embed + +let template = embed::text("templates/page.html") +``` + +### `bytes(path: Str) [Byte]` + +Embed one file as bytes. Each evaluation returns a fresh list, so mutating one +result does not modify a later result. + +```ard +use ard/embed + +fn icon() [Byte] { + embed::bytes("assets/icon.png") +} +``` + +### `fs(patterns: [Str]) embed::FS` + +Embed files selected by a non-empty static list of patterns and return an +immutable filesystem. + +```ard +use ard/embed + +let assets = embed::fs([ + "public", + "templates/*.html", + "all:public/.well-known", +]) +``` + +Patterns use slash-separated Go-style path matching: + +- `*` matches within one path segment; +- `?` matches one non-separator character; +- character classes such as `[a-z]` are supported; +- a directory match includes its complete subtree; +- recursively selected names beginning with `.` or `_` are excluded by default; +- the `all:` prefix includes those hidden names; and +- recursive `**` patterns are not supported. + +Every pattern must select at least one regular file. Overlapping matches are +deduplicated. Paths cannot escape the package, cross a nested Ard package or Go +module boundary, or select symbolic links and reserved directories. + +## `embed::FS` + +`embed::FS` is an immutable, opaque filesystem. Runtime paths are unrooted and +slash-separated; `"."` denotes the filesystem root. Invalid or missing paths +return `Error` values. + +### `read_file(path: Str) [Byte]!Error` + +Read a file as a fresh byte list. + +### `read_text(path: Str) Str!Error` + +Read a file as text. Returns an error when its contents are not valid UTF-8. + +### `read_dir(path: Str) [embed::DirEntry]!Error` + +Read the immediate children of a directory, sorted by name. + +### `stat(path: Str) embed::FileInfo!Error` + +Return metadata for a file or directory. + +### `sub(path: Str) embed::FS!Error` + +Return a filesystem rooted at an embedded directory. The path must identify a +directory. + +```ard +use ard/embed + +let public = embed::fs(["public"]).sub("public").expect("embedded public files") +let index = public.read_text("index.html").expect("embedded index") +``` + +On the Go target, `embed::FS` implements `io/fs.FS` and can be passed directly +to compatible Go APIs: + +```ard +use ard/embed +use go:net/http + +let public = embed::fs(["public"]).sub("public").expect("embedded public files") +let handler = http::FileServer(http::FS(public)) +``` + +## Supporting types + +### `embed::DirEntry` + +| Field | Type | Description | +| --- | --- | --- | +| `name` | `Str` | Entry name relative to the directory being read. | +| `is_dir` | `Bool` | Whether the entry is a directory. | + +### `embed::FileInfo` + +| Field | Type | Description | +| --- | --- | --- | +| `name` | `Str` | Base name, or `"."` for the filesystem root. | +| `is_dir` | `Bool` | Whether the path identifies a directory. | +| `size` | `Int?` | File size in bytes; `none` for directories. | + +## Resource limits + +- 16 MiB per file +- 64 MiB per embedded filesystem +- 128 MiB and 10,000 selected files per program + +Embedded contents are recoverable from the executable. Do not embed passwords, +tokens, private keys, or other secrets. + +For patterns, generated artifacts, and a complete example, see +[Embedded files](/guide/embedding/).