Package bootstrapping - #5768
Draft
StachuDotNet wants to merge 32 commits into
Draft
StachuDotNet wants to merge 32 commits into
StachuDotNet wants to merge 32 commits into
Conversation
fetch-seed curled one hardcoded url and then cp'd over data.db whatever was there, so running it with local work in the store discarded it. it now takes --from a file, --url, or --url + --commit, and refuses to replace an existing store without --force. _post-start fetches a seed when there is none and we are in seed mode. the default stays disk: in seed mode the planner correctly routes a packages/ edit nowhere, so until dark edit <module> exists there would be no way to change package code at all. said in config/dev next to the setting. two of the build-planning tests assumed disk mode without saying so and failed under seed. both pin the mode now, and each gained a seed-mode counterpart.
locally the seed is the one store guaranteed not to contain the package you just authored, so testing against it tests the wrong thing. copy data.db instead, through sqlite .backup because the store is WAL and cp drops the tail. CI keeps using the seed: it has no store of its own and wants the pinned one.
nothing wrote an Unbind. a name could be bound and rebound but never ended, so one authored by mistake lived forever and greeted strangers as a constraint. one op. the content is untouched and stays reachable by hash, so callers go on working -- a reference points at content, not at a name. that is the claim the command prints, so there is a test for it: author a caller, end the callee's name, the caller still returns 4242. confirmExplicit, not confirm: a stray Return should not end a name. swept in the four shapes like every registered command.
after the flip there is no packages/ tree to grep, so this is the replacement. search looks at names; this looks at source. nothing stores source: the store holds ops and text comes from the pretty printer. so it is cached, keyed by content hash, which never needs invalidating because content is immutable. the cache lives beside the store like credentials.db -- it is derived and per-install, and Seed.export strips by a denylist, so a table added to the store would ride out in every seed. takes an optional module to search under. unscoped it renders everything the first time, which is minutes; scoped it is the common case and it is quick. it searches RENDERED source, so /// docs are in and // asides are not: the parser does not keep them. said in --help, because a search tool you cannot trust a negative answer from is worse than none -- which is also why it now reports items it could not render instead of counting them as non-matches.
a new command joins the registry sweep the day it is registered, and the sweep
runs everything with a bogus argument. unscoped grep renders 6,604 items, so
adding it quietly cost nine minutes of suite time.
it now says what the search would cost and how to avoid it: name a module, or
pass --all to accept the wait. the sweep takes the refusal in a second.
sweep 9m48 -> 59s
a guard test caught this: locations is main's projection, so grep enumerating it directly searched MAIN while standing on a branch, and said nothing was wrong. exactly the class the SCM silo exists to stop. the enumerating read belongs in the silo beside liveBindingFor, so allLiveBindings lays the chain overlay over main: a branch's SetName replaces main's entry, and a name the chain unbound is dropped even though main holds it. the main-scoped half is marked as the silo's rule requires. tested both directions: a branch finds its own work, main does not.
f# only CALLS those seventeen; it never takes one apart. so the frozen contract is the name and the signature, and the newest committed binding can win -- which is what would make the store the source for the cli's own entry points rather than for everything except them. types stay pinned: a DRecord the kernel builds carries its type's hash. the hook lives in PackageRefs and is installed by LibDB, because LibDB depends on LibExecution and not the reverse. main's committed projection only: resolving entry points through a branch overlay would run a branch's parser the moment you stood on one. OFF unless DARK_REFS_BY_NAME=1, and it falls back to the pinned hash for any name the store does not bind, so with the flag off this changes nothing. it also does nothing useful yet: Darklang is a reserved owner, so nothing can rebind those names anyway. that is the same question as who may fold on the server, and it is written up in the worklist.
…folds
reservedOwners refused every write under Darklang that was not trusted seeding,
and effective=0 kept the server from folding what it was pushed. both were the
same defence -- a name binds last-writer-wins across the store, so anything that
could write could rebind the names the kernel resolves through, including the
router a server serves. both were also the reason the store could not become the
source: the only sanctioned writer to Darklang.* was the reload from packages/
that this arc deletes.
given up deliberately. the replacement is an op's authority checked at fold
time, and it does not exist yet, so for now write access is rebind access. the
comments say so where the checks used to be.
effective was carrying two meanings and only one was given up. a pushed op is
folded now, but it is still not this store's draft -- op_owners says who pushed
it, and that is what keeps discard from deleting a peer's work. two tests
covered exactly that and both now assert the split rather than the flag.
the 17 entry points resolve from the store by default. safe because a candidate
is checked against the pinned version's SIGNATURE and a mismatch falls back,
loudly, so a wrong edit degrades rather than bricks the cli. proven both ways:
compatible rebind -> GOOD:2 the store's printer ran
wrong return type -> warning + 2 the built-in one ran
crosses a process and a machine boundary -- a real server on a port, a real client with its own store, a real push over http -- because the whole question is whether the SERVER's projection moved, and neither half can be faked in-process. it asserts both halves of what effective=0 used to mean: the op is folded, so the browser can see a package the server hosts; and it is NOT the server's draft, so a discard leaves what its clients pushed alone. the second is the one that would be data loss, and nothing else covers it end to end. in the CI subset. also drops a comment in relay-routes that said the relay never folds what it hosts.
a revert is a rebinding, not a recovery. every version is still in the store because content is content-addressed and never deleted, so pointing the name at an old hash is the whole operation -- which is why it is symmetric, and why dark log still lists the version you moved off. "at that commit" means that commit or any ancestor, walked through commits.parent. you name a commit to mean a point in time, and the version you want was usually bound by something earlier. that ancestry is also what made this buildable: i had parked revert as needing commit ordering that did not exist, and the column was there. short hashes resolve like git's, and an ambiguous prefix is refused rather than guessed: two commits sharing a prefix is rare, and picking one would revert to the wrong history. the two store reads live in the SCM silo, marked main-scoped: a past binding is a fact about what main committed, and a branch has committed nothing.
The package server hands out its op log as a file now: `/seed/latest.db`, `/seed/<commit>.db`, and `/seed/meta` for the four fields a pin needs without the twelve megabytes. `latest.db` resolves to the head and is served as the cut AT that head, so it is the same thing as asking for that commit by name. Cuts are cached beside the store keyed by commit, which never needs invalidating: commits are immutable and ops only append. A short prefix resolves like git's; an ambiguous one is refused, because a newest-wins tie would pin a package set nobody chose. Every seed carries a `store_meta` stamp: `format` (the op-blob layout, which is what a migrator keys on), `cut_at`, `kernel`, `at`. `cut_at` names a commit even when no cut was asked for, read back after the cut, so a seed always says what it is a cut of. `Releases.runPending` reads it at open: a store from a NEWER format is refused, one carrying none is stamped. The asymmetry is the point -- a store behind this build is the migrator's job, one ahead is not recoverable by reading harder. Three bugs in `exportAt`, all of them only reachable on a process that cuts more than one seed, which is what a server is: - it checkpointed the source through a READ-ONLY connection and then copied the file. The moment there is actually a WAL to fold in, that fails with a disk I/O error. It goes through `Sqlite.Backup.toFile`, the online-backup API, which also means it exports the store a test repointed LibDB at rather than the config one. - `connStringFor` sets `Pooling=true`, and a pooled connection outlives its `Close`, so the second cut got a handle to a file the first had deleted and replaced. Backup targets are unpooled now. - the seed was left in WAL mode, so the `.db` alone was not the whole database and whether a fetch got everything depended on when a checkpoint ran. It ends in `journal_mode=DELETE`; `LibDB.Sqlite` puts a store back into WAL at open. Folding is not running. `evaluateAllValues` now skips a value whose bindings are all hosted, so a pushed `val` is folded, browsable and servable in a seed, and never executed here -- whoever fetches it evaluates it under their own policy. "All" and not "any": content is shared, so a value this store authored can also arrive by push from a peer who wrote the same thing, and anything locally bound is still evaluated. `op_owners` is empty on an instance, so a client selects exactly what it selected before. Two gates, both crossing a machine boundary because the questions are about what a SERVER did. `seed-serving` tests the pin property by moving the store underneath it: cut at a commit, push, delete the cache so the cut is recomputed, ask again, same ops and no later commit. `server-folds` gained the evaluation half, which needed a real push and then a RESTART, since the startup grow is what would have run the body.
`package-set.txt` records which package set this kernel was tested against. Two fields, `commit` and `format`, and deliberately no URL: a pin says WHICH package set, not where to get it, two mirrors serving the same commit are interchangeable, and an address in the repo is a deployment detail that ages badly. It comes from `DARK_SEED_URL` or `--url`. It ships `commit unset`, which every reader treats as "build the package set from packages/" -- exactly what happens today. So this is a mechanism with a one-line switch rather than a cutover, which is the only responsible way to land a half that has never run against a deployed server. `scripts/packages/pin head | <commit> | --show | --unset` writes it, and checks every pin before writing: `/seed/meta?commit=` makes the server resolve the commit and answer what a cut at it would carry, so a bad pin is refused at pin time instead of at the next CI run. `scripts/build/prepare-package-set` is the one place that answers "where does this build's package set come from", and CI's three package-reloading jobs call it instead of `reload-packages`. It does NOT fall back to reloading when the server is unreachable: a build that silently used the tree instead of the pin is the failure the pin exists to prevent. `seed-db` gained a cache keyed on the pin rather than on the week, because a pinned package set is immutable. `scripts/run-local-exec refs generate` is the piece that makes a fetched seed usable: the kernel resolves its entry points by hash, and that file is a projection of the store, not something the seed carries. The fill path wrote it as a side effect of reloading; this is the same step for a store that has no packages/ to reload. Against the plan, which wanted `package-ref-hashes.txt` tracked in git. It should not be, and this is what makes that the better answer: it is derived, tracking it taxes every rebase (two branches that both touch packages conflict, and GitHub cannot regenerate it), and the endgame reason for tracking -- no packages/ to regenerate from -- is answered by regenerating from the pinned store instead. What git records is one line of pin.
An op's id comes from `Hashing.computeOpRowId` over the decoded op, not over its
bytes, and that divides format changes cleanly:
- a change under LibSerialization/Binary/* moves the LAYOUT and nothing's
identity. The migration is a blob rewrite: decode with the old reader,
re-encode with the new writer, every id untouched, projections re-folded.
That is `dark store upgrade`.
- a change under LibSerialization/Hashing/* moves every op id, item hash and
commit hash at once, which is a whole-log re-mint. Not built. StoreUpgrade.fs
ends with the six steps it needs and the two properties that make it worth
the care.
The migrator refuses rather than guessing which it faces: it recomputes each
op's id from the decoded op and compares it with the id the store filed it
under. Disagreement means the store's ids were minted by different hashing, and
a blob rewrite would be a lie. The gate forges exactly that, which is the most
important assertion in it.
The backup lands first, through the online-backup API, so the rollback target
exists before anything is touched. The rewrite and the format stamp are one
transaction, through `executeTransactionSync` rather than hand-written
BEGIN/COMMIT: connections are pooled, so a BEGIN and the statements after it are
not guaranteed to be on the same one, and the transaction would silently cover
nothing.
`validateVersion` accepts any version up to this build's and refuses only newer.
An old blob can be read because every historical reader stays in the binary; a
newer one cannot be read by trying harder.
A build reading an OLDER format than the store gets a note, not a refusal, and
the note names a file to move rather than a verb to run. Refusing to open would
refuse the rollback too, and the projections hold newer blobs as well, so such a
build dies resolving the name of whatever command you typed. A `mv` needs no
working binary. The note comes from `growIfNeeded`, which every process that
opens the store passes through; the first version of it lived only in
`Releases.runPending`, which a dev build never reaches, and the gate is what
showed that.
`DARK_FORMAT_VERSION` is the synthetic bump the gate runs on. One build writes
the version named and reads every version up to it, so the layout is identical
and a migration across it is exactly the format-only case. It is the only way to
exercise a migrator while one format exists, and having it mechanised before the
first real bump is the point.
Also a serialization corpus, `serialization-artifacts/corpus/v<N>/`, ten types,
never regenerated. Different question from the golden files beside it: a golden
is regenerated whenever a format change is intended, so the moment the format
moves the old bytes are gone and nothing checks they can still be read. This
decodes a blob a past build wrote, re-encodes with today's writer, decodes that,
and compares, which is the migration's core operation. At the current version it
also asserts byte-exactness, as a prefix so a test value can be appended without
destroying history; that catches a layout change made without bumping the
version, which is the one format bug that corrupts silently.
`store-upgrade` goes in the CI gate subset rather than staying local-only. It is
the migrator's only test, and a migration that has quietly stopped working is
discovered at the worst possible moment.
…e set comes from `dark docs relay` said the server never folds what it hosts. It does now, into main, like any other instance; what it does not do is RUN it. The old text had that exactly backwards, which is worse than silence on a page people read to decide whether pointing a machine at a server is safe. It now also carries the `/seed/*` routes and the honest sentence: a push here can rebind what this server serves, and the write secret is currently the whole of the answer. `dark docs packages` gains the two answers to "where does this build's package set come from", the pin commands, and what a store says about itself. `AGENTS.md`: the `package-ref-hashes.txt` gotcha now says it is a projection of the store and names `refs generate`, which is how a seed-mode store produces it with no `packages/` to reload. And a section next to the build loop on the package set, which says plainly that the pinned path is written and unverified.
The policy is per INSTANCE, and a gate is its own instance, so this one was depending on whatever the dev instance happened to allow. It passed or failed by machine. The symptom is worth knowing, because it names nothing: the LSP writes a debug log, a denied effect raises, and a raise inside the server's request loop takes the loop down. So every request goes unanswered and the gate reports the first assertion after that, which was about branch names. `LspServer.logFilePath` is a hardcoded absolute container path. That is its own bug -- it ignores the rundir, and on an installed machine the directory cannot exist -- and not this branch's to fix; granting it by name is the honest way to depend on it until it moves.
`dark backups` does not list the copy an upgrade takes, so after one went wrong there was nothing that named it. `dark store` now points at `store rollback <n>` whenever the format is above 1. The verb knows its own filename and says plainly when there is no copy to put back, so this is a pointer rather than a second source of truth about what exists.
Stachu's call, against my recommendation, and his reason is the better one: a kernel entry point changing identity should be visible in review, and `assert-clean-worktree` is what makes carrying the regenerated file non-optional. There is a second argument I had missed, and it is what turns this from a tradeoff into a check. On the PINNED path, `prepare-package-set` regenerates this file from the FETCHED store. If that differs from the committed one, the pin's package set is not the one this kernel was built against -- and with the file tracked, the build jobs' `assert-clean-worktree` says so. Untracked, that mismatch is silent, and it is exactly the failure a pin exists to prevent. The cost is real and stands: two branches that both touch packages conflict here, and GitHub's rebase cannot regenerate it. `AGENTS.md` now says how to resolve one -- regenerate, never hand-merge, since the lines are content hashes and picking sides is meaningless. Checked that a reload reproduces the committed file byte for byte, so the worktree stays clean unless one of the 206 actually moves.
189 of the 206 pinned refs are types, so the 17 fns that resolved by name were the small half of the coupling between the kernel and the package set. What made this look impossible was a claim in `PackageRefs` that turned out to be half right: a `DRecord` the kernel builds does carry its type's hash, so a store whose version of that type has a different SHAPE would hand Dark a value it cannot typecheck. That is what the declaration check is for. What the claim missed is which way the coupling runs. All 44 `fromDT` conversions match `DEnum(_, _, …)` -- F# never reads the type name it receives, only the one it writes. So the hash is write-only, used to tag values rather than recognise them, which makes it a lookup by name that happens to be cached in a file rather than a contract that has to be. The net is the fn net with the comparison swapped: shapes instead of signatures, descriptions stripped so a doc edit does not cost the store its binding. The case it exists for is the one with NO pin -- a type a branch has just authored, which F# on the same git branch wants to reference. There is nothing to compare against, so the store's answer is taken. Caching is not a nicety. These sit under Option and Result construction, so a raw query per call measured at 46% more allocation on the reference workload (9.4 -> 13.6 MB) with the wall clock barely moving, which is why time would not have caught it. Two layers: `Caching.withCache` for the store lookup, cleared by the fold, and a memo in the ref closure against a generation the fold bumps, so a resolved ref is one int compare. `record` stays off the hit path; it is a `Map.add` into the generator's lookup table and calling it per resolution allocates a map node per Option construction. Lands at about 6.5% over, isolated by running the same workload with `DARK_REFS_BY_NAME=0`. WHAT THIS DOES NOT DO, and it is the reason the arc is not finished: the resolver reads `locations`, which is main's projection and has no branch column. Authoring two items on a branch leaves `locations` with zero rows for them and `op_branches` with two. So a type authored on a branch is invisible here, and the coordination case this exists for does not work yet. The generator has the same blind spot, for the same reason. Both are named in the design note.
The two reference each other in both directions -- the kernel names 17 fns and 189 types, the package code calls builtins -- and a kernel and a package set are compatible only if both resolve. Nothing checked either direction, and one of them could not be checked at all. DIRECTION ONE, and three defects in the way. `selectBranch` did not invalidate the caches. A one-shot `dark` selects its branch before resolving anything and never notices; the LSP, the REPL and any daemon would answer for the branch they started on, forever. My bug, from adding the memo earlier today. `PackageRefsGenerator` read `locations`, which is main's projection and has no branch column -- authoring two items on a branch leaves zero rows there and two in `op_branches`. So standing on a branch it wrote a hash file describing main, silently omitting the items the branch exists to add, which is the case where the file matters most: it is what lets somebody else build the F# that references them. `refs generate` in LocalExec never selected the stored branch at all. With those fixed, type refs resolve through the branch overlay. Fns deliberately do not: resolving one through an overlay means the binary runs a branch's parser the moment you stand on that branch, which is not something you opted into by switching. A type declaration is inert -- resolving it decides what a value is tagged with and executes nothing -- so a branch may lend the kernel its types and may not lend it its code. DIRECTION TWO needed a primitive that did not exist. `package_dependencies` is package-to-package only; `PackageItem.fnPackageHash` answers `None` for a builtin, so the call was dropped on the floor and a store could not say which kernel it needed. `package_builtin_deps` records it, filled by the same AST walk that fills the other -- one walk, because walking twice to ask two questions about the same nodes is how the two answers drift apart. That replaces the two builtin discipline tests that grep `.dark` text off disk and evaporate the day `packages/` goes. Strictly better than they were: the real call graph rather than a regex. `refs check` asks both at once, wired into every build after a compile or a reload. At build time and completely, because the ref closures are lazy -- an unresolvable ref is otherwise found whenever some code path happens to reach it, which can be a different day and an unrelated command. The case it exists for is checking out somebody's git branch without their package work. All 206 kernel refs resolve, and all 734 builtins this package set calls exist in this kernel. Both halves verified by breaking them: an F# ref to a type nothing binds, and a row claiming the set calls a builtin that was deleted. An EMPTY `package_builtin_deps` is refused rather than passed -- it is a projection, so a store that got the table from a release step without re-folding would otherwise report success having asked nothing, which is exactly how the tests it replaces would have quietly stopped covering anything. The raise for an unresolvable ref now names the type and the three ways out, instead of saying the hash file is stale, which was usually untrue and never actionable.
Today `git clone && scripts/dev/start` works from the repo alone. After the flip it needs the network and a live server, which is a robustness regression I had not considered at all and which matters most for exactly the person with the least: a contributor on a plane, or anyone when the server is down. A seed cut at a COMMIT never changes, so it is worth keeping once per machine. With this, a machine needs the network once per pin; a second clone, a re-clone, or working offline is a file copy. Only for a PINNED fetch -- `latest` is not immutable and must not be cached under a name that claims it is. And when there is nothing to fall back to, the failure names the way out (`--from <path>`) rather than being a bare curl error. Tested three ways: cold with the server up, a second clone with the server killed, and a third with the cache deleted too.
`tryResolve` accepted any non-empty hash, so a pin naming content the store no longer holds counted as resolving. That is the failure measured this morning: a stale type hash produces no error and blank output at runtime. The check now verifies the hash against the set of hashes the store holds content for, asked once rather than per ref, so it is caught at build time instead. And when the check fails and git is on a branch with a same-named dark branch, say the command: git is on `add-foo` and there is a dark branch called `add-foo`, but you are on dark main. Try `dark switch add-foo`. That is the F#/package coupling made visible at the one moment it matters, rather than a rule somebody has to have read. Deliberately NOT an auto-switching "dark branch follows git branch" tier. Every ordering I worked through either overrides an explicit `dark switch` or fails to let one stick, and it is not a thing I can test on a person. A hint that carries the fix is the version worth shipping without that. Finding the git branch walks UP for `.git` rather than assuming the rundir sits directly inside the repo: true for the dev rundir, false for a test's, and silent either way.
`/seed/<commit>.db` is anonymous -- it has to be, since fetching a pinned package set is the one thing a contributor with no access must be able to do -- and it kept a ~12MB file per commit asked for. The commits are discoverable through the equally public `/sync/pull`, so a stranger could walk a store's history and make the server keep a file for every commit in it. Eight cuts, oldest-by-mtime evicted, which is least-recently-asked-for because a cut is only written when it is missing. Housekeeping failures are ignored: refusing to serve a seed because a stale file would not delete is the wrong trade. The first version raised inside the handler and turned every seed request into a 500, because `8L` is an Int64 while `List.length`, `List.drop` and `getModifiedTime` are all `Int`. The probe counted files on disk and never looked at the status code, so it reported the cuts as present while the endpoint was erroring. It asserts the codes now.
An F# change can reference package code, and then the two have to travel, merge and be abandoned together. Git already does that for the F#; this lets it do the same for the package half, so one PR carries both and reviewing or dropping it is one act. scripts/packages/bundle export|import|show Export is EXPLICIT, like `git add`: it writes into your tree and that should be something you asked for. Forgetting is caught rather than silent -- `refs check` fails the build for anyone whose F# names package code they do not have, and names the branch to switch to when there is one. Import is AUTOMATIC, from `prepare-package-set`, because checking out the git branch IS asking for that branch's package code. Idempotent: ops are content-addressed, so the second import says "up to date". Going back to dark main REMOVES the bundle rather than leaving it. A bundle that outlives its branch is how somebody imports work that was merged weeks ago. Measured: a type plus a function is about a kilobyte of JSON, so the bundle is smaller than the `package-ref-hashes.txt` diff it travels with, and `bundle show` gives a reviewer the branch and the op count without fetching anything. Stachu's shape was "the PR names a dark branch and the merge button coordinates"; this is the other half of that, not an alternative to it. The NAME is what merge coordination acts on; the BUNDLE is what lets somebody with no server access participate at all.
The seed is fetched and installed before the cache is written, so a cache that cannot be written costs the next fetch a download and nothing else. Failing there instead turns an unwritable home directory into a failed build, which is what it did: the `seed-serving` gate downloaded 12.3MB successfully and then died on `mkdir: /home/dark/.darklang/seeds: Permission denied`. Same lesson as the server's cache prune, paid twice in one afternoon: housekeeping must not be able to fail the thing it is housekeeping for. Reads are best-effort too -- an unreadable cached file falls through to the network rather than erroring.
`bundle show --source` renders what the bundle changes, with the full
declarations, by importing into a DISPOSABLE store. Reviewing a stranger's
branch should not mean taking their ops into the store you work in, and a PR
reviewer is exactly the person with the least reason to trust the bundle in
front of them.
"review-me" changes 2 names vs its parent:
Darklang.LanguageTools
+ Reviewed [type] dd5ab310
type Reviewed =
{ a: Int64
b: String }
+ useReviewed [fn] 4ac94fb1
let useReviewed (): Int64 = 7L
And `bundle export` warns when your main has moved past the pin. A bundle
carries the BRANCH's ops; if your branch rests on commits only you have, the
bundle is not enough for anyone else, who holds the pinned set plus this file
and nothing more. CI says so too, as a failed `refs check`, but by then it is
somebody else's red build.
Asking `--branch main` explicitly, because `dark log` shows a BRANCH's ops when
you are standing on one -- the bare form reported "your main is at AddFn".
The replacement for opening a `.dark` file, and after the flip there is no file to open. It renders every declaration directly under the module, hands you $EDITOR, and applies the result through `dark module` -- which validates all of them before saving any, so a half-finished edit lands nothing rather than half. The design question was deletion by omission, and it is where this command would lose work. `dark module` adds and updates; it does not unbind. So leaving a declaration out of the file could have meant "remove it" -- and one mis-parse that dropped an item from the RENDER would then quietly end a name. It does not guess: omissions are named back to you and left alone, and `dark remove` stays the way a name ends, in one op, confirmed, visible in `dark status`. The report only fires after an apply that landed. A parse error lands nothing at all, and saying "left alone" about it reads as though the rest went in. Render order is types, then values, then fns, each alphabetical -- deterministic, or a render-edit-save round trip churns the file for nobody. Everything else is the single-item path's, which already got the risky parts right: a non-zero editor exit applies nothing and keeps your file, and a rejection prints where your edit still is. A whole module is more to lose. One existing assertion said "a module is refused"; it now checks that a module with no terminal points at the file form. That test caught the behaviour change the day it happened, which is what it is for.
`refs check` ran as a `scripts/build/compile` action, and CI never runs that script -- it calls `prepare-package-set` and then builds. So the one invariant this whole scheme rests on went unchecked in exactly the place it matters most. It runs wherever the package set is ESTABLISHED now, on both the pinned and the unpinned path, which is the same reason `prepare-package-set` exists at all: one place answers "where does this build's package set come from", and it should answer "and do the two halves agree" in the same breath.
A kernel entry point's identity moved, and it is visible here because the file is tracked -- which is the argument Stachu made this morning, paying off the same day. `dark edit <module>` changed `Packages.Edit.execute`; the registry references it, `Cli.executeCliCommand` references the registry, and a reference is to CONTENT, so the hash walks up. Nothing about the entry point itself changed, which is exactly the kind of move that is invisible without a diff to look at. Reproducible: a fresh reload produces the same file byte for byte.
Stachu's call, and it only became available this afternoon. Once type refs resolve from the store BY NAME, the file is a fallback rather than a contract: between pin bumps the committed copy is simply correct, because anything it is stale about resolves by name anyway. So writing it on every reload stopped being necessary -- and writing it on every reload is what made every pair of package-touching branches conflict in a 206-line generated file. That was the cost accepted this morning when the file was first tracked. It no longer has to be paid. A pin bump now produces one reviewable diff naming every kernel identity that moved, which is the signal the file was tracked for in the first place. `generateWith` splits the two halves, because they have different cadences: the hashes must be in MEMORY before values are evaluated, on every reload, or PackageRefs lookups resolve to nothing during it. Only the FILE waits for the pin. Verified by moving a hash and watching nothing happen: changed a package file that shifts `Cli.executeCliCommand`, reloaded, and the file stayed byte-identical while `refs check` still passed 206/206 -- which is the property that makes this safe.
Stachu's call: locally anyone writes anything; on the shared server you may publish freely to your OWN namespace, and `Darklang.*` goes through a branch and a review. So somebody who has just logged in and written a function can share it without a branch, a PR or a person -- and the namespace the server resolves its own router through still cannot be rebound by a stranger. This is `reservedOwners` reborn at the one edge where it belongs. It used to sit on every write path including local authoring, which was wrong and is why it was deleted this morning. THE THING THAT MADE IT REAL, and it only showed up by running it: the first version refused every push, including the newcomer's own-namespace one. `dark push` sends the whole log, and every client's log carries the `Darklang.*` baseline it was seeded with, so the raw batch always binds reserved names. Only ops the server does NOT already have can change anything -- the rest are content-addressed no-ops -- so those are what gets checked. Without that this rule locks everyone out on their first push. A refusal is a 403 naming the item, not a 500. A client reading "the server broke" has every reason to retry forever. `scmReservedBindings` answers before storing so the handler can say so properly; the rule is still enforced inside `storeOpsWithOwner` as the backstop. Three cases in the `server-folds` gate: your own namespace accepted, a reserved binding refused and named, and the same work on a BRANCH accepted -- because a branch is isolated, nobody runs it, and review is what moves it to main.
The workflow nobody had written down, and the thing a person actually reaches for: adding a builtin and calling it from Dark, referencing a new package type or function from F#, what your coworker has to do to build your branch, trying the two-machine shape locally, publishing, the pin, and what happens when the op-log format changes. Plus a "things that will bite" section, which is the part I would want if I picked this up cold: `dark log` on a branch shows ops rather than commits, a bundle outliving its branch, a local main drifting ahead of the pin, and two branches adding the same name with different shapes -- a merge conflict git cannot see. Written for both modes. Everything in it works today except where it says otherwise, which matters because the pin ships unset and the reload is still what happens.
The test lock and why a broad pkill is the wrong way to clear it, Sqlite's @p0 parameters, the registry sweep taxing the suite, the Dark syntax traps, and measuring the artifact people actually run. These were in a working document that has now been folded away; they belong where agents read them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Production's package store becomes the source of truth: the server serves a seed of it, a dev container and CI fetch that seed instead of parsing text, F# talks to it by pinned hashes and a few frozen names, and a format change migrates the store forward.
packages/is still the source a build uses and is not deleted here; deleting it is a separate, later, one-way step.Everything with real blast radius lands switched off. Seed mode is opt-in, the package pin ships
commit unset(which means "build frompackages/", i.e. today), and the store migrator is a no-op until the first format bump.The five things the plan said must exist before the flip
The plan called these the things that must exist before the flip -- "the day the text goes, these are what people reach for instead" -- and this is what each is now.
Done.
dark modulealready validated many declarations and saved them in one batch, so this is render, edit, apply through that: all of them validated before any are saved, so a half-finished edit lands nothing rather than half. It does NOT author a deletion for an omitted declaration, deliberately -- that is the one place this command could lose work, because a single mis-parse dropping an item from the render would quietly end a name. Omissions are named back to you and left alone;dark removeends a name.Done. Cached by content hash, which never needs invalidating. Reads through the branch overlay, not
locations-- a guard test caught it searching main from a branch.Done, and the plan was wrong about the cascade: there is none and none is needed. A reference points at CONTENT, so a rename cannot reach callers at all.
Done. I had parked this as "semantically ambiguous" because I assumed commit ordering did not exist;
commits.parentis right there, so ancestry is a recursive CTE. It means the commit or any ancestor, which is the useful reading.Done. One op,
confirmExplicitso a stray Return cannot end a name. The claim it prints -- the name ends, the content and its callers do not -- is tested rather than asserted.How to actually do any of this
docs/package-workflow.md, added in this PR. It covers the things a person reaches for: adding abuiltin and using it from Dark, referencing a new package type or function from F#, what your
coworker has to do to build your branch, trying it locally before it goes anywhere, publishing, the
pin, format changes, and the handful of things that will bite.
dark docs packagesis the shortversion from inside the CLI.
What merging this does NOT give you yet
Worth being plain, because the five above are the prerequisites and not the flip.
Merging this changes nothing about the daily workflow.
package-set.txtshipscommit unset, seed mode is not the default, andDARK_CONFIG_PACKAGES_SOURCE=disk. So: still.darkfiles, still a package reload on every change, exactly as today.Three things have to happen to get the workflow the plan describes, and only the third is mine:
/seed/*has only ever run against a local server. This is the real blocker.scripts/packages/pin head --url <server>-- and flipDARK_CONFIG_PACKAGES_SOURCEtoseed.packages/(8.J), which is the one-way door and deliberately untouched here.And an honest caveat about step 2: the flipped workflow -- two people, seed mode, pinned, F# referencing package code -- has never been run end to end, because it needs a server. Every PIECE of it has: a container starting in seed mode with no store and no reload, a branch bundle moving between two stores, the kernel resolving a branch-authored type, both directions of the compatibility check. The integration has not.
The remaining path, in order
From merging this to two people developing Dark with no
.darkfiles and no package reload. Only the first needs anything that does not exist.1. Stand up a package server. (yours, an afternoon)
dark serve Darklang.Matter.routerwith a write secret, a volume, and a name. It is the same binary you already ship. This is the only real blocker: the pinned path has only ever run against a local server, so it is also the first time any of it is exercised for real.2. Seed it and pin to it. (10 minutes)
CI's
seed-dbjob then fetches at the pin instead of reloading, cached on the pin so a server outage only blocks a re-pin.3. Run the integration once, deliberately. (an hour, and I would not skip it)
Two clones, seed mode, pinned: author a type on a dark branch in one, reference it from F#, bundle it, build it in the other. Every piece is tested; this is the first time they run as one thing, and it is the cheapest moment to find out something is awkward.
4. Flip the default. (one line)
DARK_CONFIG_PACKAGES_SOURCE=seedinconfig/dev. This is the point at which the reload stops. Measured in a container: a.darkchange goes from an 11s reload to 0.238s of nothing, and an.fschange drops the reload entirely.Note what changes for you here:
packages/**.darkedits stop doing anything, because the store is the source. You author withdark fn,dark type,dark edit <module>from then on. That is why the five verbs had to exist first.5. Live on it for a week. (free, and the real test)
The plan's own risk list says it plainly: "If
dark edit <module>is not good enough on a Friday, the pressure to bring the text back will be real." Better to find that withpackages/still in git.6. Delete
packages/(8.J -- yours, one-way)Only after 5. Nothing in this PR touches it.
The honest summary: steps 1 and 2 are half a day and give you a pinned CI. Step 4 is the line where the reloading stops. Step 6 is the one you cannot undo.
What two people working together gets, once flipped
This is the case the plan did not cover and the one that took the most work today. An F# change can reference package code that only exists in somebody's store, and those two have to merge together and vanish together if abandoned.
Your coworker checks out the branch;
prepare-package-setimports the bundle; their kernel resolves your type. Verified across two stores, with the negative controls: on the branch it resolves, on main it does not, and with by-name resolution off it does not.If they forget, they are told at BUILD time, completely, rather than by whatever command trips over it first:
The decision that unblocked it
Inserts.reservedOwnersrefused every authored or synced write underDarklang, on all three write paths. It was the same defence three separate items were blocked by, from three directions: a pushed op rebinding the server's ownDarklang.Matter.router, the store rebinding the kernel's seventeen fn refs, and (after the flip) anyone editingDarklang.Stdlib.*at all, since the only sanctioned writer was the reload that the flip deletes.It is gone, on all three paths, and
effective = 0with it. Anyone who has access gets access; the security model is deliberately deferred rather than answered.Inserts.fskeeps a comment recording what it defended, and the server's/statspage says plainly that a push here can rebind what the server itself resolves and that the write secret is currently the whole of the answer.The server folds, and serves a seed
A push now folds into the server's main like any other arriving op, so
/m,/pand the seed routes show hosted packages rather than only what the binary shipped with.The bug that nearly shipped:
effectivewas carrying two meanings, "fold this" and "this is mine". Only the first was given up, so with it flipped a pushed op looked like the server's own uncommitted draft anddark discardwould have deleted a peer's work. The discriminator moved toop_owners, which records who pushed each op and is what the flag always meant; every draft and WIP clause now excludes it. Two existing tests caught this, which is what they were written for.Folding is not running.
evaluateAllValuesskips a value whose bindings are all hosted, so a pushedvalis folded, browsable and servable in a seed, and never executed on the server. Whoever fetches it evaluates it on their own machine under their own policy. "All hosted" and not "any": content is shared, so a value the server authored can also arrive by push from a peer who wrote the same thing, and anything locally bound is still evaluated.op_ownersis empty on an instance, so nothing about a client changes.New routes:
/seed/latest.db,/seed/<commit>.db,/seed/meta.latest.dbresolves to the head and is served as the cut AT that head, so it is the same bytes as asking for that commit by name. Cuts are cached beside the store keyed by commit and never need invalidating, because commits are immutable and ops only append. A short prefix resolves like git's; an ambiguous one is refused, since a newest-wins tie would pin a package set nobody chose./seed/meta(optionally?commit=) answers the fields a pin needs without the twelve megabytes.The pin
package-set.txtrecords which package set this kernel was tested against. Two fields,commitandformat, and deliberately no URL: a pin says which package set, not where to get it, and two mirrors serving the same commit are interchangeable. The address comes fromDARK_SEED_URLor--url.scripts/packages/pin head | <commit> | --show | --unsetwrites it, and checks every pin before writing it: the server resolves the commit and answers what a cut at it would carry, so a bad pin is refused at pin time instead of at the next CI run.scripts/build/prepare-package-setis the one place that answers "where does this build's package set come from", and CI's three package-reloading jobs call it instead ofreload-packages. It does not fall back to reloading when the server is unreachable: a build that silently used the tree instead of the pin is the failure the pin exists to prevent.seed-dbgained a cache keyed on the pin rather than on the week, because a pinned package set is immutable.scripts/run-local-exec refs generateis what makes a fetched seed usable. The kernel resolves its entry points by hash, and that file is a projection of the store rather than something a seed carries; the reload used to write it as a side effect.package-ref-hashes.txtis tracked, as the plan asked. I argued against it and was wrong twice over. On the pinned pathprepare-package-setregenerates it from the FETCHED store, so a difference from the committed file means the pin's package set is not the one this kernel was built against, andassert-clean-worktreeis what says so; untracked, that mismatch is silent, which is the exact failure a pin exists to prevent. And a stale pinned TYPE hash makesdark eval "1L + 1L"print blank with no error, so untracked, a branch switch left you carrying the other branch's hashes and failing silently. A stale FN hash no longer matters, because the by-name net resolves it from the store.The rebase cost stands: two branches that both touch packages conflict here.
AGENTS.mdsays how to resolve one, which is to regenerate rather than hand-merge, the lines being content hashes.Kernel entry points resolve from the store, with a net
DARK_REFS_BY_NAMEnow defaults on. F# only calls the seventeen fn refs and never takes one apart, so the frozen contract is the name and the signature, and the newest committed binding can win.The net is why the default could flip: a candidate's signature is compared against the pinned version's (parameter types, return type, type-param count), and a mismatch falls back to the pin with a warning naming the fn. The pin is what "what this build expects" means, and both versions are in the store because content is never deleted, so nothing further had to be pinned.
=0remains the escape hatch for a store broken in a way the signature check does not catch.Types stay pinned by hash, and that is not a choice: a
DRecordthe kernel builds carries its type's hash, so a store whose newest version of that type had a different shape would hand the kernel a value it cannot read.Main's committed projection only. Resolving entry points through a branch overlay would mean the binary ran a branch's parser the moment you stood on one.
Moving a store between formats
An op's id comes from
Hashing.computeOpRowIdover the decoded op, not from its bytes, and that divides format changes cleanly:LibSerialization/Binary/*moves the layout and nothing's identity. The migration is a blob rewrite, ids untouched, projections re-folded. Built, behinddark store upgrade.LibSerialization/Hashing/*moves every op id, item hash and commit hash at once, which is a whole-log re-mint. Not built.LibDB/StoreUpgrade.fsends with the six steps it needs and the two properties that make it worth the care.The migrator refuses rather than guessing which it faces: it recomputes each op's id from the decoded op and compares it with the id the store filed it under. Disagreement means the store's ids were minted by different hashing, and a blob rewrite would be a lie.
The backup lands first, through SQLite's online-backup API, so the rollback target exists before anything is touched. The rewrite and the format stamp are one transaction: outside it, a crash between the two leaves a store whose blobs and whose claim about them disagree, which is worse than either failure alone.
validateVersionnow accepts any version up to this build's and refuses only newer. An old blob can be read because every historical reader stays in the binary; a newer one cannot be read by trying harder.A build that reads an older format than the store gets a note rather than a refusal, and the note names a file to move rather than a verb to run. Refusing to open would refuse the rollback too, and the projections hold newer blobs as well, so such a build dies resolving the name of whatever command you typed. A
mvneeds no working binary.Every seed says what it is
store_metacarriesformat,cut_at,kernelandat, written by the cut because it is the only thing that knows.cut_atnames a commit even when no cut was asked for, read back after the cut, so a seed always says what it is a cut of.The kernel and the package set check that they agree
They reference each other both ways: the kernel names 17 fns and 189 types, the package code calls builtins. A kernel and a package set are compatible only if both resolve, and that single invariant decides whether a pin can be bumped, whether an F# change can merge before or after a package change, and what removing a builtin has to do first.
scripts/run-local-exec refs checkasks both, wherever the package set is established.Type refs resolve from the store by name now, as the fns already did, and the net is the fn net with the comparison swapped: declarations instead of signatures. The case it exists for is the one with no pin, which is a type a branch has just authored and which F# on the same git branch wants to reference.
Direction two needed a primitive that did not exist.
package_dependenciesis package-to-package only, andPackageItem.fnPackageHashanswersNonefor a builtin, so the call was dropped and a store could not say which kernel it needed.package_builtin_depsrecords it, filled by the same AST walk. That also replaces the two builtin discipline tests that grep.darktext off disk and evaporate the daypackages/goes -- the real call graph rather than a regex.A git branch carries the package work it needs
scripts/packages/bundle export|import|show. An F# change can reference package code, and then the two have to travel, merge and be abandoned together; git already does that for the F#. Export is explicit, likegit add. Import is automatic fromprepare-package-set, because checking out the git branch is asking for that branch's package code. Returning to dark main removes the bundle, since one that outlives its branch is how somebody imports work merged weeks ago.bundle show --sourcerenders what it changes, with full declarations, by importing into a disposable store: reviewing a stranger's branch should not mean taking their ops into the store you work in.Measured: a type plus a function is about a kilobyte of JSON.
The five verbs
dark remove <name>authors theUnbindnothing wrote before. One op; the content stays reachable by hash and callers go on working, because a reference points at content rather than a name. That claim is tested rather than asserted.dark revert <name> <commit>puts a name back to what it held at a commit or any ancestor.commits.parentis the chain, and it exists because a commit's id is derived over its parent the way git's is. Short hashes resolve like git's; an ambiguous prefix is refused.dark grep <pattern>searches the source of what is live, wheresearchlooks at names. Nothing stores source, so it is rendered from the AST and cached by content hash, which never needs invalidating. An unscoped search against a cold cache is refused rather than working silently for a minute.dark renameverified: it authors both ops and a caller keeps working across it. Plan correction: there is no propagation cascade and none is needed, because a reference points at content, so a rename cannot reach callers at all.dark edit <module>opens every declaration under a module and applies them in one batch, all validated before any are saved. Leaving one OUT does not end it: omissions are named back to you and left alone, because guessing the other way is where this command would lose work -- one mis-parse dropping an item from the render would quietly unbind a name.dark removestays the explicit way a name ends.dark storereports the stamp, and carries the upgrade and rollback verbs.Three SQLite bugs, all only reachable on a long-lived process
The server cuts more than one seed per process, and that is the first thing in the tree that does.
exportAtcheckpointed the source through a read-only connection and then copied the file. The moment there is actually a WAL to fold in, that fails with a disk I/O error. It goes throughSqlite.Backup.toFilenow, which also means it exports the store a test repointed LibDB at rather than the config one.connStringForsetsPooling=true, and a pooled connection outlives itsClose, so the second cut got a handle to a file the first had deleted and replaced. Backup targets go through an unpooled connection string..dbalone was not the whole database and whether a fetch got everything depended on when a checkpoint happened to run. It ends injournal_mode=DELETE.Tests run against your store
In seed mode the test store is a copy of
data.dbrather thanseed.db, the seed being the one store guaranteed not to contain the package you just authored. Throughsqlite3 .backup, notcp, because the store is WAL and what a plain copy drops is exactly the new work. CI keeps the seed: it has no store of its own and wants the pinned one.What is covered, and where
Three new gates, all of them crossing a boundary the F# suite structurally cannot:
server-folds: a push reaches the server's own projection, stays out of its draft, and is never run there. Needed a real push from another store and then a restart, since the startup grow is the half that would have evaluated the body.seed-serving: a seed is fixed by the commit it names, follows the pin, and grows into a working store. The pin property is tested by moving the store underneath it: cut at a commit, push, delete the server's cache so the cut is recomputed, ask again.store-upgrade: a synthetic format bump carried in place and rolled back.DARK_FORMAT_VERSIONmakes one build write a version on while reading both, which is the only way to exercise a migrator when only one format exists. The most important assertion in it is a forged op filed under an id its own content does not produce, which must stop the upgrade dead.And a serialization corpus:
serialization-artifacts/corpus/v<N>/, ten types, never regenerated. Different from the golden files beside it, which are regenerated whenever a format change is intended and so stop covering the old format the moment it moves. This decodes a blob a past build wrote, re-encodes with today's writer, decodes that, and compares, which is the migrator's core operation. At the current version it also asserts byte-exactness as a prefix, which catches a layout change made without bumping the version.Untested here, and it matters
The pinned path has never run against a deployed server, only against a local one in the
seed-servinggate. Nothing was pushed or deployed from this branch, so the fetch-at-pin half of CI is written and unverified. It lands inert (commit unset), so it changes nothing until someone pins.Not done, as judgement rather than blockage
ParseErrorthinning. Thirteen numeric error types into one touches around a hundred F# sites and changes what every numeric parse returns, so every Dark caller matching on it breaks. It is also identity-affecting, so it moves every pinned hash.package-ref-hashes.txt. Argued against above; it changes everyone's rebases, which is not mine to spend.