Experiment: structural editor in CLI -- inspired by -classic, composed, extensible (, incomplete) - #5754
Draft
StachuDotNet wants to merge 346 commits into
Draft
Experiment: structural editor in CLI -- inspired by -classic, composed, extensible (, incomplete)#5754StachuDotNet wants to merge 346 commits into
StachuDotNet wants to merge 346 commits into
Conversation
… the undo history Text mode is the escape hatch from structure editing; Ctrl-O is the escape hatch from text mode. It writes the buffer to a temp file, runs `$EDITOR` on it with the real terminal, and reads it back when the editor exits. `Step.Launch` reads exactly like what this needs, and is the wrong thing: the string it takes is looked up in the CLI's COMMAND TABLE, so `vi /tmp/x` is a command by that name and there isn't one. It exists for launching other dark commands. So there is a new step, `RunProgram`, through Component -> SubAppAction -> the loop, which spawns a real program with `Process.runInteractive` -- written for precisely this, and unused until now. Down, run, back up; no page stack, because the program is not a page and this page is still where you are when it exits. `onResume` runs `refresh` now. A resume is a transition, and a program launched this way is launched because it changes something. Without it the file could only be read back on the next keystroke, so the screen would show the stale buffer until you touched a key. What is verified and what isn't: the CLI runs in a sandbox with its own filesystem view and cannot see a script written from outside it, so I could not stand up a fake editor that rewrites the file. `EDITOR=/bin/true` runs and the buffer comes back unchanged; `EDITOR=/bin/rm` gives "could not read /tmp/dark-tmp-...", with the buffer left intact. That is the whole chain -- written to a real file, handed to a real program, re-read afterwards -- but a human in vim is not something anything here can reach, and the testfile suite can pin none of it. Separately, found by writing a program in the terminal and then pressing undo: applying a WHOLE-document text edit built a FRESH editor, which threw the undo history away, so everything typed before the trip was unreachable and `u` answered "nothing to undo". Splicing a sub-node always kept history, because that branch committed an edit. The whole document is just the root path, so they are one branch now.
`|` at the end of an enum case's last field went in as a literal character:
`| Circle of Int|`. `enclosingListItem` finds the NEAREST enclosing list, and
inside a case that is the case's FIELDS, drawn with `EachWrapped(" of", ...)`
which leads its items with nothing. The list that draws a `|` is the enum's CASES,
one further out.
The enclosing lists are tried from the inside out now, and the first one that both
leads its items with the character typed and has the caret at the end of ITS item
wins. `addSibling` grew a targeted form that takes the list to add to rather than
finding the nearest itself.
The four ways "at the end of an item" can look are pulled out into `atEndOfItem`
and spelled out there, because knowing only one of them is how this shipped broken
the first time: in a node's text, after a node, after a CONTAINER whose last node
is inside it, and in a slot holding a partial.
The two keys now do different things in the same spot, which is the point: `|`
adds a case, Enter adds a field. Both pinned, side by side.
…ipboard gets tests Generating an editor for an enum handed you a stub. `fluid generate Darklang.Stdlib.Option.Option` emitted `pieces = [ Piece.Dim "?Option" ]` and nothing else; records were fine, which is why it went unnoticed. The cause is narrow. `Derive.specsFor` already derives an enum properly -- the type-level spec that owns the hole, plus one spec per case with real layout pieces -- and the command took the head of that list, which for an enum is the type-level one whose pieces are the placeholder. So the module's editable part changed nothing you could see, while the per-case layouts it was meant to start from went unmentioned. It emits one `val` per case now, named for the case, each starting from that case's own derived spec through a new `specForCase`. That is how discovery works: it finds each Spec value by type, so several values in a module are several editors. Pinned by running what it emits through the PARSER, for a record and two enums, since the point of `generate` is source someone pastes into their packages. Also: yank and put had no tests at all. A whole editing feature, reachable only by driving the thing by hand, and what it printed was the fully qualified type name (`yanked Darklang.LanguageTools.ProgramTypes.Expr.EInt` for copying a `1`). It uses the short name the breadcrumb already uses, and there are pins now for copy, put over a hole, undo taking the hole back, yanking again replacing what is held, and the empty-clipboard message.
`v` on a package value reported `can't print that yet: unr`. Sliced mid-word at the panel's fixed width, with nothing to say it had been cut, and widening the terminal does not widen the panel. An error you cannot read is worse than no error: it reads in full now. The actions list had the same shape of bug -- it took the first seven and stopped, so seven actions read as all of them. It says `+N more, : for all`, and the value says `+N more`, the way the completion popover already did. Worth naming, because it is a class and not two bugs: the panel IS the help, so anything it silently drops is something the user is told does not exist.
`fnNameResolution` refused any `builtin:` hash outright, so `let length (list:
List<'a>) : Int = Builtin.listLength list` -- the shape of every stdlib wrapper --
was a function Fluid could draw and could not save. Refusing was the right answer
while the name could not be rebuilt: better "not saved" than a function quietly
rewritten without its body. The name can be rebuilt now.
The document already carried `builtin:listLength_v0` for functions. For VALUES it
dropped the version, so a builtin value could not be rebuilt even in principle --
`v` on one reported "unresolved value name: Builtin.scmMainBranchId". Both carry
it, one `builtinFromHash` parses it back, and each side rebuilds
`FQ*Name.Builtin { name; version }` with no package location, which is what the
printer expects when it writes `Builtin.listLength`.
The pins that asserted the refusal say "same" now, with the comment rewritten to
say what changed rather than deleted: they recorded a deliberate limitation, and
the limitation is gone.
Measured after, rather than assumed: 300 real types and 300 real values round-trip
identically, and of 300 real functions, 202 are identical and 98 differ only in the
`(state)` parens the printer puts on a parameter reference, plus the re-wrapping
removing them causes. Nothing structural in any of the 900. The parens are not a
save bug -- `saveFnBody` prints to source and hands it to `applySource`, which
re-parses, and the parser lowers a parameter reference back to EArg.
`,` was written into editor.dark, in two places, so it was the `,` of a list and
nothing else. The character that separates items is something the SPEC draws --
`,` between list elements, `;` between a record's fields -- so the kernel asks the
projection, the way it already asks what is drawn BEFORE each item for a match's
`|`. Same inside-out walk over the enclosing lists, same "at the end of an item"
test. The hardcoded `,` stays as a deliberate fallback, so nothing that worked
stops working.
Two things came out of driving it rather than reasoning about it:
A separator typed between items that ALREADY EXIST and are not finished means "on
to the next one", not "put another one here". `;` after `width = 80` was inserting
a third field into a two-field record, which its type does not allow. A list is the
other way round -- `[1, 2]` and a `,` after the `1` is a new element -- and there
the next item has nothing left to fill, so it still inserts. The rule reads off
that difference rather than knowing what a record is.
Then it landed the caret on the next field's NAME, because `firstEditable` of a
record field is its name and the name is already written, so typing `24` put digits
in `height`. It goes to the first HOLE, which is what "on to the next one" means.
`Size { width = 80; height = 24 }` now types exactly the way the source reads.
`"`, `{` and `}` were written into `handleStringI`, so a language that interpolates
with something else could not say so. They are `Syntax` fields now --
`stringDelimiter`, `interpolationOpen`, `interpolationClose` -- alongside
`operatorChars`, `closingDelimiters` and `separators` on the registry.
`stringDelimiter` is deliberately not folded into `closingDelimiters`: those are
closers you can type past, and a `'` typed inside a string is a literal apostrophe
rather than the end of anything.
The custom-syntax registry in the tests -- the one that spells operators with `@`
-- interpolates with brackets, and the pins say all three halves: `{` in Darklang,
`[` in that language giving the same document, and `{` in that language now being
an ordinary brace. The third is the one worth having, because it is easy to add a
new spelling and leave the old one quietly working underneath, which would mean the
kernel still knew.
That is the last of the five things the extensibility audit listed as fixed in the
kernel. All five are on the registry now; what remains there is the kernel naming
ProgramTypes cases, which is fine while PT is stable.
`Derive.pieceSource` knew fifteen of the twenty-four pieces and had no fallback, so it was a crash waiting for the first spec that used one of the other nine. Nothing had hit it because derived specs only ever use the fifteen. This follows from the reason for keeping `OperandSlot`, `ParenLowSlot` and `FieldTargetSlot` as their own cases rather than folding them into `ParenIfSlot`: specs are DATA, which is why they can be derived, generated and stored -- and that is only true if all of them can be written out. Seven more print properly now. The two that carry a predicate cannot: a function has no source. They emit a marked comment saying a person has to write the predicate back, so what comes out will not compile. That is the point -- better than emitting something that looks right and means something else.
Two small honesty fixes, both found by using the editor rather than reading it. `v` on a return type answered "can't print that yet: not an expression node: ...TypeReference.TInt". Nothing is wrong with printing -- `v` runs code, and a type is not something you can run -- so the prefix blamed the wrong thing. It says "can't evaluate that:" now; the inner message was already right. Tab was missing from the NAV key list. It moves between HOLES, which is how you reach the next thing that needs filling rather than the next node in reading order, and it was listed for INSERT only -- so the panel said it did not exist in the mode you are in when you open a document. That is the third of these in three ticks, after the value text that was cut mid-word and the actions list that stopped at seven without saying so, and they are one bug: the panel IS the help, so anything it drops or mislabels is a thing the user is told is not there.
`Piece.Table` renders a list of records as a grid rather than as a line of text --
the axis the extensibility audit calls the interesting one -- and it was reachable
only from the tests. That is a poor place to keep the answer to "can a table be
seen as a table".
item qty note
"widget" 3 "in stock"
"a much longer name" 12 "back order"
"nut" 100 ""
Every column as wide as its widest cell, which is the thing a Wadler group cannot
do: a column's width depends on its SIBLINGS and a group only ever knows its own.
It also needed something the app did not have. `initialWith` and
`launchInRegistry` open a document with a registry OTHER than the one the branch
derives, and every projection that is not Darklang's brings its own specs -- so
without them the only documents openable were Darklang's, which quietly limited
the whole malleability story to what the kernel already knew about.
The demo grid renders; the question worth pinning is whether you can work in it. Motion down a column and across a row, editing a cell, and adding a row, which is the operation a table is for. A new row comes out as three holes with the columns re-measured around them. The motion pins compare the caret PATH rather than the rendered grid, because the render cannot tell the grid, row 0 and cell 0 apart: the caret marker sits at the same column for all three, so a first probe that diffed the rendering said "j and l do nothing". A comparison that cannot distinguish two things will happily report they are the same. Adding a row is Enter then Enter, because Enter in NAV means "edit" (the same as `i`) and the second one grows the list. The panel said "Enter accept / next item" for INSERT, where "next item" was carrying a lot of weight. It says "accept / add an item", since growing a list is what you reach for most in a table.
`Test.keyOf` ends with `| _ -> printable s`, so a token it does not recognise gets
TYPED. Probing backspace I wrote `<Backspace>`; the token is `<BS>`, and it went in
as fourteen characters, one of which is `<`, which starts an operator:
width = (80 < Backspace) >< Backspace >~ ⟨right⟩
The script ran, produced output, and an expectation written from that output would
have passed. A pin could read as "backspace does X" while exercising nothing of the
sort, forever. That is the worst shape a test bug can take.
`Test.run`, `runFrom` and `PT.Test.program` answer `UNKNOWN TEST KEY: <X>` now, in
the place pins read from. `Test.drive` -- which two dozen pins call directly, and
which returns a State rather than a string -- SKIPS the token and records it in the
message instead of typing it: a key that does nothing is easy to notice while
writing a test; fourteen characters quietly changing the document is not.
Only angle-bracketed strings are judged, so ordinary text and paste bursts still
type. Checked every `<...>` string already in the fluid testfiles first: each is
either a known token or an expected OUTPUT (`<type>` in a rendered enum field,
`<56789abc>` in a clipped line), so nothing existing was secretly broken.
The Backspace pins themselves are what the probe says, not what I assumed. It does
nothing in NAV, because it is an INSERT key and the caret is on a node; I asserted
the opposite twice before checking.
…ing in
The pinned fuzzer's alphabet predates every character rule moved onto the registry
today. It drew from `1 2 x y l e f u m a ( [ " + * | space .` and the motion keys,
so it had never once typed `;`, `,`, `>`, `{`, `}`, `$` or `'` -- which is to say it
had never touched the separator rule, the lead-punct rule, the revision that keeps
`||` typeable, or the interpolation brackets. A fuzzer that cannot reach the new
code is not evidence about the new code.
Wider alphabet now, and 24 keys per session rather than 14. A 400-seed, 40-key run
over the same alphabet came back with nothing but the four expected tags, which is
what says these numbers can stay small: the suite grew by about thirty seconds.
It drew as `let ?pattern = t`. The LPTuple spec was registered with `simpleExpr` rather than `simpleLetPat`, so it lived under the EXPRESSION kind and was never found for a LetPattern node. The three lines above it in the same list use `simpleLetPat` and read identically at a glance. Nothing was lost -- the pattern is carried whole and lowered back correctly the entire time -- but `?pattern` is the render for "this slot is missing", about a slot that was there, and a destructuring you cannot see or edit is its own kind of bad in an editor. Nothing caught it because nothing could. No pin builds a tuple let-pattern and no fuzz seed can reach one: you cannot type it, since `(` in a pattern slot has no rule and opens a tuple EXPRESSION elsewhere. It exists only in code that was PARSED, so only opening real code finds it. This came from walking to the bottom of `Darklang.Cli.runInteractiveLoop`, which also says vertical scrolling holds up on the largest function in the store. Pinned at two, three, nested and with a wildcard, plus the three let-patterns that always worked so the fix cannot quietly move them.
The property behind the tuple-pattern bug rather than another example of it. A node whose kind the registry cannot find a spec for is drawn by the fallback rather than by its own rules, which is how `let (a, b) = t` came out as `let ?pattern = t`: a render meaning "this slot is missing", about a slot that was there. Checked as a probe over 250 real functions, 250 types and 250 values -- zero in all three, so the tuple pattern was alone. That sweep fetches 750 package items and is far too slow for the suite, so the property is pinned over parsed sources chosen to reach the awkward corners: tuple and nested let-patterns, enum and list and tuple match patterns, interpolation, field-access chains, pipes, records, record updates, lambdas, dicts. A spec registered under the wrong kind now fails a test rather than turning up as a `?` on someone's screen.
`if b then 1` is valid Darklang (`EIf(_, _, _, None)`) and Fluid drew it as
`if b then 1 else ⟨else⟩`. It lowered back correctly the whole time, so nothing was
lost, but someone reading the screen saw an unfinished branch that is not in the
program.
The fix is the DATA MODEL, and it took two wrong designs to see that. First I
called it an unavoidable trade-off, because the hole is also the authoring
affordance -- type `if`, Tab into the else, fill it. Then I reached for a
`WhenFilled` piece that draws its contents only when a slot is filled. Both fail
for one reason: `Field.One(Slot.Empty _)` cannot say whether an absent thing is
MISSING or merely UNWRITTEN, and those want different renderings.
A Many field of zero or one says exactly that. A parsed else-less `if` carries zero
branches and draws none; a freshly typed one carries a single empty branch, so
`else ⟨else⟩` is still there to Tab into; settling without filling it lowers to
None, so it saves else-less and reopens drawing nothing. Same shape as a doc
comment, for the same reason.
Five places assumed a single else slot: `toDoc`, `fromDoc`, the `emptyIf` template,
`wrapInIf`, and the if-to-match conversion, whose false arm is now what is inside
the branch node.
The pin that recorded the old behaviour ("shows the optional hole, dimly labelled")
says the new truth with the reason, rather than being deleted -- the old choice was
deliberate, so changing it had to be too. The five pins for FRESHLY TYPED ifs all
still pass untouched, which is the half a rendering rule would have lost.
Re-ran the audit after: complete functions drawn with a hole in them, across 250
real functions, 0. The else-less ifs were the whole finding.
`EFnName` ran its name through the same shortening the printer uses; `EValue`
drew the full one. So a single call could come out with one of each:
Stdlib.Cli.UI.Colors.colorize Darklang.Stdlib.Cli.UI.Colors.yellow
Same drift as the earlier name-shortening fix, in a path that fix did not
reach: two routes to one decision, one of them missed.
This changes what the document STORES, not just what it draws, so the round
trip is the thing to check: 300 of 300 real values still lower back to the
value they came from. Drawn-vs-printed parity over 250 real functions moves
from 50 token-differing to 33, identical from 81 to 84.
Chased the last token-level differences between what Fluid draws and what the printer writes over 250 real functions. Three were not cosmetic: the drawn form was source that means something else, or that will not parse. - A nullary function. `()` in a declaration parses as ONE parameter named `_` of type Unit. Fluid drew the general `(name: Type)` shape, so it came out `(_: Unit)`, which the parser then rejects. The printer has special-cased this since it started emitting source you could parse back. Now it is a case of the param kind with its own spec, so a param HOLE still finds the ordinary spec at the type level and nothing about editing changes. - A string literal. The expressions inside `$"..."` broke like any other call, and the newline and its indentation landed inside the string's VALUE. New `Piece.Flat` / `Layout.Doc.Forced`: laid out flat whatever the width, because overflowing the line is the lesser harm. That alone did not fix it -- a Group inside a Forced one re-ran its own fits test and broke anyway -- so Group now respects an inherited Flat mode, which is what Wadler's recursive `flatten` does and what this renderer was missing. Under an ordinary Group it changes nothing: the enclosing fits already measured the whole flat content. - A reserved field name. A field called `type` must be written ``type`` or it lexes as the keyword. Fluid drew the stored text. New `Piece.FieldText`, and it calls the printer's `formatFieldName`, so there is one answer to "how is this name written" rather than two that can drift. Where it leaves parity, across 250 real functions: 95 byte-identical, 136 differing only in parens or layout, and 19 differing in tokens -- all 19 the known string-escaping bug, which is still blocked on there being no escape decoder outside F#. Nothing else is left. Found on the way, not fixed here: the PRINTER breaks inside `$"..."` too, which puts a newline into a string literal in emitted source. That is a shared path this loop cannot validate without the full suite.
…business First slice of doc 09: collapse the printer's and Fluid's precedence into one place. New `Darklang.LanguageTools.Precedence` owns the operator table, the two named powers, the swallowing-construct list, and both parenthesisation rules. The printer and the editor are both consumers. Two things were wrong with where it lived. The arrow pointed the wrong way. Fluid already called the printer's helpers rather than copying them, which stopped the drift, but it meant a structural editor had to ask a PRINTER what `*` binds like. The type checker was reaching into the printer to find out how `&&` is written, for the same reason. The CLEANUP note on `infixPowerOfText` said this and said moving it was a bigger change than sharing it; this is that change. The operator SPELLINGS moved with the table, not as scope creep but because binding power is keyed by the written form -- the editor sees an operator while it is still text and has no `Infix` value yet. Splitting the two is what forced everything to reach through the printer in the first place. And one duplicate was still live: the six swallowing constructs were written twice, as `exprPower`'s arms over an Expr and as `nodePower`'s arms over a document node, with nothing making them agree. That is now one list keyed by constructor name, which is the part the two shapes share, plus a reading for each shape. A seventh swallowing construct is one edit. The printer keeps every name it had, as delegations, so no caller outside changed. `selfDelimitingPower` and `swallowingPower` are gone from it: nothing referenced them any more. VERIFIED BEYOND THE FILTERED SUITE, because this touches a shared path that suite does not cover. Printed and drew all 11,863 package items before and after: byte-for-byte identical everywhere except the twenty-odd functions this commit edits. A pure refactor should move nothing, and it moved nothing.
… a stray dot
Big item (2). The record LITERAL turned out to be done already -- completion on
the type name builds the shell with the declaration's field names in it, `;`
adds a field, `}` closes -- so this is what was still missing around it.
`{ r with x = 1 }` could be drawn and lowered but never typed. Nothing claimed
`{`, because the literal cannot use it: `ERecord` needs a resolved type hash, so
it has to go through completion. That leaves `{` free for the one record form
that needs nothing resolved, and it reads in source order: `{`, the record, the
fields.
Which immediately hit the wall 05-open-questions predicted: Tab walked past the
field NAME and landed in the value, so the update could be built and the field
could not be named. The note guessed this would need a caret hook on the wrap
rule. It didn't. Tab walks the cells the LAYOUT styles as holes, and an empty
text slot was drawing as dim decoration -- while `;` landed on that same slot,
because adding a sibling lands structurally. Two answers to "is this
unfinished", and the layout's was wrong. An empty text slot is a hole now,
everywhere, and the record case falls out.
Separately, `displayName` mangled every single-segment name.
`parsePackageLocation "Colour"` reads a bare name as an OWNER with an empty
name, and shortening owner="Colour"/name="" hands back `Colour.` -- so
`Colour.Red` drew as `Colour..Red` and `Foo { a = 1 }` drew as `Foo. { ... }`.
That is what any type declared in the module you are standing in looks like, and
what a type you have just typed looks like before it resolves. A name with no
module path has nothing to shorten.
Left alone deliberately: records always break one field per line, even when they
would fit and the printer packs them. That is the "MORE VERTICAL" decision from
tick 19, not drift.
Checked past the pins: 300 values and 300 types still round-trip, and drawn-vs-
printed over 250 real functions is unmoved at 95 identical / 136 layout / 19
escaping. Walked the whole flow live in the TUI as well, since the field-naming
half only exists through a terminal.
Big item (3), step 4. Steps 1-3 of the UI redesign had already landed: completions are a caret popover, the always-on inspector collapsed into the status line, and `?` toggles the panel. What step 4 asked for was the surfaces that make editing here better than editing text -- the VALUE, the TYPE, and the docs. `v` already evaluates. This is the other two. The panel now carries DOCS for the node under the caret: its name, its signature, and its doc comment. Every package item has stored a description since the beginning and nothing in the editor has ever displayed one. It works off the HASH the document already stores, not off the name. A shortened name can match two items; a hash matches one, so this cannot be a near-miss. Only nodes carrying a hash have an answer -- functions, values, records, enums, and builtins -- and everything else says nothing. That is the honest boundary: it is a lookup, not inference, so there is no "TYPE" line pretending to know the type of an arbitrary subexpression. Standing on a CALL resolves to its function, because an argument hole is where the caret usually is when you want to know what the arguments are. Doc markers are unwrapped on the way out -- `<param fn>` reads as `fn` -- by splitting on `<` and putting back anything that isn't one of the known tags, so a `List<'a>` written in a description survives. The lookup is passed IN to the widget rather than done there, for a reason worth keeping: that widget draws whatever language the registry describes, and a package is a Darklang thing. It must not know what one is. Noticed and left alone: on a hole in NAV the completion popover is drawn but the keys the status line advertises can't pick from it -- Enter just switches to INSERT. Arguably a preview, arguably against the redesign's "one thing at a time". Not changing it on a guess.
…g text Big item (4). Checked the extensibility audit against the code before building anything, because it is old. Axis 2 (your own syntax) and axis 4 (your own keys) were already marked done. Axis 3 still says "NOT YET, and this is the interesting one" and that is stale: `Piece.Table` landed, and all three things the audit said a spreadsheet needs and a Wadler document cannot give you are there -- columns measured across siblings, headers drawn once, a caret that moves in two dimensions. So this tick is the three things that were wrong once you sat in one. Down off the LAST row walked sideways into the next column. The row-motion helper returned one None for "not in a table" and the same None for "in one, with no row that way", so the second fell through to reading order -- which walks along the row. That is what `l` means, so two keys did one thing and one of them lied about its direction. Fixing it turned up a pin asserting the opposite for `k`, with a reason: off the first row it steps OUT to the row, "same shape as `h` from the first column". That is a real decision, not an accident, and it is half of a rule the earlier pins had already settled without naming: BACKWARDS at an edge steps out, so you are never stuck; FORWARDS at an edge sticks, so nothing runs off the end. `l` off the last column already stuck. `j` was the one case that did neither. So the fix is asymmetric on purpose, and the rule is now written down where the code is. A row that is itself a hole drew its placeholder once PER COLUMN, so clearing a row filled the width with `⟨row⟩` and re-measured every column against a placeholder that is not a cell. Drawn once now, in the first column. And filling a text cell needed a `"` first, which is not a key anyone would guess, with a bare word left sitting as a partial that can never resolve. A slot that expects the prim String KIND now takes typing as a string. That is scoped by the expectation, not by the keystroke: a Darklang expression hole expects an Expr, so a letter there still starts a variable, which is pinned.
Found by opening a real function and looking at it. In NAV the current node is shown in reverse video, and the code asked each cell "are you under the caret" and reversed the ones that said yes. Only TEXT carries a tag, though -- punctuation, keywords and the spaces between them do not -- so selecting a node highlighted its names and literals and left its `(`, its `->` and its spaces plain. On a leaf that is invisible, which is why it survived. On a fifteen-line `let` it was forty scattered words in reverse video instead of a block you could see. The selection is a range per row now: everything between the first and last selected cell. A subtree's cells are consecutive in reading order, so there is nothing foreign to catch in the middle. What it still does not cover is punctuation OUTSIDE the first and last tagged cell -- the `|>` in front of a pipe step, the `)` closing a lambda. Fixing that means tagging punctuation, which would make it a caret target, and that is a much bigger change than the thing it would tidy. Pinned with a marker theme rather than real SGR, so the pin says which cells are selected and nothing about colour.
Editing `ProgramTypes.Expr` drew `| EUnit of Darklang.LanguageTools.ID`. The source says `| EUnit of ID`, and so does the printer when it is standing in that module. On that type the prefix was enough noise to break lines that fit, turning `| EDict of ID * List<(Expr * Expr)>` into three. The registry carries the module the document lives in, next to the syntax it is spelled with, and every editor target already knows its own location, so there is nothing to look up. The document still STORES the full name, which is the identity; the shortening happens on the way to the screen, through the same `shortenName` the printer uses rather than a second answer. This is the CLEANUP the old comment on `displayName` named: "the fn editor KNOWS which module it is showing, and Context.forModule exists for exactly that". Checked the round trip, since this changes what is drawn for every qualified name: 300 types and 300 values still lower back unchanged, and drawn-vs-printed over 250 real functions is where it was.
Pressing `v` on an expression that fails scrolled the editor by two rows and left the frame drawn two rows low, with two stale rows above it. Reproducible every time on anything that errors. The cause is not in the layout. `Builtin.cliEvaluateExpression` prints the call stack on the error path as well as returning the error, and that print goes straight to the terminal the editor is holding. The renderer then diffs the next frame against a screen that has moved. I did not take the print out. It is genuinely useful at a prompt, and the call stack is not in the returned error, so removing it would lose information to fix a display bug somewhere else. The right shape was already in the CLI kernel, one case over: a page that hands the terminal to `$EDITOR` rebuilds its surface on resume rather than diffing against a stale one. New `Step.Repaint` says the same thing without leaving the page, and `v` answers it. That covers the general case too, not just the error path: evaluating `Stdlib.printLine "hi"` prints, and should. Checked by hand in a terminal, which is the only place this exists. The `v` path cannot be pinned -- it reaches a builtin the test host does not carry, so the call fails before the Step is chosen -- so what is pinned is the other half: `j` and `?` still answer `Stay`, and `?` opens the same panel without running anything.
Same bug as `v`, found the same way: made a real edit to a real function and pressed Ctrl-S. The save worked, and then "Propagated to 39 dependents:" and a grouped list of them were written over the frame, and the footer. Propagation reporting is right at a prompt, and it is not the editor's to suppress. Every branch that actually saves answers `Step.Repaint` now, so the editor takes the screen back afterwards. `Scratch` still answers `Stay`, because nothing ran and nothing wrote. While there: pinned that escaped strings survive the document. `"a\nb"` DRAWS as two rows, which is the last token-level difference between what Fluid draws and what the printer writes, and I had been treating it as possibly destructive. It is not. The body is stored decoded, `fromDoc` hands back the same `EString`, and the printer escapes it again on the way out. Confirmed end to end as well: edited a function containing `"\n"`, saved it, and read it back out of the store with the escape intact.
Thinking about what this does to the CLI rather than to the editor. `dark edit <name>` has opened the structural editor for a while; `dark fn <name>` made you type a whole definition onto a command line, or pipe it in on stdin. So the CLI said structural editing is for code that already exists, which is backwards: holes and completion are worth most on a blank page. `dark fn|type|val <name>` with nothing after it opens the editor on a blank one now, and saving CREATES it. There is no new save path: it reconstructs source and hands it to the same `applySource` an edit goes through, which is what adds a declaration to a module. The inline and stdin forms are untouched, because that is what a script or an agent wants; being the only way was the problem. Then everything that flow ran into, which was most of the work here: - A blank function had no parameters and no way to add one. `(` is an action on the fn spec now -- so it shows in the palette and the panel, and a spec binding its own key is the extension point doing its job. A lone `()` is REPLACED rather than appended to, since `() (x: Int)` is not a shape the language has. And the blank shape is `()` rather than an empty list, because a function with no parameter list is not a function. - The caret went past a new parameter's NAME into its type. Landing walked `holes`, which skips text on purpose -- "is this program complete" is a different question from "where does the caret go next" -- so a parameter with no name was not on its list. New `blanks` answers the second question. - An action left the caret on the node you were already on, so the next keystroke went nowhere. It lands in the hole the action made, and starts typing. "wrap in let" was already doing this by accident through `lastEditable`; the three actions now agree. - Escape left `Int` as a partial that draws exactly like a resolved type and refuses to lower: "not saved: unresolved: Int" under a screen reading `(n: Int): Int`. A spec's `resolve` is for what no menu can list; `Int` is a CANDIDATE. Settling takes an exact candidate now, which is the rule the menu already used one step earlier. - Enter in a list added the sibling before settling the item, so in a list whose items are TYPED rather than picked -- an enum's cases -- every case but the last stayed a partial and `| Red | Green | Blue` would not save. Settle, then add. And a case and a field are names, which no menu lists, so those two specs needed a `resolve` of their own. - `edit` and `fluid` were in the command table and in no help GROUP, so `dark help` listed neither of the two commands this is all about.
Last tick made `dark fn <name>` open the editor on a blank function. It looked right and it would not save: "not saved: could not parse the reconstructed source", under a screen reading `let triple (n: Int): Int = n * 3`. Saving reconstructs source with the printer, and the printer names a definition from its HASH -- it looks up the locations that hash has and picks one. A function being CREATED has no hash and is in no store, so that lookup found nothing and fell back to `<hash:>`. `let <hash:> (n: Int): Int =` is a debug affordance, not source, and the parser said so. `preferredLocation` is already the caller saying which location it is showing a definition under, and `saveFnBody` was already passing it. It now fills in when the hash resolves to nothing. For a stored item it never runs: a stored item has a location by construction. Found by doing the whole thing in a terminal, which is the only place it exists -- the programmatic probes stopped at `fromDoc` and the printer, and the printer is where it broke. Now: create it, save it, `dark view` it, and `dark eval "Stachu.Scratch.triple 7"` says 21. VERIFIED PAST THE FILTERED SUITE, because this is the shared printer again. Printed every package function, type and value and compared against the same dump from before this branch's printer work: every difference is a file this branch edits, or one of the content-addressing artifacts where `type X = String` and `val n = 100` collapse to one entity under several names.
Escape in NAV ended the editor. So did `q`. Both threw away everything typed since the last save and said nothing about it, and Escape is a key you press to back out of a menu, not to close a file. The first press says what is at stake and the second goes through. Any other key in between cancels it, so the state never survives more than one keystroke. This is the only confirmation in the editor and I would rather it stayed that way, but losing work silently is worth one key. Dirtiness is the document compared against the document as of the last SAVE, not a count off the undo stack, so undoing back to where you started reads as clean -- which is the truth, and the undo-stack version would have lied. Found by pressing Escape twice by accident while probing the create flows, which is exactly how a user would find it. Also verified this tick, since the last one only did functions: creating a TYPE and a VALUE work end to end too. `dark type Stachu.Scratch.Colour`, pick enum, type two cases, Ctrl-S; `dark val Stachu.Scratch.limit`, 56, Ctrl-S; then `dark view` shows both and `dark eval` returns them. A new function can take the new type as a parameter and match on it.
`dark edit <fn>` opens on the function SHELL, and text mode edits an EXPRESSION, so `^E` -- which the status line advertises -- answered "can't edit as text: not an expression node: Darklang.Fluid.PT.Fn.Function" at exactly the moment you arrive. You had to know to navigate into the body first. It falls through to the nearest expression inside the selection now: the body, for a function or a value. A type declaration still has none, and still says so, which is the right answer for a type. Found by trying it on the first function I opened. Also worth recording: the string escaping that draws wrong in the structural view comes through text mode correctly, because the printer escapes on the way out. Text mode is the one place that bug does not bite. Wrote the design for type-directed match-pattern completion into 05-open-questions rather than half-building it: `match c with` offers true/false/() when `c` is a parameter declared as a user enum, and offering that enum's cases wants scope to carry TYPES and a new spec hook for context-driven candidates. Both are real vocabulary additions and neither is a thing to rush at the end of a tick.
`saveType` prints what `fromDoc` rebuilds and applies that as source, so anything the document does not hold is erased on the next save. That is how the enum-field labels were lost, and it was still true of every DESCRIPTION: a record field's, an enum case's, an enum field's, all rebuilt as "" whatever they had been. Carried in `_doc`, hidden the way `_hash` is. Carried rather than shown, because there is nowhere in the drawing for it yet, and the drawing is pinned unchanged so the day that stops being true is visible. Nothing in the store has one: 0 of 1,054 types, because the parser does not capture a field's `///`. So this loses nothing today, which is exactly why it is cheap to do now. Found by opening `Search.SearchQuery` in a terminal and noticing its field comments were not on screen. They are not in the store either, which took a sweep to establish rather than an assumption -- the first guess was that Fluid was dropping them, and Fluid was faithful to a store that had nothing.
`q` and Escape both mean "leave", and with unsaved work the first press says what is at stake and the second goes through. Any key at all cancels the pending discard, and `afterKey` clears the flag for exactly that reason, with a comment saying so. It did not clear the WARNING. So the line kept saying the next key would throw your work away long after the next key had come and gone: Escape out of INSERT sets it, and it then sat there through navigating, editing and evaluating. The one message in the editor that must be true only for one keystroke was the one that never went away. Only cleared when the flag was actually set, so a message written by anything else is not this one's to take. Found by driving a real package value in a terminal: I pressed `v` to evaluate a field and the footer was telling me I was about to discard my work. Nothing in the harness would have shown it, because nothing in the harness reads the message line two keystrokes after the key that set it.
`^arrows` is exactly seven characters and the column was seven, so it touched its own description: `^arrowsmove along its list`. Introduced by me, when the panel's hard-coded key rows became data. The row it replaced had a literal space after `^arrows` and therefore sat one column right of every other row; eight lines them all up and keeps the gap. The reference already used eight, which is the tell I did not read. Found while re-measuring the panel's overflow for something else. The stale check in that probe searched for the OLD marker wording and reported no overflow, so I dumped the panel to see why, and the run-together row was the first thing on the screen. Two wrong things, one of them mine and neither the one I was looking for.
`Dict<String, Int64>` could not be typed through its bracket. `Dict ` was fine.
Settling a partial put the caret at the LAST editable position in the node it
settled to, and committing the same name from the menu puts it at the FIRST. Two
ways to settle one name, two different landings, and a dict type is the only thing
in the language with enough holes to tell them apart: `Dict<` settled the
candidate, landed in the VALUE, and the `<` that had done the settling went into
the value as text.
`landAfterCommit` is the one that lands first, and it falls through to
`lastEditable` when the settled node has no blanks -- which is the case the comment
there is about, and it is unchanged. Settling `Size` still leaves the caret after
the name so the `{` you type next builds the record; settling a variable still
leaves it where an operator can follow.
That closes the type side. Of the five shapes that did not type back a day ago --
a parameterised type, a nested one, a function type, a dict's two arguments, and
one that crashed the editor -- none is left.
`<` on a function shell did nothing at all, so a generic signature could be read and never written: `let inc<'a> (n: 'a): 'a` swallowed the `<'a>` and gave `let inc (n: 'a): 'a`, silently. Three things, and the middle one is why the first looked like it had worked. No action claimed `<`. `(` has had one since the shell was written, `add a parameter`, and its sibling was never added. An ACTION rather than `opensItems`, because the caret on a fresh shell is ON THE NODE rather than at the end of a text slot, and `opensItems` is for the second of those. A blank shell had no `typeParams` FIELD at all, where `toDoc` gives every function one. Two constructors for one kind disagreeing about its shape. `Node.setField` only REPLACES a field and does not add one, so the new action ran, set its message, and changed nothing: it took twenty minutes and a node dump to see that the count had stayed at zero. The name slot had no text class, so the `'` the piece already draws went in as text and `<'a>` came out `<''a>`. Still not typeable and already recorded: what comes after the `>`. The caret lands in the return type rather than on the node, so the `(` of the parameter list opens a tuple there. That is the same `(` as the two-parameter signature.
`let () = 1` could not be typed. The `)` went in as text and the whole binding came out "unresolved: =". `(` opens a tuple in a let pattern the way it does everywhere, and the LPTuple spec had no CLOSER, so nothing turned an untouched one into unit and nothing collapsed a one-element one to a grouping. The match pattern next door has had that rule since tuples landed; the let pattern is its sibling and never got it. A unit binding is what you write for a call you want only for its effect, and the printer emits it, so this was a printed form that could not be typed back. Found by walking the let-pattern vocabulary: four constructors, three fine. That is the sixth vocabulary walked and the fifth to have exactly one member nobody had asked about.
Seventh vocabulary walked, and this one is a finding rather than a fix.
The pipe-step hole's `holeRules` is empty. Every other hole in the language builds
something from a character: an expression hole takes `[`, `(`, `"`, `'`, a digit
and a `-`; a match pattern takes five; a type hole takes three; a definition hole
takes two as of this week. A pipe step takes none.
Three of the five stage shapes are reachable anyway, through the name. The two the
printer BRACKETS are not, and both are blocked on the same `(`:
xs |> (fun x -> x) ERR unresolved: (funx->x)
xs |> (+) 1 ERR unresolved: (+)1
Pinned as the wrong answer they currently give, so the day it changes is visible,
which is the device the two-parameter signature already uses. A lambda stage is
still reachable by name and that is pinned beside them, so this is a gap in the
printed form rather than a missing capability.
What `(` should MEAN there is a real question and I am not guessing at it: the
next character decides between a lambda and an infix section, and the printer
brackets the lambda precisely because a bare one would swallow the stages after
it.
Eighth vocabulary, and the claim that sent me to it was half wrong. The notes have said "`EArg`/`ESelf` and a few rare int widths are opaque" for weeks. Asked directly: no width is opaque. All eleven have their own kind, draw as themselves, and type back, suffix and all. Only `EArg` and `ESelf` are, and they are opaque because they have no source spelling rather than because anyone overlooked them. `EveryIntegerWidthIsItsOwnKind` already pinned the first half of that -- each has a kind rather than falling through to Opaque. Whether each could be TYPED is a different question and nothing had asked it. Fifth stale claim in this branch's notes that turned out to be wrong on being re-RUN rather than re-read. Five for five, still. Also written down, since I had the answer in front of me: what would fix the two that remain. An `EArg` has no spelling on its own, but the document it sits in is the function shell, which knows the parameter's name -- the printer resolves it exactly that way. So a parameter reference could draw as the name and edit as one.
The other half of the note I half-retired last tick, and I believed this half while disproving the other one. `EArg` and `ESelf` are opaque only in a body drawn WITHOUT its function shell, which is a harness situation rather than one a person can reach. `namedArgs` replaces `EArg i` with the i-th parameter's name and `ESelf` with the function's, as EVariables, before the body ever becomes a document. So in the shell a parameter reference is an ordinary variable node: it draws as `n`, the caret goes into it, and a rename is a rename. On save the body is printed with those names and the parser re-lowers them in context. Pinned by node KIND rather than by the drawing, because the drawing is what made the note plausible: `earg0` and `n` both look like text until you ask what is underneath. That retires the last of the stale claims I set out to re-run. Six for six: every written-down limitation in these notes that I have executed rather than read has been wrong or already fixed. Which follows, on reflection -- the ones that were right got fixed and the note stayed behind.
The name slot is editable and what you typed into it was thrown away. `fromDoc` hands back the name the document spells; all three save paths -- fn, type, value -- bound it to `_name` and dropped it, then printed the source through `packageFn`, which names a definition from its hash's locations. So you could retype a function's name, watch the drawing follow along, press save, and get "saved not" under a screen reading `let renamedToThis`. Two halves. `saveLocation` decides where a save goes: the document's name wins. An empty name slot is a hole, not a rename to "". Then the printer had to be willing to write it. `pickLocation` refuses a preferred name the hash is not bound to, on the grounds that spelling a reference by a name that does not resolve is a lie. That is right for a reference and wrong at the definition site, where the name is not being resolved but written, and a rename is by definition a name the hash has never been bound to. `atDefinition` now takes `preferredLocation` first, falling through to the lookup and then to `<hash:>`. No existing caller changes: everything that sets it (Fluid's saves, `dark show`) passes a location it looked the item up by, so it was already winning the scoring. A rename BINDS the new name and does not unbind the old, because `SetName` is the only op there is. That is the store's rename semantics rather than something the editor gets to decide, and it is written down next to the code. The three save paths were the same eight lines three times and carried the same bug three times; they are now `saveLocation` / `saveContext` / `applyAt` plus one `*SourceFor` each. Splitting the source from the writing is what lets the pins check the decision without a store round trip. Found while sweeping the whole function SHELL against the printer across the store, which is the drawing a person actually opens. 56 of 5,380 differ, all four categories already known.
Found by renaming a function in a terminal. Pressing `?` for the key panel typed a `?` into the
name, because the caret opens there and the slot took any character at all:
let !before (n: Int): Int =
which draws perfectly and then comes back "could not parse the reconstructed source" at save time,
blaming the reconstruction for a character the editor should never have accepted. Making renaming
work is what made this reachable: before, the name never got as far as the source.
Every name slot in a declaration had it -- the function, the value, the type, a parameter, a record
field, an enum case, an enum field label, a type variable, and a record field in an expression. The
pattern and let-binding names had the class already, and the shells, whose name goes into the source
header, did not. One `PT.nameClass ()`, nine sites.
Checked the keys that share those keystrokes rather than assuming, since a character class is a
refusal and a refusal is what breaks a key someone else claimed:
< at the end of a name still opens the type parameters -- `opensItems` is asked first
: after a parameter name still steps over the colon the spec draws
( at the end of a name refused now, with a message; before it went in as text
One test moved, and it is the one that was written to move: `fluid-fn/line711` pins the WRONG answer
for the two-parameter signature the printer writes and the editor cannot type, "so the day it changes
is visible". The `:` used to land in a type variable's name, giving `('m: * Int)`; it is refused now
and the nonsense is `('m * Int)`. Still nonsense, still pinned, with the reason next to it.
Measured on a quiet box, one keystroke end to end:
Bool.not, 3 lines 124 ms first key, ~123 ms after
tryMergeListItems, 45 1089 ms first key, ~1540 ms after
No first-key penalty either way, so it is document size and nothing lazy. A second and a half per
keystroke on a 45-line function is not an editor.
Where it goes, with the state built once and the work repeated twenty times:
one layout (rowsFor), 45 lines 225 ms
the painting on top of it 18 ms
one layout, 3 lines ~ 0 ms
Editor.candidates (Nav) ~ 0 ms
Widget.inspector 3 ms
The layout is the whole cost, and it was computed FIVE times for one keystroke on the same state:
`caretRow` and `caretCol` in `afterKey`, then `view`, `popover` and `caretScreenPos` in the render.
Nobody wrote that, it accumulated -- each of those functions politely asks for what it needs.
So the ones that need the layout take it, and the two callers work it out once. Five to two:
1540 ms -> 1090 ms per keystroke
238 ms one whole App.view
The `st`-only forms stay for callers with a single question, where a second layout is not being paid
for.
`afterKey` and the render still lay out the same state at the same width, twice, and that is worth
another 225ms. Reusing it needs a cache with an invalidation rule -- the view is also called on resize
and refresh, when `afterKey` has not run, and a stale layout puts the caret in the wrong place -- so
it is left alone rather than guessed at. The remaining ~630ms is outside Fluid, in the framework's
diff and write.
No new test: `fluid-app.dark` asserts on the rows `App.view` draws, popover and clip marker included,
and a layout threaded wrong changes them.
There was no forward delete. Only Backspace, so the only way to remove a character was from the right
of it: replacing a name meant walking the caret to the end and backspacing through the whole word.
Renaming `before` to `after` in a terminal cost eleven keystrokes, which is what made it worth
writing down.
`TextEdit.removeAt` was already there and `Delete` was already a key the stdin layer decodes. Nothing
in the editor had ever used either.
Deliberately none of Backspace's structural rules. Those all answer one question -- there is nothing
behind the caret in this text, so what did you mean? -- and they answer it well: collapse the empty
construct, merge the list items, drop the operator. Forward off the END of a text run is a different
question with no obvious answer. What does deleting the `)` you are sitting in front of mean?
Inventing an answer there is how a key ends up doing something surprising, so it says so instead.
i <Del><Del><Del> after -> after nine keystrokes, no arrow walking
i <Del> X -> Xot the caret stays, the text closes up under it
i <Right><Right><Right> <Del> -> refused, with the reason
i <Del> <C-z> -> not one keystroke, one undo
Listed in the INSERT key panel, which is where someone goes looking for it.
`|` at the end of an arm's right-hand side starts the next arm, and typing it AGAIN in that fresh arm
decides the operator was meant after all: the arm goes away and `||` or `|>` is built on what came
before. That revision only fired when the fresh arm was the LAST one.
arm 1 of 2, type |> | > -> <rhs> an arm whose pattern is ">"
arm 1 of 2, type || a bare new arm
the last arm, either correct
Typing `|` at the end of the final arm appends, so the item was last and the guard held. The same keys
at the end of any earlier arm insert in the middle, and it did not. So a pipe was typeable in the last
arm of a match and in no other, which is not a rule anyone could have learned.
The guard was `i == length - 1` and nothing needed it. `followsTheLeadPunct` is what tells an arm you
just made with `|` from one made with Enter; `fresh` and `i > 0` say the item is empty and has
something before it to operate on. Where that item sits in the list was never part of the question.
Worth saying why fifteen pins missed it. `BarStartsTheNextArm` covers `||`, `|>`, or-patterns, arms
ending in a string, a list, an unsettled partial, and a separated list keeping its comma -- and every
one of them builds its match by TYPING, so the arm `|` makes is always appended at the end and the
broken case is unreachable from that harness. The new pins parse the match instead, so the caret can
sit in an arm with something after it.
Both keys have worked for a long time and neither appeared anywhere on screen. Driving the editor in a terminal I walked a 285-character doc comment with 135 presses of Right, because nothing said End existed. Added to the INSERT key panel, next to the forward-delete that came from the same complaint one tick ago. Adding the row broke nothing and changed no count, which is its own finding: the whole INSERT key list is asserted as `(Stdlib.List.length (insertKeyHelp ()) > 0) = true`. A row could be deleted and nothing would notice. That is the shape AGENTS.md warns about for `notSweepable`, one level over. So the pins are on the BEHAVIOUR rather than the wording. They are line keys, not text-run keys: in `[11, 22, 33]` the whole list is one line and End goes past it; in a match each arm is its own line and End stops at the end of that arm. Both shapes are here, including the two I misread as bugs before checking where my own keystrokes had landed.
NAV had `u` and `^R`. INSERT had `^Z` and `^Y`. So redo worked under its Ctrl name in NAV and undo
did not, and anyone who learned `^Z` one mode over got silence. Nothing in NAV claims either
character, so both modes answer to both spellings now.
Found by sweeping an invariant worth having: one keystroke, then undo, gives the document back
exactly. 40 real store functions, 4 caret depths, 7 keys, 1,120 checks. Real store code rather than
documents the harness types for itself, because the last two bugs both turned on that difference.
It took three runs to get a number worth believing, and the first two were the harness:
160 failures, every one `x` -- `^Z` is not bound in NAV
113 failures, every one `o` -- `o` drops into INSERT, where `u` is the letter u
0 of 1,120 -- with `^Z` bound in both modes
So the sweep found nothing about undo and one real thing about keys. The invariant holds.
The tell for a harness failure is the same every time and worth writing down: the failures are too
uniform, every one the same key, and the fix would have to be in code you have just read and found
correct. `clearNode` goes through `commit`, which snapshots; that was the sentence that said to go
look at my own keystrokes instead.
34 minutes for the cli/fluid subtree, and the reason was not the harness, the database or the
scheduler. It was the collector.
sequenced ~34 min
parallel, server GC 4:23 1,880 tests, all passing
Parallel was off for a measured reason, recorded in the runner: 1702s wall against 807s sequenced,
and 64,720s of per-test CPU against 787s. Eighty times the CPU for twice the clock is not a
preference, it is a bug worth re-opening.
What found it: the parallel ceiling ignored the worker count. 200 independent CPU-only cases took
5.37s with 8 Expecto workers and 5.52s with 32, against 23.1s sequenced. A ceiling that does not move
with workers is a serial section, not a scheduling limit. This interpreter allocates as hard as
anything in the tree and the test binary was using the WORKSTATION collector: one heap, and every
thread stopped for every collection.
200 CPU-only cases, parallel, workstation GC 5.4s
the same, server GC 2.06s
The whole subtree went from 311% CPU to about 2900%. So `ServerGarbageCollection` on the test project,
and parallel becomes the default in the runner with `--sequenced` to put it back. The `/cloud`
testfiles that share DB state mark themselves `testSequenced`, so the flag does not race them.
Two things tried and dropped for measuring nothing, noted so they are not tried again:
`ThreadPool.SetMinThreads` (the pool was never the constraint) and sharing `packageFnInstrCache`
across the run (617s against 610s).
Also in here, and separate: `Registry.find` reversed the whole 117-spec list on every lookup, and a
miss did it twice. It runs for every node the layout draws. An index on the Registry, keyed by a
string that spells the case apart from the type name -- `Kind.toString` writes a case with a dot, so
`A.B` the type and `A` at case `B` collide, which is fine for a message and not for a table.
1,000 lookups 0.56s -> 0.01s
one layout, 60 rows 225ms -> 140ms
50 editor drives 4.17s -> 2.82s
That one is for the editor rather than the suite: lookups scale with nodes in a document, and a test
types `1 + 2` while the editor opens a 45-line function. It lands on keystroke latency.
The index arrived with a bug the full filter caught and two single files did not: it was built with
`Stdlib.Dict.set`, which REFUSES a key it already holds, and shadowing a built-in spec with a
discovered one is exactly that duplicate. `setOverridingDuplicates` is what the reversed scan meant.
Startup was ~16s of every run and 11.9s of it was reloading packages from disk. Most runs do not need
it: editing a TESTFILE does not change a package, and that is most of what a test-writing loop does.
So a pristine snapshot is taken right after a reload, stamped with the state of `packages/`, and
copied back when nothing there has moved. A 64MB copy against twelve seconds.
fluid-app, wall
37.5s sequenced, as it was this morning
21.6s parallel
9.3s parallel, snapshot reused
A snapshot rather than reusing the database where it lies, because tests WRITE to the store --
authoring cases add declarations -- and the next run has to start where this one did. That is what
the delete is for and this keeps it: the copy is taken before any test has run.
The stamp covers every `.dark` under `packages/` by path, size and mtime, plus `--published`, so a
package edit invalidates it and a testfile edit does not.
Whole cli/fluid subtree, 1,880 tests, all passing: 4:25 wall, against ~34 minutes this morning.
Matching stays case-insensitive, so `d` still reaches `Dict` and nothing becomes unreachable. The case
decides the ORDER instead:
typed d dict divide Dict DateTime Decimal
typed D Dict DateTime Decimal dict divide
typed '' declared order, untouched
Types and constructors are always capitalised in this language and keywords never are, so the case
you typed is a real signal about which you meant. It moves the slot Enter takes, not the contents of
the menu.
Applied to the EXACT matches as well as the prefix ones, which took a second pass to notice: `dict`
typed in full is an exact match for both `dict` and `Dict` case-insensitively, and the spelling you
used should win.
Decided rather than discovered. This was flagged from reading `labelMatches`, not from anyone being
bitten by it, and the menus where both cases exist are rare.
The printer writes `let inc (n: Int) (m: Int): Int =` and typing it back gave nonsense. `)` closes the
first parameter and lands in the return slot; `(` there is a tuple type, because a function may return
one, and nothing can tell the two apart at the moment the bracket arrives.
The `:` can, and it arrives one keystroke later. A tuple type has no colon in it anywhere, so a colon
typed into the first member of a tuple standing in a function's return slot can only mean the thing
that was never a tuple.
(n: Int) (m: Int): Int let inc (n: Int) (m: Int): Int =
(a: Int) (b: Int) (c: Int) three of them, so not a one-off
(n: Int): (m: the moment it happens, caret in the type
(n: Int): (Int * String) a real tuple return type, untouched
A revision rule beside `leadPunctRevision` and `statementPipeRevision`, which are the same shape: the
last key made one thing, this key says you meant the other, and the first thing goes away rather than
being left behind. Narrow on purpose -- the tuple must be the whole return slot, its first member a
bare name, and the rest still empty -- so a tuple somebody is actually filling in cannot trigger it.
This replaces a pin written to record the wrong answer "so the day it changes is visible".
Also here, and it should have been in the previous commit: `rankCompletions` itself. I committed its
tests while the implementation sat unstaged, then reverted that file to undo something else and took
the implementation with it. Four pins have been sitting on HEAD with nothing behind them since. The
lesson is not to split one file's work across two commits with `git stash`.
`(` in a pipe stage went into the name as a literal character, so `xs |> (1 + 2)` came out as a stage
called `(1+2)` and the two forms the printer BRACKETS could not be typed back at all.
It groups, the same meaning it carries in every other slot. Grouping a position that holds exactly one
thing is a no-op, so it is swallowed rather than drawn -- which is what the language does with it:
xs |> (Stdlib.Int.add 1) the parser reads this
xs |> Stdlib.Int.add 1 and prints back this
The one bracket a stage keeps is a lambda's, and the lambda draws its own. All four printed forms type
back now, including `xs |> (fun x -> x)`.
That needed a new kind of rule rather than a special case. `HoleFill` could only name a node to build,
and the answer here is "not an error, and not a thing", so `HoleFill.Absorb` says it: a spec may
declare that a character is swallowed in one of its holes. Punctuation a language writes and does not
mean has somewhere to be written down now, which is the spec-level character rules the extensibility
audit asks for, arriving because a real case wanted it.
The operator section `(+)` is still out of reach and stays pinned as a gap. It is a different feature:
a way of NAMING an operator, not a bracket around a stage.
`APipeStageHasNoCharacterRules` is renamed and reframed, because "it has none" was its whole premise.
Those two pins are why this got done properly -- they recorded the wrong answers on purpose, and my
own probes would not have found them.
Already true, not pinned. `l` uniquely prefixes `let`, so with nothing bound a space commits the
construct; bind something called `l` and the same key is that variable, and a space after it takes an
argument the way any settled name does.
nothing bound l 2 -> let 2 = <value>
l bound l -> l
l bound l 2 -> l 2
Keyed on an exact match against a name in scope rather than on any competing candidate, so an ordinary
keyword prefix is untouched: `l ` is still `let` with `list` in scope, because nothing is named `l`.
The existing module covers the other half -- an ambiguous prefix like `f ` settling as a variable
rather than jumping to `fun` -- and had nothing for the scoped case. Three pins, no code.
Everything else in Fluid projects a program: an expression, a declaration, a function shell. This projects what a program produced. The design doc names it as the missing half (7.D, "editing a runtime value of type T needs Dval <-> Node"), and it is why a value of your own type could be drawn in the demo and saved nowhere. Both directions, in packages/darklang/fluid/value.dark: - `ofDval` draws a value. No new specs: `Derive` already builds one for any type from its declaration and `Prims` covers the leaves, so a value document draws with what is there. - `toDval` rebuilds it, refusing rather than guessing. A hole names the slot it is in; a dict and an opaque handle say why they cannot come back at all. Two things worth a look: The kind carries the FULL location name, not the shortened one the printer writes. That name is a KEY -- `Derive.specsForName` hands it to `parsePackageLocation`, and `Option` does not resolve where `Darklang.Stdlib.Option.Option` does. Shortening is a display decision and belongs on the other side. Integer width rides in a hidden `_w` slot, the same `_`-prefix convention `_hash` and `_doc` use. On screen `1` is `1` whichever width it came from; going back is where it matters, and without it an Int8 came back an Int -- a type error at the far end and nothing at all on screen. Also fixes `Registry.addAll`, which appended to `specs` and never touched `index`. Since `index` is what `find` reads, a spec added that way was invisible to every lookup, and an unfindable spec looks identical to one that was never written. Found by adding derived specs for a value's types and watching them do nothing. Not done yet: an entry point that opens a value (so this is reachable rather than only testable), and a save target -- all four `Target` cases are code, so a runtime value still has nowhere to go.
The conversion landed last commit and nothing could reach it. This is the door.
dark fluid eval "Stdlib.List.head [1, 2]"
opens `Some(1)` in the editor: not the source that computed it, the value. Every other entry point
opens code, which is the whole reason this one is worth having. Most people never touch code and
everyone touches data.
Three pieces:
- `Builtin.cliEvaluateToDval`, a sibling of `cliEvaluateExpression`. That one has the raw result in
hand and renders it to a string before returning, because printing is what every caller so far
wanted. Fluid wants the value; a rendering of one is text it would have to parse back.
- `Value.typeNamesIn`, the walk that says which types a document mentions.
- `App.registryForValue`, which derives an editor for each of them.
That last pair is the part I would look at. A value's types are only knowable from the VALUE:
`Some([1, 2])` needs an editor for `Option` and nothing said so up front, the way a function shell
knows its own parameter types from its declaration. So the entry point walks the document it just
built and derives what it finds. `Derive` is still not a registry fallback -- wiring it in would send
every unknown kind to the store -- so this asks for the specs it needs instead.
A prim contributes nothing, which is the distinction the walk exists to make: a list is drawn by a
spec that already ships, and only a named type has a declaration to derive from.
Save refuses, and now says why. A runtime value gets `Target.RuntimeValue` rather than `Scratch`, so
Ctrl-S answers "a runtime value has nowhere to save to yet: every target is code" instead of
suggesting `fluid fn <name>`, which was wrong advice. That refusal is the honest state of it: this is
a value inspector you can edit in, not yet a value you can commit. Where a value SAVES to is a design
question and it is next.
On testing it: the testfile harness registers every builtin group except `CliHost`, so nothing that
evaluates can run there at all. That would have left the entry point unpinned over one builtin call,
so `valueTargetFor` is split out with everything downstream of the evaluator in it, and that is
pinned. The evaluating half was driven by hand in a terminal.
`c` on an `Int8` field and retyping the digits gave you back a plain `Int`: right on screen, wrong type underneath, nothing to see. Found by driving it in a terminal, which is where the last two went too. The cause is that `c` CLEARS the slot to a hole before you type, so by the time a digit arrives the old node is gone and there is nothing to copy a width from. I tried copying it across a same-kind `Edit.Replace` first and threw that away: `c` never produces one, and the rotations that do already carry their fields by construction, so it was dead code that looked like a fix. The width belongs to the SLOT. A hole carries the declared type of the field it is filling, `Derive` puts it there, and `intFromDigit` already receives the `Expect` and was ignoring it. So `Prims` reads the width off the expectation, and that is the only place the `_w` convention is written now -- the editor and `Value.ofDval` were two writers of it, and two writers of the same convention is how they came to disagree. Nothing is written for a plain `Int`, so PT documents have exactly the leaves they had. A hidden field that is always present is a field every structural comparison has to know about. Pinned with `Acme.Log`, a record whose editor is derived from a declaration with an `Int8` field, so nothing in Darklang is doing the work. One case is left alone and pinned as such: a value opened on its own has no slot above it, so `fluid eval "7y"` retyped whole really has nowhere to get the width from. Replacing the entire document is saying something else goes here, and inventing an `Int8` because that is what used to be there would be a guess. Every case that matters has a field above it.
Found by opening real values rather than by reading anything. `ofDval` kinded a tuple
`Kind.named "Tuple"`, so the value entry point went looking for a package type called `Tuple` to
derive an editor from, found none, and fell back to the unknown-kind drawing:
Tuple { items = [1, "two"] } where the printer writes (1, "two")
A program's tuple is `Expr.ETuple` and has always drawn correctly. A runtime tuple is a different
node and nothing had ever drawn one. Same for a dict and for the handles with no written form at all.
They are PRIMS now, with specs in `Prims`, which is also what stops them being sent to the store as
type names to derive from.
The dict is the sharper one. `dictPairNode` took its key as a `String`. That compiled, and every
probe I could write passed, and it crashed the first time a real dict reached it:
expects String, but got Dval (DString("j"))
A runtime dict key is a whole value, and Dark's own `Dict` is keyed by text, so a dict cannot be
CONSTRUCTED from Dark at all. No probe could reach the branch; it took opening a dict that already
existed. Dark checks a call when it runs it, so a branch nothing ever runs is a branch nothing ever
checks. The key is a slot holding its own document now, which is why a string key draws its own
quotes and comes out `{ "j": 2 }` the way the printer writes it. An empty one is `{}` rather than
`{ }`, through `Piece.WhenEmpty`.
Pinned as agreement with the printer rather than as strings I typed, since a literal expectation only
ever agrees with what I believed that day. 24 values swept by hand, no diffs.
One case disagrees and it is pinned AS a disagreement: there are two printers, and on an enum the
value repr writes `Some(1)` where the source printer writes `Option.Some(1)`. Fluid draws a value so
it follows the first, and `dark eval` and `dark fluid eval` agree. Deleting the case I disagree with
is how a comparison stops being one, so it fails loudly if they are ever reconciled.
[Log { reps = 20; note = "am" }, Log { reps = 15; note = "pm" }]
is a spreadsheet that has been written out as source. `Piece.Table` has been able to draw the grid
since early on and was reachable only from a hand-written demo kind, which is a poor place to keep
the answer to "can a table be seen as a table". A record declaration already says what the columns
are, so this needed nothing written: the columns ARE the field names, the rows ARE the records, and
their slots are named the same way.
mode size mtimeSec
493 1024 1757000000
493 2048 1757000000
`Derive` emits two specs for a record type now, the record and a list of them. The list kind is
`[T]`, bracketed so it cannot collide with a real type, and it has to be a kind of its own because a
spec is found BY kind and a list of integers must go on drawing as a list.
This is the one place a value document deliberately draws something the printer would not write, so
there are two guards on it. It only fires when every row is a record of the SAME type, since a grid
is a claim that every row has the same columns. And it goes both ways: "see as a list" on the grid,
"see as a grid" on a list whose rows agree, as spec ACTIONS rather than keys the kernel knows, so a
DSL with its own tabular type gets the same switch by declaring it. The toggle is a kind change and
nothing else, so the two drawings are two readings of one document rather than two documents. The
value it came from and the value it goes back to are the same `DList` either way.
Worth its own note: the bracketed kind broke the breadcrumb, which read `StatResult]`. There were
FOUR copies of "take the last dotted segment as a display name" and one new bracket made three of
them wrong at the same moment. That is the drift t88 found in the two name shorteners, arriving
again in the same shape. There is one now, `Kind.shortLabel`, in the kernel, and `Derive.shortName`
is a call to it.
Driven live before pinning: the grid pads its columns, the caret lands in cells, a row cleared to a
hole draws its placeholder once rather than per column, and undo puts it back.
Filling a table in is the case the value document is for, and it was two keystrokes longer than it
should have been. `o` gave you `⟨Error⟩` -- a row with no cells -- and you had to press `{` to turn
it into one before you could type.
Now:
o a new row, ⟨errno⟩ ⟨message⟩
7 the first cell
Tab the next one
oops and it is a string, no quotes needed
and it lowers to a real record in the right place with the right types in the right fields.
What was in the way is a good rule with an exception. `newItemForEach` gives a HOLE whenever the
item's kind has something to type or choose, so that arriving there means something, and a derived
record has a `{` rule. A table is the exception, for a reason the projection already admits: a row
with no cells draws its placeholder once in the first column and every column is then measured
against something that is not a cell. So `newItemForSlot` asks whether the parent's spec draws this
slot as a Table, off the pieces, and nothing in the kernel knows what a grid is.
One thing I changed and put back. An untouched row is now a FILLED node, so Enter on a blank row
adds a second blank row. I made `itemIsBlank` recognise a node equal to its spec's template, which
fixed that and broke `OfferedCasesCanBePicked`: an untouched match ARM is filled in exactly the same
way, and Enter there starting the next arm is pinned and deliberate. The pin was right and I was
wrong. Enter means "start the next item" wherever you are, and a second rule for one key costs more
than the empty row it saves. Pinned as the behaviour, with the reasoning, so the next person does not
try it either.
Also pinned: Enter is an insert-mode key, so Escape first and it does nothing -- which is why the
flow above never leaves insert until the row is done. And a half-filled row refuses to lower, naming
the cell, because a table saved with a gap in it is worse than a table not saved.
`x` means "clear this, I will put something else here". In a grid what you put back is a ROW, and a
row cleared to a bare hole kept its line in the table with no cells in it: the placeholder drew in
the first column, every column was then measured against something that is not a cell, and you had
to press `{` to turn it back into a row before you could type in it.
Same rule as the new-row change and the same reason, through the same `drawsSlotAsTable` question, so
there is one answer to "is this slot a table" and not two. Clearing is not deleting: a CELL still
clears to a hole, which is what a hole means there -- one value missing from a row that is otherwise
filled in.
That broke a pin, correctly. `ATableBehavesLikeATable` asserted the placeholder-once drawing by
reaching it with `x`, and `x` no longer produces a hole row. The behaviour it was really about still
matters, since a document can arrive holding one however it was made, so that pin now builds the hole
row by hand. A pin that tests a behaviour only through the one route that used to produce it stops
testing it the day the route changes, and says nothing when it does.
Also pinned, having checked rather than assumed: column-wise motion already works on a DERIVED grid,
which reaches the spec by a `[T]` kind nobody wrote. `l` into the first cell, `l` across, `j`/`k`
down and up one column, `h` back out at the left edge, `l` sticking at the right. It was only ever
pinned against the hand-written demo.
And that a table does NOT break: a cell too wide for the pane runs long and the pane scrolls, because
breaking a cell would put the columns out of alignment, which is the one thing a table is for.
Every other construct in Fluid breaks when it does not fit; this is the one that must not, so it is
worth a pin saying so.
"`|` after a match arm's rhs to start the next arm (ambiguous with a pipe)" has been on the list of
things still to do. It is not ambiguous and it works:
| at the end of an arm's rhs starts the next arm
|> anywhere a pipe, by the two-character-operator rule
| inside a string a character
Three meanings for one key, resolved by where the caret is, correct in every case I could construct,
and not pinned anywhere. So this pins them, which is the actual work: behaviour that is right by
accident is behaviour that changes by accident.
One real edge, pinned as the answer rather than left to be found. At the END of a string that is an
arm's right-hand side, `|` starts the arm, so a trailing bar cannot be typed into that string; the
same keystroke in the same visible position means something different because of an ancestor.
Starting the next arm is the common act by a wide margin, and the workaround is to type the bar
before the last character, but the asymmetry is exactly the thing structural editing is meant to
avoid, so it is worth writing down.
Two more items off that list, checked rather than assumed and neither needing code:
Horizontal clipping on a long line. The pane already scrolls sideways with the caret, `<` at the left
edge and `>` at the right, and `fluid-app.dark` has pinned it all along. My first attempt looked like
a failure and was a key-ordering mistake of my own.
Doc comments not shown anywhere. They are drawn in the document body and wrapped in the panel's DOCS
section. What is true is the lexer bug that makes a multi-line one arrive as a single long line, and
that needs a full run this loop cannot do.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A few days ago, I tasked an agent w/ creating a structural editor for Dark, inspired by classic-dark. It's been busy and here's where it's at. I haven't reviewed the source or this PR description, but the UX I've tested locally is neat, and directionally pretty cool, for anyone who's still typing code. I'm not sure if there's any use for this, but thought I'd share because it's neat.
This adds
packages/darklang/fluid, a projectional editor for Darklang written in Darklang, andmakes it the way you write package code from the CLI:
dark edit <name>opens what is there, anddark fn|type|val <name>with nothing after it opens a blank one.It is an experiment, not a product decision. The question it is trying to answer is whether one
data description of a node kind can drive the projection, the editing rules and eventually the
printing, instead of the three hand-written copies of the grammar we keep in sync by hand.
The bet
Classic Fluid knew about each node kind in four places: the tokenizer, the caret mapping in each
direction, and the typing rules. Adding a construct meant editing all four and hoping. We still
write the grammar three times today, in two languages: the pretty-printer (
Expr -> Doc), theF# parser (
text -> Expr), and now an editor.So the thing being tested here is composition. A node kind is one
Specvalue: a list ofPiecessaying how it is drawn, plus the rules for what typing a character in a hole means, what completes
there, what actions apply. Everything else is a fold over that data.
The projection is one interpreter over
Piece. The caret is resolved from the laid-out treerather than tracked, so there is no second mapping to keep in step. The editing rules ride on the
same value as the drawing, so a kind cannot be drawable and un-editable.
Two things follow that I did not expect to get for free, and they are the reason I think the bet
is worth continuing:
Derivebuilds a fullSpecfrom aProgramTypes.TypeDeclaration, so a record you declare today is editable today.Piece.Tabledrawsa list of records as a grid, and the caret moves in two dimensions inside it.
What it does today
Building an expression. Completion is at the caret, a call arrives with one labelled hole per
parameter, and Tab walks them:
Completion that reads the types the document already carries.
optionsis declaredAgentOptions, so.offers that record's fields:Pick
writeMode, Tab to the pattern, and the menu is the cases of the enum THAT FIELD isdeclared to be, ahead of the patterns that fit anything:
Four places ask this: a match's cases, a field access, a record update, a record literal.
All four go through one resolver that reads DECLARED types off the document and the store,
recursing through field accesses so
a.b.cresolves. Nothing here infers, and where thedocument stops knowing (a call's return type, an unannotated let) the menu falls back to
what fits anything. A wrong guess could only produce a worse menu, never a wrong program,
which is what makes reading declarations good enough without a type checker in the loop.
A whole function, opened from the package store with
dark edit. The signature, the type params,the return type and the doc comment are nodes too, not just the body.
?opens a panel that sayswhat the node under the caret IS, looked up by the hash the document already carries:
A type declaration, edited as a tree. Names are written the way someone standing in that module
would say them, which is the printer's own rule rather than a second one:
And the projection that is not a line of text.
fluid demo tableis a list of records with twospecs of its own and no Darklang code in it. Columns are measured across siblings before the row
is emitted, which is the one thing a Wadler document cannot do for you, and j/k walk a column
while h/l walk a row:
Values, not just code
The newest piece, and the one that changed what I think this is for. Everything above projects a
PROGRAM. This projects what a program produced.
Dval <-> Node, both directions, with a round trip pinned over leaves, every integer width, float,list, tuple, record, enum and nested enum. No specs were written for any of it: a value's types are
only knowable from the VALUE, so the entry point walks the document it just built and derives an
editor for each type it meets from that type's declaration.
Which means a list of records is a table, from a real value, with the columns read off the record's
declaration:
The printer writes that same value as 12 lines. And you can fill one in:
ostarts a row, Tab walksthe cells, and a cell you have not reached yet is a labelled hole like anywhere else.
This is the one place the editor deliberately draws something the printer would not write, so there
are two guards on it. It only fires when every row is a record of the SAME type, since a grid is a
claim that every row has the same columns. And it goes both ways, as spec ACTIONS rather than keys
the kernel knows, so a DSL with its own tabular type gets the switch by declaring it.
Why this is the interesting half. Most people do not write code, and most code that does get
written is now generated. Both push the human's job the same way: away from authoring text, toward
supplying and correcting structured things. A structural editor over VALUES is the shape of that
job. A daily log, a form somebody asked you to fill in, a record extracted by a model with two
fields it was unsure about. In each case the type IS the form, there is nothing to keep in sync
with it, holes are typed so an invalid value is unreachable, and every node has a path a model can
point at. The same component that edits a function body edits your expense report.
It cannot be saved yet, and that is the open design question rather than an oversight. All four
save targets are code: a function body, a type declaration, a value's body expression. A runtime
value is none of those, and Ctrl-S says exactly that. Everything past "inspect a result" above needs
an answer to it.
Four things this found that reading the code would not have, all of them by opening something real:
1.5and rebuilt1.5.0. Invisible to any pin that compares documents.Registry.addAllnever updated the index it is looked up by, so a spec added that way wasinvisible to every lookup, which looks exactly like a spec that was never written.
crashed on the first real dict: a dict cannot be CONSTRUCTED from Dark, so no probe could reach
the branch. Dark checks a call when it runs it, and a branch nothing runs is a branch nothing
checks.
con anInt8and retyping gaveback a plain
Int: right on screen, wrong underneath, becausecclears to a hole before youtype and the old node is gone.
And one that is the same drift this PR is about. The grid's kind is
[T], and that one new bracketbroke THREE separate copies of "take the last dotted segment as a display name" at the same moment.
There is one copy now, in the kernel.
What it does to the CLI
The claim underneath this is that structural editing is how you WRITE Darklang, not a mode you opt
into. If that is true then it is not one more command, and most of what follows is about making
the rest of the CLI agree with it rather than about the editor.
Making something and changing it are the same door.
dark edit <name>has opened the editorfor a while.
dark fn <name>made you type a whole definition onto a command line, or pipe it inon stdin. So the CLI said "structural editing is for code that already exists", which is exactly
backwards: holes and completion are worth most on a blank page. Now
dark fn|type|val <name>withnothing after it opens the editor on a blank one:
(adds a parameter and lands on its name; Tab walks the rest.()rather than an empty parameterlist, because a function with no parameter list is not a function, so the blank shape is one you
could save as it stands.
The inline and stdin forms are untouched. They are what a script or an agent wants, and they were
never the problem; being the ONLY way was.
There is one save path. Creating, changing, and applying a text edit all reconstruct source and
hand it to
ModuleCommand.applySource, the same onedark moduleuses. No separate create routeto keep in step with the edit route.
The editor is not the only door out.
--textoneditstill opens$EDITOR,Ctrl-Einsidedoes the same on the selected node, and
dark edit <name> <file>pairs withdark view --rawfora script. A structural editor you cannot leave is worse than one you can.
Anything that writes to the terminal while the editor holds it is the editor's problem.
Evaluating an expression prints; applying an edit reports what it propagated to. Both are right at
a prompt, and neither is the editor's to suppress, so a new
Step.Repaintin the CLI kernel takesthe screen back afterwards. Without it,
von a failing expression scrolled the frame two rows andCtrl-S painted "Propagated to 39 dependents:" over the footer.
Quitting cannot lose work by accident. The editor keeps the document it opened and compares,
so
qon an unchanged document exits as it always did, andqon a changed one saysunsaved: ^S to save, or press again to discard. Two presses still throws the edit away, which is the rightdefault for a scratch buffer; one press does not.
editandfluidare indark helpnow. Both were in the command table and in no group, sohelp listed neither. They sit next to the commands that make things.
What I deliberately did NOT do:
dark evalwith no argument could open a scratch editor and run what you build. It is cheap andit fits, and
valready runs the selection from inside, so I would rather one person use thecreate flow before I add a second entrance.
nav/ls/viewdo not offer "edit this" as a keystroke. At a promptdark edit <name>isthe same thing with fewer moving parts.
It turned into a check on the printer, which I did not expect
Two total functions over the same tree, compared over the whole package store. Where they disagree,
one of them is wrong, and it was usually not the editor. Six printer bugs came out of it:
Worth saying because it is the argument for the shared precedence table, and it is not the argument
I expected. Sharing the rule did not make the two projections agree. It made the places where they
disagree LEGIBLE, which is how those six were found.
Of 57 remaining differences over the store, all 57 are classified rather than sampled: 48 are the one
deliberate bracketing above, 4 a match guard's break point, 4 the field-target indent (fixed), and 1
the printer packing an argument list greedily where the editor's group is all-or-nothing. Nothing
mechanical is left; what remains is two decisions and a taste.
Malleability, concretely
This is the part I most want reviewed as an idea rather than as code, because it is the whole
point of the experiment. Four axes, and where each one actually stands:
Derive.specForNamebuilds aSpecfrom a typedeclaration.
fluid generate <Type>prints that spec out as editable Dark source, and anypackage
valof typeDarklang.Fluid.Specis discovered by type and shadows the built-in:The two things that made this usable rather than theoretical are small.
?on any node namesits KIND, in full, because the kind string is the only handle on which editor you are replacing
and guessing it silently gets you nothing. And
generatestarts from the built-in BY NAMErather than printing a whole
Specout, so an edited copy keeps the hole rules, wrap rules,actions and binders it never mentioned. Changing how something LOOKS should not mean restating
how it behaves.
Your own syntax. Works. The operator characters, the closing delimiters, the item
separators, the string delimiter and the interpolation brackets all live on the
Registryrather than as constants in the kernel. A DSL that separates its items with
;says so.A table seen as a table. Works, and no longer only as a demo: any list of records drawn
from a real value IS a grid, with the columns read off the record's declaration and "see as a
list" / "see as a grid" toggling between the two readings as spec actions. What it still is not
is a spreadsheet: a cell is a node, so there is no formula referring to another cell, no range
selection, no column operations. None of that is blocked by the projection; they are actions and
a selection model.
Your own keys. Works. A spec's
Action.keyis consulted before the kernel's dispatch, so akind can rebind a key on its own nodes. Only
:is reserved, because the palette listseverything and has to keep working whatever a spec has bound.
What is NOT yours: the kernel still names about 23
ProgramTypescases in about 40 places. Thatis fine while PT is the only real client, and it is why a second language's specs get the generic
behaviour rather than the good behaviour.
Changes made
New, and self-contained:
packages/darklang/fluid/(15 files).doc.darkis the tree,spec.darkthe vocabulary,layout.darka Wadler/Leijen renderer whose cells carry a tag back tothe node they came from,
project.darkthe interpreter,editor.darkthe keystroke kernel,pt.darkthe Darklang client,derive.darkthe type-declaration-to-spec generator,app.darkthe TUI.
Shared code I had to touch, and the reason:
packages/darklang/languageTools/precedence.darkis new. The operator table, the operatorspellings, the two named powers, the swallowing-construct list and both parenthesisation rules
moved here out of
PrettyPrinter.ProgramTypes. Precedence is a fact about the language, andwhile it lived in the printer the editor had to ask a PRINTER what
*binds like, and the typechecker had to ask it how
&&is written. The printer keeps every name it had, as delegations.packages/darklang/prettyPrinter/programTypes.darkis that delegation, plus two removedconstants nothing referenced any more.
packages/darklang/cli/{core,loop,component}.darkgain aRunProgramstep, so a page can handthe terminal to
$EDITORand come back.Launchwas the wrong thing: it looks its string up inthe command table.
packages/darklang/cli/packages/module.darkgainsapplySource, which isexecutewith theprinting removed, so the editor's save can apply an edit without writing to stdout.
backend/src/LibExecution/ProgramTypesToDarkTypes.fs, two lines: a lambda's variables werestill declared as
(Int64 * String)tuples after let patterns replaced them, so a lambda cameback from the round trip with its parameters mangled.
Worth special review
I checked it the other way: printed and drew every package function, type and value before
and after (11,863 sections of output), and it is byte-for-byte identical except the functions
that commit edits. A pure refactor should
move nothing, and it moved nothing. Still, this is the change I would look at first.
would swallow what follows it in a keyword position (
match (a |> b) with) where the printer doesnot. I checked all six pairs against the parser: every one parses bare and survives a
print-parse-print round trip, so the brackets are not load-bearing. They are kept because
if match x with | _ -> b then 1 else 2is a sentence you have to parse yourself. Two earlierdivergences (records always breaking, an
ifalways vertical) were argued and reversed; both nowagree with the printer exactly.
you disagree. It is not a layout difference, it is a different shape: a list of records drawn as a
table rather than as source. My case is that a value document is not source, filling a table in is
the case it exists for, and both guards are real. The alternative is opt-in only, which costs a
keystroke on every value that wants it and means the default view of a spreadsheet is not a
spreadsheet.
Dval <-> Noderefuses rather than guesses. A hole names its slot, a dict says why it cannotcome back, an opaque handle says what it was. Inventing a zero for an empty integer cell would be
indistinguishable from one you typed.
EFnNameandEValueand the full name everywhereelse. That is defensible (the identity travels in the hash) but it is two policies in one file,
and the comment saying "shortening what is STORED is wrong" sits a few lines above the code that
does it.
What is not there
"line\nbreak"draws as two rows. Over 250real package functions, what the editor draws matches the printer's tokens except 19, and all 19 are
this. The clean fix wants the document to store source-form text, which needs an escape decoder,
which today only exists in F# in
LibParser/Lexer.fs. I did not want a third copy of the escapetable.
in my notes argues the shared core is the token sequence plus the precedence table, and that the
parser keeps a hand-written spine for recovery and the offside rule. This PR does the precedence
half of that and nothing else.
vevaluates the selected subexpression on demand, andfluid evalopens a wholevalue, but there is no trace list, no 404 list, none of the thing that made classic Dark's editor
what it was. That is a runtime-and-traces subsystem, not layout, and this deliberately leaves a
place for it rather than pretending to have it.
f(x)works everywhere, so nothing isunreachable, but
let a = f xputs thexin the let's BODY, because a space in a slot with anext hole means "next hole" and that beats "open an argument". Both rules are load-bearing. It is
the form the PRINTER writes, so reading code back in produces it constantly, and it is a decision
rather than a fix.
not the
|>in front of a pipe step or the)closing a lambda; covering those means taggingpunctuation, which would make it a caret target.
(, and that is a spec ACTION, so it shows inthe palette and in the panel. It is not discoverable from the drawing itself, which is the part
of the create flow I am least happy with.
Testing
The pins are Dark testfiles under
backend/testfiles/execution/cli/fluid-*.dark, in tiers:the kernel (edits, caret, history) with no Darklang knowledge; the PT client; the shells for
functions, types and values; the extension points (a foreign language's specs, a grid, custom
keys); and a deterministic key fuzzer that asserts random keys on random documents never crash and
always round-trip.
The type-directed completion pins run against REAL store types rather than fixtures, because a
fixture would only prove the resolver reads a record I built for it.
AgentOptionsand itsWriteModeare ordinary CLI code, and the pin walks both hops: the record's fields, then theenum's cases behind the field it picked.
Three properties are checked against the real package store rather than against fixtures, because
that is the only way I trusted them: every node kind the editor draws has a spec (750 items), no
complete program draws with a hole (250 functions), and 900 real items survive the round trip
through the document with nothing structural lost.
./scripts/run-backend-tests --filter tests/LibExecution/All/testfiles/execution/cli/fluid, 2,008assertions.