Skip to content

65 Virgin.root scripts fail round-trip idempotence (string->newScriptObject); latent normalization divergence #878

Open
Open
65 Virgin.root scripts fail round-trip idempotence (string->newScriptObject); latent normalization divergence#878
Labels
priority/p2Medium priority - nice to havescope/medium1-2 weeks, requires design worktype/bugDefect in existing functionality, needs fixworkstream/database-architectureDatabase format, migration, serialization

Description

@jsavin
jsavin
opened on Aug 11, 2026

Rationale

65 scripts in Virgin.root are NOT round-trip idempotent through string() → newScriptObject. Bytes change on reinstall (still compile true; cosmetic drift, not data-destroying). These represent latent normalization divergences of the #866 class — worth characterizing before any mass re-export/import workflow. Relevant to #869's guest-DB verifier and the ODB<->.ut sync process.

Current State

Round-trip non-idempotency identified (2026-08-10):

  • 65 scripts in Virgin.root fail round-trip idempotence
  • string(script) → newScriptObject produces different bytes
  • Scripts still compile (cosmetic drift only)
  • Examples: tcp.im (Jabber/AIM drivers), Manila/xmlStorageSystem callbacks, html macros, soap.rpc, table.tableToXml, sys.openUrl
  • Notable: script.newScriptObject and op.newOutlineObject THEMSELVES are on the list

Impact:

  • ODB<->.ut sync cannot guarantee byte-for-byte fidelity
  • Mass re-export/import workflows risk silent drift
  • Normalization differences accumulate
  • Pre-existing condition; latent risk for future ODB edits

Severity: Low (scripts still compile; cosmetic not functional)

Desired Outcome

Characterize the 65 non-idempotent scripts before any mass workflow:

  1. Document the normalization divergences (whitespace, comment format, line endings, etc.)
  2. Identify root causes in newScriptObject normalization
  3. Establish conventions for acceptable drift
  4. Update ODB_SCRIPT_EDITING.md with normalization expectations

Recommendation

  1. Preserve list: Full list of 65 scripts from langscan comment-strip scanner ignored single-quoted char constants: 0xC7 inside 'x' literals corrupts scripts cumulatively on install (was misattributed to newScriptObject size/reinstall) #866 investigation (already captured in langscan comment-strip scanner ignored single-quoted char constants: 0xC7 inside 'x' literals corrupts scripts cumulatively on install (was misattributed to newScriptObject size/reinstall) #866 branch records)
  2. Characterize drift: For representative samples, show before/after bytes and identify patterns
  3. Document convention: ODB_SCRIPT_EDITING.md should note that round-trip fidelity is NOT guaranteed for cosmetic properties
  4. Impact assessment: Determine if verify_virgin_root_sync.sh covers only Virgin.root, not guest-DB mirrors; silent drift accumulates #869's guest-DB verifier should ignore cosmetic drift (recommended)

Exit Criteria

Related Context

Note

This is a low-priority characterization issue. The 65 scripts still compile after round-trip; the issue is fidelity and predictability for future sync workflows.

Activity

  1. jsavin
    added
    priority/p2Medium priority - nice to have
    workstream/database-architectureDatabase format, migration, serialization
    scope/medium1-2 weeks, requires design work
    type/bugDefect in existing functionality, needs fix
    on Aug 11, 2026

  2. jsavin commented on Aug 11, 2026

    @jsavin
    jsavin
    on Aug 11, 2026
    OwnerAuthor

    Full script list (from the #866 corpus sweep)

    Recording the actual paths here so the data survives teardown of the investigation worktree -- the sweep JSON in /tmp was the only copy.

    Method: every script in Virgin.root round-tripped via string(adr^) -> script.newScriptObject, gated on script.compile, with an installer-unrepairable negative control. Checked 2996 scripts (post-fix run, at the #866 fix).

    NOT-IDEMPOTENT (65) -- read-back differs from install, but still compiles

    These are cosmetic/latent: compile is true before AND after, so nothing is broken today. Listed for the normalization-divergence investigation.

    • system.agents.StatusMessage
    • system.menus.handlers.repl.clear
    • system.menus.handlers.repl.exit
    • system.menus.handlers.repl.help
    • system.menus.handlers.repl.keycodes
    • system.verbs.apps.AnArchie.examples.findAndOutline
    • system.verbs.apps.AnArchie.utilities.fileToList
    • system.verbs.apps.BBEdit.examples.insertTextAndSave
    • system.verbs.apps.BBEdit.examples.replaceAllString
    • system.verbs.apps.Manila.examples.getMessage
    • system.verbs.apps.Manila.examples.getTemplate
    • system.verbs.apps.Manila.windowTypes.dialogs.wizard.callbacks.customRenderer
    • system.verbs.apps.Manila.windowTypes.dialogs.wizard.callbacks.getValue
    • system.verbs.apps.Manila.windowTypes.dialogs.wizard.callbacks.setValue
    • system.verbs.apps.amazon.macros.box
    • system.verbs.apps.google.macros.box
    • system.verbs.apps.trackback.getAutodiscoveryXml
    • system.verbs.apps.trackback.respondToPing
    • system.verbs.apps.xmlStorageSystem.nodeTypes.user.data.dialogs.wizard.callbacks.getValue
    • system.verbs.apps.xmlStorageSystem.nodeTypes.user.data.dialogs.wizard.callbacks.setValue
    • system.verbs.builtins.html.data.standardMacros.opmlToBlogroll
    • system.verbs.builtins.html.data.standardMacros.spamFreeMailto.form
    • system.verbs.builtins.html.dialog.draw
    • system.verbs.builtins.html.getLink
    • system.verbs.builtins.op.newOutlineObject
    • system.verbs.builtins.radio.prefs.popupMenuPref
    • system.verbs.builtins.rootUpdates.listPendingUpdates
    • system.verbs.builtins.script.newScriptObject
    • system.verbs.builtins.soap.rpc.client
    • system.verbs.builtins.soap.rpc.server
    • system.verbs.builtins.sys.openUrl
    • system.verbs.builtins.sys.setFrontApp
    • system.verbs.builtins.table.tableToXml
    • system.verbs.builtins.tcp.examples.getFromListen
    • system.verbs.builtins.tcp.getMail
    • system.verbs.builtins.tcp.im.builtinDrivers.aim.code.core.recv_event
    • system.verbs.builtins.tcp.im.builtinDrivers.aim.code.incoming.im_in
    • system.verbs.builtins.tcp.im.builtinDrivers.aim.code.incoming.start
    • system.verbs.builtins.tcp.im.builtinDrivers.aim.code.sampleCallbacks.chat_in
    • system.verbs.builtins.tcp.im.builtinDrivers.aim.code.sampleCallbacks.start
    • system.verbs.builtins.tcp.im.builtinDrivers.jabber.code.everyMinute
    • system.verbs.builtins.tcp.im.builtinDrivers.jabber.code.examples.pubsub.pubsubSubscribe
    • system.verbs.builtins.tcp.im.builtinDrivers.jabber.code.examples.pubsub.pubsubUnsubscribe
    • system.verbs.builtins.tcp.im.builtinDrivers.jabber.code.examples.rpc.xmlrpcCall
    • system.verbs.builtins.tcp.im.builtinDrivers.jabber.code.getPresenceInfo
    • system.verbs.builtins.tcp.im.builtinDrivers.jabber.code.handleMessage
    • system.verbs.builtins.tcp.im.builtinDrivers.jabber.code.handlers.iqRegister
    • system.verbs.builtins.tcp.im.builtinDrivers.jabber.code.handlers.iqVersion
    • system.verbs.builtins.tcp.im.builtinDrivers.jabber.code.handlers.presence
    • system.verbs.builtins.tcp.im.builtinDrivers.jabber.code.idTables.create
    • system.verbs.builtins.tcp.im.builtinDrivers.jabber.code.idTables.delete
    • system.verbs.builtins.tcp.im.builtinDrivers.jabber.code.messages.presence
    • system.verbs.builtins.tcp.im.builtinDrivers.jabber.code.openConnection
    • system.verbs.builtins.tcp.im.builtinDrivers.jabber.code.thread
    • system.verbs.builtins.tcp.im.builtinDrivers.jabber.code.waitForIqReply
    • system.verbs.builtins.tcp.im.callbacks.startup
    • system.verbs.builtins.tcp.im.rpi.server
    • system.verbs.builtins.tcp.im.startAll
    • system.verbs.builtins.tcp.im.stopAll
    • system.verbs.builtins.webBrowser.protocols.otherhandlers.mailto
    • system.verbs.builtins.webserver.standardMacros.refererFilter
    • system.verbs.builtins.webserver.util.pathToAddressExt
    • system.verbs.builtins.window.show
    • system.verbs.builtins.xml.plist.tableToPlist
    • system.verbs.builtins.xml.rss.formatDrivers.feed.compile

    For reference -- PRE-COMPILE-FAIL (12), tracked separately

    Broken in the stored DB independent of install. Not part of this issue; classified by mechanism in the #866 thread (LF-single-node corruption, missing kernel verbs, and genuine syntax errors including one deliberate fixture).

    • system.verbs.apps.Filemaker.examples.transactions
    • system.verbs.builtins.quickTime.isPlaying
    • system.verbs.builtins.quickTime.open
    • system.verbs.builtins.quickTime.play
    • system.verbs.builtins.quickTime.stop
    • system.verbs.builtins.string.isValidUrl
    • system.verbs.builtins.string.nthChar
    • system.verbs.builtins.userland.cleanRoot
    • system.verbs.builtins.userland.oldstuff.Frontier3toAretha
    • system.verbs.builtins.webBrowser.openURL
    • system.verbs.builtins.webserver.data.cgis.samples.syntaxError
    • system.verbs.builtins.window.quickTime
  3. jsavin
    added 6 commits that reference this issue on Aug 16, 2026

    fix(odb): P0 -- mbar compare was vacuous; skip-region offsets were wr…

    dae566d

    fix(odb): P1 -- name-keyed matching (hfirstsort is insertion order) +…

    c1e774f

    fix(odb): P2 batch -- resource teardown, float bytes, subtree depth, …

    801e41b

    fix(odb): make the ut-sync guard provable; define never-stored-both-s…

    405a347

    test(odb): commit behavioral fixtures; every compare path proves DETE…

    06ebb59

    feat(odb): --diff-roots value-level equality walker (root-build Step …

    6520a89
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority/p2Medium priority - nice to havescope/medium1-2 weeks, requires design worktype/bugDefect in existing functionality, needs fixworkstream/database-architectureDatabase format, migration, serialization

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions