Optimize compilation, indexing, and memory performance improvements

ts/docs/architecture/memory/current-system.md

@@ -149,6 +149,19 @@ See [History and forgetting](derived-views.md#history-and-forgetting).
 Coverage: [source acquisition](https://github.com/microsoft/TypeAgent/blob/main/ts/packages/memory/service/test/sourceAcquisition.spec.ts),
 [maintenance foundation](https://github.com/microsoft/TypeAgent/blob/main/ts/packages/memory/service/test/maintenanceFoundation.spec.ts).
 
+Durable acquired batches coalesce eligible distinct new sources into detached
+two-member generations, waiting at most 25 ms for the second durable admission.
+One compatible index copy/append and manifest publication serve both jobs;
+standalone ingestion and unsupported/replacement/asset/guarded cases retain
+singleton behavior. Each member keeps its own job, progress and retry identity.
+Cancellation or a stale generation discards the uncommitted candidate and retries
+unaffected members; clear cancels captured admissions. Forget/clear and close
+drain actual provider writes, even when a provider ignores abort. Physical batch
+reads and atomic publications are serialized per batch to avoid Windows
+read/rename contention without weakening durable reads.
+Coverage: [batch ingestion](https://github.com/microsoft/TypeAgent/blob/main/ts/packages/memory/service/test/batchIngestion.spec.ts),
+[KnowPro citation persistence](https://github.com/microsoft/TypeAgent/blob/main/ts/packages/memory/service/test/markdownPersistence.spec.ts).
+
 ## Search and answer semantics
 
 | Surface                         | Result                                                                          |
@@ -206,13 +219,87 @@ The service-owned store holds source manifests and revisions, event ledgers,
 ingestion jobs and derived index generations. Revision assets and acquired-batch
 state are stored separately from Views.
 
+Canonical document revision text is stored in corpus-local content-addressed
+JSON-string blobs under `revision-text`. Version-1 corpus manifests hold digest
+and byte-length references instead of inline content. New blobs are verified
+before atomic manifest replacement; existing immutable revision references do
+not rewrite retained text. Startup verifies referenced blobs and collects
+unreferenced blobs/staging files. Missing/corrupt canonical text is an error, not
+a reason to rebuild it from a derived index.
+
+Manifest references are the only retention authority. Shared text survives while
+another retained source/revision references it; forgetting the final reference
+physically removes the blob before reopening reads. Corpus clear now has a
+durable purge intent, like source forget, and restart finishes interrupted
+cleanup. Incompatible canonical formats require explicit reset/reimport; no
+canonical data is silently migrated or deleted. This removes text rewrite
+amplification, not metadata scaling or in-memory text retention. Atomic
+replacement does not establish power-loss durability.
+
+KnowPro derived generations publish small descriptors referencing shared
+immutable JSON nodes under their root's `.segments`. Copying a generation no
+longer copies all retained node payloads. Message and related-term vectors live
+in those nodes rather than a monolithic binary file. Text-only loading discards
+segmented vectors along with their index metadata before identity validation.
+Compatible retain-history replacements without acquired assets append only
+changed content and persist active revision
+eligibility; inactive knowledge/vector candidates are excluded before TopN.
+Historical reactivation retains message ordinals without re-extraction.
+Replace-without-history and unsupported custom indexes still rebuild. Explicit
+reindexing compacts inactive indexed revisions. Full hydration, JSON traversal,
+metadata work and linear vector search remain.
+
+Canonical events publish a versioned `events.jsonl` descriptor over immutable
+`event-segments` payloads, a monotonic sequence and independent eligibility
+revision. Cold single-event lookup reads its body plus suppression metadata,
+not all event bodies, but still scans filenames. Listing/search load the
+required event set. Appending an event incrementally extends a compatible
+derived index; deletion invalidates the old index even if a later append
+restores the previous count. Interrupted physical event/index purge withholds
+reads until recovery finishes. Older canonical event formats require explicit
+reset/reimport; derived indexes can be rebuilt.
+
+Explicit document reindexing captures the source revisions under a short corpus
+reservation, prepares a new generation outside the reservation, then publishes
+under a second reservation. Publication rejects a changed source or index
+generation instead of overwriting newer work. Readers keep using the committed
+generation while preparation runs; publication waits for existing readers.
+Document search repair follows the same detached path and coalesces concurrent
+readers' repair requests. Repair failures are returned to callers and can be
+retried. Other lock-held maintenance and mutation paths retain their existing
+index preparation behavior.
+
+Forgetting and clearing drain detached provider writes before purging their
+candidate directories. Close cancels publication and waits for those writes too.
+A provider that ignores cancellation can therefore delay cleanup; the service
+does not report successful cleanup while that provider could still restore
+removed files.
+
+Cleanup distinguishes sources captured from the committed corpus from new
+sources being ingested. A concurrent reindex retains an ingestion's detached
+generation until preparation finishes, then the ingestion fails its stale-index
+publication check. Forget still cancels any preparation containing the forgotten
+source, including an uncommitted addition, and clear cancels all preparations.
+
 Each corpus's `views` directory contains a private bare repository,
 `view-history.git`. Its committed snapshots hold typed Views, generated bases,
 edits, conflicts, publication and maintenance state, and Topics. The ref
 `refs/heads/typeagent-history` marks the commit boundary. Per-revision derived
 search indexes are kept apart from it, and the conversation registry and replay
 metadata are not a second content store.
 
+Full-fidelity per-View records in Git let history queries decode only the
+selected View's distinct version arrays, rather than every corpus snapshot.
+Ordering, revision filtering, editorial metadata and head guards are unchanged.
+Legacy derived records rebuild under lock once; Git metadata traversal still
+scales with commits. History privacy replacement removes the records and old
+object database together.
+
+Topic read I/O receipts use the actual committed state blob byte length. The
+Topic-only byte length is cached by corpus and commit, rather than serializing
+the full state again for every query. This changes neither the committed format
+nor Topic eligibility or continuation checks.
+
 See [View history](derived-views.md#history-and-forgetting) and the
 [service operational guide](https://github.com/microsoft/TypeAgent/blob/main/ts/packages/memory/service/README.md).
 

ts/docs/architecture/workflows/macros.md

@@ -168,6 +168,24 @@ provider flows use. CPU and time, guest memory, stack, JSON size and broker call
 all have limits. The guest reaches the outside world only through declared
 capability and semantic adapters, never through ambient process APIs.
 
+An owned `RecipeExecutor` keeps at most eight workers warm, with at most 64
+queued admissions. This preserves the eight-way concurrency measured by the
+offline runner; higher concurrency waits rather than creating unbounded workers.
+Admission time is included in the execution deadline, and excess queued work
+fails explicitly. `performance.workerQueueMs` and `performance.workerReused` in
+the run result report admitted worker waits and reuse. Idle workers are unrefed
+and retire after 30 seconds.
+
+Workers retain the initialized QuickJS WASM module, not a guest runtime or
+context. Every invocation allocates and disposes fresh guest state before
+reporting success. Failure, timeout, crash, invalid protocol or failed cleanup
+retires the worker. Asset reconfiguration retires idle workers immediately and
+active workers after their captured invocation. Closing an executor cancels
+active and queued work, waits for termination, and rejects new work. The native
+plugin owns an executor for its local service; fixture validation owns a
+single-worker executor for one private fixture batch and closes it even on
+failure. The standalone execution helper uses a shared, idle-expiring executor.
+
 - **Symbolic** programs make no model calls when they run again.
 - **Hybrid** programs make only the semantic calls they declare, with bounds,
   through the configured provider. Results are schema-validated, calls can be
@@ -185,6 +203,56 @@ writes, unsupported schemas and changed contracts all fail with an error.
 Classifying a tool's effect **doesn't grant permission**. Native hooks, content
 exclusions and the usual host and backend authorization still apply.
 
+Capability preflight uses batches of at most four concurrent inspections and
+reports failures in declaration order. Each native execution host captures one
+immutable tool inventory and reuses its approved descriptors only within that
+binding. Live session/account/workspace/generation checks still run before
+inspection and dispatch; permission or tool-change events invalidate the binding.
+Failed inventory discovery is not cached. The inventory is not a permission
+cache, and each dispatch still goes through native authorization.
+
+Guest requests and replies are correlated by request ID, so independent
+explicitly read-only calls can complete out of order. At most four reads are
+active per invocation. Semantic and effectful calls are serialized barriers
+between read groups; this does not enable provider-flow concurrency. Denial or
+cancellation stops unsent work, aborts supported in-flight waits, and suppresses
+late replies. Already dispatched calls may still be running when a native tool
+has no exact cancellation API.
+
+Durable journal mutations remain serialized, including each pre-dispatch
+authorization/revision check and flush. Journals now append flushed checkpoints
+instead of renaming the full accumulated run on every phase. An exclusive run
+claim preserves identity; first-call order and latest-phase maps rebuild from
+the journal. A partial final append is ignored conservatively, malformed
+complete records fail, and uncertain dispatched work is never automatically
+retried. Failed append prevents later dispatch; pending mutations drain before
+a run returns.
+
+The active file-backed manager uses small per-macro catalog pointers and
+immutable version/run payloads. Learning and child-authoring jobs use immutable
+content-addressed records behind versioned pointer maps, lazy rollback
+snapshots and dirty tracking. Independent owners serialize mutations using
+SQLite OS-lock transactions plus revision checks; process death releases locks.
+Existing SQLite catalog transactions are unchanged. Immutable records publish
+from flushed staging files without overwrite. Missing payloads fail closed;
+startup resumes tombstone cleanup and collects abandoned staging/revisions.
+Older canonical catalogs, job maps and journals require explicit reset/reimport,
+not silent migration or reset. Pointer maps still scale with retained jobs;
+startup and journal reconstruction are not constant-time.
+
+Authoring workers use bounded IPC notification waits, not one-second polling.
+Notifications are hints: reconnect and wake reconcile durable status and claims.
+Observe-only/disconnected workers cannot acquire a lease. Cancellation registers
+the wait before reading status, avoiding missed completions; empty/already-leased
+scans do not self-notify. A failed connection uses a bounded backoff. None of
+these storage changes grants execution permission or bypasses live dispatch
+authorization.
+
+Learning loads and verifies the physical recorded trace in one evidence pass,
+including private result sidecars and before/after source integrity checks. It
+does not perform a redundant preliminary trace/evidence load or skip physical
+verification on a cache hit.
+
 The optional TypeAgent action adapter runs in the same host. It swaps the recorded
 protocol scopes and IDs for stable schema and action dependencies plus concrete
 parameters. At run time it resolves fresh contracts and scopes, and it keeps the

ts/packages/agent-flows/src/execution/scriptExecutor.ts

@@ -67,6 +67,7 @@ export function createScriptExecutor(config: ScriptExecutorConfig): {
                 ...blockedGlobalsOverride,
             };
 
+            let timeoutId: ReturnType<typeof setTimeout> | undefined;
             try {
                 const fn = new Function(
                     ...Object.keys(sandbox),
@@ -77,7 +78,7 @@ export function createScriptExecutor(config: ScriptExecutorConfig): {
 
                 const timeout = options?.timeout ?? defaultTimeout;
                 const timeoutPromise = new Promise<never>((_, reject) => {
-                    setTimeout(
+                    timeoutId = setTimeout(
                         () => reject(new Error("Script execution timeout")),
                         timeout,
                     );
@@ -112,6 +113,8 @@ export function createScriptExecutor(config: ScriptExecutorConfig): {
                     error: errorMessage,
                     message: `Script execution failed: ${errorMessage}`,
                 };
+            } finally {
+                if (timeoutId !== undefined) clearTimeout(timeoutId);
             }
         },
     };

ts/packages/agent-flows/test/scriptExecutor.spec.ts

@@ -0,0 +1,42 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { jest } from "@jest/globals";
+import { createScriptExecutor } from "../src/execution/scriptExecutor.js";
+
+describe("script execution timers", () => {
+    const executor = createScriptExecutor({
+        apiParamName: "api",
+        defaultTimeout: 100,
+    });
+
+    beforeEach(() => jest.useFakeTimers());
+    afterEach(() => jest.useRealTimers());
+
+    test.each([
+        ["async () => 42", true],
+        ['async () => { throw new Error("failed"); }', false],
+        ['() => { throw new Error("failed"); }', false],
+        ["not valid script", false],
+    ])("clears timers after %s", async (source, success) => {
+        const result = await executor.execute(source, {}, {});
+        expect(result.success).toBe(success);
+        expect(jest.getTimerCount()).toBe(0);
+    });
+
+    test("still times out a pending script and clears the timer", async () => {
+        const result = executor.execute(
+            "async (api) => api.pending()",
+            { pending: () => new Promise<never>(() => {}) },
+            {},
+            { timeout: 20 },
+        );
+        expect(jest.getTimerCount()).toBe(1);
+        await jest.advanceTimersByTimeAsync(20);
+        expect(await result).toMatchObject({
+            success: false,
+            error: "Script execution timeout",
+        });
+        expect(jest.getTimerCount()).toBe(0);
+    });
+});

ts/packages/copilot-macros/README.AUTOGEN.md

@@ -3,7 +3,7 @@
 
 <!-- AUTOGEN:DOCS:START -->
 
-<!-- AUTOGEN:DOCS:HASH:sha256=7a02d5b2282eeb56a16414ffa0e54ace2fea0fa542a0c4793b6cbfe1b3275cfd -->
+<!-- AUTOGEN:DOCS:HASH:sha256=d8c1cefd673a898cde1cfccd35875ff8191363bb2f286381ec95215f42d73197 -->
 <!-- AUTOGEN:DOCS:SOURCE: ./README.md (hand-written documentation; this file is the AI-generated companion) -->
 
 # @typeagent/copilot-macros — AI-generated documentation
@@ -64,10 +64,10 @@ External: `@tootallnate/quickjs-emscripten`, `typescript`, `zod`
 - [./src/macroExpressions.ts](./src/macroExpressions.ts)
 - [./src/macroFeatures.ts](./src/macroFeatures.ts)
 - [./src/macroLearning.ts](./src/macroLearning.ts)
-- _…and 29 more under `./src/`._
+- _…and 31 more under `./src/`._
 
 ---
 
-_Auto-generated against commit `b31d4f4e81c76a93c35bb43c24687b93d537f84d` on `2026-10-16T23:09:45.978Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter @typeagent/copilot-macros docs:verify-links` to spot-check._
+_Auto-generated against commit `6c5d8dc89190a3602376bdb4cc9d4b87930587ac` on `2026-10-20T17:40:06.178Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter @typeagent/copilot-macros docs:verify-links` to spot-check._
 
 <!-- AUTOGEN:DOCS:END -->

ts/packages/copilot-macros/README.md

@@ -484,6 +484,12 @@ The focused specs cover definition and validation, contained program execution,
 catalog lifecycle, approval, persistence, cancellation, schema drift, and
 run-record sanitization.
 
+Persistence fixtures use `RevisionRecords` to load and commit learning jobs by
+job ID. `learning.json` is a revision manifest, not an inline `jobs` array;
+editing the manifest directly does not update its immutable job records.
+Restart validation persists rejected child contexts as failed job revisions,
+including the instruction to re-record the task.
+
 ## Limitations
 
 - Single-trace induction is deliberately narrow.

ts/packages/copilot-macros/src/contracts.ts

@@ -83,6 +83,9 @@ export interface MacroAuthoringClaim {
     cwd: string;
     sessionId: string;
     workerId: string;
+    /** Durable reconciliation happens before and after this bounded IPC wait. */
+    waitMs?: number;
+    observeOnly?: boolean;
 }
 
 export interface MacroAuthoringRequest {

ts/packages/copilot-macros/src/macroAuthoringRuntime.ts

@@ -916,13 +916,35 @@ function validMacroParameters(
     });
 }
 
+const compiledMacroGrammars = new Map<
+    string,
+    {
+        grammar: ReturnType<typeof loadGrammarRules>;
+        nfa: ReturnType<typeof compileGrammarToNFA>;
+    }
+>();
+const maxCompiledMacroGrammars = 128;
+
 function compileMacroGrammar(rules: string[]) {
+    const source = rules.join("\n");
+    const cached = compiledMacroGrammars.get(source);
+    if (cached) {
+        compiledMacroGrammars.delete(source);
+        compiledMacroGrammars.set(source, cached);
+        return cached;
+    }
     try {
-        const grammar = loadGrammarRules(
-            "learned-macros.agr",
-            rules.join("\n"),
-        );
-        return { grammar, nfa: compileGrammarToNFA(grammar, "macros") };
+        const grammar = loadGrammarRules("learned-macros.agr", source);
+        const compiled = {
+            grammar,
+            nfa: compileGrammarToNFA(grammar, "macros"),
+        };
+        compiledMacroGrammars.set(source, compiled);
+        if (compiledMacroGrammars.size > maxCompiledMacroGrammars) {
+            const oldest = compiledMacroGrammars.keys().next().value;
+            if (oldest !== undefined) compiledMacroGrammars.delete(oldest);
+        }
+        return compiled;
     } catch {
         throw new Error(
             "Macro grammar rejected (stage=compile; matchCount=unknown).",