From 89044864c253d7a10c8ba5e5826f7142b454f92b Mon Sep 17 00:00:00 2001 From: Matthew Lee Date: Thu, 28 May 2026 23:49:36 -0500 Subject: [PATCH 1/4] docs(dev-docs): onboarding for the Keyman ecosystem Add a dev-docs/ folder with five files aimed at two audiences: * External tool builders -- AI extensions for keyboard authoring or submission triage, lint bots, instrumented test rigs, etc., lodged in other repos and consuming Keyman's formats and CLI/library surface. Read keyboard-anatomy.md and external-tooling.md. * Keyman contributors -- read repository-map.md for the architecture and migration-guide.md for what's old, what's new, and what's still in flight. The migration story covers the Delphi removal (umbrella issue #4599), LDML keyboards including the still-open touch read side (#7238 / epic-ldml), the LDML visual editor scaffolding (#12798 / epic-ldml-editor), and KeymanWeb's Core-via-WASM model. Files: * dev-docs/README.md -- entry point that routes the two audiences and surfaces the LDML-first generation strategy. * dev-docs/repository-map.md -- architecture tour with 5 Mermaid diagrams (top-level layout, four-subsystems graph, keystroke sequence, keyboard build flow, build orchestration), tech-stack table, cross-platform matrix. * dev-docs/migration-guide.md -- Delphi removal scorecard, LDML layer-by-layer gap analysis, KeymanWeb modernization, links to the March 2026 roadmap and the relevant epic-* issue labels. * dev-docs/keyboard-anatomy.md -- file-format reference for a Keyman keyboard project (.kpj, .kmn, LDML XML, .kvks, .keyboard_info, .kps, .kmp, .kmx, .kmx+), with schema locations. * dev-docs/external-tooling.md -- operational guide for tools that drive Keyman from outside: kmc CLI, @keymanapp/kmc-* npm packages, schema-based validation, cross-file consistency contract, 9-point triage checklist for keyboard submissions, and the approach for instrumented forks of kmc-* / Keyman Core for desktop-vs-touch comparison work. Co-Authored-By: Claude Opus 4.7 --- dev-docs/README.md | 88 ++++++ dev-docs/external-tooling.md | 543 +++++++++++++++++++++++++++++++++++ dev-docs/keyboard-anatomy.md | 244 ++++++++++++++++ dev-docs/migration-guide.md | 396 +++++++++++++++++++++++++ dev-docs/repository-map.md | 344 ++++++++++++++++++++++ 5 files changed, 1615 insertions(+) create mode 100644 dev-docs/README.md create mode 100644 dev-docs/external-tooling.md create mode 100644 dev-docs/keyboard-anatomy.md create mode 100644 dev-docs/migration-guide.md create mode 100644 dev-docs/repository-map.md diff --git a/dev-docs/README.md b/dev-docs/README.md new file mode 100644 index 00000000000..53f7b74c069 --- /dev/null +++ b/dev-docs/README.md @@ -0,0 +1,88 @@ +# Keyman Developer Onboarding + +Welcome. This folder is the entry point for newcomers to the Keyman +ecosystem. There are two distinct audiences here: + +* **People building tools *around* Keyman from external repos** — AI + extensions for keyboard authoring, submission-triage bots for + [keymanapp/keyboards](https://github.com/keymanapp/keyboards), + validation tools, instrumented test rigs. You're consuming Keyman's + formats and CLI/library surface; you're not modifying Keyman itself. + Start at [keyboard-anatomy.md](keyboard-anatomy.md) and + [external-tooling.md](external-tooling.md). + +* **People contributing to Keyman itself** — runtime engines, the + developer tooling, KeymanWeb, the Delphi-removal effort. You'll + modify code in this repo. Start at [repository-map.md](repository-map.md) + and [migration-guide.md](migration-guide.md). + +The canonical user-facing documentation is at +[help.keyman.com](https://help.keyman.com). The canonical build +instructions are under [docs/build/](../docs/build/). This `dev-docs/` +folder is *only* for the "where is everything, what shape is it, and +why" questions. + +## The docs + +### For external tool builders + +> **Strategic frame, read this first.** The CLDR LDML keyboard format +> is the future-canonical Keyman keyboard format. The spec is complete +> for both desktop and touch keyboards. Keyman's *read* side for LDML +> touch isn't finished yet — runtime still drives off the legacy +> multi-file setup — but LDML is forward-compatible, so the right move +> for a generation tool is to emit LDML XML as the canonical source and +> only produce the transitional `.kmn`+`.kvks`+`.keyman-touch-layout` +> bundle when the deliverable has to run on touch *today*. See +> [external-tooling.md § The strategic frame](external-tooling.md#the-strategic-frame-for-tools-generating-keyboards) +> and [migration-guide.md § LDML status](migration-guide.md#status--what-works-today-vs-whats-still-in-flight) +> for the full picture. + +1. **[keyboard-anatomy.md](keyboard-anatomy.md)** — Reference for what + files make up a Keyman keyboard project (`.kpj`, `.kmn`, LDML XML, + `.kvks`, `.keyboard_info`, `.kps`, `.kmp`, ...), what each does, and + where the JSON schemas live for validation. + +2. **[external-tooling.md](external-tooling.md)** — How to drive the + Keyman compiler and engine programmatically: the `kmc` CLI, the + `@keymanapp/kmc-*` npm package surface, schema-based validation, + the cross-file consistency contract that any generator must enforce + today, triage criteria for keyboard submissions, and the approach + for instrumented forks of the compiler or engine. + +### For Keyman contributors + +3. **[repository-map.md](repository-map.md)** — A guided tour of the + top-level directory structure, what each major component does, and + how they connect. Includes Mermaid diagrams of the component graph, + build dependencies, and the cross-platform matrix. External tool + builders should read this too if they want to understand what their + tool is hooking *into*. + +4. **[migration-guide.md](migration-guide.md)** — Keyman has been + actively modernizing for years, replacing Delphi components with + C++ (Keyman Core), TypeScript (the kmc compiler family and + KeymanWeb), and embracing LDML as the canonical keyboard format. + This guide explains what has already moved, what's still in flight, + and where the long-running migration branches live. + +## Then look at the build docs + +Once you've internalized the layout, the standard build instructions live at: + +* [docs/build/windows.md](../docs/build/windows.md) — the canonical Windows + flow (Delphi 10.3 / Pro) +* [docs/build/windows-d12.md](../docs/build/windows-d12.md) — fallback for + contributors stuck on Delphi 12 Community Edition (the only free Delphi + tier today); covers IDE-based workflow plus helper scripts +* `linux/`, `mac/`, `android/`, `ios/` — platform-specific READMEs in each + top-level directory + +## Help / Getting Stuck + +* [SIL/Keyman Community Forum](https://community.software.sil.org/c/keyman) +* [Keyman roadmap blog](https://blog.keyman.com/category/roadmap/) — the + long-form context for the migrations described in `migration-guide.md` +* [Issue tracker](https://github.com/keymanapp/keyman/issues) — search for + the component or migration branch you're touching before filing new issues +* [CONTRIBUTING.md](../CONTRIBUTING.md) (at repo root) for PR conventions diff --git a/dev-docs/external-tooling.md b/dev-docs/external-tooling.md new file mode 100644 index 00000000000..477a4c93b9f --- /dev/null +++ b/dev-docs/external-tooling.md @@ -0,0 +1,543 @@ +# External Tooling Guide + +For people building tools *around* Keyman in other repositories: AI +extensions that assist keyboard development, automated submission +triage for the [keymanapp/keyboards](https://github.com/keymanapp/keyboards) +repo, lint bots, exhaustive test rigs, instrumented forks of the +compiler or engine for documentation work. + +This guide is for *you*. It assumes you know what a Keyman keyboard *is* +(see [keyboard-anatomy.md](keyboard-anatomy.md) first if not) and +explains how to drive the Keyman pipeline from external code. + +## The strategic frame for tools generating keyboards + +Two facts shape every tooling decision here: + +1. **The CLDR LDML keyboard spec is complete, including touch.** Layers + (hardware + touch), forms, flicks, longpress, multi-tap, transforms — + all defined. A spec-correct LDML keyboard with full touch markup is + a perfectly valid document today. +2. **Keyman's *read side* for LDML touch isn't complete yet.** Compiler + has TODOs, the Core C API doesn't expose layer/form data to hosts, + and the KeymanWeb / mobile OSKs still drive off the legacy + `.keyman-touch-layout` JSON, not kmx+. The runtime gap is detailed + in [migration-guide.md § LDML](migration-guide.md#what-s-not-working-yet-the-touch-gap-all-layers) + and tracked under + [#7238](https://github.com/keymanapp/keyman/issues/7238) and the + [`epic-ldml` label](https://github.com/keymanapp/keyman/issues?q=label%3Aepic-ldml). + Realistic timeline: v21-ish (months out, funding-dependent). + +The implication for an AI extension that generates keyboards: **emit +LDML XML as the canonical source of truth**, even though today Keyman +can't yet run all of it natively on mobile/web. The LDML is +forward-compatible — when Keyman closes the read side, the same +generated keyboards become natively runnable with no source rewrite. + +### Recommended generation strategy + +```mermaid +flowchart LR + src[AI extension intent
name, language, target, layout, rules] + ldml[LDML XML
canonical source] + desk{Target
= desktop only?} + done[Ship LDML alone
compile with kmc-ldml] + bundle[Also emit parallel
.kmn + .kvks +
.keyman-touch-layout] + ship[Ship LDML + bundle
kmc build runs
the .kpj path today] + + src --> ldml + ldml --> desk + desk -- yes --> done + desk -- no, touch needed --> bundle + bundle --> ship + + style src fill:#28a,color:#fff + style ldml fill:#46c,color:#fff + style done fill:#284,color:#fff + style ship fill:#a52,color:#fff +``` + +Concretely: + +* **For desktop-only keyboards:** generate LDML XML, compile with + `kmc build`. Done. No consistency contract to manage. +* **For keyboards that need to run on touch today** (mobile, web): + generate the LDML XML *and* a derived `.kmn` + `.kvks` + + `.keyman-touch-layout` bundle. Treat the LDML as the source of + truth; treat the bundle as a transitional deliverable that exists + only because the runtime can't read LDML touch yet. When Keyman + catches up, you delete the bundle and regenerate. + +The cross-file consistency contract documented below is for the +**transitional bundle** — keeping its files in sync with the LDML and +with each other. It exists because today's Keyman runtime requires it, +not because the LDML format does. + +### Two practical tactics + +1. **Start from a kmc-generated scaffold and mutate consistently.** Run + `kmc generate keyboard ...` to produce a known-good starting set, + then modify with strict cross-file invariants (see § Cross-file + consistency contract below). kmc owns the source of truth for "what + a fresh project looks like" and your tool inherits future kmc + improvements automatically. +2. **Validate the LDML XML against the CLDR spec independently of + Keyman** — that way the canonical source is correct even if your + transitional bundle has gaps. Use the CLDR keyboard schema directly + (see [§ Validating metadata with JSON Schemas](#validating-metadata-with-json-schemas)). + +## The two integration surfaces + +```mermaid +graph LR + subgraph EXT[Your external tool] + yt[AI extension / triage bot] + end + subgraph CLI[CLI surface] + kmc[kmc
command-line] + end + subgraph LIB[Library surface] + npm[@keymanapp/kmc-*
npm packages] + end + subgraph CORE[Native surface] + core_so[Keyman Core
libkmcore.so / .dll / .dylib] + wasm[Keyman Core
WebAssembly] + end + + yt --shell out--> kmc + yt --import--> npm + yt --FFI / Emscripten--> core_so + yt --import wasm--> wasm + + style EXT fill:#28a,color:#fff + style CLI fill:#284,color:#fff + style LIB fill:#46c,color:#fff + style CORE fill:#a52,color:#fff +``` + +* **CLI surface** — shell out to `kmc` (or `kmcomp` for legacy `.kmn`-only + work). Easiest, least coupling, but you pay process-startup cost per + invocation and you parse text output. +* **Library surface** — import the `@keymanapp/kmc-*` packages from npm + in a Node.js tool. Get TypeScript types, programmatic results, no shell + parsing. Recommended for any non-trivial tool. +* **Native surface** — for tools that need to *execute* keyboards (not + just compile them), link Keyman Core as a shared library (desktop / + server) or import the WebAssembly build (Node.js or browser). + +## The `kmc` CLI + +`kmc` is the unified TypeScript entry point. Subcommands available: + +``` +kmc analyze Analyze a keyboard / project +kmc build Compile a keyboard, project, model, or package +kmc copy Duplicate a project with a new id +kmc generate Scaffold a new project (keyboard, model, etc.) +kmc message Print info about a kmc compiler message (warning/error) +``` + +Install (Node 20+): + +```bash +npm install -g @keymanapp/kmc +``` + +Or use directly from a checkout: + +```bash +node /path/to/keyman/developer/src/kmc/build/kmc.js +``` + +Get help: + +```bash +kmc --help +kmc --help +``` + +### Compiling a keyboard programmatically + +```bash +# Compile a single .kmn or LDML XML file +kmc build path/to/keyboard.kmn + +# Compile an entire .kpj project (produces .kmx + .js + .kmp) +kmc build path/to/keyboard.kpj + +# JSON output for machine consumption +kmc build --log-format json path/to/keyboard.kpj +``` + +Exit code is 0 on success, non-zero on errors. Use +`--log-format json` to get a structured stream of messages your tool +can parse. + +### Analyzing a keyboard + +```bash +# Lint / static analysis +kmc analyze path/to/keyboard.kpj + +# Get the rendered Osk image and other artifacts +kmc analyze osk-char-use path/to/keyboard.kmn +``` + +### Scaffolding a new keyboard + +```bash +kmc generate keyboard \ + --author "Your Name" \ + --copyright "(c) Your Org" \ + --bcp47 "und-Latn" \ + --name "my_keyboard" \ + --license "MIT" \ + --target "any" \ + --description "Demo keyboard" +``` + +Result is a directory with `.kpj`, `.kmn`, `.keyboard_info`, `.kps`, +`.kvks` (basic touch layout), `welcome.htm`, `README.md`, +`HISTORY.md`, `build.sh`. It compiles out of the box. + +This is your starting point for any "reliably produce a Keyman developer +folder that compiles" tool. + +## The `@keymanapp/kmc-*` library surface + +Each `kmc-*` directory under `developer/src/` is published as an npm +package. Use them as a library when you're already in a Node.js tool +and want structured results instead of shelling out. + +Notable packages and their roles: + +| Package | What it does | +|---|---| +| `@keymanapp/kmc` | The unified CLI (drives the others) | +| `@keymanapp/kmc-kmn` | Compile `.kmn` → `.kmx` (delegates to native kmcmplib) | +| `@keymanapp/kmc-ldml` | Compile LDML XML → `.kmx+` (TypeScript-native) | +| `@keymanapp/kmc-package` | Build `.kmp` package from `.kps` | +| `@keymanapp/kmc-keyboard-info` | Validate / compile `.keyboard_info` | +| `@keymanapp/kmc-analyze` | Static analysis / lint | +| `@keymanapp/kmc-copy` | Duplicate a project under a new id | +| `@keymanapp/kmc-generate` | Scaffold new projects | +| `@keymanapp/kmc-model` | Compile lexical models | +| `@keymanapp/kmc-model-info` | Validate `.model_info` metadata | +| `@keymanapp/common-types` | TypeScript types for `.kpj`, `.keyboard_info`, etc., plus generated JSON-schema validators | + +Example: compiling a keyboard from a Node script. + +```ts +import { KmnCompiler } from '@keymanapp/kmc-kmn'; +import { CompilerCallbacks, NodeCompilerCallbacks } from '@keymanapp/common-types'; + +const callbacks: CompilerCallbacks = new NodeCompilerCallbacks(); +const compiler = new KmnCompiler(); +await compiler.init(callbacks); + +const result = await compiler.runCompiler('path/to/keyboard.kmn', /*infile_str=*/null, { + shouldAddCompilerVersion: true, +}); + +if (!result) { + // compile failure: read messages from callbacks + console.error(callbacks.messages); +} +// result.kmx is the compiled binary +``` + +The exact API may evolve; check each package's `src/` and exported `index.ts` +for the current shape. + +## Cross-file consistency contract + +For tools generating or mutating a `.kmn`-based keyboard project (the +non-LDML path), these values must stay aligned across the project files. +Most validation failures and weird runtime behaviors come from breaking +one of these. + +### Identity + +The **keyboard ID** is the most-replicated value. Treat it as the +canonical key everywhere: + +| Where | How it appears | +|---|---| +| Directory name | `/` (convention) | +| File names | `.kpj`, `.kmn`, `.kps`, `.kvks`, `.keyman-touch-layout`, `.ico`, `.keyboard_info` | +| `.kpj` | The `` references files by relative path; the ID is implicit in the filenames | +| `.keyboard_info` | The filename stem IS the ID; not repeated inside the JSON | +| `.kps` | Lists output artifacts (`.kmx`, `.js`, `.kvk`) with `.` names | +| Compiled `.kmx`/`.kmp` | Identity baked into the binaries; tools downstream key off it | + +Rules: lowercase, ASCII letters / digits / underscores. Must be unique +across the [keymanapp/keyboards](https://github.com/keymanapp/keyboards) +catalog if you plan to submit. + +### Display name + +The human-readable keyboard name appears in two places: + +| Where | Form | +|---|---| +| `.kmn` | `store(&NAME) 'Display Name'` | +| `.kps` | `Display Name` | + +(`.keyboard_info` does *not* carry the name; consumers display the +package metadata.) + +### Versions — three different things, don't confuse them + +This is the easiest place to mess up: + +| Concept | `.kmn` | `.keyboard_info` | `HISTORY.md` | +|---|---|---|---| +| Keyboard version (what you ship) | `store(&KEYBOARDVERSION) '1.2.3'` | (no field) | top-most heading: `1.2.3 (YYYY-MM-DD)` | +| Min Keyman engine version | `store(&VERSION) '14.0'` | (no field) | n/a | +| Schema/file format version | n/a | n/a (implicit in schema) | n/a | + +A bump to the keyboard version touches `.kmn` AND `HISTORY.md`. The +engine min-version (`&VERSION`) only changes when you start using +features that require a newer Keyman. + +### Copyright / Author / License + +| Where | Form | +|---|---| +| `.kmn` | `store(©RIGHT) '© Your Name'` | +| `.kps` | `…Your Name` | +| `.keyboard_info` | `"license": "mit"` (SPDX id) | +| `LICENSE.md` | full license text | +| `README.md` / `welcome.htm` | display only | + +The `.keyboard_info` license is an SPDX identifier (e.g. `"mit"`, +`"apache-2.0"`) and is what triage uses. The `LICENSE.md` must be the +matching text. + +### Language tags (BCP-47) + +| Where | Form | +|---|---| +| `.keyboard_info` | `"languages": ["en", "und-Latn", "am"]` (array of BCP-47) | +| `.kmn` (optional) | `store(&BCP47CODE) 'en'` for the primary language | + +The `.keyboard_info` array is the authoritative list. Each entry must +be a valid BCP-47 tag — use `@keymanapp/langtags` or `kmc analyze` to +validate. + +### Resource references inside `.kmn` + +Each "embedded" resource needs the file to exist alongside the `.kmn`: + +| Store | Points at | Required if | +|---|---|---| +| `store(&BITMAP)` | `.ico` (or `.bmp`) | Always (or omit if no icon) | +| `store(&VISUALKEYBOARD)` | `.kvks` | Desktop OSK / mobile touch support | +| `store(&LAYOUTFILE)` | `.keyman-touch-layout` | Mobile touch (newer touch layout format) | + +If a store points at a missing file, `kmc build` fails. If a `.kvks` +exists but isn't referenced, the OSK is silently omitted from the +package — the failure mode that bites AI generators hardest. + +### Targets / platforms + +| Where | Form | Notes | +|---|---|---| +| `.kmn` | `store(&TARGETS) 'any'` or `'desktop mobile'` etc. | Space-separated list | +| `.keyboard_info` (optional) | `"platformSupport": { "windows": "full", ... }` | More granular per-platform support | + +If `.kmn` says `'mobile'` but there's no `.kvks` / `.keyman-touch-layout`, +`kmc build` warns or errors depending on flags. + +### Quick check: does the generated project compile + validate? + +A scaffolded keyboard from a healthy generator passes all of: + +```bash +kmc build path/to/.kpj # exit 0 = compile clean +kmc analyze path/to/.kpj # exit 0 = lint clean +# JSON Schema validate .keyboard_info +ajv validate -s common/schemas/keyboard_info/keyboard_info.schema.json \ + -d path/to/.keyboard_info +``` + +Treat all three as required checks in your AI extension's +generate-keyboard pipeline. + +## Validating metadata with JSON Schemas + +For triage tools that just want to *validate* keyboard submissions +without compiling, the schemas live at: + +``` +common/schemas/keyboard_info/keyboard_info.schema.json # .keyboard_info +common/schemas/keyboard-info-source/ # source-side +common/schemas/displaymap/displaymap.schema.json # display maps +common/schemas/kvk/kvk.schema.json # visual keyboards +common/schemas/package/package.schema.json # package metadata +common/schemas/kpj/kpj.schema.json # project files +``` + +Pre-generated validators (ajv-style) and TypeScript types are at: + +``` +common/web/types/obj/schemas/*.schema.validator.{cjs,mjs} # runtime validators +common/web/types/src/schemas/*.schema.ts # TS types +``` + +If your tool is Node-based, install `@keymanapp/common-types` and +import: + +```ts +import { KeymanFileTypes, validateKeyboardInfoFile } from '@keymanapp/common-types'; +const { valid, errors } = await validateKeyboardInfoFile('path/to/x.keyboard_info'); +``` + +If your tool isn't Node-based, point any JSON Schema validator (ajv-cli, +jsonschema for Python, etc.) at the `.schema.json` files directly. + +## Triage criteria for keymanapp/keyboards submissions + +The [keymanapp/keyboards](https://github.com/keymanapp/keyboards) repo +expects each submitted keyboard to clear these gates. A triage bot +should check them in order: + +1. **Compiles.** `kmc build path/to/keyboard.kpj` returns exit 0. +2. **Metadata is schema-valid.** `.keyboard_info` validates against the + schema; `.kps` validates against the package schema. +3. **Identifiers match.** The `id` in `.keyboard_info`, the filename of + the `.kmn`/`.xml`, and the `id` attribute in `.kpj` are consistent. +4. **License is recognized.** `license` in `.keyboard_info` is one of + the accepted SPDX identifiers (MIT, Apache-2.0, etc.). +5. **Target language tags are valid BCP-47.** Each `bcp47` in + `.keyboard_info` is a valid tag (use `kmc analyze` or + `@keymanapp/langtags`). +6. **Tests pass** (when present). If `kmc analyze` reports test + failures, the keyboard isn't ready. +7. **No bundled binaries the source doesn't justify.** The package + shouldn't ship a `.kmx` if there's no matching `.kmn`/`.xml` source. +8. **Documentation present.** A non-empty `welcome.htm` (or + `welcome.md`) and a `HISTORY.md`. +9. **Version monotonic.** If updating an existing keyboard, version in + `.keyboard_info` must increase. + +Most of these are programmatically checkable via the `@keymanapp/kmc-*` +packages plus the JSON Schemas. Layered approach for a triage bot: + +```mermaid +flowchart TD + pr[New PR / submission] + s1{Schemas valid?} + s2{kmc build OK?} + s3{kmc analyze OK?} + s4{Custom checks
license, ids, version?} + pass[Auto-approve / queue for human] + fail[Bot comment with specifics] + + pr --> s1 + s1 -- yes --> s2 + s1 -- no --> fail + s2 -- yes --> s3 + s2 -- no --> fail + s3 -- yes --> s4 + s3 -- no --> fail + s4 -- yes --> pass + s4 -- no --> fail +``` + +## Forking the compiler or engine for instrumentation + +If your goal is to exhaustively test keyboards or compare behavior +(e.g. desktop vs touch), the practical approach is to fork a `kmc-*` +package or Keyman Core itself, add instrumentation, and run your +keyboard corpus through it. + +### Instrumenting `kmc` for compile-time analysis + +* Fork `developer/src/kmc-kmn/` (for `.kmn` keyboards) or + `developer/src/kmc-ldml/` (for LDML). +* Add hooks in the compiler's `CompilerCallbacks` interface — log every + rule, every grouping, every transform. +* Build with `developer/build.sh build:kmc-kmn` (or `:kmc-ldml`); run + via your fork's CLI. +* Keep the patched files small and isolated so you can rebase the fork + onto upstream periodically. + +### Instrumenting Keyman Core for runtime comparison + +For runtime behavior comparisons (e.g. does this keyboard produce the +same output on desktop and touch?): + +* Fork `core/src/` and add logging or trace hooks where the + `km_core_state_process_event()` function dispatches events. +* Build with `core/build.sh build:x64` (or `:wasm` for browser-side). +* For desktop testing: link your instrumented `libkmcore` into a small + C++ harness that feeds simulated key events. +* For touch testing: build to WASM, load in a Node.js harness with + `@keymanapp/keyman/engine/web/test` helpers, simulate touch input. + +The `core/tests/` directory has examples of how Keyman Core is exercised +from C++ test harnesses — these are good starting points for your own +harness. + +### Comparing desktop vs touch behavior + +The behavior difference between desktop and touch usually comes from: + +* The `.kvks` (OSK layout) — different keys / modifiers exposed +* `&platform()` rules in `.kmn` (or LDML transforms scoped to platforms) +* The host's event handling — what the OS / browser passes to the engine + +For exhaustive comparison, instrument both contexts using the same input +corpus: + +```mermaid +graph TD + corpus[Keyboard test corpus] + desk[Desktop harness
libkmcore + simulated kbd events] + touch[Touch harness
WASM kmcore + simulated tap events] + diff[Diff tool
compare action streams] + + corpus --> desk + corpus --> touch + desk --> diff + touch --> diff + diff --> report[Comparison report
per-keyboard, per-input] +``` + +The `web/src/test/` directory has fixtures and helpers for the touch +side; `core/tests/` has fixtures for the desktop side. Reusing the same +keyboard files in both harnesses is the easy part — generating an +equivalent simulated-input stream is the harder design problem. + +## Don't reinvent these + +Things the `kmc` / Keyman Core ecosystem already does — don't roll your +own: + +* **`.kmn` parsing**: use `kmc-kmn`'s parser (or the C++ parser in + `kmcmplib` if you need native code). +* **LDML XML parsing**: use `kmc-ldml`'s parser (it handles the + CLDR-specific quirks). +* **`.keyboard_info` validation**: use the schemas + validators in + `common/schemas/` and `common/web/types/`. +* **Language-tag handling**: use `@keymanapp/common-types` / + `@keymanapp/langtags`; the BCP-47 + IANA subtag stuff is non-trivial + and the team has invested in getting it right. + +## Pointers + +* [keyboard-anatomy.md](keyboard-anatomy.md) — file format reference +* [repository-map.md](repository-map.md) — where things live inside Keyman +* [migration-guide.md](migration-guide.md) — what's old, what's new, + what to bias toward +* [help.keyman.com/developer/](https://help.keyman.com/developer/) — + authoritative user-facing developer docs (the Keyman language + reference, the touch-layout designer guide, etc.) +* [common/test/keyboards/](../common/test/keyboards/) — small, + feature-focused example keyboards +* [keymanapp/keyboards](https://github.com/keymanapp/keyboards) — the + production keyboard catalog; the data your triage tools will work with +* [keymanapp/lexical-models](https://github.com/keymanapp/lexical-models) + — predictive text models (separate from keyboards but use the same + `kmc-*` ecosystem) diff --git a/dev-docs/keyboard-anatomy.md b/dev-docs/keyboard-anatomy.md new file mode 100644 index 00000000000..eaf8f92a421 --- /dev/null +++ b/dev-docs/keyboard-anatomy.md @@ -0,0 +1,244 @@ +# Keyboard Project Anatomy + +A reference for tools that read, write, validate, or generate Keyman +keyboard projects. If you're building something that produces Keyman +keyboards (a scaffolder, an AI-assisted authoring extension, a triage +bot), start here. + +For the Keyman *runtime* — how a compiled keyboard actually runs on the +user's machine — see [repository-map.md](repository-map.md). For *why* +some of these formats exist (and which are being phased out), see +[migration-guide.md](migration-guide.md). + +## File extensions at a glance + +| Extension | Role | Hand-edit? | Notes | +|---|---|---|---| +| `.kpj` | Project file (XML) | Mostly — Tike maintains it | Lists the source files; references metadata | +| `.kmn` | Keyman keyboard source | Yes | The historical proprietary format | +| `.xml` (LDML) | LDML keyboard source | Yes | The modern open-standard format ([CLDR keyboards](https://cldr.unicode.org/index/keyboard-workgroup)) | +| `.kvks` | Visual / on-screen keyboard layout (XML) | Yes (or via Tike) | Touch + desktop OSK definitions | +| `.kvk` | Compiled visual keyboard binary | No | Output of `.kvks` | +| `.keyboard_info` | Keyboard metadata (JSON) | Yes | Schema-validated; published to keyman.com | +| `.kps` | Package source (XML) | Mostly — Tike maintains | Lists what goes into the package | +| `.kmp` | Package binary (ZIP) | No | The single deliverable: contains kmx, js, kvk, fonts, docs | +| `.kmx` | Compiled keyboard binary | No | Output of `.kmn` compilation; runs in Keyman Core | +| `.kmx+` | LDML-extended kmx (same `.kmx` filename) | No | Output of LDML compilation; runs in Keyman Core | +| `.js` | KeymanWeb keyboard | No | Output of compilation; runs in browser | +| `.html` | Welcome / help page | Yes (optional) | Shipped inside the `.kmp` | +| `HISTORY.md` | Version history | Yes | Convention; rendered on keyman.com | +| `README.md` | Documentation | Yes | Optional | +| `fonts/*.ttf` `*.otf` | Embedded keyboard fonts | n/a | Shipped inside the `.kmp` | + +## Source → compiled outputs + +```mermaid +flowchart LR + subgraph SRC[Source files] + kpj[.kpj
project] + kmn[.kmn
source] + ldml[LDML XML
source] + kvks[.kvks
OSK layout] + kps[.kps
package source] + info[.keyboard_info
metadata] + end + subgraph CC[kmc compiler] + kmc[kmc CLI
TypeScript] + kmnc[kmcmplib
C++] + end + subgraph OUT[Compiled outputs] + kmx[.kmx /
.kmx+] + js[.js] + kvk[.kvk] + kmp[.kmp
package] + end + + kpj --> kmc + kmn --> kmnc + kmnc --> kmx + kmn --> kmc + kmc --> js + ldml --> kmc + kvks --> kmc + kvks --> kvk + kps --> kmc + info --> kmc + + kmx --> kmp + js --> kmp + kvk --> kmp + info --> kmp + + style SRC fill:#46c,color:#fff + style CC fill:#284,color:#fff + style OUT fill:#a52,color:#fff +``` + +## File-by-file deep dive + +### `.kpj` — Keyman project file + +XML, root element ``. Lists every file in the +project: source `.kmn` / LDML XML files, the `.kvks` OSK layout, the +package source `.kps`, the metadata `.keyboard_info`, the documentation +pages, the fonts. + +Hand-editing is sometimes necessary (e.g. adding a file Tike didn't pick +up); for generated projects, follow the structure of one of the +canonical examples (see [Examples in the repo](#examples-in-the-repo)). + +### `.kmn` — Keyman source + +The historical Keyman keyboard source format. Defines rules in a +domain-specific syntax: + +``` +store(&NAME) 'My Keyboard' +store(&VERSION) '1.0' +begin Unicode > use(main) +group(main) using keys +'a' > 'A' +``` + +Reference: . New keyboards +should be authored in LDML XML instead where possible — see § below and +[migration-guide.md § LDML](migration-guide.md#migration-2-ldml-keyboards). +`.kmn` is the today-required deliverable for touch keyboards because +Keyman's LDML read side isn't yet complete for touch (the CLDR spec +itself is — the gap is purely on Keyman's side). + +### LDML XML — Keyman keyboard authored in CLDR LDML + +**The future-canonical keyboard source format.** LDML is the Unicode CLDR +open standard for keyboards. The spec covers everything Keyman needs +including full touch markup (layers/forms/flicks/longpress/multi-tap), +so a spec-correct LDML keyboard is a complete authoring artifact — +hand-editable XML, no parallel files needed. + +File extension is `.xml`. Compiles to `.kmx+` (a `.kmx` with extra +LDML-specific sections inside). References: +* [Unicode CLDR keyboard workgroup](https://cldr.unicode.org/index/keyboard-workgroup) +* Keyman compiler: `developer/src/kmc-ldml/` +* Keyman runtime: `core/src/ldml/` + +**What works today**: desktop keyboards compile and run end-to-end from +LDML. + +**What doesn't work today**: touch keyboards authored in LDML compile, +but Keyman doesn't yet read the LDML touch markup through to the OSK on +mobile/web — the runtime still drives off the legacy `.keyman-touch-layout` +JSON. See [migration-guide.md § LDML touch gap](migration-guide.md#what-s-not-working-yet-the-touch-gap-all-layers) +for details. + +**Practical guidance**: +* For desktop-only keyboards, prefer LDML XML. No consistency contract. +* For touch keyboards, generate LDML *and* a parallel `.kmn` + `.kvks` + + `.keyman-touch-layout` bundle. Treat LDML as the source of truth; + treat the bundle as transitional until Keyman closes the read gap. + See [external-tooling.md § Recommended generation strategy](external-tooling.md#recommended-generation-strategy). + +### `.kvks` — Visual keyboard source + +XML, defines the layout of the on-screen keyboard for *both* desktop OSK +and mobile touch keyboards. Each key is positioned, sized, and bound to +a `.kmn`/LDML rule via key codes or named keys. Reference: +. + +Compiles to the binary `.kvk` shipped inside the `.kmp`. + +### `.keyboard_info` — Keyboard metadata + +JSON document describing the keyboard for the keyboards.keyman.com +catalog and the Keyman app stores. Required fields include name, id, +license, version, supported platforms, supported languages. + +**Schema** (validate against this): +[`common/schemas/keyboard_info/keyboard_info.schema.json`](../common/schemas/keyboard_info/keyboard_info.schema.json). + +Auto-validators (CommonJS + ESM) are generated and live under +`common/web/types/obj/schemas/keyboard_info.schema.validator.{cjs,mjs}`. +TypeScript types live under +`common/web/types/src/schemas/keyboard_info.schema.ts`. + +### `.kps` — Package source + +XML describing what goes into the final `.kmp` package: keyboards, +visual keyboards, documentation files, fonts, install behavior. +Reference: . + +### `.kmp` — Package (deliverable) + +ZIP file containing the compiled `.kmx`/`.kmx+` (one per keyboard), +`.js` (for KeymanWeb), `.kvk` (compiled OSK), `kmp.json`, fonts, and +optional `welcome.htm` / `readme.htm`. This is the single artifact a user +or store installs. + +Inspection: any zip tool works. The package's identity, version, and +contents are described in `kmp.json` at the root of the archive. + +### `.kmx`, `.kmx+`, `.js` + +Binary compiled outputs. Don't hand-edit. Documented for reference: +* `.kmx` format: [docs/file-formats/kmx-file-format.md](../docs/file-formats/kmx-file-format.md) +* `.kmx+` (LDML extensions): [docs/file-formats/kmx-plus-file-format.md](../docs/file-formats/kmx-plus-file-format.md) +* `.js` (KeymanWeb): not documented in repo; generated by `kmc-kmn` / + `kmc-ldml` to be loaded by KeymanWeb at runtime. + +## Schemas + +Programmatically validating keyboard metadata / project structure uses +the JSON Schemas under `common/schemas/`: + +| Schema | Validates | Path | +|---|---|---| +| `keyboard_info.schema.json` | `.keyboard_info` files | `common/schemas/keyboard_info/keyboard_info.schema.json` | +| `package.schema.json` | `.kps` package metadata | `common/schemas/package/` | +| `kpj.schema.json` (where applicable) | `.kpj` project files | `common/schemas/kpj/` | +| `displaymap.schema.json`, `kvk.schema.json`, etc. | Various subformats | `common/schemas/` | + +The TypeScript-generated validators (used by `kmc` internally) live +under `common/web/types/`. External tools can either: +* call the JSON Schemas directly with their own validator (ajv, etc.), or +* import the `@keymanapp/common-types` npm package for the + pre-generated validators and TS types. + +## Examples in the repo + +The most useful starting points for tooling work: + +| Location | What it is | +|---|---| +| `common/test/keyboards/baseline/` | A directory full of small `.kmn` keyboards, each illustrating one Keyman language feature (deadkeys, virtual keys, options, etc.) | +| `common/test/keyboards/text_selection_tests_keyboard_9073/` | A complete project with `.kpj`, `.kmn`, `.keyboard_info`, `build.sh` | +| `common/test/keyboards/platform-rules/` | Demonstrates platform-specific rule behavior (desktop vs touch) | +| `web/src/test/manual/web/keyboards/diacritic_rota/` | A KeymanWeb test keyboard with full `.keyboard_info` | + +To see a *complete*, real-world example, point at the +[keymanapp/keyboards](https://github.com/keymanapp/keyboards) repo — +that's where production keyboards live and where AI submission-triage +tools will spend most of their time. Layout: one keyboard per +subdirectory under `release///`. + +## Minimum viable keyboard project + +For a generation tool, this is the smallest set of files that compiles +to a valid `.kmp`: + +``` +my_keyboard/ +├── my_keyboard.kpj XML +├── my_keyboard.kmn store(&NAME), begin Unicode > use(main), +│ group(main) using keys, at least one rule +├── my_keyboard.keyboard_info JSON validated against keyboard_info.schema.json +└── my_keyboard.kps XML referencing the above +``` + +Plus optional but normal: +* `my_keyboard.kvks` — touch / OSK layout +* `welcome.htm` — first-run welcome page +* `HISTORY.md` — version history +* `fonts/*.ttf` — embedded fonts + +The `kmc generate` subcommand (see +[external-tooling.md](external-tooling.md#scaffolding-a-new-keyboard)) +will produce a working scaffold of this set. diff --git a/dev-docs/migration-guide.md b/dev-docs/migration-guide.md new file mode 100644 index 00000000000..f6345f509ed --- /dev/null +++ b/dev-docs/migration-guide.md @@ -0,0 +1,396 @@ +# Migration Guide: The Modernization in Flight + +Keyman is in the middle of several long-running migrations. As a contributor, +knowing where each migration is on the timeline is the difference between +modifying code that ships for another decade and modifying code that is +weeks away from deletion. This guide is the map. + +Three things to understand before you write any non-trivial patch: + +1. **The Delphi removal** — replacing Delphi components with C++ and TypeScript. +2. **LDML keyboards** — replacing the proprietary `.kmn`/`.kmx` keyboard + format with the open Unicode CLDR LDML keyboard standard. +3. **KeymanWeb modernization** — folding the historical KeymanWeb codebase + onto the shared Keyman Core (C++ compiled to WebAssembly) and a modern + TypeScript host. + +These are tracked across several long-lived branches and a multi-year +roadmap. The [Keyman roadmap blog](https://blog.keyman.com/category/roadmap/) +is the authoritative high-level source; this document summarizes the state +as understood from the codebase and links into specific issues and source +trees you can read for yourself. + +## Migration 1: Delphi removal + +### Why + +The original Keyman Developer IDE and the Windows Engine were written in +Delphi starting in the 1990s. Delphi was a practical choice then — strong +Win32 integration, a mature VCL UI toolkit. Today the licensing situation +is hostile to open-source contribution: the only freely-available tier +(Community Edition) was crippled in 10.4+ to remove command-line +compilation, leaving contributors stuck on the no-longer-distributed 10.3 +CE or on a paid Professional license. + +The Keyman team has been [tracking removal](https://github.com/keymanapp/keyman/issues/4599) +since 2022. Progress has been steady but slow as team capacity has +shrunk, so the long tail keeps stretching. + +### Scorecard (as of 2026 mid-year) + +```mermaid +graph LR + subgraph DONE[Migrated] + kmcomp[kmcomp.exe
.kmn compiler] + kmanalyze_old[kmanalyze
analysis tool] + kmconvert_old[kmconvert
keyboard converter] + kmdecomp_old[kmdecomp] + runtime[Keyboard runtime
processing] + lang[Language code utilities] + end + subgraph DELPHI[Still Delphi] + tike[TIKE
Keyman Developer IDE] + kmshell[kmshell
Keyman Desktop UI] + engine[Windows Engine
keyman.exe, kmcomapi.dll, ...] + setup_dev[Keyman Developer setup] + codegen[Build/codegen tools
devtools, build_standards_data, ...] + end + subgraph NEW[New stack] + kmcmplib[kmcmplib
C++] + kmcli[kmc & kmc-*
TypeScript] + core[Keyman Core
C++] + end + + kmcomp -.->|replaced by| kmcmplib + kmcomp -.->|wrapped by| kmcli + kmanalyze_old -.->|replaced by| kmcli + kmconvert_old -.->|replaced by| kmcli + runtime -.->|replaced by| core + lang -.->|replaced by| kmcli + + style DONE fill:#284,color:#fff + style DELPHI fill:#a52,color:#fff + style NEW fill:#46c,color:#fff +``` + +**Migrated away from Delphi:** + +| Old (Delphi) | New stack | Where to find the new code | +|---|---|---| +| `kmcomp.exe` — the `.kmn` keyboard compiler | C++ (`kmcmplib`) + TS wrapper (`kmc-kmn`) | `developer/src/kmcmplib/`, `developer/src/kmc-kmn/` | +| `kmanalyze` — analysis tool | `kmc-analyze` (TS) | `developer/src/kmc-analyze/` | +| `kmconvert` — generation/conversion | `kmc-generate`, `kmc-copy`, `kmc-keyboard-info` (TS) | `developer/src/kmc-{generate,copy,keyboard-info}/` | +| `kmdecomp` — decompiler | (none — marked Unsupported) | — | +| Keyboard runtime engine | Keyman Core (C++) | `core/` | +| Build-time language tag utilities | `kmc` family + `Keyman.System.LanguageCodeUtils` shim | `developer/src/kmc*/` | + +**Still Delphi (no replacement yet):** + +| Component | Path | Why still Delphi | +|---|---|---| +| Tike — the Keyman Developer IDE | `developer/src/Tike/` | Largest VCL app in the project; replacement is a major design exercise | +| Keyman for Windows shell | `windows/src/desktop/kmshell/` | Deeply integrated with Windows TSF / shell | +| Windows Engine | `windows/src/engine/{keyman,kmcomapi,insthelper,tsysinfo,tsysinfox64}/` | COM API + TSF text service plumbing | +| Desktop helpers | `windows/src/desktop/{kmbrowserhost,kmconfig,setup,insthelp}/` | UI / installer infrastructure | +| Developer installer | `developer/src/setup/` | WiX-driven Delphi setup wizard | +| Build tools | `common/windows/delphi/tools/{devtools,build_standards_data,sentrytool,certificates,test-klog,buildunidata}/` | Codegen used by the Delphi build cascade | + +**About the "C# migration"** — there is no Delphi → C# migration in flight. +The repo has 6 `.cs` files in a tiny test sandbox (`windows/src/support/NetInputBoxTest/`) +and nothing else. The actual modernization targets are **C++** (Keyman +Core, kmcmplib, Windows C++ engine pieces like `keyman32.dll`/`kmtip.dll`) +and **TypeScript** (the `kmc-*` compiler suite, KeymanWeb). + +### Working with code that's mid-migration + +Two practical rules for new contributors: + +* **Prefer the new stack for new functionality.** If you're adding a new + compiler feature, add it to `kmc-*` (TypeScript) or `kmcmplib` (C++), + not to Tike or kmconvert. +* **Resist the urge to refactor Delphi.** Touching `developer/src/Tike/` + or `windows/src/engine/keyman/` to "clean up" is often wasted effort + because the long-term plan is to retire the code. Fix the immediate bug, + flag the deeper issue in a comment + issue, move on. + +### Building Delphi components today + +If you must work in the Delphi tree, see the build doc that matches your +Delphi version: + +* [docs/build/windows.md](../docs/build/windows.md) — the canonical flow + (Delphi 10.3 / Pro) +* [docs/build/windows-d12.md](../docs/build/windows-d12.md) — the + workaround for Delphi 12 Community Edition contributors + +## Migration 2: LDML keyboards + +### What's changing + +Keyman's historical keyboard formats — `.kmn` (source) and `.kmx` +(compiled binary) — predate the Unicode CLDR project's LDML keyboard +standard. The team is adding first-class support for LDML XML keyboards +so that: + +* Keyboards authored for LDML work natively in Keyman everywhere +* Keyman authors can publish keyboards to the CLDR repository +* Tooling between Keyman and other CLDR-aware projects becomes shared + +### Where it lives + +* **Compiler**: `developer/src/kmc-ldml/` — TypeScript module that compiles + LDML XML directly to a `.kmx` (the existing binary format that Keyman + Core runs). +* **Core runtime**: `core/src/ldml/` and `core/include/ldml/` — the C++ + LDML processor inside Keyman Core. Same C API as `.kmx` keyboards; the + binary format `kmx+` ([docs/file-formats/kmx-plus-file-format.md](../docs/file-formats/kmx-plus-file-format.md)) + carries the LDML data through to runtime. +* **Tests**: `core/tests/unit/ldml/` (runtime) and + `developer/src/kmc-ldml/test/` (compiler). + +### Status — what works today vs what's still in flight + +LDML support is shipping in pieces. The +[March 2026 roadmap](https://blog.keyman.com/2026/03/keyman-roadmap-march-2026/) +lays out the next several versions: + +* **v19 (current)**: "web-core" — bring LDML keyboard support to mobile, + tablet, and embedded web. This is where most LDML work landed in + shipped releases. +* **v20**: an **LDML keyboard visual editor** (built as a VSCode plugin — + see [epic-ldml-editor scaffolding, issue #12798](https://github.com/keymanapp/keyman/issues/12798)), + plus the Delphi-to-C++ Windows-UI migration. The visual editor is the + long-term replacement for hand-editing LDML XML. +* **v21**: rebuild the on-screen keyboard rendering using web technologies + so the touch/OSK experience is consistent across platforms. +* **v22**: deprecate `.js` keyboards in favor of LDML everywhere. + +#### What works today (master) + +* `.kmn` → `.kmx`: fully working via `kmcmplib` +* LDML XML → `.kmx+`: working for the **desktop** keyboard layer. + Compile via `kmc build path/to/keyboard.xml`. +* Keyman Core runtime executes both `.kmx` and `.kmx+` keyboards. +* `core/src/ldml/` handles transforms, markers, deadkeys, syllable-based + scripts (with some known edge cases under + [active LDML-runtime work](https://github.com/keymanapp/keyman/issues?q=label%3Aepic-ldml+is%3Aopen)). + +#### What's NOT working yet (the touch gap, all layers) + +**The CLDR LDML keyboard spec itself is complete for touch** — layers +(hardware and touch), forms, flicks, longpress, multi-tap, transforms, +markers, all defined. The gap is entirely on **Keyman's read side**: an +LDML keyboard with full touch layer markup is a valid spec-compliant +document today, but Keyman doesn't yet consume that touch markup +end-to-end. + +A consequence for tool builders: an AI extension *can* generate +spec-correct LDML touch keyboards today, and those keyboards will +become natively runnable on mobile/web when Keyman finishes the read +side — no source rewriting needed at that point. Generating to the +LDML spec is a forward-compatible bet even though runtime delivery +today still requires a parallel `.kmn` + `.kvks` set. + +That said, fully consuming the LDML touch markup end-to-end on mobile +and the browser through the kmx+ pipeline requires **four** layers of +implementation, not just the compiler: + +| Layer | State | Where to look | +|---|---|---| +| kmc-ldml writing LAYR data | **Partial** | TODOs in `developer/src/kmc-ldml/src/compiler/` (see table below) | +| kmx+ binary carrying LAYR | Done | Format defined; runtime parser at `core/src/kmx/kmx_plus.cpp:659+` | +| Keyman Core C API exposing layout to hosts | **Missing** | No `km_core_keyboard_get_layer` / `get_form` in `core/include/` | +| KeymanWeb / Mobile OSK consuming kmx+ layout | **Missing** | KeymanWeb OSK still drives off the legacy `.keyman-touch-layout` JSON spec; mobile engines the same | + +The interim today: kmc-ldml's +`visual-keyboard-compiler.ts` emits a legacy `.kvk` from the LDML source +so a touch-LDML keyboard *can* render through the existing OSK path. But +that's a stopgap — it's not "LDML XML drives the touch experience +natively through kmx+ on mobile and web", which is what the design aims +for and what the roadmap's v21 OSK rebuild promises to deliver +cross-platform. + +#### Compiler-side gaps (kmc-ldml) + +The specific TODOs in current master, all in +`developer/src/kmc-ldml/src/compiler/`: + +| File | TODO | Effect | +|---|---|---| +| `metadata-compiler.ts:52` | `dpString: 'desktop'` hardcoded | `&TARGETS` store always says `'desktop'` even if the source declares touch layers | +| `keys.ts:240` | "do nothing if only touch layers" | A touch-only LDML keyboard produces no keymap | +| `keys.ts:148` | "TODO-LDML: } else { touch?" | Touch layer validation skipped | +| `layr.ts:60` | "does not validate touch layers yet" | Layer count gate doesn't enforce touch | +| `visual-keyboard-compiler.ts:7` | "This is an interim solution until Keyman Core supports interrogation of the KMX+ data for OSK" | Today's stopgap is to emit a legacy `.kvk` from the LDML source rather than carry touch through in kmx+ | + +Tracking issue: **[#7238 — support touch layouts in kmc-ldml](https://github.com/keymanapp/keyman/issues/7238)** +(labelled `epic-ldml`, milestone A19S1). + +For the broader LDML editor / authoring story, see the +[`epic-ldml-editor` label](https://github.com/keymanapp/keyman/issues?q=label%3Aepic-ldml-editor) +and the umbrella scaffolding issue [#12798](https://github.com/keymanapp/keyman/issues/12798). + +#### Cheapest contributions if you want to accelerate + +In order of leverage (smallest to largest): + +1. **`metadata-compiler.ts:52` one-liner** — emit `'desktop touch'` + when the source declares touch layers instead of the current + hardcoded `'desktop'`. Doesn't fix runtime, but lets correctly-built + LDML keyboards declare their actual platform support. Probably a + 1-line PR + tests. +2. **Finish `keys.ts` touch handling** — the `keys.ts:148, 240` TODOs. + A few hundred lines; lands "touch-aware LDML compilation" without + needing runtime changes. +3. **Finish `layr.ts:60` validation** — modest PR. + +Steps 1–3 close the compiler-side gap. The runtime gap (Core C API + +KeymanWeb/Mobile OSK refactors) is the v21-scale effort; not a small +PR. If a contributor wanted to make LDML touch keyboards *compile +correctly today* (with the legacy `.kvk` runtime stopgap still doing +the actual touch rendering), the three items above are the path. + +### What this means for you as an intern + +* **For desktop-only keyboards**: emit LDML XML. No consistency + contract; works today end-to-end. Bias new test keyboards toward + LDML. +* **For keyboards that need to run on touch today** (mobile, web): + emit LDML XML as the canonical source AND a derived `.kmn` + `.kvks` + + `.keyman-touch-layout` bundle. Treat the LDML as authoritative; the + bundle is transitional until Keyman closes the read gap. See + [external-tooling.md § Recommended generation strategy](external-tooling.md#recommended-generation-strategy). +* The CLDR LDML spec is complete for touch — your generated LDML is + forward-compatible. When Keyman finishes the read side, you delete + the transitional bundle and the same LDML just works. +* If you encounter an LDML-related bug at runtime, look in `core/src/ldml/` + (the C++ runtime); at compile time, look in `developer/src/kmc-ldml/`. +* `kmx+` files are still `.kmx` files on disk — same Keyman Core entry + point. The "+" refers to additional LDML-specific binary sections. + +## Migration 3: KeymanWeb modernization + +### What changed + +Original KeymanWeb was a large standalone JavaScript engine — a parallel +implementation of the Keyman keyboard runtime in JavaScript, separate +from the desktop/mobile engines. Two engines meant two divergent feature +sets, two bug surfaces, two test matrices. + +The modernization replaces the JS-side keyboard engine with **Keyman Core +compiled to WebAssembly**. The TypeScript code in `web/` is now a host +that integrates with browser inputs (DOM events, OSK rendering, language +switcher) and delegates *all* keyboard logic to Keyman Core via the WASM +binding. + +```mermaid +graph LR + subgraph WEB[Modern KeymanWeb / web/] + host[TypeScript host
DOM, OSK, UI] + wasm[Keyman Core
compiled to WASM] + end + user[Browser keyboard event] + page[Page text input] + + user --> host + host --> wasm + wasm --> host + host --> page + + style WEB fill:#83a,color:#fff +``` + +### Where it lives + +* **Host**: `web/src/` — TypeScript organized into: + * `web/src/engine/` — the WASM-binding glue and runtime helpers + * `web/src/app/` — browser entrypoints (KeymanWeb embedded library, + keyman.com app integration) + * `web/src/test/` — Karma + Mocha test suites +* **WASM core**: built from `core/` via `core/build.sh build:wasm` and + consumed by `web/`'s build at link time. Lives in `web/build/engine/` + after build. +* **Build**: `web/build.sh build` is the single entry point. Internally + it manages the emsdk toolchain, node packages, and the wasm dependency. + +### Status + +The Core-via-WASM model is shipping in 17+ release lines. Active work +focuses on feature parity edge cases (touch-keyboard interactions, +predictive text, IME interaction) and removing remaining legacy paths +inherited from the original JS engine. Search the issue tracker for +`web/` and `kmw` for current work. + +### What this means for you + +* When you debug a KeymanWeb keyboard bug, ask first: is the bug in the + keyboard logic (Keyman Core / WASM) or in the host (DOM event handling, + rendering)? The split is more useful than the old "look anywhere in the + KeymanWeb monolith." +* If a behavior differs between desktop Keyman and KeymanWeb, the bug is + *probably* in the web host (`web/src/`), not in Keyman Core — Core is + shared. + +## Long-running migration branches + +The three migrations above don't always land directly on `master`. Some +larger work happens on long-lived branches that periodically merge back. +Conventional name prefixes you'll see in `git branch -r`: + +* `feat/core/...` — runtime features in `core/`, often coordinated with + LDML or platform-engine work +* `feat/developer/...` — compiler / Developer-side features (kmc-* etc.) +* `feat/web/...` — KeymanWeb modernization work +* `chore/windows/...` — incremental cleanups in the Windows / Delphi tree +* `auto/...` — automated branches (version bumps, etc.); not for human + contribution + +Before starting a non-trivial feature, search the open issues by `epic-*` +label first, then `git branch -r | grep -i ` against +`upstream/` to see if a long-running branch already covers it. + +## Roadmap and decision context + +The Keyman team publishes high-level direction on the +[Keyman roadmap blog](https://blog.keyman.com/category/roadmap/). The +current public roadmap is the +[**March 2026 update**](https://blog.keyman.com/2026/03/keyman-roadmap-march-2026/) — +read it before making bets on what to invest effort in. Headlines: + +| Version | Theme | Highlights | +|---|---|---| +| **19** (current) | LDML / web-core | CLDR keyboards on mobile, tablet, embedded web | +| **20** | Compiler & UI modernization | **Delphi → C++ for Windows UI**, LDML visual editor (VSCode plugin, [#12798](https://github.com/keymanapp/keyman/issues/12798)), compiler rewrite | +| **21** | Touch/OSK rebuild | On-screen keyboard rebuilt with web tech for cross-platform consistency | +| **22** | Distribution & deprecation | KeymanWeb on npm; deprecate `.js` keyboards in favor of LDML; CLDR predictive text wordlists | + +All version targets are funding-dependent (the roadmap is explicit about +this). + +**Umbrella tracking issues to watch:** + +* [#4599 — Delphi removal](https://github.com/keymanapp/keyman/issues/4599) + (filed 2022, umbrella) +* [#7238 — LDML touch layout compilation](https://github.com/keymanapp/keyman/issues/7238) + (epic-ldml, see § Migration 2 above) +* [#12798 — VSCode plugin scaffolding](https://github.com/keymanapp/keyman/issues/12798) + (epic-ldml-editor, milestone 20.0) +* [`epic-ldml` label](https://github.com/keymanapp/keyman/issues?q=label%3Aepic-ldml) + — open work on the LDML keyboard format support +* [`epic-ldml-editor` label](https://github.com/keymanapp/keyman/issues?q=label%3Aepic-ldml-editor) + — open work on LDML authoring tooling +* [`epic-keyman-developer` label](https://github.com/keymanapp/keyman/issues?q=label%3Aepic-keyman-developer) + — broader Developer-side work + +## TL;DR for new contributors + +* **New features**: write them in the new stack (C++ for runtime, TS for + tooling). Don't add to Delphi unless you specifically have to. +* **Bug fixes**: fix them where the bug is. If it's in Delphi-bound code, + patch Delphi. Don't refactor surrounding code. +* **LDML**: it's the future format. Bias new test keyboards toward LDML + XML, not `.kmn`. +* **KeymanWeb**: the engine *is* Keyman Core. The host is TypeScript. + Debug accordingly. +* **When in doubt**: check the roadmap blog, search issues, ask in the + [SIL/Keyman forum](https://community.software.sil.org/c/keyman). diff --git a/dev-docs/repository-map.md b/dev-docs/repository-map.md new file mode 100644 index 00000000000..09379c1a888 --- /dev/null +++ b/dev-docs/repository-map.md @@ -0,0 +1,344 @@ +# Repository Map + +This is a tour of the Keyman monorepo. By the end of it you should know +where to find each major component, what tech stack it uses, which +platforms it targets, and how the components connect to each other. + +If you only read one section, read [Top-level layout](#top-level-layout) and +[The four big subsystems](#the-four-big-subsystems). + +## What Keyman is, in three sentences + +Keyman is an open-source cross-platform input system that lets people type in +any of the world's languages. It runs on Windows, macOS, Linux, iOS, Android, +and in web browsers — same keyboard files everywhere. The repo contains the +runtime engines for each platform, the developer tooling for authoring +keyboards, the Keyman Developer IDE, the KeymanWeb JavaScript engine, and a +shared C++ keyboard-processing core. + +## Top-level layout + +```mermaid +graph TD + root[keyman/] + root --> core[core/
Keyman Core
C++ shared runtime] + root --> common[common/
Shared code & tools] + root --> developer[developer/
Keyman Developer
compiler + IDE] + root --> web[web/
KeymanWeb
browser engine] + root --> windows[windows/
Windows engine + desktop] + root --> mac[mac/
macOS] + root --> linux[linux/
Linux] + root --> ios[ios/
iOS] + root --> android[android/
Android] + root --> oem[oem/
OEM builds
e.g. FirstVoices] + root --> docs[docs/
Build & format docs] + root --> resources[resources/
Build scripts & data] + root --> devdocs[dev-docs/
You are here] + + style root fill:#444,color:#fff + style devdocs fill:#28a,color:#fff + style core fill:#284,color:#fff +``` + +## The four big subsystems + +```mermaid +graph LR + subgraph SC[Shared C++] + kmcore[core/
Keyman Core] + end + subgraph DV[Developer tooling] + kmcmplib[kmcmplib
.kmn compiler library] + kmc[kmc & kmc-*
CLI compiler suite] + tike[Tike
Developer IDE] + end + subgraph RT[Runtime engines] + eng_win[Windows Engine
Delphi + C++] + eng_mac[macOS Engine
Swift + ObjC] + eng_lin[Linux Engine
C++] + eng_ios[iOS Engine
Swift] + eng_and[Android Engine
Java/Kotlin + C++] + end + subgraph WB[KeymanWeb] + kmweb[KeymanWeb
TypeScript + WASM] + end + + kmcore -.shared lib.-> eng_win + kmcore -.shared lib.-> eng_mac + kmcore -.shared lib.-> eng_lin + kmcore -.shared lib.-> eng_ios + kmcore -.shared lib.-> eng_and + kmcore -.wasm.-> kmweb + + kmcmplib --> kmc + kmc --> tike + kmcmplib -.compiles .kmn.-> tike + + style SC fill:#284,color:#fff + style DV fill:#46c,color:#fff + style RT fill:#a52,color:#fff + style WB fill:#83a,color:#fff +``` + +* **Shared C++** — `core/` is a single shared keyboard-processing engine + used by *every* runtime. When you press a key, the platform-specific + engine translates that into Keyman Core API calls; Core runs the keyboard + logic and returns the resulting text actions; the engine then injects + text into the focused application using platform APIs. + +* **Developer tooling** — under `developer/src/`. The historical Delphi + IDE (Tike) and the modern TypeScript CLI compiler family (`kmc` and the + ten `kmc-*` modules) live side by side. The `.kmn` keyboard compiler + itself is C++ (`kmcmplib`). + +* **Runtime engines** — one per platform under `windows/`, `mac/`, + `linux/`, `ios/`, `android/`. Each is a thin shell over Keyman Core that + handles the platform's keyboard hooks, IME registration, OS integration, + and configuration UI. + +* **KeymanWeb** — under `web/`. The same Keyman Core code compiled to + WebAssembly via Emscripten, plus a TypeScript host that integrates with + HTML inputs and embeds in Keyman web apps and `keyman.com`. + +## Tech stack per directory + +| Directory | Primary language(s) | Build system | Platforms | +|---|---|---|---| +| `core/` | C++ | Meson + Ninja | Cross (Win, Mac, Linux, WASM) | +| `common/` | Mixed (C++, TS, Delphi, data) | Various | Shared | +| `common/web/` | TypeScript | npm + esbuild | Cross | +| `common/windows/delphi/` | Delphi | build.sh / Delphi IDE | Windows only | +| `developer/src/kmcmplib/` | C++ | Meson | Cross | +| `developer/src/kmc*/` | TypeScript | npm + esbuild | Cross | +| `developer/src/Tike/` | Delphi (VCL) | build.sh + Delphi IDE | Windows only | +| `developer/src/kmconvert/`, `setup/` | Delphi | build.sh + Delphi IDE | Windows only | +| `web/` | TypeScript + WASM | build.sh + npm | Browsers | +| `windows/src/engine/` | Delphi + C++ | build.sh + Delphi/VS | Windows | +| `windows/src/desktop/` | Delphi | build.sh + Delphi IDE | Windows | +| `mac/` | Swift, Objective-C | Xcode | macOS | +| `linux/` | C++, Python | meson, debhelper | Linux | +| `ios/` | Swift | Xcode | iOS | +| `android/` | Java, Kotlin, C++ | Gradle | Android | +| `oem/firstvoices/` | Platform-specific | Per-platform | Cross | +| `resources/` | Bash, data files | Plain | Build infra | + +## How a keystroke flows (runtime) + +```mermaid +sequenceDiagram + participant User + participant OS as OS Input Layer + participant Engine as Platform Engine
(e.g. windows/src/engine) + participant Core as Keyman Core
(core/, C++) + participant App as Target App + + User->>OS: Press key + OS->>Engine: Keyboard hook callback + Engine->>Core: km_core_state_process_event() + Core->>Core: Run active keyboard rules
(.kmx or LDML) + Core-->>Engine: List of TActionItem
(emit char, backspace, ...) + Engine->>OS: SendInput / TSF actions + OS->>App: Text typed +``` + +The platform engine never implements keyboard logic itself; it always +delegates to Keyman Core. That's why a single keyboard file works across +Windows, macOS, Linux, iOS, Android, and the web. + +## How a keyboard is built (developer flow) + +```mermaid +flowchart LR + src[Source
.kmn or LDML XML] + kmc[kmc
TypeScript CLI] + kmcmplib[kmcmplib
C++ compiler] + kmx[.kmx
Keyman binary] + js[.js
KeymanWeb bundle] + kmp[.kmp
Package zip] + + src --> kmc + kmc -.delegates .kmn.-> kmcmplib + kmcmplib --> kmx + kmc --> js + kmc --> kmp + kmx --> kmp + js --> kmp +``` + +The TypeScript `kmc` CLI is the modern entry point. For `.kmn` source it +delegates to the C++ `kmcmplib`. For LDML XML source, `kmc-ldml` handles +compilation directly in TypeScript. The single `.kmp` package contains +the platform binaries plus metadata (`keyboard_info`, fonts, docs). + +## Subsystem deep-dives + +### `core/` — Keyman Core + +The cross-platform shared C++ runtime. Implements the keyboard-rule +processor in a host-agnostic way. Exposed through a stable C API +(`km_core_*` functions) so every platform engine and KeymanWeb can call +the same logic. + +* **Build**: `core/build.sh build:x64` / `:x86` / `:wasm` / per-platform + targets. Output goes to `core/build///src/libkmcore-*.{dll,so,dylib}`. +* **Tests**: `core/tests/` with `kmnkbd`, `ldml`, `kmx`, `kmx_plus` + unit-test directories. +* **API**: see `core/docs/api.md` and the public headers in `core/include/`. + +### `developer/src/` — Keyman Developer + +The keyboard authoring side. Two halves: + +* **Modern compiler stack** (TypeScript): `kmc/` is the unified CLI; + `kmc-kmn/` delegates `.kmn` to `kmcmplib`; `kmc-ldml/` handles LDML + keyboards; `kmc-package/`, `kmc-keyboard-info/`, `kmc-analyze/`, + `kmc-copy/`, `kmc-generate/`, `kmc-model/`, `kmc-model-info/` cover the + other authoring tasks. +* **Legacy Delphi IDE**: `Tike/Tike.dproj` is the visual authoring tool, + built with VCL. `setup/` is the Developer installer. `kmconvert/` is a + legacy conversion tool — marked Legacy and being phased out in favor of + the kmc-* family. See [migration-guide.md](migration-guide.md). + +### `web/` — KeymanWeb + +The browser-side keyboard engine. Same Keyman Core, compiled to +WebAssembly via Emscripten and wrapped in a TypeScript host. Used by +keyman.com web apps, by sites that embed the keyman.js script, and by +Keyman Developer's preview mode. + +* **Build**: `web/build.sh build` (needs node + emsdk; sets up both). +* **Output**: `web/build/app/`, `web/build/engine/`, `web/build/publish/` + for the release bundles. +* **Tests**: `web/build/test/` after `build`, runnable via Karma. + +### `windows/src/engine/` — Windows runtime engine + +A mix of Delphi (the COM API, configuration helpers, TSF text service +plumbing) and C++ (the actual input hook, `keyman32.dll`, the TSF text +processor `kmtip.dll`). For a tour of which dproj does what, see +`windows/src/engine/engine.groupproj` — the five Delphi projects build +together as a group; the C++ pieces (`keyman32`, `kmtip`, `keymanhp`, +`kmrefresh`, `mcompile`, `testhost`) build standalone via msbuild. + +### `windows/src/desktop/` — Keyman for Windows (the user-facing app) + +Delphi VCL apps: `kmshell` is the configuration tray app; `kmconfig` is +the legacy settings UI; `kmbrowserhost` embeds CEF for the +in-app browser; `setup` is the installer; `insthelp` is the install helper. + +### `mac/`, `linux/`, `ios/`, `android/` + +Each runs Keyman Core under the hood, with a platform-native shell: + +* **macOS** (`mac/Keyman4MacIM/`): Objective-C / Swift Input Method bundle. +* **Linux** (`linux/ibus-keyman/`): IBus engine module written in C++, + packaged via the `keyman-config` Python helpers. +* **iOS** (`ios/keyman/`): Swift app + KeymanEngine framework. +* **Android** (`android/KMAPro/`): Kotlin/Java app + KMEA library; + Keyman Core compiled for NDK is loaded via JNI. + +### `common/` — Shared anything-and-everything + +Anything used by more than one platform lives here. Highlights: + +* `common/web/` — shared TypeScript helpers (`keyman-version`, `langtags`, + `types`, `utils`). +* `common/windows/delphi/` — Delphi units shared by every Windows-Delphi + project (engine, desktop, developer). Plus bundled third-party Delphi + libraries: JCL, JVCL, mbcolor, dcpcrypt, CEF4Delphi. +* `common/windows/cpp/` — Windows-specific C++ headers. +* `common/include/`, `common/test/` — cross-platform shared includes and + test fixtures. +* `common/tools/` — small cross-platform helpers like `hextobin`. + +### `resources/`, `build/`, `docs/` + +* `resources/build/` — the shared bash-builder framework (`builder-full.inc.sh` + etc.) that every project's `build.sh` sources. Read this once when you + start writing a new `build.sh`. +* `resources/standards-data/` — language tag and ISO data used to generate + the BCP-47 Pascal modules. +* `docs/` — build instructions, file-format references, history. + +## Build orchestration + +Every top-level directory has a `build.sh` that obeys the same conventions. +A typical invocation is: + +```bash +.//build.sh [:] [options...] +``` + +Where: +* `` is `clean`, `configure`, `build`, `test`, `publish`, `install`, + `api`, or `edit`. +* `` is component-specific (e.g. `./web/build.sh build:engine`). + +Read `/build.sh --help` to see available targets. Most scripts inherit +the cross-platform builder framework from `resources/build/builder-*.inc.sh`, +so they all have the same look-and-feel. + +```mermaid +graph TD + root_build[./build.sh
top-level orchestrator] + root_build --> core_b[core/build.sh] + root_build --> dev_b[developer/build.sh] + root_build --> web_b[web/build.sh] + root_build --> win_b[windows/src/build.sh] + root_build --> linux_b[linux/build.sh] + root_build --> mac_b[mac/build.sh] + root_build --> ios_b[ios/build.sh] + root_build --> and_b[android/build.sh] + + dev_b --> kmc_b[developer/src/kmc/build.sh] + dev_b --> kmcmplib_b[developer/src/kmcmplib/build.sh] + dev_b --> tike_b[developer/src/Tike/build.sh] + dev_b --> setup_b[developer/src/setup/build.sh] + + win_b --> eng_b[windows/src/engine/build.sh] + win_b --> desk_b[windows/src/desktop/build.sh] + eng_b --> k32_b[engine/keyman32/build.sh] + eng_b --> kmtip_b[engine/kmtip/build.sh] + eng_b --> ihelp_b[engine/insthelper/build.sh] + + style root_build fill:#444,color:#fff +``` + +## Cross-platform matrix + +Which directories actually build on which OS: + +| Directory | Win 11 | macOS | Linux | +|---|---|---|---| +| `core/` | ✓ (x86, x64, ARM64, WASM) | ✓ (universal) | ✓ | +| `common/` | ✓ | ✓ | ✓ | +| `developer/src/kmcmplib/` | ✓ | ✓ | ✓ | +| `developer/src/kmc*/` | ✓ | ✓ | ✓ | +| `developer/src/Tike/`, `setup/`, `kmconvert/` | Delphi only | — | — | +| `web/` | ✓ | ✓ | ✓ | +| `windows/` | ✓ | — | — | +| `linux/` | — | — | ✓ | +| `mac/`, `ios/` | — | ✓ | — | +| `android/` | ✓ (Java/Kotlin/NDK) | ✓ | ✓ | + +When the `docs/build/windows.md` doc says "the following projects cannot be +built on Windows: Keyman for Linux / Keyman for macOS / Keyman for iOS" — +this matrix is what it's reflecting. + +## Where to start as an intern + +Depending on what you're working on: + +* **Keyboard runtime work**: start with `core/` and one of the platform + engines (most likely `windows/src/engine/` or `linux/ibus-keyman/`). +* **Compiler / authoring work**: start with `developer/src/kmc/` and + follow the dependency from there into the `kmc-*` modules. +* **Web work**: start with `web/` and the TypeScript app under + `web/src/app/`. +* **Mobile**: `android/KMAPro/` or `ios/keyman/`. +* **Delphi UI work (Tike, kmshell)**: read [migration-guide.md](migration-guide.md) + *first* — the long-term direction matters before you invest time in + Delphi changes. + +Whatever you touch first, read its `build.sh --help` and the `README.md` +in that directory (most components have one). From 1c74ddf26454663e776a83dba01aa6bc665e7a6d Mon Sep 17 00:00:00 2001 From: Matthew Lee Date: Thu, 28 May 2026 22:26:47 -0500 Subject: [PATCH 2/4] feat(windows): Delphi 11/12 source compat + Delphi 12 CE dev workflow Delphi Community Edition (10.4 and 12) does not allow command-line compilation -- only the Delphi IDE can drive dcc32. This blocks the repo's build.sh chain for any Keyman component that requires Delphi, which means contributors without a paid Delphi Professional license cannot build Engine, Desktop, or Developer from CLI. Community Edition is currently the only free way to do Keyman Windows development. This branch captures a working Delphi-IDE-based workflow for Delphi 12 CE, plus source patches that bring Delphi 11/12 compatibility to the existing tree without breaking Delphi 10.3. The changes here target the still-Delphi UI/engine layer (TIKE, kmshell, the engine COM/TSF plumbing). The compiler stack (kmcmplib, kmc-*, Keyman Core) is already Delphi-free and is not touched by any of these patches. Context: keymanapp/keyman#4599 (filed 2022) tracks the ongoing work to remove Delphi dependencies from Keyman. Until that migration completes, this branch provides a bridge for contributors building locally on the free Delphi tier. CI safety: with KEYMAN_DELPHI_VERSION unset, all build-script defaults match the current Delphi 10.3 CI behavior. All source patches are additive -- they introduce VER350/VER360 (Delphi 11/12) IFDEF cases without altering the VER330 (10.3) paths, or add explicit casts that the older compiler would have applied implicitly. Build scripts: * resources/build/win/{configure,delphi}_environment.inc.sh: make DELPHI_VERSION configurable via the KEYMAN_DELPHI_VERSION env var, defaulting to 20.0 (existing 10.3 behavior preserved). * windows/src/engine/engine.groupproj: drop the inst\insthelper reference. The path doesn't resolve (insthelper.dproj lives at windows/src/engine/insthelper, not .../inst/insthelper); IDE "Build All" was failing on it. build.sh-based builds were unaffected. * .gitignore: add patterns for per-arch version*.res, meson wraplock files, and dev-workflow local artifacts. Docs: * docs/build/windows.md: one-line pointer to windows-d12.md from the "Delphi 10.3 Community no longer available" warning, so future CE users find the workaround instead of dead-ending. * docs/build/windows-d12.md: full IDE-workflow guide -- prerequisites, library Search Path registry setup, source patches, build order (including chicken-and-egg patterns), install-and-overlay run workflow, debugging, troubleshooting catalog, and an upstreaming guide for which patches are safe to PR back. Scripts: * scripts/preflight-resources.ps1: generates manifest.xml and .res files per Delphi project, equivalent to build.sh's rc-step. * scripts/run-codegen.ps1: runs devtools and build_standards_data codegen subcommands plus the tsysinfox64 -> .bin -> tsysinfo_x64.res chicken-and-egg. * scripts/install-build-env.ps1: one-time toolchain installer (Chocolatey, VS 2022, nvm/node, Emscripten, release tools). * overlay-dev-builds.ps1: copies dev binaries onto an installed Keyman 19, equivalent to what `build.sh install` would do. All four scripts auto-detect the repo root via $PSScriptRoot and honor $env:KEYMAN_ROOT as an override. Delphi 11/12 source compat (Keyman-owned, additive on 10.3): * common/windows/delphi/tools/devtools/SourceRootPath.pas: add VER350 ('22.0') and VER360 ('23.0') cases to DelphiMajorVersion. * common/windows/delphi/web/Keyman.System.HttpServer.Base.pas: extend the VER330 tripwire to accept VER340/VER350/VER360. * common/windows/delphi/components/FixedTrackbar.pas: extend the nested IFNDEF tripwire similarly. * common/windows/delphi/general/CleartypeDrawCharacter.pas: use integer-return comparison for EnumFontFamiliesEx on VER340/VER350/VER360 (was VER340 only; RTL return type changed in 10.4+). * common/windows/delphi/general/JsonUtil.pas: pass `[]` options arg to TJSONAncestor.ToChars on VER350/VER360 (Delphi 11+ added a required Options parameter). Vendored third-party patches (local-only; refresh from upstream when those projects ship Delphi 12-compatible releases). Each hunk is annotated with a "Keyman local patch" comment. * developer/src/ext/jedi/jcl/jcl/source/common/JclSynch.pas: add explicit BOOL() casts to Win32-API Boolean arguments (Delphi 12 tightened implicit Boolean->BOOL conversion at call sites); switch JclWin32.CreateMutex to Winapi.Windows.CreateMutex to match the file's existing pattern and avoid 12's unit-resolution ambiguity. * developer/src/ext/jedi/jvcl/jvcl/run/JvComponent.pas: skip the OldCreateOrder property check on VER350/VER360 (Embarcadero removed the property in Delphi 11; modern behavior is equivalent to OldCreateOrder=True). * developer/src/ext/mbcolor/mxs.inc: add VER350/VER360 cases that set DELPHI_5_UP through DELPHI_10_UP. Without this, HTMLColors.pas silently drops `Variants` from its uses clause and breaks with "Undeclared identifier: 'null'". Co-Authored-By: Claude Opus 4.7 --- .gitignore | 14 + .../delphi/components/FixedTrackbar.pas | 4 + .../delphi/general/CleartypeDrawCharacter.pas | 4 +- common/windows/delphi/general/JsonUtil.pas | 4 + .../delphi/tools/devtools/SourceRootPath.pas | 8 + .../web/Keyman.System.HttpServer.Base.pas | 6 + .../jedi/jcl/jcl/source/common/JclSynch.pas | 18 +- .../ext/jedi/jvcl/jvcl/run/JvComponent.pas | 8 + developer/src/ext/mbcolor/mxs.inc | 22 + docs/build/windows-d12.md | 992 ++++++++++++++++++ docs/build/windows.md | 5 + overlay-dev-builds.ps1 | 102 ++ .../build/win/configure_environment.inc.sh | 6 +- resources/build/win/delphi_environment.inc.sh | 5 +- scripts/install-build-env.ps1 | 63 ++ scripts/preflight-resources.ps1 | 334 ++++++ scripts/run-codegen.ps1 | 308 ++++++ windows/src/engine/engine.groupproj | 18 +- 18 files changed, 1894 insertions(+), 27 deletions(-) create mode 100644 docs/build/windows-d12.md create mode 100644 overlay-dev-builds.ps1 create mode 100644 scripts/install-build-env.ps1 create mode 100644 scripts/preflight-resources.ps1 create mode 100644 scripts/run-codegen.ps1 diff --git a/.gitignore b/.gitignore index 7b872195880..d66507ce9c5 100644 --- a/.gitignore +++ b/.gitignore @@ -19,6 +19,7 @@ /windows/src/**/*.identcache /windows/src/**/*.vcxproj.user /windows/src/**/version.res +/windows/src/**/version*.res /windows/src/**/*.pch /windows/src/**/*.wixobj /windows/src/**/*.sbr @@ -186,3 +187,16 @@ lcov.info # flag file for build script .configured + +# Local artifacts from the Delphi 12 CE dev workflow (see docs/build/windows-d12.md) +/delphi-library-paths.backup*.reg +/local-delphi-12-patches.patch +/configure.log +/core-build.log +/cpp-engine-build.log +/install-build-env.log +/kmcmplib-build.log +/web-build.log + +# Meson wrap dependency lock files +**/subprojects/.wraplock diff --git a/common/windows/delphi/components/FixedTrackbar.pas b/common/windows/delphi/components/FixedTrackbar.pas index 50cb444a55f..8d3c525d842 100644 --- a/common/windows/delphi/components/FixedTrackbar.pas +++ b/common/windows/delphi/components/FixedTrackbar.pas @@ -70,10 +70,14 @@ procedure TTntFixedDrawGrid.WMEraseBkgnd(var Message: TMessage); {$MESSAGE WARN 'Not yet checked against Delphi 10.4'} {$IFNDEF VER330} {$IFNDEF VER320} +{$IFNDEF VER350} +{$IFNDEF VER360} {$MESSAGE ERROR 'Check that this fix is still applicable for a new version of Delphi. Checked against Delphi 10.2, 10.3' } {$ENDIF} {$ENDIF} {$ENDIF} +{$ENDIF} +{$ENDIF} procedure TTntFixedDrawGrid.MouseDown(Button: TMouseButton; Shift: TShiftState; X, Y: Integer); diff --git a/common/windows/delphi/general/CleartypeDrawCharacter.pas b/common/windows/delphi/general/CleartypeDrawCharacter.pas index 6d1a51976de..44562012ed2 100644 --- a/common/windows/delphi/general/CleartypeDrawCharacter.pas +++ b/common/windows/delphi/general/CleartypeDrawCharacter.pas @@ -591,11 +591,11 @@ function TestFont(FFontName: string): Boolean; StrPCopy(lf.lfFaceName, FFontName); //'Code2000'); hdc := GetDC(0); //FPlane0FontName := 'Code2000'; -{$IFDEF VER340} +{$IF Defined(VER340) or Defined(VER350) or Defined(VER360)} if EnumFontFamiliesEx(hdc, lf, @EnumFallbackFonts, 0, 0) <> 0 then {$ELSE} if EnumFontFamiliesEx(hdc, lf, @EnumFallbackFonts, 0, 0) then -{$ENDIF} +{$IFEND} begin FPlane0FontName := FFontName; Result := True; diff --git a/common/windows/delphi/general/JsonUtil.pas b/common/windows/delphi/general/JsonUtil.pas index ae2b9b0e9c6..16a90bc729d 100644 --- a/common/windows/delphi/general/JsonUtil.pas +++ b/common/windows/delphi/general/JsonUtil.pas @@ -52,7 +52,11 @@ function JSONToString(obj: TJSONAncestor; ReplaceSlashes: Boolean = False): stri begin builder := TStringBuilder.Create; try +{$IF Defined(VER350) or Defined(VER360)} + obj.ToChars(builder, []); +{$ELSE} obj.ToChars(builder); +{$IFEND} Result := builder.ToString; finally builder.Free; diff --git a/common/windows/delphi/tools/devtools/SourceRootPath.pas b/common/windows/delphi/tools/devtools/SourceRootPath.pas index 360477641e7..c8372c3c6e2 100644 --- a/common/windows/delphi/tools/devtools/SourceRootPath.pas +++ b/common/windows/delphi/tools/devtools/SourceRootPath.pas @@ -14,11 +14,19 @@ interface {$IFDEF VER340} const DelphiMajorVersion = '21.0'; {$ELSE} +{$IFDEF VER350} +const DelphiMajorVersion = '22.0'; +{$ELSE} +{$IFDEF VER360} +const DelphiMajorVersion = '23.0'; +{$ELSE} ERROR: must define Delphi version {$ENDIF} {$ENDIF} {$ENDIF} {$ENDIF} +{$ENDIF} +{$ENDIF} const DelphiBasePath = 'C:\Program Files (x86)\Embarcadero\Studio\' + DelphiMajorVersion + '\'; diff --git a/common/windows/delphi/web/Keyman.System.HttpServer.Base.pas b/common/windows/delphi/web/Keyman.System.HttpServer.Base.pas index d1daf087c39..8df4e732056 100644 --- a/common/windows/delphi/web/Keyman.System.HttpServer.Base.pas +++ b/common/windows/delphi/web/Keyman.System.HttpServer.Base.pas @@ -46,7 +46,13 @@ function CrackUTF8ZeroExtendedString(CommandType: THTTPCommandType; const p: str // Indy's UTF8 handling of URLs is *completely* broken. // We may need to check this with updated versions of Delphi {$IFNDEF VER330} +{$IFNDEF VER340} +{$IFNDEF VER350} +{$IFNDEF VER360} ERROR! Check if this is still needed with Delphi update +{$ENDIF} +{$ENDIF} +{$ENDIF} {$ENDIF} SetLength(s, p.Length); diff --git a/developer/src/ext/jedi/jcl/jcl/source/common/JclSynch.pas b/developer/src/ext/jedi/jcl/jcl/source/common/JclSynch.pas index f73b1722b09..eb405224249 100644 --- a/developer/src/ext/jedi/jcl/jcl/source/common/JclSynch.pas +++ b/developer/src/ext/jedi/jcl/jcl/source/common/JclSynch.pas @@ -937,11 +937,15 @@ function TJclCriticalSectionEx.TryEnter: Boolean; //== { TJclEvent } =========================================================== +// Keyman local patch: Delphi 11/12 compat (vendored JCL). Explicit BOOL() casts +// added below to satisfy Delphi 12's stricter implicit-Boolean->BOOL conversion. +// Backwards-compatible with Delphi 10.3 (BOOL is an ordinal-preserving typecast). +// On JCL refresh from upstream: re-apply if upstream hasn't picked up the cast. constructor TJclEvent.Create(SecAttr: PSecurityAttributes; Manual, Signaled: Boolean; const Name: string); begin inherited Create; FName := Name; - FHandle := {$IFDEF HAS_UNITSCOPE}Winapi.{$ENDIF}Windows.CreateEvent(SecAttr, Manual, Signaled, PChar(FName)); + FHandle := {$IFDEF HAS_UNITSCOPE}Winapi.{$ENDIF}Windows.CreateEvent(SecAttr, BOOL(Manual), BOOL(Signaled), PChar(FName)); if FHandle = 0 then raise EJclEventError.CreateRes(@RsSynchCreateEvent); FExisted := GetLastError = ERROR_ALREADY_EXISTS; @@ -952,7 +956,7 @@ constructor TJclEvent.Open(Access: Cardinal; Inheritable: Boolean; begin FName := Name; FExisted := True; - FHandle := {$IFDEF HAS_UNITSCOPE}Winapi.{$ENDIF}Windows.OpenEvent(Access, Inheritable, PChar(Name)); + FHandle := {$IFDEF HAS_UNITSCOPE}Winapi.{$ENDIF}Windows.OpenEvent(Access, BOOL(Inheritable), PChar(Name)); if FHandle = 0 then raise EJclEventError.CreateRes(@RsSynchOpenEvent); end; @@ -980,7 +984,7 @@ constructor TJclWaitableTimer.Create(SecAttr: PSecurityAttributes; begin FName := Name; FResume := False; - FHandle := CreateWaitableTimer(SecAttr, Manual, PChar(Name)); + FHandle := CreateWaitableTimer(SecAttr, BOOL(Manual), PChar(Name)); if FHandle = 0 then raise EJclWaitableTimerError.CreateRes(@RsSynchCreateWaitableTimer); FExisted := GetLastError = ERROR_ALREADY_EXISTS; @@ -1000,7 +1004,7 @@ constructor TJclWaitableTimer.Open(Access: Cardinal; Inheritable: Boolean; FExisted := True; FName := Name; FResume := False; - FHandle := OpenWaitableTimer(Access, Inheritable, PChar(Name)); + FHandle := OpenWaitableTimer(Access, BOOL(Inheritable), PChar(Name)); if FHandle = 0 then raise EJclWaitableTimerError.CreateRes(@RsSynchOpenWaitableTimer); end; @@ -1048,7 +1052,7 @@ constructor TJclSemaphore.Open(Access: Cardinal; Inheritable: Boolean; begin FName := Name; FExisted := True; - FHandle := {$IFDEF HAS_UNITSCOPE}Winapi.{$ENDIF}Windows.OpenSemaphore(Access, Inheritable, PChar(Name)); + FHandle := {$IFDEF HAS_UNITSCOPE}Winapi.{$ENDIF}Windows.OpenSemaphore(Access, BOOL(Inheritable), PChar(Name)); if FHandle = 0 then raise EJclSemaphoreError.CreateRes(@RsSynchOpenSemaphore); end; @@ -1075,7 +1079,7 @@ constructor TJclMutex.Create(SecAttr: PSecurityAttributes; InitialOwner: Boolean begin inherited Create; FName := Name; - FHandle := JclWin32.CreateMutex(SecAttr, InitialOwner, PChar(Name)); + FHandle := {$IFDEF HAS_UNITSCOPE}Winapi.{$ENDIF}Windows.CreateMutex(SecAttr, BOOL(InitialOwner), PChar(Name)); if FHandle = 0 then raise EJclMutexError.CreateRes(@RsSynchCreateMutex); FExisted := GetLastError = ERROR_ALREADY_EXISTS; @@ -1086,7 +1090,7 @@ constructor TJclMutex.Open(Access: Cardinal; Inheritable: Boolean; const Name: s inherited Create; FName := Name; FExisted := True; - FHandle := {$IFDEF HAS_UNITSCOPE}Winapi.{$ENDIF}Windows.OpenMutex(Access, Inheritable, PChar(Name)); + FHandle := {$IFDEF HAS_UNITSCOPE}Winapi.{$ENDIF}Windows.OpenMutex(Access, BOOL(Inheritable), PChar(Name)); if FHandle = 0 then raise EJclMutexError.CreateRes(@RsSynchOpenMutex); end; diff --git a/developer/src/ext/jedi/jvcl/jvcl/run/JvComponent.pas b/developer/src/ext/jedi/jvcl/jvcl/run/JvComponent.pas index 1877ee48efe..fab8609e2c8 100644 --- a/developer/src/ext/jedi/jvcl/jvcl/run/JvComponent.pas +++ b/developer/src/ext/jedi/jvcl/jvcl/run/JvComponent.pas @@ -123,8 +123,16 @@ constructor TJvForm.Create(AOwner: TComponent); finally Exclude(FFormState, fsCreating); end; + // Keyman local patch: Delphi 11/12 compat (vendored JVCL). The + // OldCreateOrder property was removed in Delphi 11; the modern semantics + // are equivalent to OldCreateOrder=True, so always call DoCreate on + // VER350+. On JVCL refresh from upstream: re-apply if not yet upstreamed. +{$IF Defined(VER350) or Defined(VER360)} + DoCreate; +{$ELSE} if OldCreateOrder then DoCreate; +{$IFEND} end; finally GlobalNameSpace.EndWrite; diff --git a/developer/src/ext/mbcolor/mxs.inc b/developer/src/ext/mbcolor/mxs.inc index 41b82ad96e5..75ca0b34e73 100644 --- a/developer/src/ext/mbcolor/mxs.inc +++ b/developer/src/ext/mbcolor/mxs.inc @@ -7,6 +7,28 @@ {$define DELPHI_10_UP} {$endif} + // Keyman local patch: Delphi 11/12 compat (vendored mbcolor). Without + // these blocks DELPHI_*_UP flags are never defined on VER350/VER360, + // which causes HTMLColors.pas to drop "uses Variants" and fail to + // resolve the Null constant. On mbcolor refresh: re-apply if needed. + {$ifdef VER350} + {$define DELPHI_5_UP} + {$define DELPHI_6_UP} + {$define DELPHI_7_UP} + {$define DELPHI_8_UP} + {$define DELPHI_9_UP} + {$define DELPHI_10_UP} + {$endif} + + {$ifdef VER360} + {$define DELPHI_5_UP} + {$define DELPHI_6_UP} + {$define DELPHI_7_UP} + {$define DELPHI_8_UP} + {$define DELPHI_9_UP} + {$define DELPHI_10_UP} + {$endif} + {$ifdef VER330} {$define DELPHI_5_UP} {$define DELPHI_6_UP} diff --git a/docs/build/windows-d12.md b/docs/build/windows-d12.md new file mode 100644 index 00000000000..3836cbcd01a --- /dev/null +++ b/docs/build/windows-d12.md @@ -0,0 +1,992 @@ +# Build Keyman on Windows with Delphi 12 Community Edition + +> [!WARNING] +> This is a workaround guide for developers using **Delphi 12 Athens Community +> Edition (CE)**. Delphi CE 11+ deliberately blocks command-line `dcc32` +> compilation: +> +> ``` +> This version of the product does not support command line compiling +> ``` +> +> Because of that, the repo's `build.sh` chain cannot drive Delphi builds +> end-to-end on CE. Everything Delphi must be opened and built in the IDE by +> hand, and the codegen / resource-compilation steps that `build.sh` would +> normally perform must be run manually first. This guide documents that +> workflow plus the local source patches needed to compile against Delphi 12. +> +> If you have a Delphi 10.3 / 10.4 Professional or Enterprise license, follow +> the canonical [windows.md](windows.md) instead; none of the workarounds in +> this file are needed. + +## Scope + +This guide covers: + +1. Prerequisites (Windows + Delphi 12 CE specific notes) +2. One-time environment setup +3. Local-only source patches required for Delphi 12 +4. Build order and the implicit cross-project dependencies +5. Helper scripts that paper over what `build.sh` would normally do +6. Running and debugging the dev build alongside an installed Keyman 19 +7. Troubleshooting common error messages + +The canonical reference for the standard (non-CE) flow is +[windows.md](windows.md). Refer to it for repository layout, the +`KEYMAN_ROOT` / `KEYMAN_CEF4DELPHI_ROOT` / `EMSCRIPTEN_BASE` env vars, +Chocolatey package list, Visual Studio workloads, and Android dependencies. +This document only describes the deltas. + +> [!IMPORTANT] +> All patches and registry edits described here are **local-only** and must be +> reverted before opening a pull request. The repo's CI runs against +> Delphi 10.3 and will not accept any of these workarounds upstream. + +## 1. Prerequisites + +Beyond the [Base](windows.md#base-dependencies) and +[Web](windows.md#web-dependencies) dependencies in `windows.md`, Delphi 12 CE +needs the following: + +* **Delphi 12 Athens Community Edition** + * Download from + https://www.embarcadero.com/products/delphi/starter/free-download + * Installs to `C:\Program Files (x86)\Embarcadero\Studio\23.0\` -- note the + `23.0` version dir, not `20.0` (10.3) or `21.0` (10.4). + * During the install wizard, select **DUnit Unit Testing Frameworks** + (required by Keyman test projects). + * Launch the IDE once after install to complete registration. The first + launch populates the per-user `BDS\23.0` registry hive used in step 4. +* **Visual Studio 2022 Community** with the C++ native desktop workload, plus + Windows 10 SDK (10.0.19041.0) and Windows 11 SDK (10.0.26100). See + [Windows Platform Dependencies](windows.md#windows-platform-dependencies) + for the full winget command. +* **Keyman 19 (official release)** installed from + https://keyman.com/desktop. This is *not* optional for Delphi 12 CE + developers -- the dev kmshell.exe runtime-discovers its install path via + `TKeymanPaths.KeymanDesktopInstallPath()`, which is hardcoded to + `C:\Program Files (x86)\Keyman\Keyman Desktop\`. Without the install, + kmshell crashes at startup with `SKApplicationTitle has had a fatal + error...` before its main form appears. See section 7 (Running). +* **Keyman Developer (official release)** is recommended in parallel for TIKE + overlay debugging -- TIKE loads templates and language data from the + installed Developer support files. +* **7-Zip** (`choco install 7zip`) for extracting the CEF libcef payload. + +## 2. One-time setup + +### 2.1 Clone the repos + +Use the layout from [windows.md](windows.md#repository-paths). This guide +assumes the keyman checkout is at `C:\Projects\keyman\keyman\` (matches the local +machine). Use the same parent directory for the CEF and emsdk checkouts so +the `KEYMAN_CEF4DELPHI_ROOT` / `EMSCRIPTEN_BASE` paths below line up. + +```bash +# Git Bash -- use forward slashes via msys +mkdir -p /c/Projects/keyman +cd /c/Projects/keyman +git clone https://github.com/keymanapp/keyman +git clone https://github.com/keymanapp/CEF4Delphi_Binary +git clone https://github.com/emscripten-core/emsdk +``` + +### 2.2 Environment variables + +```cmd +:: Standard Windows cmd / PowerShell -- backslash paths +SETX KEYMAN_ROOT "C:\Projects\keyman\keyman" +SETX KEYMAN_CEF4DELPHI_ROOT "C:\Projects\keyman\CEF4Delphi_Binary" +SETX EMSCRIPTEN_BASE "C:\Projects\keyman\emsdk\upstream\emscripten" +``` + +`SETX` is persistent but does not affect the current shell -- open a fresh +shell to pick them up. Also add `\windows\lib` to `PATH` so +Delphi's design-time packages can resolve. + +### 2.3 Emscripten + +```bash +# Git Bash +cd /c/Projects/keyman/emsdk +emsdk install 3.1.58 +emsdk activate 3.1.58 +cd upstream/emscripten +npm install +``` + +### 2.4 node.js via nvm-windows + +```cmd +nvm install 20.16.0 +nvm use 20.16.0 +``` + +### 2.5 CEF4Delphi_Binary -- switch to the pinned tag and extract libcef + +`common/windows/cef-checkout.sh` normally does this, but if you're driving +Delphi from the IDE you'll want to do it once by hand. Some payload files +exceed GitHub's 100 MB limit and ship as `.zip` files inside the repo; they +must be extracted in place. + +```powershell +$cefVer = (Get-Content C:\Projects\keyman\keyman\common\windows\CEF_VERSION.md).Trim() +# e.g. 89.0.18 + +Push-Location $env:KEYMAN_CEF4DELPHI_ROOT +git reset --hard +git clean -fd +git fetch origin "v$cefVer" +git switch "v$cefVer" + +if (-not (Test-Path .\libcef.dll)) { + Get-ChildItem *.zip | ForEach-Object { + & 'C:\Program Files\7-Zip\7z.exe' x -y $_.FullName + Remove-Item $_.FullName + } +} +Pop-Location +``` + +### 2.6 Install Keyman 19 official + +Install Keyman 19.0 from https://keyman.com/desktop. After install verify +`C:\Program Files (x86)\Keyman\Keyman Desktop\kmshell.exe` exists and that +`HKLM\SOFTWARE\WOW6432Node\Keyman\Keyman Engine` is populated. The overlay +workflow in section 7 depends on these. + +### 2.7 Delphi 12 IDE Library Search Paths + +`build.sh` injects `dcc32 -I/-U/-R` flags that the IDE does not see. Without +them, opening any of the Keyman .dproj files in Delphi triggers `F1026 File +not found: jvcl.inc` / `jedi.inc` / `jcl.inc` / etc. + +These paths live in the per-user registry under +`HKCU\Software\Embarcadero\BDS\23.0\Library\Win32` (and `...\Win64`), in the +`Search Path` value, semicolon-separated. **Close Delphi first**; the IDE +caches the value at startup. + +```powershell +# Elevated PowerShell -- back up first +reg export 'HKCU\Software\Embarcadero\BDS\23.0\Library' ` + C:\Projects\keyman\keyman\delphi-library-paths.backup.reg /y + +$key32 = 'HKCU:\Software\Embarcadero\BDS\23.0\Library\Win32' +$key64 = 'HKCU:\Software\Embarcadero\BDS\23.0\Library\Win64' + +$paths = @( + # Keyman common includes (mirrors DELPHIINCLUDES in delphi_flags.inc.sh) + 'C:\Projects\keyman\keyman\common\windows\delphi\ext\cef4delphi\source', + 'C:\Projects\keyman\keyman\common\windows\delphi\ext\dcpcrypt', + 'C:\Projects\keyman\keyman\common\windows\delphi\ext\jwa\Win32API', + 'C:\Projects\keyman\keyman\common\windows\delphi\ext\sentry', + 'C:\Projects\keyman\keyman\developer\src\ext\mbcolor', + 'C:\Projects\keyman\keyman\developer\src\ext\scfontcombobox', + # JCL / JVCL source roots (needed for TIKE) + 'C:\Projects\keyman\keyman\developer\src\ext\jedi\jcl\jcl\source\common', + 'C:\Projects\keyman\keyman\developer\src\ext\jedi\jcl\jcl\source\prototypes', + 'C:\Projects\keyman\keyman\developer\src\ext\jedi\jcl\jcl\source\vcl', + 'C:\Projects\keyman\keyman\developer\src\ext\jedi\jcl\jcl\source\windows', + 'C:\Projects\keyman\keyman\developer\src\ext\jedi\jcl\jcl\source\include', # jcl.inc + 'C:\Projects\keyman\keyman\developer\src\ext\jedi\jvcl\jvcl\design', + 'C:\Projects\keyman\keyman\developer\src\ext\jedi\jvcl\jvcl\run', + 'C:\Projects\keyman\keyman\developer\src\ext\jedi\jvcl\jvcl\common', # jvcl.inc + 'C:\Projects\keyman\keyman\developer\src\ext\jedi\jvcl\jvcl\resources', # JvConsts.res + 'C:\Projects\keyman\keyman\developer\src\ext\jedi\jedi', # jedi.inc + 'C:\Projects\keyman\keyman\developer\src\ext\jedi' # parent for {$I jedi\jedi.inc} +) + +foreach ($key in @($key32, $key64)) { + $cur = (Get-ItemProperty -Path $key -Name 'Search Path' ` + -ErrorAction SilentlyContinue).'Search Path' + $merged = (@($cur -split ';') + $paths | + Where-Object { $_ } | + Select-Object -Unique) -join ';' + Set-ItemProperty -Path $key -Name 'Search Path' -Value $merged +} +``` + +Restore via `reg import C:\Projects\keyman\keyman\delphi-library-paths.backup.reg`. + +Note: only the **Search Path** value was set, not Browsing Path or Debug DCU +Path. Ctrl+click navigation to JCL/JVCL source and step-into-source debugging +may need the same paths added there manually via the IDE +(Tools > Options > Language > Delphi > Library). + +## 3. Local-only source patches + +These patches are required to compile against Delphi 12 (VER360) but are +**not yet upstreamable** -- they coexist with Delphi 10.3 (VER340) by +extending IFDEF chains rather than replacing the older guard. The patches +must be reverted before any PR. + +> [!NOTE] +> In some branches `windows/src/engine/engine.groupproj` carries a stale +> reference to `inst\insthelper\insthelper.dproj` -- a path that doesn't +> resolve because the real file is at +> `windows\src\engine\insthelper\insthelper.dproj` (no `inst\` parent). +> If your tree has the broken reference, either patch `engine.groupproj` +> to drop the `inst\` segment or move the file. See section 3.3 and the +> `[[delphi-12-local-patches]]` memory. + +### 3.1 Build-script: select Delphi version via `KEYMAN_DELPHI_VERSION` + +This branch makes the Delphi version configurable via the +`KEYMAN_DELPHI_VERSION` environment variable. It defaults to `20.0` +(Delphi 10.3, the CI target) so default builds are unaffected. For +Delphi 12 set the env var to `23.0`: + +```powershell +# PowerShell (persistent for your user) +[Environment]::SetEnvironmentVariable('KEYMAN_DELPHI_VERSION', '23.0', 'User') +``` + +```bash +# Git Bash (per-shell) +export KEYMAN_DELPHI_VERSION=23.0 +``` + +Without this override, `build.sh` flows that *prepare* the Delphi +environment (without invoking `dcc32`) look for +`C:\Program Files (x86)\Embarcadero\Studio\20.0\bin\rsvars.bat`, which +does not exist on a Delphi-12-only machine. + +This env-var approach is backwards-compatible with the existing Delphi +10.3 CI: when `KEYMAN_DELPHI_VERSION` is unset, both +`resources/build/win/configure_environment.inc.sh` and +`resources/build/win/delphi_environment.inc.sh` fall through to the +historical default of `20.0`. + +### 3.2 keyman.exe uiAccess strip (LOCAL ONLY, not committed) + +This patch is **deliberately not committed** to this branch — it would +change release-signed-binary semantics in a way upstream CI doesn't want. +Apply it manually as a working-tree-only edit if you need to overlay an +unsigned dev `keyman.exe`. Run after a fresh checkout, before building +keyman.dproj: + +```powershell +$f = 'C:\Projects\keyman\keyman\windows\src\engine\keyman\manifest.in' +(Get-Content $f) -replace 'uiAccess="true"', 'uiAccess="false"' | Set-Content $f -Encoding UTF8 +``` + +Then regenerate manifest.xml / manifest.res via +`scripts/preflight-resources.ps1`, do a **Clean + Build** of keyman.dproj +in Delphi IDE (see [[delphi-incremental-build-res-cache]] — Build alone +won't re-link the new manifest.res), and run overlay-dev-builds.ps1. + +**Why:** Windows refuses to launch any unsigned binary that declares +`uiAccess="true"`, returning `Access is denied` (error 8235). The overlay +workflow (section 7) replaces the signed installed keyman.exe with an +unsigned dev build, so uiAccess must be turned off for the overlaid dev +binary to launch. Trade-off: keyboard injection into elevated apps stops +working until uiAccess is restored and the binary is signed via Keyman's +test-cert pipeline. + +To revert (keep your tree CI-clean): `git checkout -- windows/src/engine/keyman/manifest.in`. + +### 3.3 engine.groupproj / insthelper path ambiguity + +There is a real path ambiguity in the engine tree: + +* The actual `insthelper.dproj` lives at + `windows\src\engine\insthelper\insthelper.dproj`. +* Historical `build.sh` plumbing (and at least one variant of + `engine.groupproj` carried in some branches) refers to it as + `inst\insthelper\insthelper.dproj` -- i.e. with an extra `inst\` + segment that does not resolve. + +In the current tree, `engine.groupproj` does not include an `insthelper` +entry at all (the group only builds keyman, kmcomapi, tsysinfo, +tsysinfox64), so "Build All Projects" on the group works -- but +`insthelper.dproj` must then be opened and built on its own. If you encounter +a tree where the groupproj does carry the broken `inst\insthelper\...` path, +the two clean workarounds are: + +(a) Patch `engine.groupproj` to drop the `inst\` segment -- change every + `inst\insthelper\insthelper.dproj` to `insthelper\insthelper.dproj` and + update the `` MSBuild attribute the same way. + +(b) Move the directory: `git mv windows/src/engine/insthelper + windows/src/engine/inst/insthelper` so the existing groupproj path + resolves. This is invasive and not recommended. + +In either case, also add `insthelper` to the `Build` / `Clean` / `Make` +`CallTarget` lists if you want "Build All Projects" to pick it up. See the +GAP note at the top of section 3 about `[[delphi-12-local-patches]]`. + +### 3.4 JCL Boolean -> BOOL casts, plus JclWin32 -> Winapi.Windows on CreateMutex + +`developer/src/ext/jedi/jcl/jcl/source/common/JclSynch.pas` + +Delphi 12 dropped the implicit `Boolean` -> `BOOL` conversion. Every Win32 +sync API call must be wrapped: + +```pascal +// - Result := CreateEvent(SecAttr, Manual, Signaled, PChar(Name)); +// + Result := CreateEvent(SecAttr, BOOL(Manual), BOOL(Signaled), PChar(Name)); +``` + +Call sites that need the cast: `CreateEvent`, `OpenEvent`, +`CreateWaitableTimer`, `OpenWaitableTimer`, `OpenSemaphore`, `CreateMutex`, +`OpenMutex`. Other JCL units likely need the same treatment -- patch +reactively. + +The `CreateMutex` site additionally needs a namespace switch -- the BOOL +cast alone is not sufficient. Per `local-delphi-12-patches.patch`: + +```pascal +// - FHandle := JclWin32.CreateMutex(SecAttr, InitialOwner, PChar(Name)); +// + FHandle := {$IFDEF HAS_UNITSCOPE}Winapi.{$ENDIF}Windows.CreateMutex( +// + SecAttr, BOOL(InitialOwner), PChar(Name)); +``` + +The reason is that `JclWin32.CreateMutex` is a `JclWin32`-namespaced +re-declaration that Delphi 12 resolves ambiguously against +`Winapi.Windows.CreateMutex`, producing a "Cardinal vs LongBool" type +mismatch that the BOOL cast can't fix. Routing the call through +`Winapi.Windows.CreateMutex` directly bypasses the ambiguity. The +`{$IFDEF HAS_UNITSCOPE}` guard preserves compatibility with older Delphis +that still use the un-namespaced `Windows` unit. + +### 3.5 JVCL `OldCreateOrder` removed in Delphi 11 + +`developer/src/ext/jedi/jvcl/jvcl/run/JvComponent.pas` (line ~126): + +```pascal +// - if OldCreateOrder then +// - DoCreate; +// + {$IF Defined(VER350) or Defined(VER360)} +// + DoCreate; +// + {$ELSE} +// + if OldCreateOrder then DoCreate; +// + {$IFEND} +``` + +`TCustomForm.OldCreateOrder` was removed in Delphi 11; modern Delphi behaves +as if it were always `True`. + +### 3.6 mbcolor: teach it about VER350/VER360 + +`developer/src/ext/mbcolor/mxs.inc` + +Add two new IFDEF blocks mirroring the existing VER340 block: + +```pascal +{$ifdef VER350} // Delphi 11 +{$define DELPHI_5_UP} +{$define DELPHI_6_UP} +{$define DELPHI_7_UP} +{$define DELPHI_8_UP} +{$define DELPHI_9_UP} +{$define DELPHI_10_UP} +{$endif} + +{$ifdef VER360} // Delphi 12 +{$define DELPHI_5_UP} +{$define DELPHI_6_UP} +{$define DELPHI_7_UP} +{$define DELPHI_8_UP} +{$define DELPHI_9_UP} +{$define DELPHI_10_UP} +{$endif} +``` + +Without this, mbcolor's `{$IFDEF DELPHI_6_UP}, Variants{$ENDIF}` silently +drops `Variants` from the `uses` clause, causing `E2003 Undeclared +identifier: 'null'` at `HTMLColors.pas:290`. + +### 3.7 devtools: map VER350 / VER360 to Studio version + +`common/windows/delphi/tools/devtools/SourceRootPath.pas` + +The constant is named `DelphiMajorVersion` (not `DelphiVersion`). The existing +baseline (not patched by us) maps VER310->'18.0', VER320->'19.0', +VER330->'20.0', VER340->'21.0'. The patch adds the two newer compilers: + +Before (baseline, abridged): + +```pascal +{$IFDEF VER310} const DelphiMajorVersion = '18.0'; +{$ELSE}{$IFDEF VER320} const DelphiMajorVersion = '19.0'; +{$ELSE}{$IFDEF VER330} const DelphiMajorVersion = '20.0'; +{$ELSE}{$IFDEF VER340} const DelphiMajorVersion = '21.0'; +{$ELSE} +ERROR: must define Delphi version +{$ENDIF}{$ENDIF}{$ENDIF}{$ENDIF} +``` + +After (with VER350 / VER360 inserted before the `ERROR:` fallback): + +```pascal +{$IFDEF VER340} const DelphiMajorVersion = '21.0'; +{$ELSE}{$IFDEF VER350} const DelphiMajorVersion = '22.0'; // 11 (NEW) +{$ELSE}{$IFDEF VER360} const DelphiMajorVersion = '23.0'; // 12 (NEW) +{$ELSE} +ERROR: must define Delphi version +{$ENDIF}{$ENDIF}{$ENDIF} +``` + +Note that the VER340->'21.0' baseline reflects upstream's choice to bump the +Studio dir per CE-license rev, not the more conventional 10.3->'20.0' +mapping. Don't "fix" the existing rows -- only append the two new ones. + +Without one of these defined, the `$ELSE` branch hits the `ERROR: must +define Delphi version` literal which manifests as `E2029 Declaration +expected`. + +### 3.8 Indy UTF-8 URL tripwire + +`common/windows/delphi/web/Keyman.System.HttpServer.Base.pas` + +Wrap the `ERROR! Check if this is still needed with Delphi update` tripwire +in nested `{$IFNDEF VER340}{$IFNDEF VER350}{$IFNDEF VER360}`. The Indy URL +encoding workaround is still required on 12; just extending the guard +preserves the workaround. + +### 3.9 FixedTrackbar tripwire + +`common/windows/delphi/components/FixedTrackbar.pas` + +Same pattern -- extend the inner `{$MESSAGE ERROR}` guard with +`{$IFNDEF VER350}{$IFNDEF VER360}`. + +### 3.10 CleartypeDrawCharacter EnumFontFamiliesEx return type + +`common/windows/delphi/general/CleartypeDrawCharacter.pas` (line ~594) + +`Winapi.Windows.EnumFontFamiliesEx` returns `Integer` starting with Delphi +10.4. The existing guard only covered VER340; extend it: + +```pascal +// - {$IFDEF VER340} +// + {$IF Defined(VER340) or Defined(VER350) or Defined(VER360)} + if EnumFontFamiliesEx(...) <> 0 then +// - {$ELSE} +// + {$ELSE} + if EnumFontFamiliesEx(...) then +// - {$ENDIF} +// + {$IFEND} +``` + +### 3.11 JsonUtil ToChars signature change + +`common/windows/delphi/general/JsonUtil.pas` (line ~55) + +Delphi 11 added a required `Options: TJSONOutputOptions` parameter to +`System.JSON.TJSONAncestor.ToChars`: + +```pascal +{$IF Defined(VER350) or Defined(VER360)} + obj.ToChars(builder, []); +{$ELSE} + obj.ToChars(builder); +{$IFEND} +``` + +### 3.12 Generated `.res` files + +The local working tree will also show binary diffs to many `.res` files +under `developer/src/`, `windows/src/desktop/`, `windows/src/engine/`. These +are produced by `rc.exe` during preflight (icons / manifest / version) -- +they are *not* patches but should be excluded from any PR. See section 5. + +## 4. Build order + +Keyman's repo has several cross-project dependencies that are **not** +expressed in `.dproj` or `.groupproj` files -- they're implicit because the +producer is a Delphi tool that emits code consumed by another Delphi +project, or because a Delphi `.res` embeds the output of another Delphi +`.exe`. Doing things in the wrong order produces fatal `dcc32` errors with +non-obvious messages. + +### CLI-buildable on Delphi 12 CE + +These do not need the IDE: + +* **Keyman Core** -- `./core/build.sh` (Rust / C++ via meson+ninja) +* **kmcmplib** -- `./developer/src/kmcmplib/build.sh build` (Rust / C++) +* **TypeScript modules** -- `kmc`, `kmc-kmn`, `kmc-ldml`, `kmc-package`, + `kmc-keyboard-info`, `kmc-analyze`, `kmc-copy`, `kmc-generate`, + `kmc-model`, `kmc-model-info` (each has its own `build.sh`) +* **KeymanWeb** -- `./web/build.sh` (TypeScript / Emscripten) +* **C++ engine pieces** -- `windows/src/engine/{keyman32,kmtip,keymanhp, + kmrefresh,mcompile,testhost}` (msbuild) + +Build these via the normal `build.sh` chain. They have no Delphi codegen +dependency. + +> [!CAUTION] +> `build.sh` cascades dependencies via `--builder-dep-parent`, so even a +> "non-Delphi" CLI build can pull a Delphi tool in -- which then fails on +> CE with `This version of the product does not support command line +> compiling`. Build the tool by hand in the IDE first, or pass `--no-deps`. + +### Delphi-IDE-only on CE + +* **All of `windows/src/engine/`** -- `keyman.dproj`, `kmcomapi.dproj`, + `insthelper.dproj`, `tsysinfo.dproj`, `tsysinfox64.dproj` + (`engine.groupproj`) +* **All of `windows/src/desktop/`** -- kmshell, kmbrowserhost, kmconfig, + insthelp, setup (`desktop.groupproj`) +* **All of `developer/src/`** -- TIKE, kmconvert, setup + (`developer.groupproj`) +* **All Delphi tools in `common/windows/delphi/tools/`** -- devtools, + build_standards_data, certificates, sentrytool, verify_signatures + +### The cross-project dependency graph + +The non-obvious dependencies, in the order they must be satisfied: + +#### (a) devtools.dproj BEFORE keyman.dproj / setup.dproj / kmshell.dproj + +`keyman.dpr` (engine) and `kmshell.dpr` (desktop) both depend on +`MessageIdentifierConsts.pas`, a ~1500-line file generated from +`windows/src/desktop/kmshell/xml/strings.xml` via +`devtools.exe -buildmessageconstants`. The file is `.gitignored` -- it does +not exist in a fresh checkout. Failure mode: + +``` +F1026 File not found: '...\windows\src\global\delphi\cust\MessageIdentifierConsts.pas' +``` + +`setup.dpr` depends on ~32 `Keyman.Setup.System.Locale..pas` files +generated from `windows/src/desktop/setup/locale/*.xml` via +`devtools.exe -buildsetupstrings`. + +#### (b) build_standards_data.dproj BEFORE TIKE.dproj + +`Keyman.System.LanguageCodeUtils` and friends reference five generated +BCP-47 registry units in `common/windows/delphi/standards/`. The biggest is +`Keyman.System.Standards.LangTagsRegistry.pas` (~2.4 MB). All five are +`.gitignored`. Failure mode: + +``` +F1026 File not found: '...\Keyman.System.Standards.BCP47SubtagRegistry.pas' +``` + +#### (c) tsysinfox64.dproj BEFORE tsysinfo.dproj + +`tsysinfo.dproj` is Win32 but embeds the Win64 `tsysinfox64.exe` as a +binary resource (`tsysinfo_x64.res`) so it can spawn the 64-bit helper +process at runtime. The default Build All order in `engine.groupproj` is +**keyman, kmcomapi, tsysinfo, tsysinfox64, insthelper** -- so tsysinfo is +attempted before the .exe it embeds exists. Failure mode: + +``` +F1026 File not found: 'tsysinfo_x64.res' +``` + +**Right order:** + +1. Build `tsysinfox64.dproj` in the IDE (Win64 / Debug). +2. Copy: `Copy-Item windows/src/engine/tsysinfox64/bin/Win64/Debug/tsysinfox64.exe windows/src/engine/tsysinfo/tsysinfox64.bin` +3. From `windows/src/engine/tsysinfo/`: `rc /nologo tsysinfo_x64.rc` +4. Now build `tsysinfo.dproj`. + +#### (d) kmcmplib (CLI build) BEFORE TIKE.dproj at runtime + +TIKE links to `kmcmplib-19.dll` (the .kmn keyboard compiler). It builds via +`./developer/src/kmcmplib/build.sh build` -- no IDE needed. TIKE compiles +fine without it, but throws "kmcmplib-19.dll not found" at runtime on any +compile action. + +#### (e) keyman.dproj BEFORE kmshell.dproj runtime + +`kmshell.exe` is the Desktop UI; the actual engine work happens in +`keyman.exe`, which kmshell spawns via the COM `IKeymanController` CLSID. +kmshell compiles without keyman.exe -- the failure is at runtime: the +dev kmshell shows "Keyman failed to start" when enabling a keyboard. + +#### (f) regsvr32 kmcomapi.dll BEFORE running kmshell + +`kmcomapi.dll` exposes the COM objects kmshell instantiates at startup. +Without registration, `CoCreateInstance` returns `REGDB_E_CLASSNOTREG` +(`Class not registered`) and kmshell dies before its main form appears. +`overlay-dev-builds.ps1` does this automatically; see section 7. + +### The canonical build order + +For a clean checkout, this is the order that actually works: + +1. CLI build everything CLI-buildable: `./core/build.sh build`, + `./developer/src/kmcmplib/build.sh build`, TypeScript modules, + KeymanWeb. The C++ engine pieces can wait. +2. Open `common/windows/delphi/tools/devtools/devtools.dproj`, build (Win32). +3. Open `common/windows/delphi/tools/build_standards_data/build_standards_data.dproj`, + build (Win32). +4. Run the codegen helper script (section 5.2). +5. Open `engine.groupproj`. Build tsysinfox64 first, run the + tsysinfox64-bin / `tsysinfo_x64.res` step from (c), then build everything + else in the group. +6. Open `desktop.groupproj`. Build kmcomapi (if not already), then setup, + insthelp, kmconfig, kmbrowserhost, kmshell. +7. Open `developer.groupproj`. Build TIKE, kmconvert, setup. +8. Run `overlay-dev-builds.ps1` (elevated) to copy the dev binaries over + the installed Keyman 19. + +## 5. Helper scripts + +Three scripts in the checkout paper over what `build.sh` would normally do +on a Professional Delphi machine. None are committed (they're untracked +local-only files). + +### 5.1 `scripts/preflight-resources.ps1` + +Runs `rc.exe` over every `.rc` file in every Delphi project, producing the +`.res` files that the `.dpr` files `{$R}` reference. Equivalent to the +`build.sh` `_prebuild_resources` phase. + +`rc.exe` lives at +`C:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0\x64\rc.exe`. + +Per-project mapping (paths under `windows\src\`): + +| Project | .rc files | +| --- | --- | +| `engine\keyman\` | version, manifest, icons, keymanmenuitem, osktoolbar, langswitch\langswitchmanager | +| `engine\kmcomapi\` | version, manifest, dialogs (needs SDK includes), kbd_noicon | +| `engine\insthelper\` | version | +| `engine\tsysinfo\` | version, manifest, tsysinfo_x64 (after build order step c) | +| `engine\tsysinfox64\` | version, manifest | +| `desktop\kmshell\` | version, manifest, icons | +| `desktop\kmbrowserhost\` | version, manifest | +| `desktop\kmconfig\` | version, manifest | +| `desktop\insthelp\` | version, manifest | +| `desktop\setup\` | version, manifest, icons (needs SDK includes) | + +> [!NOTE] +> `mcompile` (`windows\src\engine\mcompile\mcompile.vcxproj`) and the other +> C++ engine components -- `keyman32`, `kmtip`, `keymanhp`, `kmrefresh`, +> `testhost` -- are NOT Delphi projects and don't need this preflight step. +> They build via msbuild (driven through their own `build.sh` chain) and +> their `.rc` files are compiled by msbuild as part of the C++ build. + +For `.rc` files that `#include ` (kmcomapi `dialogs.rc`, setup +`icons.rc`, engine `testhost.rc`), pass the Windows SDK include paths: + +```powershell +& $rc /nologo ` + /I "C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\um" ` + /I "C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\shared" ` + /I "C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\winrt" ` + dialogs.rc +``` + +`manifest.rc` references `manifest.xml`, which the script generates from +`manifest.in` by substituting `$VersionWin` with the four-part version from +`VERSION.md`: + +```powershell +# Four-part version is the contents of VERSION.md with '.0' appended +# (e.g. if VERSION.md is 19.0.241, $ver becomes 19.0.241.0). +$ver = "$((Get-Content C:\Projects\keyman\keyman\VERSION.md).Trim()).0" +$content = (Get-Content manifest.in -Raw) -replace '\$VersionWin', $ver +# Write WITHOUT BOM -- rc.exe / Delphi tolerate it but the preflight script +# does it this way and it avoids surprise diffs. +[System.IO.File]::WriteAllText( + (Join-Path (Get-Location) 'manifest.xml'), + $content, + (New-Object System.Text.UTF8Encoding $false)) +& $rc /nologo manifest.rc +``` + +> [!NOTE] +> Prefer running `scripts/preflight-resources.ps1` (section 5.1 below) +> rather than the snippet above -- it already does the no-BOM write +> correctly for every project. The snippet is documentation, not a +> recommended workflow. + +> [!IMPORTANT] +> After regenerating any `.res`, in Delphi: **right-click the project -> +> Clean, then Build**. An incremental Build silently embeds the stale +> cached `.res` and you'll spend an hour wondering why your icon didn't +> update. + +### 5.2 `scripts/run-codegen.ps1` + +Drives `devtools.exe` and `build_standards_data.exe` to produce the +generated `.pas` files that `engine.groupproj`, `desktop.groupproj`, and +`developer.groupproj` depend on. Must be run after step 3 in section 4 and +before step 5. + +```powershell +$DEVTOOLS = 'C:\Projects\keyman\keyman\common\windows\delphi\tools\devtools\bin\Win32\Debug\devtools.exe' + +# (a) MessageIdentifierConsts.pas -- engine/desktop dependency +& $DEVTOOLS -buildmessageconstants ` + C:\Projects\keyman\keyman\windows\src\desktop\kmshell\xml\strings.xml ` + C:\Projects\keyman\keyman\windows\src\global\delphi\cust\MessageIdentifierConsts.pas + +# (b) Per-locale Pascal files for setup.dpr +# Trailing backslashes on both args are required (mirrors +# windows/src/desktop/setup/build.sh:37) +Push-Location C:\Projects\keyman\keyman\windows\src\desktop\setup +& $DEVTOOLS -buildsetupstrings 'locale\' '.\' +Pop-Location + +# (c) Locale index for kmshell support data overlay +& $DEVTOOLS -buildlocaleindex ` + C:\Projects\keyman\keyman\windows\src\desktop\kmshell\locale ` + C:\Projects\keyman\keyman\windows\src\desktop\kmshell\locale\index.xml +``` + +```powershell +$BSD = 'C:\Projects\keyman\keyman\common\windows\delphi\tools\build_standards_data\bin\Win32\Debug\build_standards_data.exe' +$DATA = 'C:\Projects\keyman\keyman\resources\standards-data' +$OUT = 'C:\Projects\keyman\keyman\common\windows\delphi\standards' + +# Five invocations -- mirrors +# common/windows/delphi/tools/build_standards_data/build.sh:41-45 +& $BSD iso6393 "$DATA\iso639-3\iso639-3.tab" ` + "$OUT\Keyman.System.Standards.ISO6393ToBCP47Registry.pas" +& $BSD suppress "$DATA\language-subtag-registry\language-subtag-registry" ` + "$OUT\Keyman.System.Standards.BCP47SuppressScriptRegistry.pas" +& $BSD subtag "$DATA\language-subtag-registry\language-subtag-registry" ` + "$OUT\Keyman.System.Standards.BCP47SubtagRegistry.pas" +& $BSD lcid "$DATA\windows-lcid-to-bcp-47\map_clean_win.txt" ` + "$OUT\Keyman.System.Standards.LCIDToBCP47Registry.pas" +& $BSD langtags "$DATA\langtags\langtags.json" ` + "$OUT\Keyman.System.Standards.LangTagsRegistry.pas" +``` + +All outputs are `.gitignored` and must be re-run after `git clean -fdx`. + +### 5.3 `overlay-dev-builds.ps1` + +Copies the freshly-built dev binaries from the source tree over the +installed Keyman 19 in `Program Files`, then re-runs +`regsvr32 kmcomapi.dll` so the COM CLSIDs point at the new DLL. + +Because section 3.2 patches `manifest.in` to set `uiAccess="false"`, the +unsigned dev `keyman.exe` is launchable from `Program Files` and the +overlay script **does** copy the dev `keyman.exe` over the installed one +(the `keyman.exe` copy line in `overlay-dev-builds.ps1` is active, not +commented out). The list of overlaid binaries: `kmshell.exe`, `keyman.exe`, +`kmcomapi.dll`, plus any DLLs in the source `windows\bin\` tree that exist +in the corresponding install dir. + +Run **elevated** after each dev build cycle. The final `regsvr32` step is at +lines 87-91 of the script: + +```powershell +& regsvr32 /s 'C:\Program Files (x86)\Common Files\Keyman\Keyman Engine\kmcomapi.dll' +``` + +After overlay, launch Keyman via the Start Menu (or the installed +`kmshell.exe` path) -- not from the dev tree -- so the install-time +`TKeymanPaths` lookup finds support files. + +> [!NOTE] +> An alternative workflow -- keep `uiAccess="true"` in `manifest.in` (i.e. +> skip the section 3.2 patch) and skip overlaying `keyman.exe` -- was +> tried and abandoned. With `uiAccess="true"`, the unsigned dev +> `keyman.exe` is refused by Windows (`error=8235`) and the engine never +> starts. The patch + overlay path documented here is the only flow that +> works without a signing cert. + +## 6. Running Keyman from the dev tree + +Because of `TKeymanPaths.KeymanDesktopInstallPath()`, the dev workflow is: + +1. Install Keyman 19 official from keyman.com (once). +2. Build dev binaries in the Delphi IDE (per section 4). +3. Run `scripts/preflight-resources.ps1` if any `.rc` content changed. +4. Run `overlay-dev-builds.ps1` from elevated PowerShell. +5. Launch Keyman from the Start Menu. + +If launching keyman.exe (the engine) directly from the dev `bin\` folder +returns `Could not find keyman.exe (error=8235)`, check that section 3.2's +uiAccess patch was applied and that the project was Clean + Built (not just +Built) after patching `manifest.in`. + +## 7. Debugging + +### 7.1 Run With Host Application + +For DLLs (kmcomapi, keymanhp) and projects that don't have a main entry +point: in Delphi, **Run > Parameters > Host Application** = +`C:\Program Files (x86)\Keyman\Keyman Desktop\kmshell.exe` (or +`...keyman.exe`). Set breakpoints in the DLL source, then Run (F9). + +### 7.2 Attach to Process + +For long-running processes (an already-launched kmshell, the TSF text +service): **Run > Attach to Process**, pick the process, then set +breakpoints. The dev tree's `.dproj` projects emit `.dcu` files with full +debug info into `bin\Win32\Debug\` and `bin\Win64\Debug\`, so as long as +you've overlaid the binary you'll get proper symbols. + +### 7.3 What you get -- and don't get -- from a vanilla install + +The installed Keyman 19 from keyman.com ships **stripped** PDBs at best; +practically, you'll see only export-table symbols when attaching. The +overlay workflow replaces the installed binaries with dev builds whose +debug info lives next to the source -- after overlay you can step through +Keyman source in the IDE. Without overlay, the IDE will load the source +file when you breakpoint but the line numbers will not match the running +binary. + +For C++ pieces (keyman32, kmtip, mcompile), use Visual Studio's Attach to +Process and load the `.pdb` from `windows/bin/`. + +## 8. Troubleshooting + +### `This version of the product does not support command line compiling` + +You're hitting the CE block on `dcc32`. Don't try to drive Delphi from +`build.sh`; build the relevant `.dproj` in the IDE by hand. If `build.sh` +pulled Delphi in as a transitive dep, pass `--no-deps` or build the +upstream Delphi project first. + +### `F1026 File not found: 'MessageIdentifierConsts.pas'` + +Codegen wasn't run. Build `devtools.dproj` in the IDE, then run section 5.2 +step (a). + +### `F1026 File not found: 'Keyman.Setup.System.Locale..pas'` (one of ~32) + +Codegen wasn't run. Build `devtools.dproj` then run section 5.2 step (b). +Trailing backslashes on the two args matter. + +### `F1026 File not found: 'Keyman.System.Standards.BCP47SubtagRegistry.pas'` + +`build_standards_data.exe` wasn't run. Build the `.dproj` then run section +5.2 (BSD block). + +### `F1026 File not found: 'tsysinfo_x64.res'` + +Group build order issue. Build tsysinfox64 first, copy the exe to +tsysinfo's source dir as `tsysinfox64.bin`, run `rc /nologo tsysinfo_x64.rc`, +then build tsysinfo. See section 4 (c). + +### `F1026 File not found: 'version.res'` or `'manifest.res'` + +Preflight resources weren't compiled. Run `scripts/preflight-resources.ps1` +or invoke `rc.exe` manually for the project. + +### `F1026 File not found: 'jvcl.inc'` / `'jedi/jedi.inc'` / `'jcl.inc'` + +Library Search Paths not in the registry. See section 2.7. Close and +reopen Delphi after editing the registry; the IDE caches the value at +startup. + +### `E2010 Incompatible types: 'Cardinal' and 'Boolean'` (JCL) + +JCL Boolean -> BOOL casts missing. See section 3.4. + +### `E2003 Undeclared identifier: 'OldCreateOrder'` (JVCL) + +JvComponent.pas not patched. See section 3.5. + +### `E2003 Undeclared identifier: 'null'` (HTMLColors) + +mbcolor's `mxs.inc` not patched for VER350/VER360 -- the `Variants` unit +was silently dropped from the `uses` clause. See section 3.6. + +### `E2029 Declaration expected` near SourceRootPath.pas + +devtools/SourceRootPath.pas hit the `{$ELSE} {$MESSAGE ERROR}` fallback. +See section 3.7. + +### `E2012 Type of expression must be BOOLEAN` near EnumFontFamiliesEx + +CleartypeDrawCharacter.pas guard not extended for VER350/VER360. See +section 3.10. + +### `F2613 Unit 'JvComponentBase' not found` + +JVCL `run/` (or `design/`) is missing from Library Search Path. See +section 2.7. + +### Delphi pops up an "Unsupported CEF version" dialog + +The CEF4Delphi_Binary checkout isn't on the tag in `CEF_VERSION.md`. See +section 2.5. Don't ignore the dialog -- the CEF4Delphi Pascal binding and +the libcef binary must match exactly. + +### `SKApplicationTitle has had a fatal error...` on kmshell launch + +kmshell can't find its strings.xml / locale files because +`KeymanDesktopInstallPath()` is hardcoded to `Program Files`, and Keyman +19 official isn't installed (or its install support files were removed by +an aborted dev experiment). Install Keyman 19 official from keyman.com. + +### `Class not registered ClassID {CF46549D-...}` on kmshell launch + +`kmcomapi.dll` isn't registered. Run elevated: + +```powershell +& regsvr32 /s 'C:\Program Files (x86)\Common Files\Keyman\Keyman Engine\kmcomapi.dll' +``` + +`overlay-dev-builds.ps1` does this automatically; if you bypassed the +overlay, do it by hand. + +### `Could not find keyman.exe (error=8235)` + +Windows blocked launch of an unsigned `uiAccess="true"` binary. Patch +`windows/src/engine/keyman/manifest.in` per section 3.2, Clean + rebuild +`keyman.dproj`, re-run `overlay-dev-builds.ps1`. + +--- + +## Upstream candidates + +If you'd like to PR some of these changes back upstream, the patches split +into two groups based on whether they touch Keyman-owned code or vendored +third-party code. + +**Upstreamable (Keyman-owned, additive, CI-safe with `KEYMAN_DELPHI_VERSION` +unset):** + +* `resources/build/win/configure_environment.inc.sh` + `delphi_environment.inc.sh` + -- `KEYMAN_DELPHI_VERSION` env-var override (default preserves 10.3 behavior). +* `common/windows/delphi/tools/devtools/SourceRootPath.pas` -- VER350/VER360 + cases for `DelphiMajorVersion`. +* `common/windows/delphi/web/Keyman.System.HttpServer.Base.pas` -- extend + VER tripwire to accept VER340/VER350/VER360. +* `common/windows/delphi/components/FixedTrackbar.pas` -- same tripwire + pattern. +* `common/windows/delphi/general/CleartypeDrawCharacter.pas` -- integer-return + comparison for `EnumFontFamiliesEx` on VER340+. +* `common/windows/delphi/general/JsonUtil.pas` -- two-arg `ToChars` on + VER350/VER360 (Delphi 11+ RTL change). +* `windows/src/engine/engine.groupproj` -- drop the broken + `inst\insthelper\insthelper.dproj` reference. + +These could ship as a single "Support compiling under Delphi 11/12" PR +without conflicting with the broader Delphi-removal effort +([keymanapp/keyman#4599](https://github.com/keymanapp/keyman/issues/4599)). + +**Local-only (vendored third-party):** + +* `developer/src/ext/jedi/jcl/jcl/source/common/JclSynch.pas` -- JCL Boolean + -> BOOL casts. The right fix is to refresh the bundled JCL from upstream + (project-jedi.org) when JCL releases a Delphi 12-compatible version. +* `developer/src/ext/jedi/jvcl/jvcl/run/JvComponent.pas` -- JVCL + OldCreateOrder removal. Same -- pull from upstream JVCL when available. +* `developer/src/ext/mbcolor/mxs.inc` -- third-party mbcolor library. + Upstream maintenance status unclear. + +These should *not* be PR'd upstream-Keyman as-is; the bundled third-party +copies should be refreshed from their respective maintainers instead. + +## Reverting before a PR + +Before opening a PR, revert all section 3 patches and remove the section +2.7 registry entries. The simplest check: + +```bash +git status # should show only intentional changes +git diff -- '*.res' # should be empty (preflight .res are local-only) +``` + +If `git diff` shows any of the files mentioned in section 3, revert them +with `git checkout -- ` before pushing. + +Restore the original Library Search Path values: + +```powershell +reg import C:\Projects\keyman\keyman\delphi-library-paths.backup.reg +``` diff --git a/docs/build/windows.md b/docs/build/windows.md index edc756340ff..a595a29f633 100644 --- a/docs/build/windows.md +++ b/docs/build/windows.md @@ -257,6 +257,11 @@ of appropriate node versions during builds. for a short time. (We are actively working to remove Delphi dependencies given the licensing issues with using it.) + * If you only have access to **Delphi 12 Community Edition**, see + [windows-d12.md](windows-d12.md) for an IDE-based workflow that papers over + the missing command-line compiler. The patches and helper scripts described + there are local-only and not required for the standard 10.3 / Pro flow. + Start Delphi IDE once after installation as it will create various environment files and take you through required registration. diff --git a/overlay-dev-builds.ps1 b/overlay-dev-builds.ps1 new file mode 100644 index 00000000000..03dad581098 --- /dev/null +++ b/overlay-dev-builds.ps1 @@ -0,0 +1,102 @@ +#Requires -RunAsAdministrator +# Overlay dev builds onto an installed Keyman 19. Equivalent of what +# `windows/src/build.sh install:engine install:desktop` would do, but standalone +# (doesn't require the build.sh cascade, which is blocked by Delphi CE). +# +# Run elevated after each Delphi/kmcmplib rebuild. + +$ErrorActionPreference = 'Continue' + +# Auto-detect repo root. Override by setting $env:KEYMAN_ROOT. +if ($env:KEYMAN_ROOT) { + $REPO = $env:KEYMAN_ROOT +} else { + # This script lives at the repo root. + $REPO = $PSScriptRoot +} + +$INSTALL_DESKTOP = 'C:\Program Files (x86)\Keyman\Keyman Desktop' +$INSTALL_DEVELOPER = 'C:\Program Files (x86)\Keyman\Keyman Developer' +$INSTALL_ENGINE = 'C:\Program Files (x86)\Common Files\Keyman\Keyman Engine' + +function Copy-IfExists($src, $dst, $label) { + if (Test-Path $src) { + $dstDir = Split-Path $dst -Parent + if (-not (Test-Path $dstDir)) { + New-Item -ItemType Directory -Path $dstDir -Force | Out-Null + } + Copy-Item $src $dst -Force + Write-Host " [$label] $src -> $dst" + } else { + Write-Host " [$label] SKIP (source missing): $src" -ForegroundColor Yellow + } +} + +function Copy-Tree($src, $dst, $label) { + if (Test-Path $src) { + if (Test-Path $dst) { Remove-Item $dst -Recurse -Force } + Copy-Item $src $dst -Recurse -Force + Write-Host " [$label] $src -> $dst (tree)" + } else { + Write-Host " [$label] SKIP (source tree missing): $src" -ForegroundColor Yellow + } +} + +Write-Host "=== Stopping any running Keyman processes ===" +Get-Process -Name kmshell, keyman, keymanmc, tike, tsysinfo, tsysinfox64, kmbrowserhost, kmconfig -ErrorAction SilentlyContinue | + ForEach-Object { Write-Host " Stopping $($_.Name) PID $($_.Id)"; Stop-Process -Id $_.Id -Force } +Start-Sleep -Milliseconds 800 + +Write-Host "" +Write-Host "=== Engine -> $INSTALL_ENGINE ===" +# NOTE: Dev manifest patched to uiAccess="false" so overlay works. See +# [[delphi-12-local-patches]] memory. Caveat: keyboard injection into +# elevated apps will not work in dev builds until uiAccess is restored +# and binary is signed. +Copy-IfExists "$REPO\windows\src\engine\keyman\bin\Win32\Debug\keyman.exe" "$INSTALL_ENGINE\keyman.exe" 'engine' +Copy-IfExists "$REPO\windows\src\engine\kmcomapi\bin\Win32\Debug\kmcomapi.dll" "$INSTALL_ENGINE\kmcomapi.dll" 'engine' +Copy-IfExists "$REPO\windows\src\engine\insthelper\bin\Win32\Debug\insthelper.dll" "$INSTALL_ENGINE\insthelper.dll" 'engine' +Copy-IfExists "$REPO\windows\src\engine\tsysinfo\bin\Win32\Debug\tsysinfo.exe" "$INSTALL_ENGINE\tsysinfo.exe" 'engine' +Copy-IfExists "$REPO\windows\src\engine\tsysinfox64\bin\Win64\Debug\tsysinfox64.exe" "$INSTALL_ENGINE\tsysinfox64.exe" 'engine' + +# C++ engine components (Win32 + x64 + arm64 where applicable) +Copy-IfExists "$REPO\windows\src\engine\keyman32\bin\Win32\Debug\keyman32.dll" "$INSTALL_ENGINE\keyman32.dll" 'engine-cpp' +Copy-IfExists "$REPO\windows\src\engine\keyman32\bin\x64\Debug\keyman64.dll" "$INSTALL_ENGINE\keyman64.dll" 'engine-cpp' +Copy-IfExists "$REPO\windows\src\engine\kmtip\bin\Win32\Debug\kmtip.dll" "$INSTALL_ENGINE\kmtip.dll" 'engine-cpp' +Copy-IfExists "$REPO\windows\src\engine\kmtip\bin\x64\Debug\kmtip64.dll" "$INSTALL_ENGINE\kmtip64.dll" 'engine-cpp' +Copy-IfExists "$REPO\windows\src\engine\mcompile\bin\Win32\Debug\mcompile.exe" "$INSTALL_ENGINE\mcompile.exe" 'engine-cpp' +Copy-IfExists "$REPO\windows\src\engine\kmrefresh\bin\Win32\Debug\kmrefresh.x86.exe" "$INSTALL_ENGINE\kmrefresh.x86.exe" 'engine-cpp' +Copy-IfExists "$REPO\windows\src\engine\kmrefresh\bin\x64\Debug\kmrefresh.x64.exe" "$INSTALL_ENGINE\kmrefresh.x64.exe" 'engine-cpp' +Copy-IfExists "$REPO\windows\src\engine\keymanhp\bin\x64\Debug\keymanhp.x64.exe" "$INSTALL_ENGINE\keymanhp.x64.exe" 'engine-cpp' + +Write-Host "" +Write-Host "=== Desktop -> $INSTALL_DESKTOP ===" +Copy-IfExists "$REPO\windows\src\desktop\kmshell\bin\Win32\Debug\kmshell.exe" "$INSTALL_DESKTOP\kmshell.exe" 'desktop' +Copy-IfExists "$REPO\windows\src\desktop\kmbrowserhost\bin\Win32\Debug\kmbrowserhost.exe" "$INSTALL_DESKTOP\kmbrowserhost.exe" 'desktop' +Copy-IfExists "$REPO\windows\src\desktop\kmconfig\bin\Win32\Debug\kmconfig.exe" "$INSTALL_DESKTOP\kmconfig.exe" 'desktop' +Copy-IfExists "$REPO\windows\src\desktop\insthelp\bin\Win32\Debug\insthelp.dll" "$INSTALL_DESKTOP\insthelp.dll" 'desktop' + +# kmshell expects its support data alongside the .exe +Copy-Tree "$REPO\windows\src\desktop\kmshell\xml" "$INSTALL_DESKTOP\xml" 'desktop-data' +Copy-Tree "$REPO\windows\src\desktop\kmshell\locale" "$INSTALL_DESKTOP\locale" 'desktop-data' + +Write-Host "" +Write-Host "=== Developer -> $INSTALL_DEVELOPER ===" +if (-not (Test-Path $INSTALL_DEVELOPER)) { + Write-Host " Keyman Developer is not installed. Skipping. (Install from keyman.com if you want TIKE wired up.)" -ForegroundColor Yellow +} else { + Copy-IfExists "$REPO\developer\src\tike\bin\Win32\Debug\tike.exe" "$INSTALL_DEVELOPER\tike.exe" 'developer' + Copy-IfExists "$REPO\developer\src\kmconvert\bin\Win32\Debug\kmconvert.exe" "$INSTALL_DEVELOPER\kmconvert.exe" 'developer' + Copy-IfExists "$REPO\developer\src\kmcmplib\build\x86\debug\src\kmcmplib-19.dll" "$INSTALL_DEVELOPER\kmcmplib-19.dll" 'developer' +} + +Write-Host "" +Write-Host "=== Re-registering kmcomapi.dll at install path ===" +if (Test-Path "$INSTALL_ENGINE\kmcomapi.dll") { + & regsvr32 /s "$INSTALL_ENGINE\kmcomapi.dll" + Write-Host " regsvr32 exit: $LASTEXITCODE (0 = success)" +} + +Write-Host "" +Write-Host "=== Done. Launch kmshell from Start Menu or:" -ForegroundColor Green +Write-Host " & '$INSTALL_DESKTOP\kmshell.exe'" diff --git a/resources/build/win/configure_environment.inc.sh b/resources/build/win/configure_environment.inc.sh index d0bce72a812..1d274c67be1 100644 --- a/resources/build/win/configure_environment.inc.sh +++ b/resources/build/win/configure_environment.inc.sh @@ -65,9 +65,11 @@ _build_vs_environment() { _locate_rsvars() { # - # Delphi Compiler Configuration - Delphi 10.3.2 + # Delphi Compiler Configuration - defaults to Delphi 10.3 (BDS 20.0). + # Override via the KEYMAN_DELPHI_VERSION environment variable to build with + # a newer Delphi installation (e.g., KEYMAN_DELPHI_VERSION=23.0 for Delphi 12). # - DELPHI_VERSION=20.0 + DELPHI_VERSION="${KEYMAN_DELPHI_VERSION:-20.0}" DCC32PATH="$(cygpath -u "$ProgramFilesx86\\Embarcadero\\Studio\\$DELPHI_VERSION\\bin")" RSVars_path="$DCC32PATH/rsvars.bat" } diff --git a/resources/build/win/delphi_environment.inc.sh b/resources/build/win/delphi_environment.inc.sh index 0465bcb6930..a68b08421c4 100644 --- a/resources/build/win/delphi_environment.inc.sh +++ b/resources/build/win/delphi_environment.inc.sh @@ -13,7 +13,10 @@ DELPHIWARNINGS=(-W-MESSAGE_DIRECTIVE -W-IMPLICIT_STRING_CAST -W-IMPLICIT_STRING_CAST_LOSS -W-EXPLICIT_STRING_CAST -W-EXPLICIT_STRING_CAST_LOSS -W-CVT_WCHAR_TO_ACHAR -W-CVT_NARROWING_STRING_LOST -W-CVT_ACHAR_TO_WCHAR -W-CVT_WIDENING_STRING_LOST -W-UNICODE_TO_LOCALE -W-LOCALE_TO_UNICODE -W-IMPLICIT_VARIANTS -W-IMPLICIT_INTEGER_CAST_LOSS -W-IMPLICIT_CONVERSION_LOSS -W-COMBINING_SIGNED_UNSIGNED64 -W-COMBINING_SIGNED_UNSIGNED64) # !ENDIF -DELPHI_VERSION=20.0 +# DELPHI_VERSION defaults to Delphi 10.3 (BDS 20.0). Override via the +# KEYMAN_DELPHI_VERSION environment variable to build with a newer Delphi +# installation (e.g., KEYMAN_DELPHI_VERSION=23.0 for Delphi 12). +DELPHI_VERSION="${KEYMAN_DELPHI_VERSION:-20.0}" DCC32PATH="$(cygpath -u "$ProgramFilesx86\\Embarcadero\\Studio\\$DELPHI_VERSION\\bin")" source "$KEYMAN_ROOT/resources/build/win/delphi_environment_generated.inc.sh" diff --git a/scripts/install-build-env.ps1 b/scripts/install-build-env.ps1 new file mode 100644 index 00000000000..146e3cc6586 --- /dev/null +++ b/scripts/install-build-env.ps1 @@ -0,0 +1,63 @@ +#Requires -RunAsAdministrator +# Installs the Keyman build environment on Windows (everything except Delphi). +# Logs to \install-build-env.log (gitignored). + +$ErrorActionPreference = 'Continue' +$ProgressPreference = 'SilentlyContinue' + +# Auto-detect repo root. Override by setting $env:KEYMAN_ROOT. +if ($env:KEYMAN_ROOT) { + $REPO = $env:KEYMAN_ROOT +} else { + # This script lives at \scripts\, so repo root is one level up. + $REPO = Split-Path $PSScriptRoot -Parent +} + +$log = Join-Path $REPO 'install-build-env.log' +Start-Transcript -Path $log -Force | Out-Null + +Write-Host "=== Step 1/5: Install Chocolatey ===" +if (-not (Get-Command choco -ErrorAction SilentlyContinue)) { + Set-ExecutionPolicy Bypass -Scope Process -Force + [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072 + iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1')) + $env:PATH = "$env:PATH;$env:ProgramData\chocolatey\bin" +} else { + Write-Host "Chocolatey already installed." +} + +Write-Host "=== Step 2/5: Install base deps (git, jq, python, ninja, pandoc, meson) ===" +choco install -y git jq python ninja pandoc meson + +Write-Host "=== Step 3/5: Install nvm-windows + node 20.16.0 ===" +choco install -y nvm +# refresh PATH so nvm is callable +$env:PATH = [System.Environment]::GetEnvironmentVariable('PATH','Machine') + ';' + [System.Environment]::GetEnvironmentVariable('PATH','User') +if (Get-Command nvm -ErrorAction SilentlyContinue) { + nvm install 20.16.0 + nvm use 20.16.0 +} else { + Write-Host "WARNING: nvm not on PATH after install. Open a new shell and run: nvm install 20.16.0; nvm use 20.16.0" +} + +Write-Host "=== Step 4/5: Install release-build tools (7zip, html-help-workshop, wixtoolset 3.14.1) ===" +choco install -y 7zip html-help-workshop +choco install -y wixtoolset --version=3.14.1 + +Write-Host "=== Step 5/5: Install Visual Studio 2022 Community (Native Desktop) ===" +Write-Host "This is a large install (~10GB) and may take 20-40 minutes." +$vsArgs = '--passive --add Microsoft.VisualStudio.Workload.NativeDesktop ' + + '--add Microsoft.VisualStudio.Component.VC.Tools.ARM64 ' + + '--add Microsoft.VisualStudio.Component.CppBuildInsights ' + + '--add Microsoft.VisualStudio.Component.Debugger.JustInTime ' + + '--add Microsoft.VisualStudio.Component.VC.ASAN ' + + '--add Microsoft.VisualStudio.Component.VC.DiagnosticTools ' + + '--add Microsoft.VisualStudio.Component.VC.TestAdapterForGoogleTest ' + + '--add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 ' + + '--add Microsoft.VisualStudio.Component.Windows11SDK.26100 ' + + '--add Microsoft.VisualStudio.Component.Windows11Sdk.WindowsPerformanceToolkit ' + + '--add Microsoft.VisualStudio.Component.Windows10SDK.19041' +winget install --id=Microsoft.VisualStudio.2022.Community -e --accept-source-agreements --accept-package-agreements --override $vsArgs + +Write-Host "=== Done. Log saved to $log ===" +Stop-Transcript | Out-Null diff --git a/scripts/preflight-resources.ps1 b/scripts/preflight-resources.ps1 new file mode 100644 index 00000000000..ccf58996616 --- /dev/null +++ b/scripts/preflight-resources.ps1 @@ -0,0 +1,334 @@ +<# +.SYNOPSIS + Generate .res files (and manifest.xml) for every Delphi project in the + Keyman repo, so the projects open and compile cleanly in the Delphi IDE. + +.DESCRIPTION + Delphi 12 Community Edition blocks CLI builds, which means build.sh's + do_build rc-step never runs locally. This script is the standalone + equivalent: walk every directory containing a .dproj, expand manifest.in + -> manifest.xml using VERSION.md, then compile every .rc next to it with + rc.exe from the Windows 10 SDK. Output .res files land alongside their + source .rc. + + Run this: + * After a fresh clone / `git clean`. + * After bumping VERSION.md (so manifest.xml is re-stamped). + * After editing any .rc / .in / referenced .bin (e.g. tsysinfox64.bin). + + After regenerating .res, in Delphi do: right-click project -> Clean, + then Build. An incremental Build silently re-embeds the stale cached .res. + + Codegen steps that depend on Delphi-built tools (devtools.exe, + build_standards_data.exe) live in scripts/run-codegen.ps1, not here. + +.PARAMETER Project + Optional. One or more directories to limit the preflight to. May be a + path relative to the repo (e.g. 'windows\src\engine\keyman') or an + absolute path. If omitted, every dir containing a .dproj is processed. + +.PARAMETER Repo + Repo root. Defaults to $env:KEYMAN_ROOT, then auto-detects from this + script's location (assumes the script lives at \scripts\). + +.EXAMPLE + .\preflight-resources.ps1 + +.EXAMPLE + .\preflight-resources.ps1 -Project windows\src\engine\tsysinfo +#> + +[CmdletBinding()] +param( + [string[]]$Project, + [string]$Repo +) + +$ErrorActionPreference = 'Continue' + +if (-not $Repo) { + if ($env:KEYMAN_ROOT) { $Repo = $env:KEYMAN_ROOT } + else { $Repo = Split-Path $PSScriptRoot -Parent } +} + +if (-not (Test-Path $Repo)) { + Write-Host "Repo root not found: $Repo" -ForegroundColor Red + exit 1 +} + +# ---------------------------------------------------------------------------- +# Locate Windows 10 SDK (rc.exe + um/shared/winrt include dirs) +# ---------------------------------------------------------------------------- + +function Find-WindowsSdk { + $sdkRoot = 'C:\Program Files (x86)\Windows Kits\10' + $includeRoot = Join-Path $sdkRoot 'Include' + if (-not (Test-Path $includeRoot)) { + throw "Windows 10 SDK not found at $sdkRoot" + } + + # Preferred versions first, then newest 10.0.* fallback + $preferred = @('10.0.26100.0', '10.0.19041.0') + $available = Get-ChildItem -Path $includeRoot -Directory | + Where-Object { $_.Name -like '10.0.*' } | + Select-Object -ExpandProperty Name + + $chosen = $null + foreach ($v in $preferred) { + if ($available -contains $v) { $chosen = $v; break } + } + if (-not $chosen) { + # newest by version sort + $chosen = $available | + Sort-Object { [version]($_ -replace '^10\.0\.', '10.0.') } -Descending | + Select-Object -First 1 + } + if (-not $chosen) { + throw "No 10.0.* SDK version found under $includeRoot" + } + + $rcCandidates = @( + (Join-Path $sdkRoot "bin\$chosen\x64\rc.exe"), + (Join-Path $sdkRoot "bin\$chosen\x86\rc.exe") + ) + $rc = $rcCandidates | Where-Object { Test-Path $_ } | Select-Object -First 1 + if (-not $rc) { + throw "rc.exe not found for SDK $chosen (looked in $($rcCandidates -join '; '))" + } + + [pscustomobject]@{ + Version = $chosen + Rc = $rc + IncUm = Join-Path $sdkRoot "Include\$chosen\um" + IncShared = Join-Path $sdkRoot "Include\$chosen\shared" + IncWinrt = Join-Path $sdkRoot "Include\$chosen\winrt" + } +} + +$Sdk = Find-WindowsSdk +Write-Host "Windows SDK: $($Sdk.Version)" -ForegroundColor Cyan +Write-Host "rc.exe: $($Sdk.Rc)" +Write-Host "" + +# ---------------------------------------------------------------------------- +# Version string for manifest.xml ($VersionWin substitution) +# ---------------------------------------------------------------------------- + +$versionFile = Join-Path $Repo 'VERSION.md' +if (-not (Test-Path $versionFile)) { + Write-Host "VERSION.md not found at $versionFile" -ForegroundColor Red + exit 1 +} +$Version4 = (Get-Content $versionFile).Trim() + '.0' +Write-Host "Version: $Version4 (from VERSION.md)" -ForegroundColor Cyan +Write-Host "" + +# ---------------------------------------------------------------------------- +# Counters +# ---------------------------------------------------------------------------- + +$script:Generated = New-Object System.Collections.Generic.List[string] +$script:Skipped = New-Object System.Collections.Generic.List[string] +$script:Failed = New-Object System.Collections.Generic.List[string] + +# ---------------------------------------------------------------------------- +# Helpers +# ---------------------------------------------------------------------------- + +function Test-NeedsSdkIncludes { + param([string]$RcPath) + + if (-not (Test-Path $RcPath)) { return $false } + $needles = @('', '', '') + $content = Get-Content $RcPath -Raw -ErrorAction SilentlyContinue + if (-not $content) { return $false } + foreach ($n in $needles) { + if ($content -match [regex]::Escape($n)) { return $true } + } + return $false +} + +function Write-ManifestXml { + param( + [string]$ProjectDir, + [string]$Version + ) + + $manifestIn = Join-Path $ProjectDir 'manifest.in' + $manifestXml = Join-Path $ProjectDir 'manifest.xml' + if (-not (Test-Path $manifestIn)) { return } + + try { + $raw = Get-Content $manifestIn -Raw + $expanded = $raw -replace '\$VersionWin', $Version + # Write UTF-8 without BOM, no trailing newline + [System.IO.File]::WriteAllText($manifestXml, $expanded, (New-Object System.Text.UTF8Encoding $false)) + $size = (Get-Item $manifestXml).Length + Write-Host (" [manifest] {0} ({1} bytes)" -f $manifestXml, $size) -ForegroundColor Green + $script:Generated.Add($manifestXml) + } catch { + Write-Host " [manifest] FAILED: $manifestIn -> $($_.Exception.Message)" -ForegroundColor Red + $script:Failed.Add("$manifestIn ($($_.Exception.Message))") + } +} + +function Invoke-RcCompile { + param( + [string]$RcPath + ) + + $dir = Split-Path $RcPath -Parent + $base = [System.IO.Path]::GetFileNameWithoutExtension($RcPath) + $resPath = Join-Path $dir "$base.res" + + $needsSdk = Test-NeedsSdkIncludes -RcPath $RcPath + $rcArgs = @('/nologo') + if ($needsSdk) { + $rcArgs += '/I'; $rcArgs += $Sdk.IncUm + $rcArgs += '/I'; $rcArgs += $Sdk.IncShared + $rcArgs += '/I'; $rcArgs += $Sdk.IncWinrt + } + # Pass just the filename so output .res lands in the rc's directory + $rcArgs += (Split-Path $RcPath -Leaf) + + Push-Location $dir + try { + $output = & $Sdk.Rc @rcArgs 2>&1 + $code = $LASTEXITCODE + } finally { + Pop-Location + } + + if ($code -ne 0) { + $msg = ($output | Out-String).Trim() + Write-Host (" [rc] FAILED: {0}" -f $RcPath) -ForegroundColor Red + if ($msg) { + foreach ($line in $msg -split "`r?`n") { + Write-Host " $line" -ForegroundColor Red + } + } + $script:Failed.Add("$RcPath (rc.exe exit $code)") + return + } + + if (Test-Path $resPath) { + $size = (Get-Item $resPath).Length + $tag = if ($needsSdk) { 'rc+sdk' } else { 'rc' } + Write-Host (" [{0,-7}] {1} ({2} bytes)" -f $tag, $resPath, $size) -ForegroundColor Green + $script:Generated.Add($resPath) + } else { + Write-Host " [rc] no .res produced for $RcPath" -ForegroundColor Yellow + $script:Skipped.Add("$RcPath (no .res produced)") + } +} + +function Invoke-Preflight { + param([string]$Dir) + + $rel = $Dir + if ($Dir.StartsWith($Repo, [System.StringComparison]::OrdinalIgnoreCase)) { + $rel = $Dir.Substring($Repo.Length).TrimStart('\','/') + } + Write-Host ("--- {0}" -f $rel) -ForegroundColor White + + # (a) manifest.in -> manifest.xml (must come before manifest.rc compile) + Write-ManifestXml -ProjectDir $Dir -Version $Version4 + + # (b) Compile every .rc under this dir (recursive picks up langswitch\*.rc etc.) + $rcFiles = Get-ChildItem -Path $Dir -Recurse -Filter '*.rc' -File -ErrorAction SilentlyContinue + + if (-not $rcFiles -or $rcFiles.Count -eq 0) { + Write-Host " (no .rc files in this project dir)" -ForegroundColor DarkGray + $script:Skipped.Add("$Dir (no .rc files)") + return + } + + foreach ($rc in $rcFiles) { + # Special case: tsysinfo_x64.rc embeds tsysinfox64.bin which is itself + # the output of building tsysinfox64.dproj in Win64/Debug. If the bin + # isn't present yet, skip with a pointer to the codegen helper. + if ($rc.Name -ieq 'tsysinfo_x64.rc') { + $binPath = Join-Path $rc.DirectoryName 'tsysinfox64.bin' + if (-not (Test-Path $binPath)) { + $msg = "tsysinfox64.bin missing -- build tsysinfox64.dproj (Win64/Debug) first, then run scripts\run-codegen.ps1 (or copy the exe to $binPath)" + Write-Host " [rc] SKIP $($rc.FullName): $msg" -ForegroundColor Yellow + $script:Skipped.Add("$($rc.FullName) ($msg)") + continue + } + } + + Invoke-RcCompile -RcPath $rc.FullName + } +} + +# ---------------------------------------------------------------------------- +# Resolve target project dirs +# ---------------------------------------------------------------------------- + +function Resolve-ProjectArg { + param([string]$Arg) + + if ([System.IO.Path]::IsPathRooted($Arg)) { + return $Arg + } + return (Join-Path $Repo $Arg) +} + +if ($Project -and $Project.Count -gt 0) { + $targetDirs = @() + foreach ($p in $Project) { + $resolved = Resolve-ProjectArg -Arg $p + if (-not (Test-Path $resolved)) { + Write-Host "Project path not found: $resolved" -ForegroundColor Red + $script:Failed.Add("$resolved (path not found)") + continue + } + $targetDirs += (Resolve-Path $resolved).Path + } +} else { + Write-Host "Discovering .dproj files under $Repo ..." -ForegroundColor Cyan + $dprojs = Get-ChildItem -Path $Repo -Recurse -Filter '*.dproj' -File -ErrorAction SilentlyContinue + $targetDirs = $dprojs | + ForEach-Object { $_.DirectoryName } | + Sort-Object -Unique + Write-Host ("Found {0} project dir(s) with .dproj files." -f $targetDirs.Count) -ForegroundColor Cyan + Write-Host "" +} + +if (-not $targetDirs -or $targetDirs.Count -eq 0) { + Write-Host "Nothing to do." -ForegroundColor Yellow + exit 0 +} + +# ---------------------------------------------------------------------------- +# Process each project dir +# ---------------------------------------------------------------------------- + +foreach ($d in $targetDirs) { + Invoke-Preflight -Dir $d +} + +# ---------------------------------------------------------------------------- +# Summary +# ---------------------------------------------------------------------------- + +Write-Host "" +Write-Host "===== Summary =====" -ForegroundColor Cyan +Write-Host (" Generated: {0}" -f $script:Generated.Count) -ForegroundColor Green +Write-Host (" Skipped: {0}" -f $script:Skipped.Count) -ForegroundColor Yellow +Write-Host (" Failed: {0}" -f $script:Failed.Count) -ForegroundColor $(if ($script:Failed.Count) { 'Red' } else { 'Gray' }) + +if ($script:Skipped.Count -gt 0) { + Write-Host "" + Write-Host "Skipped items:" -ForegroundColor Yellow + foreach ($s in $script:Skipped) { Write-Host " - $s" -ForegroundColor Yellow } +} + +if ($script:Failed.Count -gt 0) { + Write-Host "" + Write-Host "Failed items:" -ForegroundColor Red + foreach ($f in $script:Failed) { Write-Host " - $f" -ForegroundColor Red } + exit 1 +} + +exit 0 diff --git a/scripts/run-codegen.ps1 b/scripts/run-codegen.ps1 new file mode 100644 index 00000000000..17b61a313c1 --- /dev/null +++ b/scripts/run-codegen.ps1 @@ -0,0 +1,308 @@ +# Run all Keyman Delphi codegen invocations needed for IDE builds. +# +# This is the standalone equivalent of the codegen steps that build.sh would +# run via its cascade -- but build.sh's full cascade is blocked by Delphi 12 +# Community Edition (no CLI builds), so we orchestrate the codegen by hand. +# +# Runs: +# devtools.exe -buildmessageconstants (MessageIdentifierConsts.pas) +# devtools.exe -buildsetupstrings (Keyman.Setup.System.Locales.pas + per-locale .pas) +# devtools.exe -buildlocaleindex (kmshell\locale\index.xml) +# build_standards_data.exe x5 (ISO6393, BCP47 subtag, BCP47 suppress, LCID, langtags) +# tsysinfox64.exe -> tsysinfox64.bin -> tsysinfo_x64.res (the chicken-and-egg) +# +# Prerequisites: devtools.dproj and build_standards_data.dproj must already +# be built in the Delphi IDE (Win32 / Debug). The script exits cleanly with a +# message if either is missing. +# +# Style: continues past individual failures, logs everything, and prints a +# success/fail/skip summary at the end. No emojis. + +$ErrorActionPreference = 'Continue' + +# Auto-detect repo root. Override by setting $env:KEYMAN_ROOT. +if ($env:KEYMAN_ROOT) { + $REPO = $env:KEYMAN_ROOT +} else { + # This script lives at \scripts\, so repo root is one level up. + $REPO = Split-Path $PSScriptRoot -Parent +} + +$DEVTOOLS = "$REPO\common\windows\delphi\tools\devtools\bin\Win32\Debug\devtools.exe" +$BSD = "$REPO\common\windows\delphi\tools\build_standards_data\bin\Win32\Debug\build_standards_data.exe" + +# Counters for the summary at the end. +$script:succeeded = 0 +$script:failed = 0 +$script:skipped = 0 + +function Resolve-RcExe { + # Locate the most recent Windows 10/11 SDK rc.exe, falling back to a known + # version if the SDK install dir cannot be enumerated. + $sdkRoot = 'C:\Program Files (x86)\Windows Kits\10\bin' + if (Test-Path $sdkRoot) { + $candidate = Get-ChildItem $sdkRoot -Directory -ErrorAction SilentlyContinue | + Where-Object { $_.Name -match '^10\.' } | + Sort-Object @{Expression={[version]$_.Name}} -Descending | + ForEach-Object { Join-Path $_.FullName 'x64\rc.exe' } | + Where-Object { Test-Path $_ } | + Select-Object -First 1 + if ($candidate) { return $candidate } + } + $fallback = 'C:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0\x64\rc.exe' + return $fallback +} + +function Invoke-Codegen { + param( + [string]$Label, + [string]$Exe, + [string[]]$ExeArgs, + [string]$ExpectedOutput, + [string]$WorkingDir + ) + + Write-Host "" + Write-Host "--- $Label ---" -ForegroundColor Cyan + Write-Host " exe : $Exe" + if ($ExeArgs) { Write-Host " args: $($ExeArgs -join ' ')" } + if ($WorkingDir) { Write-Host " cwd : $WorkingDir" } + if ($ExpectedOutput) { Write-Host " out : $ExpectedOutput" } + + if (-not (Test-Path $Exe)) { + Write-Host " STATUS: SKIP (exe not found)" -ForegroundColor Yellow + $script:skipped++ + return + } + + $pushed = $false + if ($WorkingDir) { + if (-not (Test-Path $WorkingDir)) { + Write-Host " STATUS: SKIP (working dir missing: $WorkingDir)" -ForegroundColor Yellow + $script:skipped++ + return + } + Push-Location $WorkingDir + $pushed = $true + } + + try { + & $Exe @ExeArgs + $exit = $LASTEXITCODE + } catch { + Write-Host " STATUS: FAIL (exception: $($_.Exception.Message))" -ForegroundColor Red + $script:failed++ + if ($pushed) { Pop-Location } + return + } + if ($pushed) { Pop-Location } + + if ($exit -ne 0) { + Write-Host " STATUS: FAIL (exit $exit)" -ForegroundColor Red + $script:failed++ + return + } + + if ($ExpectedOutput) { + if (Test-Path $ExpectedOutput) { + $size = (Get-Item $ExpectedOutput).Length + Write-Host " STATUS: OK ($size bytes)" -ForegroundColor Green + $script:succeeded++ + } else { + Write-Host " STATUS: FAIL (expected output not found)" -ForegroundColor Red + $script:failed++ + } + } else { + Write-Host " STATUS: OK (exit 0)" -ForegroundColor Green + $script:succeeded++ + } +} + +Write-Host "=== Keyman Delphi codegen ===" +Write-Host "REPO : $REPO" +Write-Host "DEVTOOLS: $DEVTOOLS" +Write-Host "BSD : $BSD" + +# ----------------------------------------------------------------------------- +# Sanity checks +# ----------------------------------------------------------------------------- +if (-not (Test-Path $DEVTOOLS)) { + Write-Host "" + Write-Host "ERROR: devtools.exe not found." -ForegroundColor Red + Write-Host " Expected: $DEVTOOLS" + Write-Host " Build devtools.dproj in Delphi IDE first (Win32 / Debug)." -ForegroundColor Yellow + Write-Host " Project : $REPO\common\windows\delphi\tools\devtools\devtools.dproj" + exit 1 +} + +if (-not (Test-Path $BSD)) { + Write-Host "" + Write-Host "ERROR: build_standards_data.exe not found." -ForegroundColor Red + Write-Host " Expected: $BSD" + Write-Host " Build build_standards_data.dproj in Delphi IDE first (Win32 / Debug)." -ForegroundColor Yellow + Write-Host " Project : $REPO\common\windows\delphi\tools\build_standards_data\build_standards_data.dproj" + exit 1 +} + +# ----------------------------------------------------------------------------- +# devtools.exe codegen +# ----------------------------------------------------------------------------- +Write-Host "" +Write-Host "=== devtools.exe ===" + +# (a) -buildmessageconstants: required by keyman.dproj and kmshell.dproj +$stringsXml = "$REPO\windows\src\desktop\kmshell\xml\strings.xml" +$messageConstantsPas = "$REPO\windows\src\global\delphi\cust\MessageIdentifierConsts.pas" +Invoke-Codegen ` + -Label 'devtools -buildmessageconstants' ` + -Exe $DEVTOOLS ` + -ExeArgs @('-buildmessageconstants', $stringsXml, $messageConstantsPas) ` + -ExpectedOutput $messageConstantsPas + +# (b) -buildsetupstrings: must run FROM the setup dir with relative paths +# "locale\" and ".\" (trailing backslashes are required -- matches +# windows/src/desktop/setup/build.sh:37). Produces +# Keyman.Setup.System.Locales.pas plus ~32 per-locale Keyman.Setup.System.Locale..pas files. +$setupDir = "$REPO\windows\src\desktop\setup" +$setupLocalesPas = "$setupDir\Keyman.Setup.System.Locales.pas" +Invoke-Codegen ` + -Label 'devtools -buildsetupstrings' ` + -Exe $DEVTOOLS ` + -ExeArgs @('-buildsetupstrings', 'locale\', '.\') ` + -WorkingDir $setupDir ` + -ExpectedOutput $setupLocalesPas + +# (c) -buildlocaleindex: produces kmshell\locale\index.xml (matches +# windows/src/desktop/kmshell/build.sh:63-68 do_build_locale_index). +$kmshellLocale = "$REPO\windows\src\desktop\kmshell\locale" +$kmshellLocaleIndex = "$kmshellLocale\index.xml" +Invoke-Codegen ` + -Label 'devtools -buildlocaleindex' ` + -Exe $DEVTOOLS ` + -ExeArgs @('-buildlocaleindex', $kmshellLocale, $kmshellLocaleIndex) ` + -ExpectedOutput $kmshellLocaleIndex + +# ----------------------------------------------------------------------------- +# build_standards_data.exe codegen +# (mirrors common/windows/delphi/tools/build_standards_data/build.sh:41-45) +# ----------------------------------------------------------------------------- +Write-Host "" +Write-Host "=== build_standards_data.exe ===" + +$dataRoot = "$REPO\resources\standards-data" +$outRoot = "$REPO\common\windows\delphi\standards" + +$iso6393Data = "$dataRoot\iso639-3\iso639-3.tab" +$iso6393Pas = "$outRoot\Keyman.System.Standards.ISO6393ToBCP47Registry.pas" +$subtagData = "$dataRoot\language-subtag-registry\language-subtag-registry" +$subtagPas = "$outRoot\Keyman.System.Standards.BCP47SubtagRegistry.pas" +$suppressData = "$dataRoot\language-subtag-registry\language-subtag-registry" +$suppressPas = "$outRoot\Keyman.System.Standards.BCP47SuppressScriptRegistry.pas" +$lcidData = "$dataRoot\windows-lcid-to-bcp-47\map_clean_win.txt" +$lcidPas = "$outRoot\Keyman.System.Standards.LCIDToBCP47Registry.pas" +$langtagsData = "$dataRoot\langtags\langtags.json" +$langtagsPas = "$outRoot\Keyman.System.Standards.LangTagsRegistry.pas" + +Invoke-Codegen ` + -Label 'build_standards_data iso6393' ` + -Exe $BSD ` + -ExeArgs @('iso6393', $iso6393Data, $iso6393Pas) ` + -ExpectedOutput $iso6393Pas + +Invoke-Codegen ` + -Label 'build_standards_data suppress' ` + -Exe $BSD ` + -ExeArgs @('suppress', $suppressData, $suppressPas) ` + -ExpectedOutput $suppressPas + +Invoke-Codegen ` + -Label 'build_standards_data subtag' ` + -Exe $BSD ` + -ExeArgs @('subtag', $subtagData, $subtagPas) ` + -ExpectedOutput $subtagPas + +Invoke-Codegen ` + -Label 'build_standards_data lcid' ` + -Exe $BSD ` + -ExeArgs @('lcid', $lcidData, $lcidPas) ` + -ExpectedOutput $lcidPas + +Invoke-Codegen ` + -Label 'build_standards_data langtags' ` + -Exe $BSD ` + -ExeArgs @('langtags', $langtagsData, $langtagsPas) ` + -ExpectedOutput $langtagsPas + +# ----------------------------------------------------------------------------- +# tsysinfox64.exe -> tsysinfox64.bin -> tsysinfo_x64.res +# +# tsysinfo.dproj (Win32) embeds the Win64 tsysinfox64 exe as a binary resource. +# engine.groupproj lists tsysinfo BEFORE tsysinfox64, so this has to be +# hand-orchestrated: build tsysinfox64 first, then drop its exe in as +# tsysinfox64.bin, then compile the .rc to produce tsysinfo_x64.res, then +# build tsysinfo. +# ----------------------------------------------------------------------------- +Write-Host "" +Write-Host "=== tsysinfox64 -> tsysinfo_x64.res ===" + +$tsysinfox64Exe = "$REPO\windows\src\engine\tsysinfox64\bin\Win64\Debug\tsysinfox64.exe" +$tsysinfoDir = "$REPO\windows\src\engine\tsysinfo" +$tsysinfox64Bin = "$tsysinfoDir\tsysinfox64.bin" +$tsysinfoRc = "$tsysinfoDir\tsysinfo_x64.rc" +$tsysinfoRes = "$tsysinfoDir\tsysinfo_x64.res" + +if (-not (Test-Path $tsysinfox64Exe)) { + Write-Host "" + Write-Host " SKIP: tsysinfox64.exe not built yet -- skipping tsysinfo_x64.res generation." -ForegroundColor Yellow + Write-Host " Build tsysinfox64.dproj in Delphi IDE first (Win64 / Debug)." -ForegroundColor Yellow + Write-Host " Expected: $tsysinfox64Exe" + $script:skipped++ +} elseif (-not (Test-Path $tsysinfoRc)) { + Write-Host "" + Write-Host " SKIP: $tsysinfoRc not found." -ForegroundColor Yellow + $script:skipped++ +} else { + Write-Host "" + Write-Host "--- copy tsysinfox64.exe -> tsysinfox64.bin ---" -ForegroundColor Cyan + Write-Host " src: $tsysinfox64Exe" + Write-Host " dst: $tsysinfox64Bin" + try { + Copy-Item $tsysinfox64Exe $tsysinfox64Bin -Force + $size = (Get-Item $tsysinfox64Bin).Length + Write-Host " STATUS: OK ($size bytes)" -ForegroundColor Green + $script:succeeded++ + } catch { + Write-Host " STATUS: FAIL ($($_.Exception.Message))" -ForegroundColor Red + $script:failed++ + } + + $rc = Resolve-RcExe + Invoke-Codegen ` + -Label 'rc.exe tsysinfo_x64.rc' ` + -Exe $rc ` + -ExeArgs @('/nologo', 'tsysinfo_x64.rc') ` + -WorkingDir $tsysinfoDir ` + -ExpectedOutput $tsysinfoRes +} + +# ----------------------------------------------------------------------------- +# Summary +# ----------------------------------------------------------------------------- +Write-Host "" +Write-Host "=== Summary ===" -ForegroundColor Cyan +Write-Host (" succeeded: {0}" -f $script:succeeded) -ForegroundColor Green +if ($script:failed -gt 0) { + Write-Host (" failed : {0}" -f $script:failed) -ForegroundColor Red +} else { + Write-Host (" failed : {0}" -f $script:failed) +} +if ($script:skipped -gt 0) { + Write-Host (" skipped : {0}" -f $script:skipped) -ForegroundColor Yellow +} else { + Write-Host (" skipped : {0}" -f $script:skipped) +} + +if ($script:failed -gt 0) { + exit 1 +} +exit 0 diff --git a/windows/src/engine/engine.groupproj b/windows/src/engine/engine.groupproj index bb2548b6103..150ea34d9cf 100644 --- a/windows/src/engine/engine.groupproj +++ b/windows/src/engine/engine.groupproj @@ -15,9 +15,6 @@ - - - Default.Personality.12 @@ -62,23 +59,14 @@ - - - - - - - - - - + - + - + From 57bcdc8fbb1733476ee69bb9bbb9f69ea4a68ed5 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 1 Jun 2026 06:47:40 +0000 Subject: [PATCH 3/4] refactor(windows): replace PS helper scripts with CE-aware build.sh interactive mode Marc's review feedback on #16039: PowerShell scripts add maintenance burden and drift from the existing shell-script build chain. Implement the agreed alternative: gate Delphi CE support by version in delphi_msbuild, make it interactive. Changes: - resources/build/win/environment.inc.sh: delphi_msbuild now checks KEYMAN_DELPHI_CE=1; when set, prints an IDE-build prompt and waits for Enter instead of invoking msbuild.exe. Pre-build (rc.exe) and post-build (binary copy, codegen) steps in build.sh run normally around the prompt. - Remove scripts/install-build-env.ps1, scripts/preflight-resources.ps1, scripts/run-codegen.ps1, overlay-dev-builds.ps1. - docs/build/windows-d12.md: rewrite sections 3.1 and 5 to document the shell-based workflow (KEYMAN_DELPHI_CE=1 + build.sh); update canonical build order, running, and troubleshooting sections accordingly. https://claude.ai/code/session_01EqVM8WwyS53pniVUHrBnmR --- docs/build/windows-d12.md | 290 +++++++++------------ overlay-dev-builds.ps1 | 102 -------- resources/build/win/environment.inc.sh | 9 + scripts/install-build-env.ps1 | 63 ----- scripts/preflight-resources.ps1 | 334 ------------------------- scripts/run-codegen.ps1 | 308 ----------------------- 6 files changed, 121 insertions(+), 985 deletions(-) delete mode 100644 overlay-dev-builds.ps1 delete mode 100644 scripts/install-build-env.ps1 delete mode 100644 scripts/preflight-resources.ps1 delete mode 100644 scripts/run-codegen.ps1 diff --git a/docs/build/windows-d12.md b/docs/build/windows-d12.md index 3836cbcd01a..b87c87a70f3 100644 --- a/docs/build/windows-d12.md +++ b/docs/build/windows-d12.md @@ -229,33 +229,41 @@ must be reverted before any PR. > to drop the `inst\` segment or move the file. See section 3.3 and the > `[[delphi-12-local-patches]]` memory. -### 3.1 Build-script: select Delphi version via `KEYMAN_DELPHI_VERSION` +### 3.1 Build-script: select Delphi version and enable CE interactive mode -This branch makes the Delphi version configurable via the -`KEYMAN_DELPHI_VERSION` environment variable. It defaults to `20.0` -(Delphi 10.3, the CI target) so default builds are unaffected. For -Delphi 12 set the env var to `23.0`: +Set both environment variables for Delphi 12 CE: ```powershell # PowerShell (persistent for your user) [Environment]::SetEnvironmentVariable('KEYMAN_DELPHI_VERSION', '23.0', 'User') +[Environment]::SetEnvironmentVariable('KEYMAN_DELPHI_CE', '1', 'User') ``` ```bash # Git Bash (per-shell) export KEYMAN_DELPHI_VERSION=23.0 +export KEYMAN_DELPHI_CE=1 ``` -Without this override, `build.sh` flows that *prepare* the Delphi -environment (without invoking `dcc32`) look for -`C:\Program Files (x86)\Embarcadero\Studio\20.0\bin\rsvars.bat`, which -does not exist on a Delphi-12-only machine. +`KEYMAN_DELPHI_VERSION` selects which Delphi installation `build.sh` looks for +(defaults to `20.0` / Delphi 10.3). Without this override, `build.sh` looks for +`C:\Program Files (x86)\Embarcadero\Studio\20.0\bin\rsvars.bat`, which does not +exist on a Delphi-12-only machine. -This env-var approach is backwards-compatible with the existing Delphi -10.3 CI: when `KEYMAN_DELPHI_VERSION` is unset, both -`resources/build/win/configure_environment.inc.sh` and -`resources/build/win/delphi_environment.inc.sh` fall through to the -historical default of `20.0`. +`KEYMAN_DELPHI_CE=1` switches `delphi_msbuild` into interactive mode. Instead of +invoking `msbuild.exe`, the script pauses and prompts: + +``` +Delphi CE: CLI compilation is not available. +Please build in the Delphi IDE now, then press Enter to continue. +``` + +This means you use the same `build.sh` commands as a Professional Delphi user; the +scripts orchestrate pre-build steps (rc compilation, codegen) and post-build steps +(binary copying) around your IDE builds automatically. + +Both variables are backwards-compatible: when unset or `KEYMAN_DELPHI_CE=0`, all +behavior is identical to the historical Delphi 10.3 CI defaults. ### 3.2 keyman.exe uiAccess strip (LOCAL ONLY, not committed) @@ -270,10 +278,11 @@ $f = 'C:\Projects\keyman\keyman\windows\src\engine\keyman\manifest.in' (Get-Content $f) -replace 'uiAccess="true"', 'uiAccess="false"' | Set-Content $f -Encoding UTF8 ``` -Then regenerate manifest.xml / manifest.res via -`scripts/preflight-resources.ps1`, do a **Clean + Build** of keyman.dproj -in Delphi IDE (see [[delphi-incremental-build-res-cache]] — Build alone -won't re-link the new manifest.res), and run overlay-dev-builds.ps1. +Then re-run `build.sh build` for `windows/src/engine/keyman` (the pre-build +`build_manifest.res` step regenerates manifest.xml / manifest.res before prompting +for the IDE build), do a **Clean + Build** of keyman.dproj in Delphi IDE +(see [[delphi-incremental-build-res-cache]] — Build alone won't re-link the +new manifest.res), and run `windows/src/engine/build.sh install` elevated (section 5.3). **Why:** Windows refuses to launch any unsigned binary that declares `uiAccess="true"`, returning `Access is denied` (error 8235). The overlay @@ -603,7 +612,7 @@ dev kmshell shows "Keyman failed to start" when enabling a keyboard. `kmcomapi.dll` exposes the COM objects kmshell instantiates at startup. Without registration, `CoCreateInstance` returns `REGDB_E_CLASSNOTREG` (`Class not registered`) and kmshell dies before its main form appears. -`overlay-dev-builds.ps1` does this automatically; see section 7. +`windows/src/engine/build.sh install` handles this automatically (section 5.3). ### The canonical build order @@ -612,170 +621,88 @@ For a clean checkout, this is the order that actually works: 1. CLI build everything CLI-buildable: `./core/build.sh build`, `./developer/src/kmcmplib/build.sh build`, TypeScript modules, KeymanWeb. The C++ engine pieces can wait. -2. Open `common/windows/delphi/tools/devtools/devtools.dproj`, build (Win32). -3. Open `common/windows/delphi/tools/build_standards_data/build_standards_data.dproj`, - build (Win32). -4. Run the codegen helper script (section 5.2). -5. Open `engine.groupproj`. Build tsysinfox64 first, run the - tsysinfox64-bin / `tsysinfo_x64.res` step from (c), then build everything - else in the group. -6. Open `desktop.groupproj`. Build kmcomapi (if not already), then setup, - insthelp, kmconfig, kmbrowserhost, kmshell. -7. Open `developer.groupproj`. Build TIKE, kmconvert, setup. -8. Run `overlay-dev-builds.ps1` (elevated) to copy the dev binaries over - the installed Keyman 19. - -## 5. Helper scripts - -Three scripts in the checkout paper over what `build.sh` would normally do -on a Professional Delphi machine. None are committed (they're untracked -local-only files). - -### 5.1 `scripts/preflight-resources.ps1` - -Runs `rc.exe` over every `.rc` file in every Delphi project, producing the -`.res` files that the `.dpr` files `{$R}` reference. Equivalent to the -`build.sh` `_prebuild_resources` phase. - -`rc.exe` lives at -`C:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0\x64\rc.exe`. - -Per-project mapping (paths under `windows\src\`): - -| Project | .rc files | -| --- | --- | -| `engine\keyman\` | version, manifest, icons, keymanmenuitem, osktoolbar, langswitch\langswitchmanager | -| `engine\kmcomapi\` | version, manifest, dialogs (needs SDK includes), kbd_noicon | -| `engine\insthelper\` | version | -| `engine\tsysinfo\` | version, manifest, tsysinfo_x64 (after build order step c) | -| `engine\tsysinfox64\` | version, manifest | -| `desktop\kmshell\` | version, manifest, icons | -| `desktop\kmbrowserhost\` | version, manifest | -| `desktop\kmconfig\` | version, manifest | -| `desktop\insthelp\` | version, manifest | -| `desktop\setup\` | version, manifest, icons (needs SDK includes) | - -> [!NOTE] -> `mcompile` (`windows\src\engine\mcompile\mcompile.vcxproj`) and the other -> C++ engine components -- `keyman32`, `kmtip`, `keymanhp`, `kmrefresh`, -> `testhost` -- are NOT Delphi projects and don't need this preflight step. -> They build via msbuild (driven through their own `build.sh` chain) and -> their `.rc` files are compiled by msbuild as part of the C++ build. - -For `.rc` files that `#include ` (kmcomapi `dialogs.rc`, setup -`icons.rc`, engine `testhost.rc`), pass the Windows SDK include paths: +2. Build devtools: `cd common/windows/delphi/tools/devtools && ./build.sh build` + (prompted to build `devtools.dproj` in IDE; press Enter when done). +3. Build build_standards_data: `cd common/windows/delphi/tools/build_standards_data && ./build.sh build` + (prompted to build `build_standards_data.dproj` in IDE; after Enter, the script + automatically runs all five codegen invocations). +4. Build engine components: `cd windows/src/engine && ./build.sh build` + (prompted for each Delphi project in dependency order; `tsysinfo.dproj` is + prompted after `tsysinfox64.dproj` because `build.sh` handles the dependency). +5. Build desktop components: `cd windows/src/desktop && ./build.sh build` + (prompted for each desktop Delphi project). +6. Build developer components: `cd developer/src && ./build.sh build` + (prompted for TIKE, kmconvert, and developer setup). +7. Overlay dev builds (elevated Git Bash): `cd windows/src/engine && ./build.sh install` + then `cd windows/src/desktop && ./build.sh install` (section 5.3). + +## 5. Shell-based CE workflow + +With `KEYMAN_DELPHI_CE=1` set (section 3.1), the existing `build.sh` scripts +handle the CE constraint interactively. At each step that would normally invoke +`msbuild` through Delphi, the script pauses and prints: -```powershell -& $rc /nologo ` - /I "C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\um" ` - /I "C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\shared" ` - /I "C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\winrt" ` - dialogs.rc +``` +Delphi CE: CLI compilation is not available. +Please build in the Delphi IDE now, then press Enter to continue (or Ctrl-C to abort). ``` -`manifest.rc` references `manifest.xml`, which the script generates from -`manifest.in` by substituting `$VersionWin` with the four-part version from -`VERSION.md`: +Pre-build steps (rc compilation, manifest generation) and post-build steps +(binary copying, codegen) run automatically around your IDE builds. -```powershell -# Four-part version is the contents of VERSION.md with '.0' appended -# (e.g. if VERSION.md is 19.0.241, $ver becomes 19.0.241.0). -$ver = "$((Get-Content C:\Projects\keyman\keyman\VERSION.md).Trim()).0" -$content = (Get-Content manifest.in -Raw) -replace '\$VersionWin', $ver -# Write WITHOUT BOM -- rc.exe / Delphi tolerate it but the preflight script -# does it this way and it avoids surprise diffs. -[System.IO.File]::WriteAllText( - (Join-Path (Get-Location) 'manifest.xml'), - $content, - (New-Object System.Text.UTF8Encoding $false)) -& $rc /nologo manifest.rc -``` +### 5.1 Pre-build resource compilation -> [!NOTE] -> Prefer running `scripts/preflight-resources.ps1` (section 5.1 below) -> rather than the snippet above -- it already does the no-BOM write -> correctly for every project. The snippet is documentation, not a -> recommended workflow. +`build_version.res()` and `build_manifest.res()` use `rc.exe` from the Visual +Studio environment -- they work on CE without modification. They run inside each +project's `build.sh build` before the IDE prompt appears. > [!IMPORTANT] -> After regenerating any `.res`, in Delphi: **right-click the project -> -> Clean, then Build**. An incremental Build silently embeds the stale -> cached `.res` and you'll spend an hour wondering why your icon didn't -> update. +> After building in the IDE following a resource change, use **right-click → +> Clean, then Build** in Delphi. An incremental Build silently embeds the stale +> cached `.res`. -### 5.2 `scripts/run-codegen.ps1` +### 5.2 Codegen (devtools.exe, build_standards_data.exe) -Drives `devtools.exe` and `build_standards_data.exe` to produce the -generated `.pas` files that `engine.groupproj`, `desktop.groupproj`, and -`developer.groupproj` depend on. Must be run after step 3 in section 4 and -before step 5. +Build the tools first via their `build.sh` scripts, which prompt for the IDE build +then run codegen automatically: -```powershell -$DEVTOOLS = 'C:\Projects\keyman\keyman\common\windows\delphi\tools\devtools\bin\Win32\Debug\devtools.exe' - -# (a) MessageIdentifierConsts.pas -- engine/desktop dependency -& $DEVTOOLS -buildmessageconstants ` - C:\Projects\keyman\keyman\windows\src\desktop\kmshell\xml\strings.xml ` - C:\Projects\keyman\keyman\windows\src\global\delphi\cust\MessageIdentifierConsts.pas - -# (b) Per-locale Pascal files for setup.dpr -# Trailing backslashes on both args are required (mirrors -# windows/src/desktop/setup/build.sh:37) -Push-Location C:\Projects\keyman\keyman\windows\src\desktop\setup -& $DEVTOOLS -buildsetupstrings 'locale\' '.\' -Pop-Location +```bash +# Git Bash (KEYMAN_DELPHI_CE=1 and KEYMAN_DELPHI_VERSION=23.0 must be exported) -# (c) Locale index for kmshell support data overlay -& $DEVTOOLS -buildlocaleindex ` - C:\Projects\keyman\keyman\windows\src\desktop\kmshell\locale ` - C:\Projects\keyman\keyman\windows\src\desktop\kmshell\locale\index.xml +# Build devtools: prompted to build devtools.dproj in IDE, then exits +cd "$KEYMAN_ROOT/common/windows/delphi/tools/devtools" +./build.sh build + +# Build build_standards_data: prompted for IDE build, then auto-runs all five +# build_standards_data.exe invocations to generate the registry .pas files +cd "$KEYMAN_ROOT/common/windows/delphi/tools/build_standards_data" +./build.sh build ``` -```powershell -$BSD = 'C:\Projects\keyman\keyman\common\windows\delphi\tools\build_standards_data\bin\Win32\Debug\build_standards_data.exe' -$DATA = 'C:\Projects\keyman\keyman\resources\standards-data' -$OUT = 'C:\Projects\keyman\keyman\common\windows\delphi\standards' - -# Five invocations -- mirrors -# common/windows/delphi/tools/build_standards_data/build.sh:41-45 -& $BSD iso6393 "$DATA\iso639-3\iso639-3.tab" ` - "$OUT\Keyman.System.Standards.ISO6393ToBCP47Registry.pas" -& $BSD suppress "$DATA\language-subtag-registry\language-subtag-registry" ` - "$OUT\Keyman.System.Standards.BCP47SuppressScriptRegistry.pas" -& $BSD subtag "$DATA\language-subtag-registry\language-subtag-registry" ` - "$OUT\Keyman.System.Standards.BCP47SubtagRegistry.pas" -& $BSD lcid "$DATA\windows-lcid-to-bcp-47\map_clean_win.txt" ` - "$OUT\Keyman.System.Standards.LCIDToBCP47Registry.pas" -& $BSD langtags "$DATA\langtags\langtags.json" ` - "$OUT\Keyman.System.Standards.LangTagsRegistry.pas" -``` - -All outputs are `.gitignored` and must be re-run after `git clean -fdx`. - -### 5.3 `overlay-dev-builds.ps1` - -Copies the freshly-built dev binaries from the source tree over the -installed Keyman 19 in `Program Files`, then re-runs -`regsvr32 kmcomapi.dll` so the COM CLSIDs point at the new DLL. - -Because section 3.2 patches `manifest.in` to set `uiAccess="false"`, the -unsigned dev `keyman.exe` is launchable from `Program Files` and the -overlay script **does** copy the dev `keyman.exe` over the installed one -(the `keyman.exe` copy line in `overlay-dev-builds.ps1` is active, not -commented out). The list of overlaid binaries: `kmshell.exe`, `keyman.exe`, -`kmcomapi.dll`, plus any DLLs in the source `windows\bin\` tree that exist -in the corresponding install dir. - -Run **elevated** after each dev build cycle. The final `regsvr32` step is at -lines 87-91 of the script: +The `-buildmessageconstants`, `-buildsetupstrings`, and `-buildlocaleindex` codegen +steps run automatically when you build their dependent projects +(`windows/src/global/delphi`, `windows/src/desktop/setup`, `windows/src/desktop/kmshell`). -```powershell -& regsvr32 /s 'C:\Program Files (x86)\Common Files\Keyman\Keyman Engine\kmcomapi.dll' +All generated `.pas` files are `.gitignored` and must be regenerated after `git clean -fdx`. + +### 5.3 Overlay dev builds + +From an **elevated** Git Bash (right-click Git Bash → Run as administrator): + +```bash +cd "$KEYMAN_ROOT/windows/src/engine" +./build.sh install + +cd "$KEYMAN_ROOT/windows/src/desktop" +./build.sh install ``` -After overlay, launch Keyman via the Start Menu (or the installed -`kmshell.exe` path) -- not from the dev tree -- so the install-time -`TKeymanPaths` lookup finds support files. +`windows/src/engine/kmcomapi/build.sh install` re-registers `kmcomapi.dll` via +`regsvr32` automatically. + +After overlay, launch Keyman via the Start Menu (or the installed `kmshell.exe` +path) -- not from the dev tree -- so `TKeymanPaths.KeymanDesktopInstallPath()` +finds the support files. > [!NOTE] > An alternative workflow -- keep `uiAccess="true"` in `manifest.in` (i.e. @@ -790,10 +717,10 @@ After overlay, launch Keyman via the Start Menu (or the installed Because of `TKeymanPaths.KeymanDesktopInstallPath()`, the dev workflow is: 1. Install Keyman 19 official from keyman.com (once). -2. Build dev binaries in the Delphi IDE (per section 4). -3. Run `scripts/preflight-resources.ps1` if any `.rc` content changed. -4. Run `overlay-dev-builds.ps1` from elevated PowerShell. -5. Launch Keyman from the Start Menu. +2. Build dev binaries via `build.sh build` with `KEYMAN_DELPHI_CE=1` (per section 4). +3. From an elevated Git Bash: `cd windows/src/engine && ./build.sh install`, then + `cd windows/src/desktop && ./build.sh install` (section 5.3). +4. Launch Keyman from the Start Menu. If launching keyman.exe (the engine) directly from the dev `bin\` folder returns `Could not find keyman.exe (error=8235)`, check that section 3.2's @@ -862,8 +789,9 @@ then build tsysinfo. See section 4 (c). ### `F1026 File not found: 'version.res'` or `'manifest.res'` -Preflight resources weren't compiled. Run `scripts/preflight-resources.ps1` -or invoke `rc.exe` manually for the project. +Pre-build resource compilation didn't run. Re-run `build.sh build` for the +project (which calls `build_version.res` / `build_manifest.res` before the IDE +prompt), or invoke `rc.exe` manually from the project directory. ### `F1026 File not found: 'jvcl.inc'` / `'jedi/jedi.inc'` / `'jcl.inc'` @@ -914,20 +842,24 @@ an aborted dev experiment). Install Keyman 19 official from keyman.com. ### `Class not registered ClassID {CF46549D-...}` on kmshell launch -`kmcomapi.dll` isn't registered. Run elevated: +`kmcomapi.dll` isn't registered. From an elevated Git Bash: + +```bash +cd "$KEYMAN_ROOT/windows/src/engine/kmcomapi" +./build.sh install +``` + +This re-registers the DLL automatically. If you want to run `regsvr32` directly: ```powershell & regsvr32 /s 'C:\Program Files (x86)\Common Files\Keyman\Keyman Engine\kmcomapi.dll' ``` -`overlay-dev-builds.ps1` does this automatically; if you bypassed the -overlay, do it by hand. - ### `Could not find keyman.exe (error=8235)` Windows blocked launch of an unsigned `uiAccess="true"` binary. Patch `windows/src/engine/keyman/manifest.in` per section 3.2, Clean + rebuild -`keyman.dproj`, re-run `overlay-dev-builds.ps1`. +`keyman.dproj`, then re-run `windows/src/engine/build.sh install` elevated. --- @@ -942,6 +874,8 @@ unset):** * `resources/build/win/configure_environment.inc.sh` + `delphi_environment.inc.sh` -- `KEYMAN_DELPHI_VERSION` env-var override (default preserves 10.3 behavior). +* `resources/build/win/environment.inc.sh` -- `KEYMAN_DELPHI_CE=1` interactive + mode for `delphi_msbuild` (no-op when unset). * `common/windows/delphi/tools/devtools/SourceRootPath.pas` -- VER350/VER360 cases for `DelphiMajorVersion`. * `common/windows/delphi/web/Keyman.System.HttpServer.Base.pas` -- extend diff --git a/overlay-dev-builds.ps1 b/overlay-dev-builds.ps1 deleted file mode 100644 index 03dad581098..00000000000 --- a/overlay-dev-builds.ps1 +++ /dev/null @@ -1,102 +0,0 @@ -#Requires -RunAsAdministrator -# Overlay dev builds onto an installed Keyman 19. Equivalent of what -# `windows/src/build.sh install:engine install:desktop` would do, but standalone -# (doesn't require the build.sh cascade, which is blocked by Delphi CE). -# -# Run elevated after each Delphi/kmcmplib rebuild. - -$ErrorActionPreference = 'Continue' - -# Auto-detect repo root. Override by setting $env:KEYMAN_ROOT. -if ($env:KEYMAN_ROOT) { - $REPO = $env:KEYMAN_ROOT -} else { - # This script lives at the repo root. - $REPO = $PSScriptRoot -} - -$INSTALL_DESKTOP = 'C:\Program Files (x86)\Keyman\Keyman Desktop' -$INSTALL_DEVELOPER = 'C:\Program Files (x86)\Keyman\Keyman Developer' -$INSTALL_ENGINE = 'C:\Program Files (x86)\Common Files\Keyman\Keyman Engine' - -function Copy-IfExists($src, $dst, $label) { - if (Test-Path $src) { - $dstDir = Split-Path $dst -Parent - if (-not (Test-Path $dstDir)) { - New-Item -ItemType Directory -Path $dstDir -Force | Out-Null - } - Copy-Item $src $dst -Force - Write-Host " [$label] $src -> $dst" - } else { - Write-Host " [$label] SKIP (source missing): $src" -ForegroundColor Yellow - } -} - -function Copy-Tree($src, $dst, $label) { - if (Test-Path $src) { - if (Test-Path $dst) { Remove-Item $dst -Recurse -Force } - Copy-Item $src $dst -Recurse -Force - Write-Host " [$label] $src -> $dst (tree)" - } else { - Write-Host " [$label] SKIP (source tree missing): $src" -ForegroundColor Yellow - } -} - -Write-Host "=== Stopping any running Keyman processes ===" -Get-Process -Name kmshell, keyman, keymanmc, tike, tsysinfo, tsysinfox64, kmbrowserhost, kmconfig -ErrorAction SilentlyContinue | - ForEach-Object { Write-Host " Stopping $($_.Name) PID $($_.Id)"; Stop-Process -Id $_.Id -Force } -Start-Sleep -Milliseconds 800 - -Write-Host "" -Write-Host "=== Engine -> $INSTALL_ENGINE ===" -# NOTE: Dev manifest patched to uiAccess="false" so overlay works. See -# [[delphi-12-local-patches]] memory. Caveat: keyboard injection into -# elevated apps will not work in dev builds until uiAccess is restored -# and binary is signed. -Copy-IfExists "$REPO\windows\src\engine\keyman\bin\Win32\Debug\keyman.exe" "$INSTALL_ENGINE\keyman.exe" 'engine' -Copy-IfExists "$REPO\windows\src\engine\kmcomapi\bin\Win32\Debug\kmcomapi.dll" "$INSTALL_ENGINE\kmcomapi.dll" 'engine' -Copy-IfExists "$REPO\windows\src\engine\insthelper\bin\Win32\Debug\insthelper.dll" "$INSTALL_ENGINE\insthelper.dll" 'engine' -Copy-IfExists "$REPO\windows\src\engine\tsysinfo\bin\Win32\Debug\tsysinfo.exe" "$INSTALL_ENGINE\tsysinfo.exe" 'engine' -Copy-IfExists "$REPO\windows\src\engine\tsysinfox64\bin\Win64\Debug\tsysinfox64.exe" "$INSTALL_ENGINE\tsysinfox64.exe" 'engine' - -# C++ engine components (Win32 + x64 + arm64 where applicable) -Copy-IfExists "$REPO\windows\src\engine\keyman32\bin\Win32\Debug\keyman32.dll" "$INSTALL_ENGINE\keyman32.dll" 'engine-cpp' -Copy-IfExists "$REPO\windows\src\engine\keyman32\bin\x64\Debug\keyman64.dll" "$INSTALL_ENGINE\keyman64.dll" 'engine-cpp' -Copy-IfExists "$REPO\windows\src\engine\kmtip\bin\Win32\Debug\kmtip.dll" "$INSTALL_ENGINE\kmtip.dll" 'engine-cpp' -Copy-IfExists "$REPO\windows\src\engine\kmtip\bin\x64\Debug\kmtip64.dll" "$INSTALL_ENGINE\kmtip64.dll" 'engine-cpp' -Copy-IfExists "$REPO\windows\src\engine\mcompile\bin\Win32\Debug\mcompile.exe" "$INSTALL_ENGINE\mcompile.exe" 'engine-cpp' -Copy-IfExists "$REPO\windows\src\engine\kmrefresh\bin\Win32\Debug\kmrefresh.x86.exe" "$INSTALL_ENGINE\kmrefresh.x86.exe" 'engine-cpp' -Copy-IfExists "$REPO\windows\src\engine\kmrefresh\bin\x64\Debug\kmrefresh.x64.exe" "$INSTALL_ENGINE\kmrefresh.x64.exe" 'engine-cpp' -Copy-IfExists "$REPO\windows\src\engine\keymanhp\bin\x64\Debug\keymanhp.x64.exe" "$INSTALL_ENGINE\keymanhp.x64.exe" 'engine-cpp' - -Write-Host "" -Write-Host "=== Desktop -> $INSTALL_DESKTOP ===" -Copy-IfExists "$REPO\windows\src\desktop\kmshell\bin\Win32\Debug\kmshell.exe" "$INSTALL_DESKTOP\kmshell.exe" 'desktop' -Copy-IfExists "$REPO\windows\src\desktop\kmbrowserhost\bin\Win32\Debug\kmbrowserhost.exe" "$INSTALL_DESKTOP\kmbrowserhost.exe" 'desktop' -Copy-IfExists "$REPO\windows\src\desktop\kmconfig\bin\Win32\Debug\kmconfig.exe" "$INSTALL_DESKTOP\kmconfig.exe" 'desktop' -Copy-IfExists "$REPO\windows\src\desktop\insthelp\bin\Win32\Debug\insthelp.dll" "$INSTALL_DESKTOP\insthelp.dll" 'desktop' - -# kmshell expects its support data alongside the .exe -Copy-Tree "$REPO\windows\src\desktop\kmshell\xml" "$INSTALL_DESKTOP\xml" 'desktop-data' -Copy-Tree "$REPO\windows\src\desktop\kmshell\locale" "$INSTALL_DESKTOP\locale" 'desktop-data' - -Write-Host "" -Write-Host "=== Developer -> $INSTALL_DEVELOPER ===" -if (-not (Test-Path $INSTALL_DEVELOPER)) { - Write-Host " Keyman Developer is not installed. Skipping. (Install from keyman.com if you want TIKE wired up.)" -ForegroundColor Yellow -} else { - Copy-IfExists "$REPO\developer\src\tike\bin\Win32\Debug\tike.exe" "$INSTALL_DEVELOPER\tike.exe" 'developer' - Copy-IfExists "$REPO\developer\src\kmconvert\bin\Win32\Debug\kmconvert.exe" "$INSTALL_DEVELOPER\kmconvert.exe" 'developer' - Copy-IfExists "$REPO\developer\src\kmcmplib\build\x86\debug\src\kmcmplib-19.dll" "$INSTALL_DEVELOPER\kmcmplib-19.dll" 'developer' -} - -Write-Host "" -Write-Host "=== Re-registering kmcomapi.dll at install path ===" -if (Test-Path "$INSTALL_ENGINE\kmcomapi.dll") { - & regsvr32 /s "$INSTALL_ENGINE\kmcomapi.dll" - Write-Host " regsvr32 exit: $LASTEXITCODE (0 = success)" -} - -Write-Host "" -Write-Host "=== Done. Launch kmshell from Start Menu or:" -ForegroundColor Green -Write-Host " & '$INSTALL_DESKTOP\kmshell.exe'" diff --git a/resources/build/win/environment.inc.sh b/resources/build/win/environment.inc.sh index 9408c6660ef..eed75080f28 100644 --- a/resources/build/win/environment.inc.sh +++ b/resources/build/win/environment.inc.sh @@ -100,6 +100,15 @@ tds2dbg() { } delphi_msbuild() { + # When KEYMAN_DELPHI_CE=1 the Delphi Community Edition blocks CLI + # compilation. Pause and prompt the developer to build in the IDE. + if [[ "${KEYMAN_DELPHI_CE:-}" == "1" ]]; then + local project="${1:-}" + builder_echo warning "Delphi CE: CLI compilation is not available." + builder_echo warning "Please build ${project} in the Delphi IDE now, then press Enter to continue (or Ctrl-C to abort)." + read -r _ + return 0 + fi run_in_delphi_env msbuild.exe "$@" "$DELPHI_MSBUILD_FLAG_DEBUG" } diff --git a/scripts/install-build-env.ps1 b/scripts/install-build-env.ps1 deleted file mode 100644 index 146e3cc6586..00000000000 --- a/scripts/install-build-env.ps1 +++ /dev/null @@ -1,63 +0,0 @@ -#Requires -RunAsAdministrator -# Installs the Keyman build environment on Windows (everything except Delphi). -# Logs to \install-build-env.log (gitignored). - -$ErrorActionPreference = 'Continue' -$ProgressPreference = 'SilentlyContinue' - -# Auto-detect repo root. Override by setting $env:KEYMAN_ROOT. -if ($env:KEYMAN_ROOT) { - $REPO = $env:KEYMAN_ROOT -} else { - # This script lives at \scripts\, so repo root is one level up. - $REPO = Split-Path $PSScriptRoot -Parent -} - -$log = Join-Path $REPO 'install-build-env.log' -Start-Transcript -Path $log -Force | Out-Null - -Write-Host "=== Step 1/5: Install Chocolatey ===" -if (-not (Get-Command choco -ErrorAction SilentlyContinue)) { - Set-ExecutionPolicy Bypass -Scope Process -Force - [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072 - iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1')) - $env:PATH = "$env:PATH;$env:ProgramData\chocolatey\bin" -} else { - Write-Host "Chocolatey already installed." -} - -Write-Host "=== Step 2/5: Install base deps (git, jq, python, ninja, pandoc, meson) ===" -choco install -y git jq python ninja pandoc meson - -Write-Host "=== Step 3/5: Install nvm-windows + node 20.16.0 ===" -choco install -y nvm -# refresh PATH so nvm is callable -$env:PATH = [System.Environment]::GetEnvironmentVariable('PATH','Machine') + ';' + [System.Environment]::GetEnvironmentVariable('PATH','User') -if (Get-Command nvm -ErrorAction SilentlyContinue) { - nvm install 20.16.0 - nvm use 20.16.0 -} else { - Write-Host "WARNING: nvm not on PATH after install. Open a new shell and run: nvm install 20.16.0; nvm use 20.16.0" -} - -Write-Host "=== Step 4/5: Install release-build tools (7zip, html-help-workshop, wixtoolset 3.14.1) ===" -choco install -y 7zip html-help-workshop -choco install -y wixtoolset --version=3.14.1 - -Write-Host "=== Step 5/5: Install Visual Studio 2022 Community (Native Desktop) ===" -Write-Host "This is a large install (~10GB) and may take 20-40 minutes." -$vsArgs = '--passive --add Microsoft.VisualStudio.Workload.NativeDesktop ' + - '--add Microsoft.VisualStudio.Component.VC.Tools.ARM64 ' + - '--add Microsoft.VisualStudio.Component.CppBuildInsights ' + - '--add Microsoft.VisualStudio.Component.Debugger.JustInTime ' + - '--add Microsoft.VisualStudio.Component.VC.ASAN ' + - '--add Microsoft.VisualStudio.Component.VC.DiagnosticTools ' + - '--add Microsoft.VisualStudio.Component.VC.TestAdapterForGoogleTest ' + - '--add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 ' + - '--add Microsoft.VisualStudio.Component.Windows11SDK.26100 ' + - '--add Microsoft.VisualStudio.Component.Windows11Sdk.WindowsPerformanceToolkit ' + - '--add Microsoft.VisualStudio.Component.Windows10SDK.19041' -winget install --id=Microsoft.VisualStudio.2022.Community -e --accept-source-agreements --accept-package-agreements --override $vsArgs - -Write-Host "=== Done. Log saved to $log ===" -Stop-Transcript | Out-Null diff --git a/scripts/preflight-resources.ps1 b/scripts/preflight-resources.ps1 deleted file mode 100644 index ccf58996616..00000000000 --- a/scripts/preflight-resources.ps1 +++ /dev/null @@ -1,334 +0,0 @@ -<# -.SYNOPSIS - Generate .res files (and manifest.xml) for every Delphi project in the - Keyman repo, so the projects open and compile cleanly in the Delphi IDE. - -.DESCRIPTION - Delphi 12 Community Edition blocks CLI builds, which means build.sh's - do_build rc-step never runs locally. This script is the standalone - equivalent: walk every directory containing a .dproj, expand manifest.in - -> manifest.xml using VERSION.md, then compile every .rc next to it with - rc.exe from the Windows 10 SDK. Output .res files land alongside their - source .rc. - - Run this: - * After a fresh clone / `git clean`. - * After bumping VERSION.md (so manifest.xml is re-stamped). - * After editing any .rc / .in / referenced .bin (e.g. tsysinfox64.bin). - - After regenerating .res, in Delphi do: right-click project -> Clean, - then Build. An incremental Build silently re-embeds the stale cached .res. - - Codegen steps that depend on Delphi-built tools (devtools.exe, - build_standards_data.exe) live in scripts/run-codegen.ps1, not here. - -.PARAMETER Project - Optional. One or more directories to limit the preflight to. May be a - path relative to the repo (e.g. 'windows\src\engine\keyman') or an - absolute path. If omitted, every dir containing a .dproj is processed. - -.PARAMETER Repo - Repo root. Defaults to $env:KEYMAN_ROOT, then auto-detects from this - script's location (assumes the script lives at \scripts\). - -.EXAMPLE - .\preflight-resources.ps1 - -.EXAMPLE - .\preflight-resources.ps1 -Project windows\src\engine\tsysinfo -#> - -[CmdletBinding()] -param( - [string[]]$Project, - [string]$Repo -) - -$ErrorActionPreference = 'Continue' - -if (-not $Repo) { - if ($env:KEYMAN_ROOT) { $Repo = $env:KEYMAN_ROOT } - else { $Repo = Split-Path $PSScriptRoot -Parent } -} - -if (-not (Test-Path $Repo)) { - Write-Host "Repo root not found: $Repo" -ForegroundColor Red - exit 1 -} - -# ---------------------------------------------------------------------------- -# Locate Windows 10 SDK (rc.exe + um/shared/winrt include dirs) -# ---------------------------------------------------------------------------- - -function Find-WindowsSdk { - $sdkRoot = 'C:\Program Files (x86)\Windows Kits\10' - $includeRoot = Join-Path $sdkRoot 'Include' - if (-not (Test-Path $includeRoot)) { - throw "Windows 10 SDK not found at $sdkRoot" - } - - # Preferred versions first, then newest 10.0.* fallback - $preferred = @('10.0.26100.0', '10.0.19041.0') - $available = Get-ChildItem -Path $includeRoot -Directory | - Where-Object { $_.Name -like '10.0.*' } | - Select-Object -ExpandProperty Name - - $chosen = $null - foreach ($v in $preferred) { - if ($available -contains $v) { $chosen = $v; break } - } - if (-not $chosen) { - # newest by version sort - $chosen = $available | - Sort-Object { [version]($_ -replace '^10\.0\.', '10.0.') } -Descending | - Select-Object -First 1 - } - if (-not $chosen) { - throw "No 10.0.* SDK version found under $includeRoot" - } - - $rcCandidates = @( - (Join-Path $sdkRoot "bin\$chosen\x64\rc.exe"), - (Join-Path $sdkRoot "bin\$chosen\x86\rc.exe") - ) - $rc = $rcCandidates | Where-Object { Test-Path $_ } | Select-Object -First 1 - if (-not $rc) { - throw "rc.exe not found for SDK $chosen (looked in $($rcCandidates -join '; '))" - } - - [pscustomobject]@{ - Version = $chosen - Rc = $rc - IncUm = Join-Path $sdkRoot "Include\$chosen\um" - IncShared = Join-Path $sdkRoot "Include\$chosen\shared" - IncWinrt = Join-Path $sdkRoot "Include\$chosen\winrt" - } -} - -$Sdk = Find-WindowsSdk -Write-Host "Windows SDK: $($Sdk.Version)" -ForegroundColor Cyan -Write-Host "rc.exe: $($Sdk.Rc)" -Write-Host "" - -# ---------------------------------------------------------------------------- -# Version string for manifest.xml ($VersionWin substitution) -# ---------------------------------------------------------------------------- - -$versionFile = Join-Path $Repo 'VERSION.md' -if (-not (Test-Path $versionFile)) { - Write-Host "VERSION.md not found at $versionFile" -ForegroundColor Red - exit 1 -} -$Version4 = (Get-Content $versionFile).Trim() + '.0' -Write-Host "Version: $Version4 (from VERSION.md)" -ForegroundColor Cyan -Write-Host "" - -# ---------------------------------------------------------------------------- -# Counters -# ---------------------------------------------------------------------------- - -$script:Generated = New-Object System.Collections.Generic.List[string] -$script:Skipped = New-Object System.Collections.Generic.List[string] -$script:Failed = New-Object System.Collections.Generic.List[string] - -# ---------------------------------------------------------------------------- -# Helpers -# ---------------------------------------------------------------------------- - -function Test-NeedsSdkIncludes { - param([string]$RcPath) - - if (-not (Test-Path $RcPath)) { return $false } - $needles = @('', '', '') - $content = Get-Content $RcPath -Raw -ErrorAction SilentlyContinue - if (-not $content) { return $false } - foreach ($n in $needles) { - if ($content -match [regex]::Escape($n)) { return $true } - } - return $false -} - -function Write-ManifestXml { - param( - [string]$ProjectDir, - [string]$Version - ) - - $manifestIn = Join-Path $ProjectDir 'manifest.in' - $manifestXml = Join-Path $ProjectDir 'manifest.xml' - if (-not (Test-Path $manifestIn)) { return } - - try { - $raw = Get-Content $manifestIn -Raw - $expanded = $raw -replace '\$VersionWin', $Version - # Write UTF-8 without BOM, no trailing newline - [System.IO.File]::WriteAllText($manifestXml, $expanded, (New-Object System.Text.UTF8Encoding $false)) - $size = (Get-Item $manifestXml).Length - Write-Host (" [manifest] {0} ({1} bytes)" -f $manifestXml, $size) -ForegroundColor Green - $script:Generated.Add($manifestXml) - } catch { - Write-Host " [manifest] FAILED: $manifestIn -> $($_.Exception.Message)" -ForegroundColor Red - $script:Failed.Add("$manifestIn ($($_.Exception.Message))") - } -} - -function Invoke-RcCompile { - param( - [string]$RcPath - ) - - $dir = Split-Path $RcPath -Parent - $base = [System.IO.Path]::GetFileNameWithoutExtension($RcPath) - $resPath = Join-Path $dir "$base.res" - - $needsSdk = Test-NeedsSdkIncludes -RcPath $RcPath - $rcArgs = @('/nologo') - if ($needsSdk) { - $rcArgs += '/I'; $rcArgs += $Sdk.IncUm - $rcArgs += '/I'; $rcArgs += $Sdk.IncShared - $rcArgs += '/I'; $rcArgs += $Sdk.IncWinrt - } - # Pass just the filename so output .res lands in the rc's directory - $rcArgs += (Split-Path $RcPath -Leaf) - - Push-Location $dir - try { - $output = & $Sdk.Rc @rcArgs 2>&1 - $code = $LASTEXITCODE - } finally { - Pop-Location - } - - if ($code -ne 0) { - $msg = ($output | Out-String).Trim() - Write-Host (" [rc] FAILED: {0}" -f $RcPath) -ForegroundColor Red - if ($msg) { - foreach ($line in $msg -split "`r?`n") { - Write-Host " $line" -ForegroundColor Red - } - } - $script:Failed.Add("$RcPath (rc.exe exit $code)") - return - } - - if (Test-Path $resPath) { - $size = (Get-Item $resPath).Length - $tag = if ($needsSdk) { 'rc+sdk' } else { 'rc' } - Write-Host (" [{0,-7}] {1} ({2} bytes)" -f $tag, $resPath, $size) -ForegroundColor Green - $script:Generated.Add($resPath) - } else { - Write-Host " [rc] no .res produced for $RcPath" -ForegroundColor Yellow - $script:Skipped.Add("$RcPath (no .res produced)") - } -} - -function Invoke-Preflight { - param([string]$Dir) - - $rel = $Dir - if ($Dir.StartsWith($Repo, [System.StringComparison]::OrdinalIgnoreCase)) { - $rel = $Dir.Substring($Repo.Length).TrimStart('\','/') - } - Write-Host ("--- {0}" -f $rel) -ForegroundColor White - - # (a) manifest.in -> manifest.xml (must come before manifest.rc compile) - Write-ManifestXml -ProjectDir $Dir -Version $Version4 - - # (b) Compile every .rc under this dir (recursive picks up langswitch\*.rc etc.) - $rcFiles = Get-ChildItem -Path $Dir -Recurse -Filter '*.rc' -File -ErrorAction SilentlyContinue - - if (-not $rcFiles -or $rcFiles.Count -eq 0) { - Write-Host " (no .rc files in this project dir)" -ForegroundColor DarkGray - $script:Skipped.Add("$Dir (no .rc files)") - return - } - - foreach ($rc in $rcFiles) { - # Special case: tsysinfo_x64.rc embeds tsysinfox64.bin which is itself - # the output of building tsysinfox64.dproj in Win64/Debug. If the bin - # isn't present yet, skip with a pointer to the codegen helper. - if ($rc.Name -ieq 'tsysinfo_x64.rc') { - $binPath = Join-Path $rc.DirectoryName 'tsysinfox64.bin' - if (-not (Test-Path $binPath)) { - $msg = "tsysinfox64.bin missing -- build tsysinfox64.dproj (Win64/Debug) first, then run scripts\run-codegen.ps1 (or copy the exe to $binPath)" - Write-Host " [rc] SKIP $($rc.FullName): $msg" -ForegroundColor Yellow - $script:Skipped.Add("$($rc.FullName) ($msg)") - continue - } - } - - Invoke-RcCompile -RcPath $rc.FullName - } -} - -# ---------------------------------------------------------------------------- -# Resolve target project dirs -# ---------------------------------------------------------------------------- - -function Resolve-ProjectArg { - param([string]$Arg) - - if ([System.IO.Path]::IsPathRooted($Arg)) { - return $Arg - } - return (Join-Path $Repo $Arg) -} - -if ($Project -and $Project.Count -gt 0) { - $targetDirs = @() - foreach ($p in $Project) { - $resolved = Resolve-ProjectArg -Arg $p - if (-not (Test-Path $resolved)) { - Write-Host "Project path not found: $resolved" -ForegroundColor Red - $script:Failed.Add("$resolved (path not found)") - continue - } - $targetDirs += (Resolve-Path $resolved).Path - } -} else { - Write-Host "Discovering .dproj files under $Repo ..." -ForegroundColor Cyan - $dprojs = Get-ChildItem -Path $Repo -Recurse -Filter '*.dproj' -File -ErrorAction SilentlyContinue - $targetDirs = $dprojs | - ForEach-Object { $_.DirectoryName } | - Sort-Object -Unique - Write-Host ("Found {0} project dir(s) with .dproj files." -f $targetDirs.Count) -ForegroundColor Cyan - Write-Host "" -} - -if (-not $targetDirs -or $targetDirs.Count -eq 0) { - Write-Host "Nothing to do." -ForegroundColor Yellow - exit 0 -} - -# ---------------------------------------------------------------------------- -# Process each project dir -# ---------------------------------------------------------------------------- - -foreach ($d in $targetDirs) { - Invoke-Preflight -Dir $d -} - -# ---------------------------------------------------------------------------- -# Summary -# ---------------------------------------------------------------------------- - -Write-Host "" -Write-Host "===== Summary =====" -ForegroundColor Cyan -Write-Host (" Generated: {0}" -f $script:Generated.Count) -ForegroundColor Green -Write-Host (" Skipped: {0}" -f $script:Skipped.Count) -ForegroundColor Yellow -Write-Host (" Failed: {0}" -f $script:Failed.Count) -ForegroundColor $(if ($script:Failed.Count) { 'Red' } else { 'Gray' }) - -if ($script:Skipped.Count -gt 0) { - Write-Host "" - Write-Host "Skipped items:" -ForegroundColor Yellow - foreach ($s in $script:Skipped) { Write-Host " - $s" -ForegroundColor Yellow } -} - -if ($script:Failed.Count -gt 0) { - Write-Host "" - Write-Host "Failed items:" -ForegroundColor Red - foreach ($f in $script:Failed) { Write-Host " - $f" -ForegroundColor Red } - exit 1 -} - -exit 0 diff --git a/scripts/run-codegen.ps1 b/scripts/run-codegen.ps1 deleted file mode 100644 index 17b61a313c1..00000000000 --- a/scripts/run-codegen.ps1 +++ /dev/null @@ -1,308 +0,0 @@ -# Run all Keyman Delphi codegen invocations needed for IDE builds. -# -# This is the standalone equivalent of the codegen steps that build.sh would -# run via its cascade -- but build.sh's full cascade is blocked by Delphi 12 -# Community Edition (no CLI builds), so we orchestrate the codegen by hand. -# -# Runs: -# devtools.exe -buildmessageconstants (MessageIdentifierConsts.pas) -# devtools.exe -buildsetupstrings (Keyman.Setup.System.Locales.pas + per-locale .pas) -# devtools.exe -buildlocaleindex (kmshell\locale\index.xml) -# build_standards_data.exe x5 (ISO6393, BCP47 subtag, BCP47 suppress, LCID, langtags) -# tsysinfox64.exe -> tsysinfox64.bin -> tsysinfo_x64.res (the chicken-and-egg) -# -# Prerequisites: devtools.dproj and build_standards_data.dproj must already -# be built in the Delphi IDE (Win32 / Debug). The script exits cleanly with a -# message if either is missing. -# -# Style: continues past individual failures, logs everything, and prints a -# success/fail/skip summary at the end. No emojis. - -$ErrorActionPreference = 'Continue' - -# Auto-detect repo root. Override by setting $env:KEYMAN_ROOT. -if ($env:KEYMAN_ROOT) { - $REPO = $env:KEYMAN_ROOT -} else { - # This script lives at \scripts\, so repo root is one level up. - $REPO = Split-Path $PSScriptRoot -Parent -} - -$DEVTOOLS = "$REPO\common\windows\delphi\tools\devtools\bin\Win32\Debug\devtools.exe" -$BSD = "$REPO\common\windows\delphi\tools\build_standards_data\bin\Win32\Debug\build_standards_data.exe" - -# Counters for the summary at the end. -$script:succeeded = 0 -$script:failed = 0 -$script:skipped = 0 - -function Resolve-RcExe { - # Locate the most recent Windows 10/11 SDK rc.exe, falling back to a known - # version if the SDK install dir cannot be enumerated. - $sdkRoot = 'C:\Program Files (x86)\Windows Kits\10\bin' - if (Test-Path $sdkRoot) { - $candidate = Get-ChildItem $sdkRoot -Directory -ErrorAction SilentlyContinue | - Where-Object { $_.Name -match '^10\.' } | - Sort-Object @{Expression={[version]$_.Name}} -Descending | - ForEach-Object { Join-Path $_.FullName 'x64\rc.exe' } | - Where-Object { Test-Path $_ } | - Select-Object -First 1 - if ($candidate) { return $candidate } - } - $fallback = 'C:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0\x64\rc.exe' - return $fallback -} - -function Invoke-Codegen { - param( - [string]$Label, - [string]$Exe, - [string[]]$ExeArgs, - [string]$ExpectedOutput, - [string]$WorkingDir - ) - - Write-Host "" - Write-Host "--- $Label ---" -ForegroundColor Cyan - Write-Host " exe : $Exe" - if ($ExeArgs) { Write-Host " args: $($ExeArgs -join ' ')" } - if ($WorkingDir) { Write-Host " cwd : $WorkingDir" } - if ($ExpectedOutput) { Write-Host " out : $ExpectedOutput" } - - if (-not (Test-Path $Exe)) { - Write-Host " STATUS: SKIP (exe not found)" -ForegroundColor Yellow - $script:skipped++ - return - } - - $pushed = $false - if ($WorkingDir) { - if (-not (Test-Path $WorkingDir)) { - Write-Host " STATUS: SKIP (working dir missing: $WorkingDir)" -ForegroundColor Yellow - $script:skipped++ - return - } - Push-Location $WorkingDir - $pushed = $true - } - - try { - & $Exe @ExeArgs - $exit = $LASTEXITCODE - } catch { - Write-Host " STATUS: FAIL (exception: $($_.Exception.Message))" -ForegroundColor Red - $script:failed++ - if ($pushed) { Pop-Location } - return - } - if ($pushed) { Pop-Location } - - if ($exit -ne 0) { - Write-Host " STATUS: FAIL (exit $exit)" -ForegroundColor Red - $script:failed++ - return - } - - if ($ExpectedOutput) { - if (Test-Path $ExpectedOutput) { - $size = (Get-Item $ExpectedOutput).Length - Write-Host " STATUS: OK ($size bytes)" -ForegroundColor Green - $script:succeeded++ - } else { - Write-Host " STATUS: FAIL (expected output not found)" -ForegroundColor Red - $script:failed++ - } - } else { - Write-Host " STATUS: OK (exit 0)" -ForegroundColor Green - $script:succeeded++ - } -} - -Write-Host "=== Keyman Delphi codegen ===" -Write-Host "REPO : $REPO" -Write-Host "DEVTOOLS: $DEVTOOLS" -Write-Host "BSD : $BSD" - -# ----------------------------------------------------------------------------- -# Sanity checks -# ----------------------------------------------------------------------------- -if (-not (Test-Path $DEVTOOLS)) { - Write-Host "" - Write-Host "ERROR: devtools.exe not found." -ForegroundColor Red - Write-Host " Expected: $DEVTOOLS" - Write-Host " Build devtools.dproj in Delphi IDE first (Win32 / Debug)." -ForegroundColor Yellow - Write-Host " Project : $REPO\common\windows\delphi\tools\devtools\devtools.dproj" - exit 1 -} - -if (-not (Test-Path $BSD)) { - Write-Host "" - Write-Host "ERROR: build_standards_data.exe not found." -ForegroundColor Red - Write-Host " Expected: $BSD" - Write-Host " Build build_standards_data.dproj in Delphi IDE first (Win32 / Debug)." -ForegroundColor Yellow - Write-Host " Project : $REPO\common\windows\delphi\tools\build_standards_data\build_standards_data.dproj" - exit 1 -} - -# ----------------------------------------------------------------------------- -# devtools.exe codegen -# ----------------------------------------------------------------------------- -Write-Host "" -Write-Host "=== devtools.exe ===" - -# (a) -buildmessageconstants: required by keyman.dproj and kmshell.dproj -$stringsXml = "$REPO\windows\src\desktop\kmshell\xml\strings.xml" -$messageConstantsPas = "$REPO\windows\src\global\delphi\cust\MessageIdentifierConsts.pas" -Invoke-Codegen ` - -Label 'devtools -buildmessageconstants' ` - -Exe $DEVTOOLS ` - -ExeArgs @('-buildmessageconstants', $stringsXml, $messageConstantsPas) ` - -ExpectedOutput $messageConstantsPas - -# (b) -buildsetupstrings: must run FROM the setup dir with relative paths -# "locale\" and ".\" (trailing backslashes are required -- matches -# windows/src/desktop/setup/build.sh:37). Produces -# Keyman.Setup.System.Locales.pas plus ~32 per-locale Keyman.Setup.System.Locale..pas files. -$setupDir = "$REPO\windows\src\desktop\setup" -$setupLocalesPas = "$setupDir\Keyman.Setup.System.Locales.pas" -Invoke-Codegen ` - -Label 'devtools -buildsetupstrings' ` - -Exe $DEVTOOLS ` - -ExeArgs @('-buildsetupstrings', 'locale\', '.\') ` - -WorkingDir $setupDir ` - -ExpectedOutput $setupLocalesPas - -# (c) -buildlocaleindex: produces kmshell\locale\index.xml (matches -# windows/src/desktop/kmshell/build.sh:63-68 do_build_locale_index). -$kmshellLocale = "$REPO\windows\src\desktop\kmshell\locale" -$kmshellLocaleIndex = "$kmshellLocale\index.xml" -Invoke-Codegen ` - -Label 'devtools -buildlocaleindex' ` - -Exe $DEVTOOLS ` - -ExeArgs @('-buildlocaleindex', $kmshellLocale, $kmshellLocaleIndex) ` - -ExpectedOutput $kmshellLocaleIndex - -# ----------------------------------------------------------------------------- -# build_standards_data.exe codegen -# (mirrors common/windows/delphi/tools/build_standards_data/build.sh:41-45) -# ----------------------------------------------------------------------------- -Write-Host "" -Write-Host "=== build_standards_data.exe ===" - -$dataRoot = "$REPO\resources\standards-data" -$outRoot = "$REPO\common\windows\delphi\standards" - -$iso6393Data = "$dataRoot\iso639-3\iso639-3.tab" -$iso6393Pas = "$outRoot\Keyman.System.Standards.ISO6393ToBCP47Registry.pas" -$subtagData = "$dataRoot\language-subtag-registry\language-subtag-registry" -$subtagPas = "$outRoot\Keyman.System.Standards.BCP47SubtagRegistry.pas" -$suppressData = "$dataRoot\language-subtag-registry\language-subtag-registry" -$suppressPas = "$outRoot\Keyman.System.Standards.BCP47SuppressScriptRegistry.pas" -$lcidData = "$dataRoot\windows-lcid-to-bcp-47\map_clean_win.txt" -$lcidPas = "$outRoot\Keyman.System.Standards.LCIDToBCP47Registry.pas" -$langtagsData = "$dataRoot\langtags\langtags.json" -$langtagsPas = "$outRoot\Keyman.System.Standards.LangTagsRegistry.pas" - -Invoke-Codegen ` - -Label 'build_standards_data iso6393' ` - -Exe $BSD ` - -ExeArgs @('iso6393', $iso6393Data, $iso6393Pas) ` - -ExpectedOutput $iso6393Pas - -Invoke-Codegen ` - -Label 'build_standards_data suppress' ` - -Exe $BSD ` - -ExeArgs @('suppress', $suppressData, $suppressPas) ` - -ExpectedOutput $suppressPas - -Invoke-Codegen ` - -Label 'build_standards_data subtag' ` - -Exe $BSD ` - -ExeArgs @('subtag', $subtagData, $subtagPas) ` - -ExpectedOutput $subtagPas - -Invoke-Codegen ` - -Label 'build_standards_data lcid' ` - -Exe $BSD ` - -ExeArgs @('lcid', $lcidData, $lcidPas) ` - -ExpectedOutput $lcidPas - -Invoke-Codegen ` - -Label 'build_standards_data langtags' ` - -Exe $BSD ` - -ExeArgs @('langtags', $langtagsData, $langtagsPas) ` - -ExpectedOutput $langtagsPas - -# ----------------------------------------------------------------------------- -# tsysinfox64.exe -> tsysinfox64.bin -> tsysinfo_x64.res -# -# tsysinfo.dproj (Win32) embeds the Win64 tsysinfox64 exe as a binary resource. -# engine.groupproj lists tsysinfo BEFORE tsysinfox64, so this has to be -# hand-orchestrated: build tsysinfox64 first, then drop its exe in as -# tsysinfox64.bin, then compile the .rc to produce tsysinfo_x64.res, then -# build tsysinfo. -# ----------------------------------------------------------------------------- -Write-Host "" -Write-Host "=== tsysinfox64 -> tsysinfo_x64.res ===" - -$tsysinfox64Exe = "$REPO\windows\src\engine\tsysinfox64\bin\Win64\Debug\tsysinfox64.exe" -$tsysinfoDir = "$REPO\windows\src\engine\tsysinfo" -$tsysinfox64Bin = "$tsysinfoDir\tsysinfox64.bin" -$tsysinfoRc = "$tsysinfoDir\tsysinfo_x64.rc" -$tsysinfoRes = "$tsysinfoDir\tsysinfo_x64.res" - -if (-not (Test-Path $tsysinfox64Exe)) { - Write-Host "" - Write-Host " SKIP: tsysinfox64.exe not built yet -- skipping tsysinfo_x64.res generation." -ForegroundColor Yellow - Write-Host " Build tsysinfox64.dproj in Delphi IDE first (Win64 / Debug)." -ForegroundColor Yellow - Write-Host " Expected: $tsysinfox64Exe" - $script:skipped++ -} elseif (-not (Test-Path $tsysinfoRc)) { - Write-Host "" - Write-Host " SKIP: $tsysinfoRc not found." -ForegroundColor Yellow - $script:skipped++ -} else { - Write-Host "" - Write-Host "--- copy tsysinfox64.exe -> tsysinfox64.bin ---" -ForegroundColor Cyan - Write-Host " src: $tsysinfox64Exe" - Write-Host " dst: $tsysinfox64Bin" - try { - Copy-Item $tsysinfox64Exe $tsysinfox64Bin -Force - $size = (Get-Item $tsysinfox64Bin).Length - Write-Host " STATUS: OK ($size bytes)" -ForegroundColor Green - $script:succeeded++ - } catch { - Write-Host " STATUS: FAIL ($($_.Exception.Message))" -ForegroundColor Red - $script:failed++ - } - - $rc = Resolve-RcExe - Invoke-Codegen ` - -Label 'rc.exe tsysinfo_x64.rc' ` - -Exe $rc ` - -ExeArgs @('/nologo', 'tsysinfo_x64.rc') ` - -WorkingDir $tsysinfoDir ` - -ExpectedOutput $tsysinfoRes -} - -# ----------------------------------------------------------------------------- -# Summary -# ----------------------------------------------------------------------------- -Write-Host "" -Write-Host "=== Summary ===" -ForegroundColor Cyan -Write-Host (" succeeded: {0}" -f $script:succeeded) -ForegroundColor Green -if ($script:failed -gt 0) { - Write-Host (" failed : {0}" -f $script:failed) -ForegroundColor Red -} else { - Write-Host (" failed : {0}" -f $script:failed) -} -if ($script:skipped -gt 0) { - Write-Host (" skipped : {0}" -f $script:skipped) -ForegroundColor Yellow -} else { - Write-Host (" skipped : {0}" -f $script:skipped) -} - -if ($script:failed -gt 0) { - exit 1 -} -exit 0 From a530ccdfbbc0509107e69c172b32f38315fdf350 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 1 Jun 2026 12:17:47 +0000 Subject: [PATCH 4/4] fix(windows): respect KEYMAN_DELPHI_VERSION in builder delphi tool detection builder_describe_platform's delphi-tool check was hardcoded to Studio\20.0, so machines with only Delphi 12 (23.0) installed had all win,delphi-gated targets silently skipped even when KEYMAN_DELPHI_VERSION=23.0 was set. Update the detection to use ${KEYMAN_DELPHI_VERSION:-20.0}, matching the same default used in configure_environment.inc.sh and delphi_environment.inc.sh. CI (unset) continues to check for 20.0 unchanged; Delphi 12 Professional users set KEYMAN_DELPHI_VERSION=23.0 and the full win,delphi build graph is enabled. CE users additionally set KEYMAN_DELPHI_CE=1 for interactive delphi_msbuild. Update windows-d12.md section 3.1 to document both Professional and CE cases. https://claude.ai/code/session_01EqVM8WwyS53pniVUHrBnmR --- docs/build/windows-d12.md | 35 +++++++++++++++++++++-------------- resources/builder.inc.sh | 3 ++- 2 files changed, 23 insertions(+), 15 deletions(-) diff --git a/docs/build/windows-d12.md b/docs/build/windows-d12.md index b87c87a70f3..b7e740c6b0e 100644 --- a/docs/build/windows-d12.md +++ b/docs/build/windows-d12.md @@ -229,28 +229,34 @@ must be reverted before any PR. > to drop the `inst\` segment or move the file. See section 3.3 and the > `[[delphi-12-local-patches]]` memory. -### 3.1 Build-script: select Delphi version and enable CE interactive mode +### 3.1 Build-script: select Delphi version -Set both environment variables for Delphi 12 CE: +`KEYMAN_DELPHI_VERSION` selects which Delphi installation `build.sh` and the +builder tool-detection look for. It defaults to `20.0` (Delphi 10.3, the CI +target), so unset == existing CI behavior. + +For any Delphi 12 installation (Professional or CE), set it to `23.0`: ```powershell # PowerShell (persistent for your user) [Environment]::SetEnvironmentVariable('KEYMAN_DELPHI_VERSION', '23.0', 'User') -[Environment]::SetEnvironmentVariable('KEYMAN_DELPHI_CE', '1', 'User') ``` ```bash # Git Bash (per-shell) export KEYMAN_DELPHI_VERSION=23.0 -export KEYMAN_DELPHI_CE=1 ``` -`KEYMAN_DELPHI_VERSION` selects which Delphi installation `build.sh` looks for -(defaults to `20.0` / Delphi 10.3). Without this override, `build.sh` looks for -`C:\Program Files (x86)\Embarcadero\Studio\20.0\bin\rsvars.bat`, which does not -exist on a Delphi-12-only machine. +Without this, `build.sh` looks for +`C:\Program Files (x86)\Embarcadero\Studio\20.0\bin\rsvars.bat` and the builder +platform system looks for `Studio\20.0\bin\dcc32.exe` to decide whether to include +`win,delphi`-gated targets -- both fail silently on a Delphi-12-only machine. -`KEYMAN_DELPHI_CE=1` switches `delphi_msbuild` into interactive mode. Instead of +**Delphi 12 Professional** — setting `KEYMAN_DELPHI_VERSION=23.0` is sufficient. +CLI compilation works normally; `build.sh` targets run end-to-end. + +**Delphi 12 Community Edition** — additionally set `KEYMAN_DELPHI_CE=1`. CE blocks +CLI compilation, so `delphi_msbuild` switches to interactive mode: instead of invoking `msbuild.exe`, the script pauses and prompts: ``` @@ -258,12 +264,13 @@ Delphi CE: CLI compilation is not available. Please build in the Delphi IDE now, then press Enter to continue. ``` -This means you use the same `build.sh` commands as a Professional Delphi user; the -scripts orchestrate pre-build steps (rc compilation, codegen) and post-build steps -(binary copying) around your IDE builds automatically. +```powershell +[Environment]::SetEnvironmentVariable('KEYMAN_DELPHI_CE', '1', 'User') +``` -Both variables are backwards-compatible: when unset or `KEYMAN_DELPHI_CE=0`, all -behavior is identical to the historical Delphi 10.3 CI defaults. +```bash +export KEYMAN_DELPHI_CE=1 +``` ### 3.2 keyman.exe uiAccess strip (LOCAL ONLY, not committed) diff --git a/resources/builder.inc.sh b/resources/builder.inc.sh index 6ec2fbcebff..7e9ea36a0e4 100755 --- a/resources/builder.inc.sh +++ b/resources/builder.inc.sh @@ -2323,7 +2323,8 @@ builder_describe_platform() { # Detect delphi compiler (see also delphi_environment.inc.sh) if builder_is_windows; then local ProgramFilesx86="$(cygpath -w -F 42)" - if [[ -x "$(cygpath -u "$ProgramFilesx86\\Embarcadero\\Studio\\20.0\\bin\\dcc32.exe")" ]]; then + local _delphi_version="${KEYMAN_DELPHI_VERSION:-20.0}" + if [[ -x "$(cygpath -u "$ProgramFilesx86\\Embarcadero\\Studio\\$_delphi_version\\bin\\dcc32.exe")" ]]; then builder_installed_tools+=(delphi) fi fi