diff --git a/.github/release-examples/workspace-preview/release.json b/.github/release-examples/workspace-preview/release.json index d8ffa5c..82f8abe 100644 --- a/.github/release-examples/workspace-preview/release.json +++ b/.github/release-examples/workspace-preview/release.json @@ -14,7 +14,8 @@ "requiredFeatures": [ "manifest/v1", "composition/v1", - "manifest-selection/v1" + "manifest-selection/v1", + "typed-input-maps/v1" ] }, "include": [ diff --git a/docs/aio-releases.md b/docs/aio-releases.md index bf11a56..1b8c2ee 100644 --- a/docs/aio-releases.md +++ b/docs/aio-releases.md @@ -77,8 +77,8 @@ parameters: Site values win over the release settings and template defaults. Values must be strings and are visible in Azure, not secret storage. See -[configure the install Site](../workspaces/iot-operations/manifests/aio-install/README.md#configure-the-site) -for all extension override maps and the saved Site route for guided users. +[Arc extension settings](../workspaces/iot-operations/manifests/aio-install/README.md#arc-extension-settings) +for all extension override maps and the typed inputs that set them. ## Pinning a Site to an AIO release @@ -167,7 +167,7 @@ This applies to every resource area a Site selects through `properties.resourceS ### Supported upgrade paths -Azure IoT Operations supports upgrade to any patch of the same minor version, or to the next minor version. Other transitions (downgrades, jumps across several minor versions, and moves between preview and GA) require uninstall and reinstall. See [Upgrade Azure IoT Operations](https://learn.microsoft.com/azure/iot-operations/deploy-iot-ops/howto-upgrade) for the authoritative rules. +Azure IoT Operations supports upgrade to any patch of the same minor version, or to the next minor version. Move across several minor versions one minor version at a time, for example from a 1.2.x release through a 1.3.x release to a 1.4.x release. A downgrade, or a move between preview and GA in either direction, requires uninstall and reinstall. See [Upgrade Azure IoT Operations](https://learn.microsoft.com/azure/iot-operations/manage-iot-ops/howto-upgrade) for the authoritative rules, and the [upgrade operation guide](../workspaces/iot-operations/manifests/aio-upgrade/README.md#choose-a-supported-upgrade-path) for the minor version of each AIO release and the broker redundancy requirement. The optional E2E workflow can exercise selected upgrades between adjacent AIO releases, such as `2608` to `2609`. A passing dispatch establishes its selected diff --git a/docs/guided-inputs.md b/docs/guided-inputs.md index b0538cc..1e5d58e 100644 --- a/docs/guided-inputs.md +++ b/docs/guided-inputs.md @@ -9,7 +9,9 @@ separate plan command. |---|---| | Install AIO on an existing Arc cluster | [AIO installation](#install-aio-from-an-existing-cluster) | | Install AIO and enable Secret Sync together | [Combined installation](#install-aio-with-secret-sync) | +| Set Arc extension configuration | [Arc extension settings](#set-arc-extension-settings) | | Enable Secret Sync without reinstalling AIO | [Enablement on an existing instance](#enable-secret-sync-on-an-existing-instance) | +| Upgrade an existing instance | [Upgrade with typed inputs](#upgrade-an-existing-instance) | Install a compatible Site Ops build and select a release containing the complete AIO workspace and its typed input contract. The source and release @@ -129,7 +131,9 @@ For a short command without secrets, supply the same named answers with repeated the input file. `--input-file` selects exactly one answer file. Repeating that option is an error, not a way to select several Sites. Strings and strict `true` or `false` booleans are parsed -according to the selected contract. Do not put secrets in process arguments +according to the selected contract. A map input takes one `NAME=KEY=VALUE` +entry for each key, as [Arc extension settings](#set-arc-extension-settings) +describes. Do not put secrets in process arguments or shell history. Input contracts that declare sensitive inputs are rejected. Duplicate and unknown answer names fail. @@ -174,6 +178,70 @@ experience, give your account a Key Vault data role as [vault access](secret-sync.md#give-your-account-access-to-the-vault) describes. +## Set Arc extension settings + +`aioExtensionSettings`, `secretStoreExtensionSettings` and +`certManagerExtensionSettings` set the configuration of the AIO, secret +store and cert-manager Arc extensions. Each is an optional map of string +keys and string values. Inline, supply one entry for each key as +`NAME=KEY=VALUE`: + +```text +siteops deploy aio-install --source "official@" --input "cluster=" --input "aioExtensionSettings=observability.metrics.exportIntervalSeconds=60" +``` + +The input name ends at the first `=` and the key ends at the second, so the +value can contain `=`. An empty value after the second `=` sets an empty +string. Keys that contain `=` need the answer file, which holds the same +maps. Quote keys and values there, because YAML reads an unquoted `on` as +true and `1.10` as 1.1: + +```yaml +apiVersion: siteops.inputs/v1 +kind: SiteInputValues +values: + cluster: "" + aioExtensionSettings: + "observability.metrics.exportIntervalSeconds": "60" +``` + +Inline entries replace answer file entries key by key, and other file +entries remain. A key repeated inline fails, even with the same value. +Keys and values must be text. Numbers, booleans, null, lists and nested maps +fail rather than being converted, so quote each key and value, and write a +nested setting as one dotted key. Keys must also be printable: no control, +format, surrogate, private use or unassigned characters, and no line or +paragraph separators. Which characters are unassigned depends on the Unicode +version of the engine's Python. Each map, from the file, inline or combined, +holds at most 64 entries, with keys of 1 to 256 characters, values of at +most 4,096 characters, and at most 64 KiB of keys and values together. +Errors name the input and, unless output is redacted, the key, never the +value. The settings appear in answer files, shell history, the Site preview +and Azure, so never put secrets in them. The private Site preview shows `***` +for values whose names look sensitive, such as the whole +`secretStoreConfigurationOverrides` map and keys that contain `token`. A +saved Site keeps the real values. + +An omitted map or `{}` adds no settings. With `aio-upgrade`, the settings +already on the cluster stay in place and no answer removes a key. +Installation writes each extension's settings without reading the +cluster's, so change the settings of an existing installation with +[aio-upgrade](../workspaces/iot-operations/manifests/aio-upgrade/README.md#change-extension-settings) +rather than by installing again. Typed answers build a new Site each time, +so they never change a saved Site. `certManagerExtensionSettings` +applies only while `enableCertManager` is true. Supplying it otherwise, even +as `{}`, fails like any inactive input. The secret store settings apply +whether or not Secret Sync is enabled. See +[Arc extension settings](../workspaces/iot-operations/manifests/aio-install/README.md#arc-extension-settings) +for the Site parameters these inputs set and the settings the installation +derives. + +These typed maps need a Site Ops engine that supports them. An older engine +refuses a package that uses them with +`This package requires unsupported Site Ops features.` Install the engine +that the content release names. Configured Sites and complete Site files +set the same maps under `parameters` without typed inputs. + ## Manual Site values without resource reads Omit `cluster` and provide `siteName`, `subscription`, `resourceGroup`, @@ -222,6 +290,32 @@ still updates the default Secret Provider Class binding. It does not prove that a secret materialized in Kubernetes. See the [Secret Sync reference](secret-sync.md) for preservation, vault permissions and functional checks. +## Upgrade an existing instance + +To upgrade one existing instance without a saved Site, supply its resource +ID, the target AIO release and whether this upgrade includes the existing +cert-manager Arc extension: + +```text +siteops deploy aio-upgrade --source "official@" --input "instance=" --input aioRelease=2609 --input "enableCertManager=" +``` + +`aioRelease` and `enableCertManager` have no defaults. Answer +`enableCertManager=true` when Scale Kit installed cert-manager and manages +it, as an installation with the default `enableCertManager` does. Answer +`false` when another owner manages cert-manager, and the upgrade neither +reads nor updates that extension. Site Ops reads the instance, its custom +location and that location's Arc cluster, as for Secret Sync enablement, +but checks no OIDC or workload identity prerequisite. The extension +settings inputs apply here too. Choose a supported transition from the +version the instance runs, never an older release, as the +[upgrade guide](../workspaces/iot-operations/manifests/aio-upgrade/README.md#choose-a-supported-upgrade-path) +describes. + +If you saved the installation's Site, change `properties.aioRelease` in that +file and upgrade with `--site-file` or `-l name=` instead. Typed +answers never change a saved Site. + ## Check the result A successful Site Ops result means the deployment operations completed. @@ -306,8 +400,9 @@ Site must be complete and cannot inherit from packaged example Sites. Configured Sites can continue to use the existing inheritance and overlay rules. -Settings outside the typed inputs belong in the saved Site, such as -[AIO extension configuration overrides](../workspaces/iot-operations/manifests/aio-install/README.md#configure-the-site). +Settings outside the typed inputs belong in the saved Site, such as a +[trust bundle you manage](../workspaces/iot-operations/manifests/aio-install/README.md#settings-scale-kit-derives). +Typed answers never change a saved Site, so edit the file itself. Use one resource group per AIO instance. A Site built from a cluster ID takes that cluster's resource group, so connect each Arc cluster in its own diff --git a/docs/manifest-reference.md b/docs/manifest-reference.md index 0714979..1dd9651 100644 --- a/docs/manifest-reference.md +++ b/docs/manifest-reference.md @@ -125,8 +125,8 @@ inputs: ``` `siteDefaults` may hold only `labels`, `properties`, and `parameters`. -Each ordinary `inputs` row declares one semantic name, a `string` or -`boolean` type, description, and a destination under `name`, +Each ordinary `inputs` row declares one semantic name, a `string`, +`boolean` or `stringMap` type, description, and a destination under `name`, `subscription`, `resourceGroup`, `location`, `labels`, `parameters`, or `properties`. An input without a default is required unless `required: false` is @@ -143,6 +143,31 @@ The DNS label format accepts lowercase letters, digits and interior hyphens, with at most 63 characters. `maxLength` supplies a further string limit. These constraints apply to defaults, file values and inline overrides. +A `stringMap` input writes a mapping of string keys to string values under +`parameters` or `properties`: + +```yaml +- name: settings + type: stringMap + required: false + description: Extension configuration settings. + sitePath: parameters.settingsOverrides +``` + +It declares no default, and `required: true` means at least one entry. Its +path cannot overlap `siteDefaults` or another input, and it cannot control a +condition. An omitted map or `{}` writes nothing. Operators supply one inline +entry for each key as `--input NAME=KEY=VALUE`, and inline keys replace answer +file keys one by one. Keys and values are never converted from other YAML +types. Keys must be printable text, with no control, format, surrogate, +private use or unassigned characters and no line or paragraph separators. +Which characters are unassigned depends on the Unicode version of the +engine's Python. Each supplied map and the combined map hold at most 64 +entries, keys of 1 to 256 characters, values of at most 4,096 characters +and at most 64 KiB of UTF-8 keys and values. A package whose contract uses +`stringMap` must declare the required feature `typed-input-maps/v1`, which +the package producer enforces. + A named `azureResourceId` input can instead derive values for existing semantic inputs. This is an optional read, not a source of Azure credentials or arbitrary provider commands. For example, after declaring diff --git a/docs/migrating.md b/docs/migrating.md index 11981a2..9492f3a 100644 --- a/docs/migrating.md +++ b/docs/migrating.md @@ -47,6 +47,12 @@ These changes affect how you install or run Site Ops after v1.0.0b7. | Install Site Ops on Windows with the PowerShell bootstrap | Review the [saved user PATH](#windows-user-path). | | Match artifact verification error text, or read `source` from failed `browse --output json` | Review [artifact verification messages](#artifact-verification-messages). | | Read `diagnostics` from `validate`, `plan` or `deploy` JSON, or render it in a pipeline | Review [warnings in JSON results](#warnings-in-json-results). | +| Want Arc extension settings without a saved Site | Nothing changes for existing use. Review [typed extension settings](#typed-extension-settings). | +| Upgrade an existing AIO instance that has no configured Site | Nothing changes for existing use. Review the [typed AIO upgrade](#typed-aio-upgrade). | +| Run v1.0.0b8 content with an earlier Site Ops engine, or produce your own packages | Install the matching engine. Review the [typed map feature](#typed-map-feature). | +| Write the YAML key `=` twice in one mapping, quoted or bare | Keep one. Review [repeated equals keys](#repeated-equals-keys). | +| Match the text of a missing typed input error | Review [missing input messages](#missing-input-messages). | +| Match the text of a template error from `plan` or `deploy` | Review [template error messages](#template-error-messages). | ### Windows user PATH @@ -110,6 +116,80 @@ If your automation treats every diagnostic record as a failure, check your templates to show the occurrence count. The v1.0.0b7 renderer shows the same warnings as single records. See [warnings in results](run-output.md#read-warnings). +### Typed extension settings + +This change is additive. `aio-install` accepts three optional typed inputs: +`aioExtensionSettings`, `secretStoreExtensionSettings` and +`certManagerExtensionSettings`. They write the existing Site parameters +`aioConfigurationOverrides`, `secretStoreConfigurationOverrides` and +`certManagerConfigurationOverrides`. Repeat `--input NAME=KEY=VALUE` once for +each setting, or put the maps in the answer file. See +[Arc extension settings](guided-inputs.md#set-arc-extension-settings). + +Existing Sites, overlays, `SITE_OVERRIDES`, parameter files and answer files +work unchanged. Scalar `--input NAME=VALUE` answers keep their meaning, +including values that contain `=`. Bundled AIO release files still set +`certManagerConfigurationOverrides`, and Site keys still replace release +keys one by one. + +### Typed AIO upgrade + +This change is additive. `aio-upgrade` accepts typed inputs for one existing +instance: its resource ID, the required `aioRelease` and `enableCertManager` +answers, the extension settings maps and optional labels. Runs with +configured Sites and `--site-file` are unchanged. See +[upgrade an existing instance](guided-inputs.md#upgrade-an-existing-instance). + +### Typed map feature + +The typed maps are additive for existing Sites and answers, and they need a +Site Ops engine that supports the package feature `typed-input-maps/v1`. An +earlier engine refuses a package that declares it, on every route, with +`This package requires unsupported Site Ops features.` Install the Site Ops +engine that the content release names. With a local checkout, typed routes on +an earlier engine fail with `type must be string, boolean or azureResourceId`, +while configured Sites and `--site-file` runs read no input contract. + +If you produce your own packages, add `typed-input-maps/v1` to +`requiredFeatures`, or `--require-feature typed-input-maps/v1`, for a +workspace whose input contract declares a `stringMap` input. The producer +rejects such a workspace without it. See +[workspace packages](workspace-packages.md#build-workspaces-declared-by-a-release). + +### Repeated equals keys + +YAML reads a bare `=` key and a quoted `"="` key as the same key. Site Ops +now rejects a mapping that repeats it in any spelling, as it already rejects +other repeated keys, with +`found a duplicate key '='. YAML keeps the last one, so the first is discarded. Merge them or rename one.` +Earlier releases kept only the last value. Keep the entry you intend and +remove the other. + +### Missing input messages + +A missing required typed input now ends with the `--input` form that +supplies it, for example: + +```text +Missing required input 'siteName'. Supply `--input siteName=`. +``` + +A boolean input shows `` instead of ``. The error code +stays `inputs.invalid`, so match the code rather than the text. + +### Template error messages + +An error for a template that is unresolved, unsupported or malformed in a +parameter value names the value's location and the supported syntax. It no +longer repeats the value: + +```text +Plan value at 'munich-dev.aio-instance.parameters.aioConfigurationOverrides.observability.metrics.exportIntervalSeconds' contains an unresolved or unsupported template. Write `{{ site. }}` or `{{ steps..outputs. }}` with one space inside each pair of braces, and name a field the Site sets. Remove `{{` from text that is not a template. +``` + +The error code stays `operation-preparation.invalid`, so match the code +rather than the text. + ## To v1.0.0b7 These changes affect workspaces, scripts and pipelines that worked with diff --git a/docs/workspace-packages.md b/docs/workspace-packages.md index 11d7700..c20e124 100644 --- a/docs/workspace-packages.md +++ b/docs/workspace-packages.md @@ -35,6 +35,7 @@ python scripts\build-workspace-package.py ` --require-feature manifest/v1 ` --require-feature composition/v1 ` --require-feature manifest-selection/v1 ` + --require-feature typed-input-maps/v1 ` --include docs ` --include README.md ` --license LICENSE ` @@ -121,7 +122,7 @@ its own package, compatibility range and licensing files: "package": "iot-operations.zip", "compatibility": { "siteops": ">=1.0.0b1,<2", - "requiredFeatures": ["manifest/v1", "composition/v1", "manifest-selection/v1"] + "requiredFeatures": ["manifest/v1", "composition/v1", "manifest-selection/v1", "typed-input-maps/v1"] }, "include": ["docs", "README.md"], "licenses": ["LICENSE", "ThirdPartyNotices.txt"] @@ -181,7 +182,9 @@ file's byte limit. Package names are unique portable ZIP filenames and reserve room for their proof names. Workspace and companion paths must exist in the selected commit. License paths must identify regular files. `include` is optional, and omitted `requiredFeatures` defaults to `manifest/v1`. -The producer also records `compiled-templates/v1`. +The producer also records `compiled-templates/v1`. It rejects a workspace +whose input contract declares a `stringMap` input unless `requiredFeatures` +includes `typed-input-maps/v1`. If a workspace contains either generated index file, both must be present and current for that exact source. Production preserves the existing generic or @@ -295,10 +298,11 @@ Each record contains `path`, `sha256` and `size`. Companion files outside the workspace affect the archive identity, not the workspace tree identity. The metadata file is excluded from its own payload inventory. -Supported required features are `manifest/v1`, `composition/v1` and -`manifest-selection/v1`. Package production also requires -`compiled-templates/v1`. Unsupported required features and incompatible -engine versions are rejected. Compatibility describes the declared engine +Supported required features are `manifest/v1`, `composition/v1`, +`manifest-selection/v1` and `typed-input-maps/v1`. Package production also +requires `compiled-templates/v1`, and `typed-input-maps/v1` when an input +contract declares a `stringMap` input. Unsupported required features and +incompatible engine versions are rejected. Compatibility describes the declared engine contract, not deployment or workload qualification. ## Template mapping diff --git a/plugins/siteops/skills/upgrade-aio/SKILL.md b/plugins/siteops/skills/upgrade-aio/SKILL.md index 515a566..5818780 100644 --- a/plugins/siteops/skills/upgrade-aio/SKILL.md +++ b/plugins/siteops/skills/upgrade-aio/SKILL.md @@ -87,11 +87,13 @@ Site that describes this installation. ## 3. Choose the target AIO release Azure IoT Operations supports an upgrade to any patch of the same minor -version, or to the next minor version. Other moves need an uninstall and +version, or to the next minor version. To cross several minor versions, +upgrade one minor version at a time, each with its own plan and deploy. A +downgrade, or a move between preview and GA, needs an uninstall and reinstall. AIO releases 2512 and 2602 are 1.2, 2603 to 2606 are 1.3, and 2607 to 2609 are 1.4. Confirm the current and target AIO releases with the user. If the move is not clearly supported, stop and point the user to -[Upgrade Azure IoT Operations](https://learn.microsoft.com/azure/iot-operations/deploy-iot-ops/howto-upgrade). +[Upgrade Azure IoT Operations](https://learn.microsoft.com/azure/iot-operations/manage-iot-ops/howto-upgrade). With the user's agreement, set the target in each Site file: diff --git a/siteops/browse_output.py b/siteops/browse_output.py index cd997c9..04de1dc 100644 --- a/siteops/browse_output.py +++ b/siteops/browse_output.py @@ -85,6 +85,8 @@ def _placeholder(answer: TypedAnswer) -> str: return f"<{label}-resource-ID>" if answer.type == "boolean": return "" + if answer.type == "stringMap": + return "=" return f"<{answer.name}>" diff --git a/siteops/cli.py b/siteops/cli.py index 89e0799..bcc92cd 100644 --- a/siteops/cli.py +++ b/siteops/cli.py @@ -1179,6 +1179,16 @@ def cmd_inputs(args: argparse.Namespace, orchestrator: Orchestrator) -> int: indent=" ", ): print(line) + if field["type"] == "stringMap": + name = _content_text(field["name"]) + for line in _wrap( + f"Site field: {_content_text(field['sitePath'])}. Keys and values " + "are text, and an omitted or empty map sets nothing. Never include secrets.", + indent=" ", + ): + print(line) + print(f' In an answer file:\n {name}:\n "": ""') + print(f' Inline, once for each key: --input "{name}=="') if "resource" in field: if field.get("fromResource"): link = field["fromResource"] @@ -2122,6 +2132,12 @@ def _auto_discover_workspace(start: Path) -> Path | None: "no limit. Overrides the manifest's `parallel` setting." ) +_INPUT_HELP = ( + "Typed answer that is not secret, as NAME=VALUE or NAME=KEY=VALUE for one " + "map entry (repeatable, overrides --input-file, map entries merge with it " + "key by key)" +) + _ROOT_EPILOG = """ Global options such as -w, --project and --approved-source go before the command. @@ -2332,7 +2348,7 @@ def main() -> None: ) p_inputs.add_argument( "--input", dest="input_values", action="append", metavar="NAME=VALUE", - help="Typed answer that is not secret (repeatable, overrides --input-file)", + help=_INPUT_HELP, ) p_inputs.add_argument( "--save-site", type=Path, action=_SingleValueOption, metavar="FILE", @@ -2499,7 +2515,7 @@ def main() -> None: ) command.add_argument( "--input", dest="input_values", action="append", metavar="NAME=VALUE", - help="Typed answer that is not secret (repeatable, overrides --input-file)", + help=_INPUT_HELP, ) command.add_argument( "--offline-content", dest="offline", action="store_true", diff --git a/siteops/guided_inputs.py b/siteops/guided_inputs.py index 7f942a1..b9d0a0c 100644 --- a/siteops/guided_inputs.py +++ b/siteops/guided_inputs.py @@ -8,6 +8,7 @@ import copy import difflib import re +import unicodedata from collections.abc import Mapping from dataclasses import dataclass from dataclasses import field as dataclass_field @@ -31,6 +32,12 @@ _VERSION = "siteops.inputs/v1" _MAX_YAML_BYTES = 128 * 1024 _MAX_INPUTS = 64 +_TYPES = ("string", "boolean", "azureResourceId", "stringMap") +# Limits for each supplied stringMap source and for the combined map, in code points and UTF-8 bytes. +_MAP_ENTRIES = 64 +_MAP_KEY_LENGTH = 256 +_MAP_VALUE_LENGTH = 4096 +_MAP_BYTES = 64 * 1024 _IDENTIFIER = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\Z", re.ASCII) _ROOT_FIELDS = {"name", "subscription", "resourceGroup", "location"} _MAPPING_FIELDS = {"labels", "parameters", "properties"} @@ -180,6 +187,56 @@ def _typed_value(value: Any, kind: str, label: str, *, required: bool) -> None: raise GuidedInputError(f"{label} must be one nonempty ARM resource ID.") +def _map_key(key: Any, label: str) -> None: + """Check one stringMap key before any message can name it. + + Printable keys exclude Unicode control, format, surrogate, private use and + unassigned characters, and line and paragraph separators. + """ + if not isinstance(key, str): + raise GuidedInputError(f"{label} keys must be text. Quote each key in YAML.") + if not 1 <= len(key) <= _MAP_KEY_LENGTH or any( + unicodedata.category(character)[0] == "C" + or unicodedata.category(character) in ("Zl", "Zp") + for character in key + ): + raise GuidedInputError( + f"{label} keys must be printable text of 1 to {_MAP_KEY_LENGTH} characters." + ) + + +def _map_entry_error(label: str, key: str, problem: str) -> GuidedInputError: + """Name the input publicly and its checked key only in private detail, never the value.""" + error = GuidedInputError(f"{label} has a value that {problem}") + error.private_message = f"{label} key '{key}' has a value that {problem}" + return error + + +def _string_map(value: Any, label: str) -> None: + """Check one supplied stringMap source, or the combined map, against its limits.""" + if not isinstance(value, Mapping): + raise GuidedInputError(f"{label} must be a mapping of string keys to string values.") + if len(value) > _MAP_ENTRIES: + raise GuidedInputError(f"{label} exceeds its limit of {_MAP_ENTRIES} entries.") + size = 0 + for key, item in value.items(): + _map_key(key, label) + if isinstance(item, Mapping): + raise _map_entry_error( + label, key, "is a mapping. Write each setting as one dotted key with a quoted value.", + ) + if type(item) is not str: + raise _map_entry_error(label, key, "is not a string. Quote each value in YAML.") + if len(item) > _MAP_VALUE_LENGTH: + raise _map_entry_error(label, key, "is longer than 4,096 characters.") + try: + size += len(key.encode("utf-8")) + len(item.encode("utf-8")) + except UnicodeEncodeError: + raise _map_entry_error(label, key, "is not valid Unicode text.") from None + if size > _MAP_BYTES: + raise GuidedInputError(f"{label} exceeds its 64 KiB limit for keys and values.") + + def _site_path(value: Any) -> tuple[str, ...]: if not isinstance(value, str): raise GuidedInputError("Input sitePath must be a dotted Site field.") @@ -254,7 +311,14 @@ def has_default(self) -> bool: return self.default is not _MISSING def validate(self, value: Any, label: str) -> None: - """Check the declared scalar type and closed string constraints.""" + """Check the declared type and closed constraints. + + A stringMap is checked for each supplied source and for the combined map. + Its `required` rule applies only to the combined map. + """ + if self.type == "stringMap": + _string_map(value, label) + return _typed_value(value, self.type, label, required=self.required) if self.type != "string": return @@ -280,8 +344,8 @@ def _condition( if not isinstance(controller, str) or controller not in earlier: raise GuidedInputError(f"{label} must reference an earlier declared input.") field = earlier[controller] - if field.when is not None or field.resource is not None: - raise GuidedInputError(f"{label} cannot reference a conditional or resource input.") + if field.when is not None or field.resource is not None or field.type == "stringMap": + raise GuidedInputError(f"{label} cannot reference a conditional, resource or map input.") _typed_value(row["equals"], field.type, f"{label} equals", required=False) return InputCondition(controller, row["equals"]) @@ -404,8 +468,10 @@ def _parse_fields(value: Any) -> tuple[InputField, ...]: if name in by_name: raise GuidedInputError(f"Contract inputs contain a duplicate name: {name}.") kind = row["type"] - if kind not in ("string", "boolean", "azureResourceId"): - raise GuidedInputError(f"{label} type must be string, boolean or azureResourceId.") + if kind not in _TYPES: + raise GuidedInputError( + f"{label} type must be string, boolean, azureResourceId or stringMap." + ) description = row["description"] if not isinstance(description, str) or not description.strip(): raise GuidedInputError(f"{label} description must be nonempty text.") @@ -421,6 +487,10 @@ def _parse_fields(value: Any) -> tuple[InputField, ...]: raise GuidedInputError(f"{label} requires sitePath or derived input fields.") if resource_role and path is not None and path[0] not in {"parameters", "properties"}: raise GuidedInputError(f"{label} resource ID sitePath must be a parameter or property.") + if kind == "stringMap" and path[0] not in {"parameters", "properties"}: + raise GuidedInputError(f"{label} stringMap sitePath must be a parameter or property.") + if kind == "stringMap" and "default" in row: + raise GuidedInputError(f"{label} stringMap cannot declare a default.") if path is not None and any( field.site_path is not None and ( path[: len(field.site_path)] == field.site_path @@ -517,10 +587,20 @@ def _validate_default_writers( continue current: Any = defaults for part in field.site_path: - if not isinstance(current, dict) or part not in current: + if not isinstance(current, dict): + if field.type == "stringMap": + raise GuidedInputError( + f"Input '{field.name}' sitePath overlaps a siteDefaults value." + ) + break + if part not in current: break current = current[part] else: + if field.type == "stringMap": + raise GuidedInputError( + f"Input '{field.name}' sitePath overlaps a siteDefaults value." + ) if isinstance(current, dict): raise GuidedInputError( f"Input '{field.name}' sitePath would replace a siteDefaults mapping." @@ -544,9 +624,13 @@ class BoundResource: @dataclass(frozen=True) class BoundInputs: - """Resolved local answers and their origins, before any provider read.""" + """Resolved local answers and their origins, before any provider read. + + A stringMap answer is a read-only mapping. Its origin is `input file`, + `inline`, or `input file and inline` when both supply entries. + """ - active_values: Mapping[str, str | bool] = dataclass_field(repr=False) + active_values: Mapping[str, Any] = dataclass_field(repr=False) sources: Mapping[str, str] resources: tuple[BoundResource, ...] = dataclass_field(default=(), repr=False) @@ -736,23 +820,49 @@ def _missing_input(self, name: str, active_values: Mapping[str, Any]) -> GuidedI f"Missing required input '{name}'. Supply it, or supply " f"{self._resource_answer(route)} to derive it." ) - return GuidedInputError(f"Missing required input '{name}'.") + if field.type == "stringMap": + return GuidedInputError( + f"Missing required input '{name}'. Supply at least one entry, such as " + f"`--input {name}==`." + ) + placeholder = "" if field.type == "boolean" else "" + return GuidedInputError( + f"Missing required input '{name}'. Supply `--input {name}={placeholder}`." + ) def bind( self, values_file: Path | None = None, inline: list[str] | None = None, ) -> BoundInputs: - """Validate and merge local answers without constructing a Site.""" + """Validate and merge local answers without constructing a Site. + + A stringMap input repeats inline once per key as `NAME=KEY=VALUE`, and + each inline key replaces the same key from the input file. + """ fields = {field.name: field for field in self.fields} - inline_values: dict[str, str | bool] = {} + inline_values: dict[str, Any] = {} for answer in inline or []: if not isinstance(answer, str) or "=" not in answer: raise GuidedInputError("Inline inputs must use NAME=VALUE format.") name, value = answer.split("=", 1) if name not in fields: raise _unknown_input(name, list(fields), "Inline inputs") + field = fields[name] + if field.type == "stringMap": + if "=" not in value: + raise GuidedInputError( + f"Inline input '{name}' must use {name}=KEY=VALUE for each entry." + ) + key, item = value.split("=", 1) + entries = inline_values.setdefault(name, {}) + _map_key(key, f"Inline input '{name}'") + if key in entries: + error = GuidedInputError(f"Inline input '{name}' repeats a key.") + error.private_message = f"Inline input '{name}' repeats key '{key}'." + raise error + entries[key] = item + continue if name in inline_values: raise GuidedInputError(f"Duplicate inline input '{name}'.") - field = fields[name] if field.resource is not None and field.resource.from_resource is not None: raise GuidedInputError( f"Input '{name}' is derived from '{self._source_input(field).name}' " @@ -770,6 +880,9 @@ def bind( else: _typed_value(value, "string", f"Input '{name}'", required=field.required) inline_values[name] = value + for name, entries in inline_values.items(): + if fields[name].type == "stringMap": + fields[name].validate(entries, f"Inline input '{name}'") file_values: dict[str, Any] = {} if values_file is not None: @@ -805,13 +918,26 @@ def bind( } effective.update(file_values) effective.update(inline_values) - active_values: dict[str, str | bool] = {} + map_sources: dict[str, str] = {} + for name, entries in inline_values.items(): + from_file = file_values.get(name) + if fields[name].type != "stringMap" or not isinstance(from_file, Mapping) or not from_file: + continue + combined = {**from_file, **entries} + fields[name].validate(combined, f"Combined input '{name}'") + effective[name] = combined + if any(key not in entries for key in from_file): + map_sources[name] = "input file and inline" + active_values: dict[str, Any] = {} sources: dict[str, str] = {} missing_required: list[str] = [] for field in self.fields: if field.when is not None: controller = active_values.get(field.when.input, _MISSING) if controller is _MISSING: + if fields[field.when.input].required: + # The missing required controller is reported below, in input order. + continue raise GuidedInputError( f"Input '{field.when.input}' is missing. It controls conditional " f"input '{field.name}'." @@ -823,12 +949,16 @@ def bind( ) continue value = effective.get(field.name, _MISSING) - if value is _MISSING or value is None: + if value is _MISSING or value is None or ( + field.type == "stringMap" and field.required and not value + ): if field.required: missing_required.append(field.name) continue + if field.type == "stringMap": + value = MappingProxyType(dict(value)) active_values[field.name] = value - sources[field.name] = ( + sources[field.name] = map_sources.get(field.name) or ( "inline" if field.name in inline_values else "input file" if field.name in file_values else "default" ) @@ -1101,11 +1231,18 @@ def build_site( field.when is None or values.get(field.when.input) == field.when.equals ): - raise GuidedInputError(f"Missing required input '{field.name}'.") + raise self._missing_input(field.name, values) continue if field.site_path is None: raise GuidedInputError("A mapped input needs a Site field.") field.validate(values[field.name], f"Input '{field.name}'") + if field.type == "stringMap": + if field.required and not values[field.name]: + raise self._missing_input(field.name, values) + # An empty map writes nothing to the Site. + if values[field.name]: + _assign(data, field.site_path, dict(values[field.name])) + continue _assign(data, field.site_path, values[field.name]) return Site.from_data(data, source="guided inputs", default_name="guided-site") @@ -1149,7 +1286,7 @@ def resolve( return self.build_site(self.bind(values_file=values_file, inline=inline)) -def _assign(data: dict[str, Any], path: tuple[str, ...], value: str | bool) -> None: +def _assign(data: dict[str, Any], path: tuple[str, ...], value: Any) -> None: current = data for part in path[:-1]: child = current.setdefault(part, {}) @@ -1161,6 +1298,30 @@ def _assign(data: dict[str, Any], path: tuple[str, ...], value: str | bool) -> N current[path[-1]] = value +def _is_contract(document: dict[str, Any]) -> bool: + """Treat a sibling as a contract by its kind or its input API namespace.""" + version = document.get("apiVersion") + return document.get("kind") == "SiteInputContract" or ( + isinstance(version, str) and version.startswith("siteops.inputs/") + ) + + +def declares_string_map(path: Path) -> bool: + """Report whether a file read as an input contract declares a stringMap input. + + Package producers use this to require the matching package feature. YAML + that is not a contract declares none, and unreadable YAML raises + GuidedInputError rather than counting as none. + """ + document = _read_yaml(path, label="Input contract") + if not isinstance(document, dict) or not _is_contract(document): + return False + inputs = document.get("inputs") + return isinstance(inputs, list) and any( + isinstance(row, dict) and row.get("type") == "stringMap" for row in inputs + ) + + def load_contract( manifest_path: Path, *, binding: MaterializedPackageBinding | None = None, ) -> InputContract | None: @@ -1179,10 +1340,7 @@ def load_contract( document = _mapping(_read_yaml(verified, label="Input contract"), "Input contract") if binding is not None: binding.require_workspace_file(path) - version = document.get("apiVersion") - if document.get("kind") != "SiteInputContract" and not ( - isinstance(version, str) and version.startswith("siteops.inputs/") - ): + if not _is_contract(document): return None _shape( document, diff --git a/siteops/package_builder.py b/siteops/package_builder.py index dd145c2..25d4a3b 100644 --- a/siteops/package_builder.py +++ b/siteops/package_builder.py @@ -14,7 +14,7 @@ import zipfile import zlib from collections.abc import Callable -from pathlib import Path +from pathlib import Path, PurePosixPath import yaml @@ -43,6 +43,7 @@ VersionProvenance, resolve_tool_from_path, ) +from siteops.guided_inputs import GuidedInputError, contract_path, declares_string_map from siteops.models import DeploymentStep, Manifest, _parse_manifest_spec from siteops.runtime import RuntimePaths from siteops.workspace_compatibility import require_engine_version @@ -148,6 +149,11 @@ def _record_bytes(relative: str, content: bytes) -> tuple[PayloadFile, int]: def _discover_template_sources(root: Path, workspace: str) -> tuple[str, ...]: + return _discover_workspace(root, workspace)[1] + + +def _discover_workspace(root: Path, workspace: str) -> tuple[tuple[str, ...], tuple[str, ...]]: + """Return the workspace's manifests and their deployment template roots.""" workspace_path = root if workspace == "." else checked_path(root, workspace, directory=True) try: reader = ContentReader(workspace_path) @@ -157,6 +163,7 @@ def _discover_template_sources(root: Path, workspace: str) -> tuple[str, ...]: "Workspace deployment entries could not be inspected for package compilation." ) manifests = {entry.path for entry in entries} + discovered = set(manifests) sources: set[str] = set() def collect(manifest_path: str) -> set[str]: @@ -197,7 +204,9 @@ def collect(manifest_path: str) -> set[str]: except (ValueError, yaml.YAMLError): if document.get("kind") == "Manifest": raise - return tuple(sorted(sources)) + else: + discovered.add(manifest_path) + return tuple(sorted(discovered)), tuple(sorted(sources)) except ArtifactError: raise except (BrowseError, OSError, ValueError): @@ -206,6 +215,44 @@ def collect(manifest_path: str) -> set[str]: ) from None +def _require_input_contract_features( + root: Path, + workspace: str, + manifests: tuple[str, ...], + paths: tuple[str, ...], + required_features: tuple[str, ...], +) -> None: + """Require the package feature that each manifest's input contract needs. + + Only the sidecar that the contract loader selects for a manifest is read, + so other files named `inputs.yaml` are ordinary content. + """ + if package.TYPED_INPUT_MAPS_FEATURE in required_features: + return + prefix = "" if workspace == "." else workspace + "/" + # Case-insensitive file systems can open a sidecar whose name differs only in case. + inventory = {path.casefold(): path for path in paths} + for manifest in manifests: + try: + sidecar = contract_path(PurePosixPath(manifest)) + except GuidedInputError: + continue + relative = inventory.get((prefix + sidecar.as_posix()).casefold()) + if relative is None: + continue + try: + maps = declares_string_map(checked_path(root, relative)) + except (GuidedInputError, OSError): + raise ArtifactError( + f"Input contract '{relative}' could not be read to check its required features." + ) from None + if maps: + raise ArtifactError( + f"Input contract '{relative}' declares a stringMap input, which requires " + f"the package feature {package.TYPED_INPUT_MAPS_FEATURE}." + ) + + def workspace_bicep_sources(root: Path, workspace: str) -> tuple[str, ...]: """Return Bicep deployment roots discovered throughout the workspace.""" return tuple( @@ -503,7 +550,8 @@ def build_package( workspace, directory=True, ) - template_sources = _discover_template_sources(snapshot, workspace) + manifests, template_sources = _discover_workspace(snapshot, workspace) + _require_input_contract_features(snapshot, workspace, manifests, paths, required_features) requires_bicep = any( Path(source).suffix.casefold() == ".bicep" for source in template_sources diff --git a/siteops/planning.py b/siteops/planning.py index 68927e1..e53b9c9 100644 --- a/siteops/planning.py +++ b/siteops/planning.py @@ -45,6 +45,12 @@ _EXTRA_CLOSER_AFTER_TEMPLATE_PATTERN = re.compile( STEP_OUTPUT_PATTERN.pattern + r"\s*\}" ) +# Template errors name the location and never the value, which can carry settings. +_TEMPLATE_REMEDY = ( + "Write `{{ site. }}` or `{{ steps..outputs. }}` with " + "one space inside each pair of braces, and name a field the Site sets. " + "Remove `{{` from text that is not a template." +) class PlanIntent(str, Enum): @@ -451,8 +457,8 @@ def _classify_plan_string( or _EXTRA_CLOSER_AFTER_TEMPLATE_PATTERN.search(value) ): raise ValueError( - f"{_value_location(path)} contains a malformed template: " - f"{value!r}." + f"{_value_location(path)} contains a malformed template. " + f"{_TEMPLATE_REMEDY}" ) stripped = value.strip() @@ -491,14 +497,14 @@ def _classify_plan_string( if "{{" in remainder: raise ValueError( f"{_value_location(path)} contains an unsupported or " - f"malformed template: {value!r}." + f"malformed template. {_TEMPLATE_REMEDY}" ) return InterpolatedValue(tuple(parts)) if "{{" in value: raise ValueError( f"{_value_location(path)} contains an unresolved or unsupported " - f"template: {value!r}." + f"template. {_TEMPLATE_REMEDY}" ) return LiteralValue(value) diff --git a/siteops/workspace_package.py b/siteops/workspace_package.py index 8b5f110..e78f4ed 100644 --- a/siteops/workspace_package.py +++ b/siteops/workspace_package.py @@ -74,11 +74,13 @@ MAX_CENTRAL_BYTES = 8 * 1024 * 1024 MAX_COMPRESSION_RATIO = 200 COMPILED_TEMPLATE_FEATURE = "compiled-templates/v1" +TYPED_INPUT_MAPS_FEATURE = "typed-input-maps/v1" SUPPORTED_FEATURES = frozenset({ "manifest/v1", "composition/v1", "manifest-selection/v1", COMPILED_TEMPLATE_FEATURE, + TYPED_INPUT_MAPS_FEATURE, }) _SHA256 = re.compile(r"[0-9a-f]{64}") logger = logging.getLogger(__name__) diff --git a/siteops/yamlio.py b/siteops/yamlio.py index f3aa505..100ff2a 100644 --- a/siteops/yamlio.py +++ b/siteops/yamlio.py @@ -34,12 +34,16 @@ def _construct_mapping(loader: _StrictLoader, node: yaml.MappingNode, deep: bool seen: set = set() for key_node, _ in node.value: - # Both are rewritten by the parent constructor after this pre-pass: - # `<<` is flattened into the mapping and `=` becomes a string. Neither - # is a key to compare here, and constructing them now would fail. - if key_node.tag in ("tag:yaml.org,2002:merge", "tag:yaml.org,2002:value"): + # The parent constructor flattens `<<` into the mapping after this + # pre-pass, so it is not a key to compare. It also turns a bare `=`, + # tagged as a YAML value key, into the string "=", so compare it as + # that string rather than constructing it, which would fail. + if key_node.tag == "tag:yaml.org,2002:merge": continue - key = loader.construct_object(key_node, deep=deep) + if key_node.tag == "tag:yaml.org,2002:value": + key = key_node.value + else: + key = loader.construct_object(key_node, deep=deep) if not isinstance(key, collections.abc.Hashable): # The predicate the standard constructor uses, so the two cannot # disagree. It reports the unhashable key with its own message. diff --git a/tests/fixtures/prepare_installed_project.py b/tests/fixtures/prepare_installed_project.py index 7df495e..16e7b69 100644 --- a/tests/fixtures/prepare_installed_project.py +++ b/tests/fixtures/prepare_installed_project.py @@ -86,6 +86,7 @@ def compilation_session(): inspection = build_package( source, archive, workspace="workspace", kit_id="example.storage", version="7", source_revision=revision, siteops_range=">=1.0.0b1,<2", + required_features=("manifest/v1", "typed-input-maps/v1") if len(sys.argv) == 3 else ("manifest/v1",), compilation_session_factory=compilation_session_factory, ) proof, trusted = root / "proof.jsonl", root / "trusted-root.json" diff --git a/tests/test_browse.py b/tests/test_browse.py index b73905b..15925b6 100644 --- a/tests/test_browse.py +++ b/tests/test_browse.py @@ -197,6 +197,39 @@ def test_typed_route_placeholders_follow_the_answer_type(tmp_path, monkeypatch): ) in plain +def test_typed_route_suggests_only_a_required_map_with_a_generic_entry(tmp_path, monkeypatch): + contract = """\ +apiVersion: siteops.inputs/v1 +kind: SiteInputContract +inputs: + - name: subscription + type: string + description: Subscription. + sitePath: subscription + - name: labels + type: stringMap + description: Required settings. + sitePath: parameters.labels + - name: extra + type: stringMap + required: false + description: Optional settings. + sitePath: parameters.extra +""" + _entry(tmp_path).with_name("inputs.yaml").write_text(contract, encoding="utf-8") + monkeypatch.setattr(browse_output, "_command_shell", lambda: "POSIX shell") + result = inspect_content(tmp_path, "example") + plain = render_browse_plain(result) + assert ( + " siteops plan example --input 'subscription=' " + "--input 'labels=='\n" + ) in plain + assert "extra=" not in plain + assert result.document()["entries"][0]["typedInputs"]["answers"][1] == { + "name": "labels", "type": "stringMap", "resourceType": None, + } + + @pytest.mark.parametrize(("contents", "status"), [ ("apiVersion: siteops.inputs/v1\nkind: SiteInputContract\ninputs: [PRIVATE_SENTINEL]\n", "unreadable"), ("customLocationName: '{{ steps.resolve.outputs.name }}'\n", "none"), diff --git a/tests/test_copilot_plugin.py b/tests/test_copilot_plugin.py index db6ce60..14b15b6 100644 --- a/tests/test_copilot_plugin.py +++ b/tests/test_copilot_plugin.py @@ -218,6 +218,20 @@ def test_upgrade_skill_maps_each_aio_release_to_its_minor_version(): assert expected in skill +@pytest.mark.parametrize("document", [ + "plugins/siteops/skills/upgrade-aio/SKILL.md", + "docs/aio-releases.md", + "workspaces/iot-operations/manifests/aio-upgrade/README.md", +]) +def test_upgrade_guidance_steps_through_minor_versions_and_links_the_current_page(document): + text = " ".join((ROOT / document).read_text(encoding="utf-8").split()) + assert "https://learn.microsoft.com/azure/iot-operations/manage-iot-ops/howto-upgrade" in text + assert "deploy-iot-ops/howto-upgrade" not in text + assert "one minor version at a time" in text + assert "Other moves need an uninstall" not in text + assert "jumps across several minor versions" not in text + + def test_deploy_skill_states_the_workspace_defaults(): workspace = ROOT / "workspaces" / "iot-operations" skill = (PLUGIN / "skills" / "deploy-aio" / "SKILL.md").read_text(encoding="utf-8") diff --git a/tests/test_guided_cli.py b/tests/test_guided_cli.py index 290aa25..bff33c8 100644 --- a/tests/test_guided_cli.py +++ b/tests/test_guided_cli.py @@ -356,6 +356,133 @@ def test_inputs_preview_redacts_site_in_automation( assert "00000000-0000-0000-0000-000000000001" not in output.out + output.err +def _map_workspace(workspace: Path) -> Path: + path = contract_path(workspace / "manifests" / "test-manifest.yaml") + contract = yaml.safe_load(path.read_text(encoding="utf-8")) + contract["inputs"].append({ + "name": "settings", "type": "stringMap", "required": False, + "description": "Extension settings.", "sitePath": "parameters.settings", + }) + path.write_text(yaml.safe_dump(contract), encoding="utf-8") + return workspace + + +def test_inputs_shows_a_map_destination_example_and_inline_form(guided_workspace, capsys): + workspace = _map_workspace(guided_workspace) + assert _invoke(["-w", str(workspace), "inputs", _manifest(workspace)]) == 0 + output = capsys.readouterr().out + assert "settings (stringMap, optional): Extension settings.\n" in output + normalized = " ".join(output.split()) + assert ( + "Site field: parameters.settings. Keys and values are text, and an omitted or empty " + "map sets nothing. Never include secrets." + ) in normalized + assert ' In an answer file:\n settings:\n "": ""\n' in output + assert ' Inline, once for each key: --input "settings=="\n' in output + assert "siteName (string, required): Site identity.\n" in output + assert "In an answer file" not in output.split("settings (stringMap")[0] + + +@pytest.mark.parametrize("command", ["inputs", "plan", "deploy", "validate"]) +def test_input_help_names_the_map_entry_form(capsys, command): + with patch.object(sys, "argv", ["siteops", command, "--help"]): + with pytest.raises(SystemExit) as stopped: + main() + assert stopped.value.code == 0 + help_text = " ".join(capsys.readouterr().out.split()) + assert "NAME=KEY=VALUE for one map entry" in help_text + assert "map entries merge with it key by key" in help_text + + +def test_map_answers_preview_through_the_command_and_errors_withhold_keys_when_redacted( + guided_workspace, tmp_path, capsys, monkeypatch, +): + workspace = _map_workspace(guided_workspace) + answers = _input_file(tmp_path / "answers.yaml") + document = yaml.safe_load(answers.read_text(encoding="utf-8")) + document["values"]["settings"] = {"from.file": "1", "replaced": "file"} + answers.write_text(yaml.safe_dump(document), encoding="utf-8") + base = ["-w", str(workspace), "inputs", _manifest(workspace), "--input-file", str(answers)] + assert _invoke([ + *base, "--input", "settings=replaced=inline", "--input", "settings=added=a=b", "--output", "json", + ]) == 0 + site = json.loads(capsys.readouterr().out)["resolution"]["site"] + assert site["parameters"]["settings"] == {"from.file": "1", "replaced": "inline", "added": "a=b"} + repeated = [*base, "--input", "settings=PRIVATE_KEY=PRIVATE_VALUE", "--input", "settings=PRIVATE_KEY=PRIVATE_VALUE"] + monkeypatch.setenv("SITEOPS_REDACT_OUTPUT", "0") + assert _invoke(repeated) == 1 + error = capsys.readouterr().err + assert "Inline input 'settings' repeats key 'PRIVATE_KEY'." in error + assert "PRIVATE_VALUE" not in error + monkeypatch.setenv("SITEOPS_REDACT_OUTPUT", "1") + assert _invoke(repeated) == 1 + error = capsys.readouterr().err + assert "Inline input 'settings' repeats a key." in error + assert "PRIVATE_KEY" not in error and "PRIVATE_VALUE" not in error + + +def _settings_template_workspace(workspace: Path) -> Path: + """Deploy the map to a template that declares it, so planning classifies each entry.""" + _map_workspace(workspace) + (workspace / "templates" / "settings.json").write_text(json.dumps({ + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "parameters": {"settings": {"type": "object", "defaultValue": {}}}, + "resources": [], + }), encoding="utf-8") + manifest = workspace / "manifests" / "test-manifest.yaml" + document = yaml.safe_load(manifest.read_text(encoding="utf-8")) + document["steps"] = [{"name": "apply", "template": "templates/settings.json", "scope": "subscription"}] + manifest.write_text(yaml.safe_dump(document), encoding="utf-8") + return workspace + + +@pytest.mark.parametrize(("output", "redacted", "codes"), [ + ("plain", False, None), + ("plain", True, None), + ("json", False, ["operation-preparation.invalid", "content.other"]), + ("json", True, ["plan.operation-preparation-failed", "content.other"]), +]) +def test_map_template_errors_name_the_key_and_never_the_value( + guided_workspace, capsys, monkeypatch, deterministic_local_tools, output, redacted, codes, +): + workspace = _settings_template_workspace(guided_workspace) + original = Orchestrator.build_plan + plans = [] + + def build_plan(self, *args, **kwargs): + plans.append(original(self, *args, **kwargs)) + return plans[-1] + + monkeypatch.setattr(Orchestrator, "build_plan", build_plan) + monkeypatch.setenv("SITEOPS_REDACT_OUTPUT", "1" if redacted else "0") + base = [ + "-w", str(workspace), "plan", _manifest(workspace), "--input", "siteName=one", + "--input", "subscription=00000000-0000-0000-0000-000000000001", "--input", "location=eastus", + "--output", output, + ] + assert _invoke([*base, "--input", "settings=plain.key=PRIVATE_VALUE {{ not_a_template }}"]) == 1 + captured = capsys.readouterr() + text = captured.out + captured.err + assert "PRIVATE_VALUE" not in text and "not_a_template" not in text + if codes is not None: + assert [item["code"] for item in json.loads(captured.out)["diagnostics"]] == codes + elif not redacted: + assert ( + "Plan value at 'one.apply.parameters.settings.plain.key' contains an unresolved or " + "unsupported template. Write `{{ site. }}` or `{{ steps..outputs. }}`" + ) in " ".join(text.split()) + + assert _invoke([*base, "--input", "settings=plain.key=PRIVATE_VALUE {{ site.name }}"]) == 0 + captured = capsys.readouterr() + assert "PRIVATE_VALUE" not in captured.out + captured.err + operation = plans[-1].plan.targets[0].operations[0] + assert operation.disposition is PlanDisposition.EXECUTE + assert isinstance(operation.details, DeploymentOperation) + (entry,) = operation.details.parameters.entries + assert resolve_plan_value(entry.value, {}) == {"plain.key": "PRIVATE_VALUE one"} + + def test_resource_plan_reads_only_selected_role_before_planning( guided_workspace, tmp_path, capsys, ): diff --git a/tests/test_guided_inputs.py b/tests/test_guided_inputs.py index 2556ace..152b10d 100644 --- a/tests/test_guided_inputs.py +++ b/tests/test_guided_inputs.py @@ -1005,7 +1005,7 @@ def test_missing_optional_controller_is_not_treated_as_false(tmp_path): ), ], ) - with pytest.raises(ValueError, match="enable"): + with pytest.raises(ValueError, match=r"^Input 'enable' is missing\. It controls conditional input 'choice'\.$"): load_contract(manifest).resolve() assert load_contract(manifest).resolve(inline=["enable=false"]).properties == { "enable": False, @@ -1243,19 +1243,24 @@ def test_multiple_dependents_can_share_an_unconditional_controller(tmp_path): } -def test_missing_controller_is_reported_in_separate_sentences(tmp_path): +def test_missing_required_controller_is_reported_in_input_order_before_its_dependents(tmp_path): manifest, _ = _manifest_and_contract( tmp_path, fields=[ - _field("subscription", "subscription", default="sub"), + _field("subscription", "subscription"), _field("location", "location", default="eastus"), _field("enable", "properties.enable", type="boolean"), _field("first", "properties.first", when={"input": "enable", "equals": True}), ], ) - with pytest.raises(ValueError) as rejected: - load_contract(manifest).resolve() - assert str(rejected.value) == "Input 'enable' is missing. It controls conditional input 'first'." + contract = load_contract(manifest) + with pytest.raises(GuidedInputError) as rejected: + contract.resolve() + assert str(rejected.value) == "Missing required input 'subscription'. Supply `--input subscription=`." + for answers in (["subscription=sub"], ["subscription=sub", "first=supplied"]): + with pytest.raises(GuidedInputError) as rejected: + contract.resolve(inline=answers) + assert str(rejected.value) == "Missing required input 'enable'. Supply `--input enable=`." def test_missing_required_name_is_safe_guided_error(tmp_path): @@ -1271,7 +1276,7 @@ def test_missing_required_name_is_safe_guided_error(tmp_path): ) with pytest.raises(GuidedInputError) as rejected: load_contract(manifest).resolve() - assert str(rejected.value) == "Missing required input 'siteName'." + assert str(rejected.value) == "Missing required input 'siteName'. Supply `--input siteName=`." assert isinstance(rejected.value, ValueError) @@ -1335,7 +1340,7 @@ def unexpected_site(*args, **kwargs): pytest.fail("An incomplete example must fail before constructing a Site or plan.") monkeypatch.setattr(Site, "from_data", unexpected_site) - with pytest.raises(ValueError, match=r"^Missing required input 'siteName'\.$"): + with pytest.raises(ValueError, match=r"^Missing required input 'siteName'\. Supply `--input siteName=`\.$"): contract.resolve(values_file=answers) @@ -1655,3 +1660,315 @@ def test_exclusive_yaml_writer_does_not_overwrite_or_create_parents(tmp_path): with pytest.raises(ValueError, match="mapping"): write_yaml_exclusive(tmp_path / "invalid.yaml", []) assert not (tmp_path / "invalid.yaml").exists() + + +def _map_field(name="settings", site_path="parameters.settings", **extra): + return { + "name": name, + "type": "stringMap", + "description": f"Enter {name}", + "sitePath": site_path, + "required": False, + **extra, + } + + +def _map_contract(tmp_path, *fields, defaults=None, **options): + manifest, _ = _manifest_and_contract( + tmp_path, + defaults=defaults if defaults is not None else {}, + fields=[ + _field("subscription", "subscription", default="sub"), + _field("location", "location", default="eastus"), + *fields, + _map_field(**options), + ], + ) + return load_contract(manifest) + + +def _map_answers(tmp_path, values, name="answers.yaml"): + path = tmp_path / name + if isinstance(values, str): + path.write_text( + f"apiVersion: {VERSION}\nkind: SiteInputValues\nvalues:\n{values}", encoding="utf-8", + ) + else: + path.write_text(yaml.safe_dump( + {"apiVersion": VERSION, "kind": "SiteInputValues", "values": values}, + allow_unicode=True, + ), encoding="utf-8") + return path + + +def test_string_map_writes_entries_and_an_omitted_or_empty_map_writes_nothing(tmp_path): + contract = _map_contract(tmp_path, defaults={"parameters": {"kept": "default"}}) + assert contract.resolve().parameters == {"kept": "default"} + empty = _map_answers(tmp_path, " settings: {}\n") + assert contract.resolve(values_file=empty).parameters == {"kept": "default"} + assert contract.bind(values_file=empty).sources["settings"] == "input file" + supplied = _map_answers( + tmp_path, ' settings:\n "metrics.interval": "60"\n "flag": ""\n', name="supplied.yaml", + ) + assert contract.resolve(values_file=supplied).parameters == { + "kept": "default", "settings": {"metrics.interval": "60", "flag": ""}, + } + + +@pytest.mark.parametrize(("value", "problem"), [ + *((value, "is not a string. Quote each value in YAML.") for value in ( + "on", "1.10", "5", "true", "null", "[PRIVATE_SENTINEL]", + )), + ("{PRIVATE_SENTINEL: x}", "is a mapping. Write each setting as one dotted key with a quoted value."), +]) +def test_string_map_values_are_never_coerced(tmp_path, value, problem): + contract = _map_contract(tmp_path) + answers = _map_answers(tmp_path, f' settings:\n "named.key": {value}\n') + with pytest.raises(GuidedInputError) as rejected: + contract.resolve(values_file=answers) + assert str(rejected.value) == f"Input 'settings' has a value that {problem}" + assert rejected.value.private_message == f"Input 'settings' key 'named.key' has a value that {problem}" + quoted = _map_answers(tmp_path, ' settings:\n "on": "on"\n "v": "1.10"\n', name="quoted.yaml") + assert contract.resolve(values_file=quoted).parameters["settings"] == {"on": "on", "v": "1.10"} + + +_UNQUOTED_KEY = "Input 'settings' keys must be text. Quote each key in YAML." +_UNPRINTABLE_KEY = "Input 'settings' keys must be printable text of 1 to 256 characters." + + +@pytest.mark.parametrize(("entry", "message"), [ + (' 5: "x"\n', _UNQUOTED_KEY), + (' on: "x"\n', _UNQUOTED_KEY), + (' 1.10: "x"\n', _UNQUOTED_KEY), + (' "": "x"\n', _UNPRINTABLE_KEY), + (f' "{"k" * 257}": "x"\n', _UNPRINTABLE_KEY), + *((f' "PRIVATE\\u{code:04x}KEY": "x"\n', _UNPRINTABLE_KEY) for code in ( + 0x07, 0x7F, 0x202E, 0x200B, 0xE000, 0x0378, 0x2028, 0x2029, + )), + ("", "Input 'settings' must be a mapping of string keys to string values."), +]) +def test_string_map_keys_are_printable_text_and_never_named_when_invalid(tmp_path, entry, message): + contract = _map_contract(tmp_path) + text = f" settings:\n{entry}" if entry else ' settings: "key=value"\n' + with pytest.raises(GuidedInputError) as rejected: + contract.resolve(values_file=_map_answers(tmp_path, text)) + assert str(rejected.value) == message + assert rejected.value.private_message is None + + +def test_string_map_keys_accept_printable_text_beyond_ascii(tmp_path): + keys = ["caf\u00e9.mode", "\u8a2d\u5b9a", "with space", "emoji.\U0001f600", "tab-free.key"] + answers = _map_answers(tmp_path, {"settings": {key: "v" for key in keys}}) + assert sorted(_map_contract(tmp_path).resolve(values_file=answers).parameters["settings"]) == sorted(keys) + + +def test_string_map_limits_count_code_points_and_utf8_bytes_in_each_file(tmp_path): + contract = _map_contract(tmp_path) + accepted = [ + {f"k{index:02}": "v" for index in range(64)}, + {"\u00e9" * 256: "\u00e9" * 4096}, + {f"k{index:02}": "v" * 4093 for index in range(16)}, + ] + assert sum(len(key) + len(value) for key, value in accepted[2].items()) == 64 * 1024 + for number, entries in enumerate(accepted): + path = _map_answers(tmp_path, {"settings": entries}, name=f"accepted-{number}.yaml") + assert contract.resolve(values_file=path).parameters["settings"] == entries + rejected = [ + ({f"k{index:02}": "v" for index in range(65)}, "exceeds its limit of 64 entries"), + ({"\u00e9" * 257: "v"}, "keys must be printable text of 1 to 256 characters"), + ({"key": "\u00e9" * 4097}, "has a value that is longer than 4,096 characters"), + ({**accepted[2], "k00": "v" * 4094}, "exceeds its 64 KiB limit"), + ({f"k{index:02}": "\u00e9" * 4096 for index in range(8)}, "exceeds its 64 KiB limit"), + ] + for number, (entries, message) in enumerate(rejected): + path = _map_answers(tmp_path, {"settings": entries}, name=f"rejected-{number}.yaml") + with pytest.raises(GuidedInputError, match=message) as failed: + contract.resolve(values_file=path) + assert str(failed.value).startswith("Input 'settings' ") + assert "\u00e9" * 8 not in f"{failed.value} {failed.value.private_message}" + + +def test_string_map_limits_apply_to_inline_entries_and_the_combined_map(tmp_path): + contract = _map_contract(tmp_path) + file_entries = {f"f{index:02}": "v" for index in range(40)} + answers = _map_answers(tmp_path, {"settings": file_entries}) + inline = [f"settings=i{index:02}=v" for index in range(24)] + assert len(contract.resolve(values_file=answers, inline=inline).parameters["settings"]) == 64 + replaced = _map_answers(tmp_path, {"settings": {f"k{index:02}": "v" for index in range(64)}}, name="full.yaml") + assert len(contract.resolve(values_file=replaced, inline=["settings=k00=new"]).parameters["settings"]) == 64 + with pytest.raises(GuidedInputError, match="^Combined input 'settings' exceeds its limit of 64 entries"): + contract.resolve(values_file=answers, inline=[*inline, "settings=i24=v"]) + with pytest.raises(GuidedInputError, match="^Inline input 'settings' exceeds its limit of 64 entries"): + contract.resolve(inline=[f"settings=i{index:02}=v" for index in range(65)]) + large_file = _map_answers( + tmp_path, {"settings": {f"f{index:02}": "v" * 4093 for index in range(9)}}, name="large.yaml", + ) + large_inline = [f"settings=i{index:02}={'v' * 4093}" for index in range(8)] + assert len(contract.resolve(inline=large_inline).parameters["settings"]) == 8 + with pytest.raises(GuidedInputError, match="^Combined input 'settings' exceeds its 64 KiB limit"): + contract.resolve(values_file=large_file, inline=large_inline) + with pytest.raises(GuidedInputError) as failed: + contract.resolve(inline=[f"settings=named={'v' * 4097}"]) + assert str(failed.value) == "Inline input 'settings' has a value that is longer than 4,096 characters." + assert failed.value.private_message == ( + "Inline input 'settings' key 'named' has a value that is longer than 4,096 characters." + ) + + +@pytest.mark.parametrize(("change", "message"), [ + ({"sitePath": "name"}, "stringMap sitePath must be a parameter or property"), + ({"sitePath": "labels.extra"}, "stringMap sitePath must be a parameter or property"), + ({"sitePath": "location"}, "stringMap sitePath must be a parameter or property"), + ({"default": {}}, "stringMap cannot declare a default"), + ({"default": {"key": "value"}}, "stringMap cannot declare a default"), + ({"sensitive": True}, "sensitive"), + ({"format": "nonEmpty"}, "format"), + ({"maxLength": 8}, "maxLength"), +]) +def test_string_map_declarations_fail_at_contract_load(tmp_path, change, message): + manifest, _ = _manifest_and_contract(tmp_path, fields=[ + _field("subscription", "subscription", default="sub"), + _map_field(**change), + ]) + with pytest.raises(GuidedInputError, match=message): + load_contract(manifest) + + +@pytest.mark.parametrize(("defaults", "site_path"), [ + ({"parameters": {"settings": {"key": "value"}}}, "parameters.settings"), + ({"parameters": {"settings": {}}}, "parameters.settings"), + ({"parameters": {"settings": "scalar"}}, "parameters.settings"), + ({"properties": {"group": "scalar"}}, "properties.group.settings"), + ({"properties": {"group": {"settings": {"nested": "value"}}}}, "properties.group.settings"), +]) +def test_string_map_cannot_overlap_site_defaults(tmp_path, defaults, site_path): + with pytest.raises(GuidedInputError, match="overlaps a siteDefaults value"): + _map_contract(tmp_path, defaults=defaults, site_path=site_path) + + +def test_string_map_beside_site_defaults_and_other_writers(tmp_path): + contract = _map_contract( + tmp_path / "beside", defaults={"properties": {"group": {"other": "kept"}}}, + site_path="properties.group.settings", + ) + assert contract.resolve(inline=["settings=k=v"]).properties == { + "group": {"other": "kept", "settings": {"k": "v"}}, + } + with pytest.raises(GuidedInputError, match="overlaps another input"): + _map_contract(tmp_path / "writer", _field("child", "parameters.settings.child", required=False)) + with pytest.raises(GuidedInputError, match="cannot reference a conditional, resource or map input"): + _manifest_and_contract(tmp_path / "controller", fields=[ + _field("subscription", "subscription", default="sub"), + _map_field(), + _field("mode", "properties.mode", when={"input": "settings", "equals": "on"}), + ]) + load_contract(tmp_path / "controller" / "manifests" / "storage" / "manifest.yaml") + + +def test_required_string_map_needs_at_least_one_entry(tmp_path): + contract = _map_contract(tmp_path, required=True) + assert contract.example()["values"] == {"settings": None} + row = next(row for row in contract.describe()["inputs"] if row["name"] == "settings") + assert (row["type"], row["status"], row["sitePath"]) == ("stringMap", "required", "parameters.settings") + placeholder = tmp_path / "example.yaml" + write_yaml_exclusive(placeholder, contract.example()) + for answers in (None, placeholder, _map_answers(tmp_path, " settings: {}\n")): + with pytest.raises(GuidedInputError) as missing: + contract.resolve(values_file=answers) + assert str(missing.value) == ( + "Missing required input 'settings'. Supply at least one entry, such as " + "`--input settings==`." + ) + assert contract.resolve(values_file=answers, inline=["settings=k="]).parameters == { + "settings": {"k": ""}, + } + + +def test_optional_string_map_is_left_out_of_the_example(tmp_path): + contract = _map_contract(tmp_path) + assert contract.example()["values"] == {} + row = next(row for row in contract.describe()["inputs"] if row["name"] == "settings") + assert (row["type"], row["status"], row["required"]) == ("stringMap", "optional", False) + assert "default" not in row + + +def test_inactive_string_map_rejects_explicit_answers_including_an_empty_map(tmp_path): + contract = _map_contract( + tmp_path, + _field("enabled", "properties.enabled", type="boolean", default=False), + when={"input": "enabled", "equals": True}, + ) + assert "settings" not in contract.resolve().parameters + for number, text in enumerate((" settings: {}\n", ' settings:\n "k": "v"\n')): + with pytest.raises(GuidedInputError, match="Input 'settings' was supplied while its condition is inactive."): + contract.resolve(values_file=_map_answers(tmp_path, text, name=f"inactive-{number}.yaml")) + with pytest.raises(GuidedInputError, match="Input 'settings' was supplied while its condition is inactive."): + contract.resolve(inline=["settings=k=v"]) + assert contract.resolve(inline=["enabled=true"]).parameters == {} + assert contract.resolve(inline=["enabled=true", "settings=k=v"]).parameters == {"settings": {"k": "v"}} + + +@pytest.mark.parametrize(("answers", "expected"), [ + (["settings=a.b=c"], {"a.b": "c"}), + (["settings=a=b=c"], {"a": "b=c"}), + (["settings=empty="], {"empty": ""}), + (["settings=one=1", "settings=two=2"], {"one": "1", "two": "2"}), +]) +def test_inline_map_entries_split_at_the_first_two_equals_signs(tmp_path, answers, expected): + assert _map_contract(tmp_path).resolve(inline=answers).parameters["settings"] == expected + + +@pytest.mark.parametrize(("answers", "message", "private"), [ + (["settings="], "Inline input 'settings' must use settings=KEY=VALUE for each entry.", None), + (["settings=PRIVATE_SENTINEL"], "Inline input 'settings' must use settings=KEY=VALUE for each entry.", None), + (["settings==PRIVATE_SENTINEL"], + "Inline input 'settings' keys must be printable text of 1 to 256 characters.", None), + (["settings=k=PRIVATE_SENTINEL", "settings=k=PRIVATE_SENTINEL"], + "Inline input 'settings' repeats a key.", "Inline input 'settings' repeats key 'k'."), + (["settings=k=one", "settings=k=PRIVATE_SENTINEL"], + "Inline input 'settings' repeats a key.", "Inline input 'settings' repeats key 'k'."), +]) +def test_inline_map_grammar_fails_without_naming_values(tmp_path, answers, message, private): + with pytest.raises(GuidedInputError) as rejected: + _map_contract(tmp_path).resolve(inline=answers) + assert str(rejected.value) == message + assert rejected.value.private_message == private + + +def test_scalar_inline_answers_keep_their_grammar_beside_a_map(tmp_path): + contract = _map_contract(tmp_path, _field("mode", "properties.mode", required=False)) + assert contract.resolve(inline=["mode=a=b", "settings=k=v"]).properties == {"mode": "a=b"} + with pytest.raises(GuidedInputError, match="Duplicate inline input 'mode'."): + contract.resolve(inline=["mode=a", "mode=a"]) + + +def test_inline_map_keys_replace_file_keys_one_by_one_with_combined_provenance(tmp_path): + contract = _map_contract(tmp_path) + answers = _map_answers(tmp_path, ' settings:\n "a": "file"\n "b": "file"\n') + bound = contract.bind(values_file=answers, inline=["settings=b=inline", "settings=c=inline"]) + assert dict(bound.active_values["settings"]) == {"a": "file", "b": "inline", "c": "inline"} + assert bound.sources["settings"] == "input file and inline" + assert contract.build_site(bound).parameters["settings"] == {"a": "file", "b": "inline", "c": "inline"} + with pytest.raises(TypeError): + bound.active_values["settings"]["d"] = "changed" + assert contract.bind(values_file=answers).sources["settings"] == "input file" + assert contract.bind(inline=["settings=a=x"]).sources["settings"] == "inline" + replaced = contract.bind(values_file=answers, inline=["settings=a=x", "settings=b=y"]) + assert replaced.sources["settings"] == "inline" + + +def test_answer_file_maps_keep_merge_keys_and_reject_literal_duplicates(tmp_path): + contract = _map_contract(tmp_path) + merged = _map_answers(tmp_path, ' settings:\n <<: {"a": "1", "b": "2"}\n "b": "3"\n') + assert contract.resolve(values_file=merged).parameters["settings"] == {"a": "1", "b": "3"} + duplicate = _map_answers(tmp_path, ' settings:\n "a": "1"\n "a": "2"\n', name="duplicate.yaml") + with pytest.raises(GuidedInputError, match="duplicate"): + contract.resolve(values_file=duplicate) + for number, keys in enumerate((("=", "="), ("=", '"="'), ('"="', "="))): + repeated = _map_answers( + tmp_path, f' settings:\n {keys[0]}: "first"\n {keys[1]}: "second"\n', name=f"equals-{number}.yaml", + ) + with pytest.raises(GuidedInputError, match="duplicate"): + contract.resolve(values_file=repeated) + single = _map_answers(tmp_path, ' settings:\n =: "bare"\n x=y: "pair"\n', name="equals.yaml") + assert contract.resolve(values_file=single).parameters["settings"] == {"=": "bare", "x=y": "pair"} diff --git a/tests/test_planning.py b/tests/test_planning.py index 96da176..04979ef 100644 --- a/tests/test_planning.py +++ b/tests/test_planning.py @@ -568,6 +568,39 @@ def test_classifier_rejects_unavailable_or_malformed_templates(value): classify_plan_value(value, {}) +@pytest.mark.parametrize( + ("value", "problem"), + [ + ("PRIVATE_VALUE { site.name }}", "a malformed template"), + ( + "{{ steps.resolve.outputs.value }} PRIVATE_VALUE {{ other }}", + "an unsupported or malformed template", + ), + ("PRIVATE_VALUE {{ other }}", "an unresolved or unsupported template"), + ], +) +def test_classifier_template_errors_name_the_location_and_never_the_value( + value, + problem, +): + available = {"resolve": OperationIdentity(target="munich", step="resolve")} + + with pytest.raises(ValueError) as rejected: + classify_plan_value( + {"settings": {"key": value}}, + available, + "munich.deploy.parameters", + ) + + assert str(rejected.value) == ( + "Plan value at 'munich.deploy.parameters.settings.key' contains " + f"{problem}. Write `{{{{ site. }}}}` or " + "`{{ steps..outputs. }}` with one space inside each " + "pair of braces, and name a field the Site sets. Remove `{{` from " + "text that is not a template." + ) + + def test_classifier_rejects_later_step_reference(): prior = OperationIdentity(target="munich", step="prior") diff --git a/tests/test_workspace_package.py b/tests/test_workspace_package.py index 23ecc8d..2b47f3d 100644 --- a/tests/test_workspace_package.py +++ b/tests/test_workspace_package.py @@ -207,13 +207,98 @@ def test_producer_target_is_checked_before_template_discovery(snapshot, tmp_path def unexpected(*args): pytest.fail("Invalid engine selection must fail before template discovery.") - monkeypatch.setattr(package_builder, "_discover_template_sources", unexpected) + monkeypatch.setattr(package_builder, "_discover_workspace", unexpected) output = tmp_path / "package.zip" with pytest.raises(ArtifactError, match="compatibility declaration|different Site Ops version"): _build(snapshot, output, engine_version=engine_version) assert not output.exists() +_MAP_CONTRACT = """\ +apiVersion: siteops.inputs/v1 +kind: SiteInputContract +inputs: + - name: settings + type: stringMap + required: false + description: Settings. + sitePath: parameters.settings +""" +# The required features a v1.0.0b7 engine supports. +_B7_FEATURES = frozenset({ + "manifest/v1", "composition/v1", "manifest-selection/v1", package.COMPILED_TEMPLATE_FEATURE, +}) + + +@pytest.mark.parametrize("relative", [ + "manifests/storage/inputs.yaml", "manifests/flat.inputs.yaml", "manifests/other/Inputs.YAML", +]) +def test_producer_requires_the_map_feature_for_a_string_map_contract(snapshot, tmp_path, relative): + contract = snapshot / "workspace" / relative + contract.parent.mkdir(parents=True, exist_ok=True) + manifest = snapshot / "workspace" / "manifests" / "storage" / "manifest.yaml" + if relative.startswith("manifests/other/"): + shutil.copyfile(manifest, contract.with_name("manifest.yaml")) + elif relative == "manifests/flat.inputs.yaml": + shutil.copyfile(manifest, contract.with_name("flat.yaml")) + contract.write_text(_MAP_CONTRACT, encoding="utf-8") + output = tmp_path / "package.zip" + with pytest.raises(ArtifactError) as rejected: + _build(snapshot, output) + assert str(rejected.value) == ( + f"Input contract 'workspace/{relative}' declares a stringMap input, which requires " + "the package feature typed-input-maps/v1." + ) + assert not output.exists() + built = _build(snapshot, output, required_features=("manifest/v1", package.TYPED_INPUT_MAPS_FEATURE)) + assert package.TYPED_INPUT_MAPS_FEATURE in built.metadata.required_features + + +@pytest.mark.parametrize(("relative", "contents"), [ + ("manifests/storage/inputs.yaml", _MAP_CONTRACT.replace("stringMap", "string")), + ("manifests/storage/inputs.yaml", "settings:\n type: stringMap\n"), + ("defaults.inputs.yml", _MAP_CONTRACT), + ("manifests/storage/other.inputs.yaml", _MAP_CONTRACT), + ("parameters/settings.inputs.yaml", _MAP_CONTRACT), + ("parameters/inputs.yaml", "kind: SiteInputContract\nkind: Parameters\n"), +]) +def test_producer_needs_no_map_feature_without_a_string_map_contract(snapshot, tmp_path, relative, contents): + (snapshot / "workspace" / relative).parent.mkdir(parents=True, exist_ok=True) + (snapshot / "workspace" / relative).write_text(contents, encoding="utf-8") + (snapshot / "docs" / "inputs.yaml").write_text(_MAP_CONTRACT, encoding="utf-8") + built = _build(snapshot, tmp_path / "package.zip") + assert package.TYPED_INPUT_MAPS_FEATURE not in built.metadata.required_features + + +def test_producer_reads_no_parameter_file_that_is_not_a_manifest_sidecar(snapshot, tmp_path): + # Larger than the contract reader's limit, and accepted by the v1.0.0b7 producer. + parameters = snapshot / "workspace" / "parameters" + parameters.mkdir() + (parameters / "inputs.yaml").write_text(f"description: {'a' * 131100}\n", encoding="utf-8") + built = _build(snapshot, tmp_path / "package.zip") + assert built.metadata.required_features == (package.COMPILED_TEMPLATE_FEATURE, "manifest/v1") + assert "workspace/parameters/inputs.yaml" in {entry.path for entry in built.metadata.files} + + +def test_producer_fails_closed_on_an_unreadable_contract_sidecar(snapshot, tmp_path): + (snapshot / "workspace" / "manifests" / "storage" / "inputs.yaml").write_text( + "kind: SiteInputContract\nkind: SiteInputContract\n", encoding="utf-8", + ) + with pytest.raises(ArtifactError, match="^Input contract 'workspace/manifests/storage/inputs.yaml' could not be read"): + _build(snapshot, tmp_path / "package.zip") + + +def test_engine_without_the_map_feature_refuses_the_package(snapshot, tmp_path, monkeypatch): + (snapshot / "workspace" / "manifests" / "storage" / "inputs.yaml").write_text(_MAP_CONTRACT, encoding="utf-8") + output = tmp_path / "package.zip" + built = _build(snapshot, output, required_features=("manifest/v1", package.TYPED_INPUT_MAPS_FEATURE)) + assert package.extract_package(output, built.sha256, tmp_path / "current") == built + monkeypatch.setitem(package.check_compatibility.__kwdefaults__, "features", _B7_FEATURES) + with pytest.raises(ArtifactError, match=r"^This package requires unsupported Site Ops features\.$"): + package.extract_package(output, built.sha256, tmp_path / "older") + assert not (tmp_path / "older").exists() + + def test_materialized_binding_resolves_manifest_name_and_revalidates_inventory( snapshot, tmp_path, @@ -576,6 +661,7 @@ def test_real_aio_entries_produce_mapped_template_roots(tmp_path): version="preview-1", source_revision="fixture", siteops_range=">=1.0.0b1,<2", + required_features=("manifest/v1", package.TYPED_INPUT_MAPS_FEATURE), compilation_session_factory=_controlled_factory( control_root, snapshot, diff --git a/tests/test_yamlio.py b/tests/test_yamlio.py index 342297a..0e3590c 100644 --- a/tests/test_yamlio.py +++ b/tests/test_yamlio.py @@ -41,6 +41,20 @@ def test_the_same_key_in_sibling_mappings_is_allowed(self): loaded = yamlio.load("dataflows:\n - name: a\n - name: b\n") assert [entry["name"] for entry in loaded["dataflows"]] == ["a", "b"] + @pytest.mark.parametrize("document", [ + "=: first\n=: second\n", + '=: first\n"=": second\n', + '"=": first\n=: second\n', + ]) + def test_a_repeated_equals_key_is_rejected(self, document): + """YAML tags a bare `=` key as a value key, then loads it as the string "=". + + The bare and quoted spellings are therefore the same key, in either order. + """ + assert yaml.safe_load(document) == {"=": "second"} + with pytest.raises(yamlio.DuplicateKeyError, match="duplicate key '='"): + yamlio.load(document) + def test_the_error_is_a_yaml_error(self): """A caller that already tolerates a parse error keeps its behavior. @@ -64,6 +78,10 @@ def test_malformed_yaml_still_raises(self): with pytest.raises(yaml.YAMLError): yamlio.load("a:\n b: 1\n c: 2\n") + def test_an_equals_key_and_keys_containing_equals_load(self): + assert yamlio.load("=: bare\nx=y: pair\n") == {"=": "bare", "x=y": "pair"} + assert yamlio.load('"=": quoted\n') == {"=": "quoted"} + def test_a_merge_key_still_resolves(self): """`<<` appears once per mapping, so the guard must not break inheritance.""" loaded = yamlio.load( diff --git a/tests/workspace/test_aio_extension_settings.py b/tests/workspace/test_aio_extension_settings.py new file mode 100644 index 0000000..ee5d79b --- /dev/null +++ b/tests/workspace/test_aio_extension_settings.py @@ -0,0 +1,450 @@ +"""AIO extension settings through typed inputs and configured Sites.""" + +import json +import re +import shlex +import subprocess +import sys +from contextlib import nullcontext +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import MagicMock, patch + +import pytest +import yaml + +from siteops.arm_resources import ArmResourceObservation +from siteops.cli import main +from siteops.command_context import CommandContext +from siteops.compilation import TemplateCompilationSession +from siteops.guided_inputs import GuidedInputError, load_contract +from siteops.orchestrator import Orchestrator +from siteops.planning import ( + DeploymentOperation, + LiteralValue, + PlanDisposition, + PlanIntent, + PlanStatus, + resolve_plan_value, +) +from siteops.results import RunResult +from tests.test_orchestrator_planning import _arm_template, _RecordingToolRunner + +MAPS = { + "aioExtensionSettings": "aioConfigurationOverrides", + "secretStoreExtensionSettings": "secretStoreConfigurationOverrides", + "certManagerExtensionSettings": "certManagerConfigurationOverrides", +} +_SCOPE = "/subscriptions/00000000-0000-0000-0000-000000000001/resourceGroups/rg-first/providers" +INSTANCE_ID = f"{_SCOPE}/Microsoft.IoTOperations/instances/external-instance" +LOCATION_ID = f"{_SCOPE}/Microsoft.ExtendedLocation/customLocations/external-location" +CLUSTER_ID = f"{_SCOPE}/Microsoft.Kubernetes/connectedClusters/arc-first" +MANUAL = [ + "siteName=plant-one", "subscription=00000000-0000-0000-0000-000000000001", + "resourceGroup=rg-existing", "location=eastus", "clusterName=existing-arc", +] + + +def _release(workspace: Path, release: str) -> dict: + return yaml.safe_load((workspace / "parameters" / "aio-releases" / f"{release}.yaml").read_text(encoding="utf-8")) + + +def _session(tmp_path: Path) -> TemplateCompilationSession: + runner = _RecordingToolRunner() + + def compile_locally(argv, timeout): + result = runner(argv, timeout) + if argv[1:3] == ("bicep", "build"): + source = Path(argv[argv.index("--file") + 1]).read_text(encoding="utf-8") + parameters = { + name: { + "type": kind if kind in {"string", "bool", "int", "object", "array"} else "object", + "defaultValue": None, + } + for name, kind in re.findall(r"^param\s+(\w+)\s+(\w+)", source, re.MULTILINE) + } + Path(argv[argv.index("--outfile") + 1]).write_text( + json.dumps(_arm_template(parameters)), encoding="utf-8", + ) + return result + + return TemplateCompilationSession( + command_runner=compile_locally, tool_resolver=lambda name: str(tmp_path / name), + ) + + +def _plan(workspace: Path, tmp_path: Path, manifest: str, monkeypatch, **selection): + session = _session(tmp_path) + monkeypatch.setattr("siteops.orchestrator.TemplateCompilationSession", lambda: session) + monkeypatch.setattr(subprocess, "Popen", MagicMock(side_effect=AssertionError("No real tools"))) + orchestrator = Orchestrator(workspace, site_config_root=selection.pop("site_config_root", None)) + result = orchestrator.build_plan( + workspace / "manifests" / manifest / "manifest.yaml", intent=PlanIntent.EXECUTABLE, **selection, + ) + assert result.status is PlanStatus.PLANNED, result.diagnostics + return result.plan + + +def _parameters(plan, step: str) -> dict: + operation = next(op for op in plan.targets[0].operations if op.identity.step == step) + assert operation.disposition is PlanDisposition.EXECUTE + assert isinstance(operation.details, DeploymentOperation) + return { + entry.key.value: entry.value for entry in operation.details.parameters.entries + if isinstance(entry.key, LiteralValue) + } + + +def _reader(): + calls = [] + related = { + INSTANCE_ID: {"extendedLocation": LOCATION_ID}, + LOCATION_ID: {"customLocations.hostResourceId": CLUSTER_ID}, + CLUSTER_ID: {}, + } + + def read(ref, *, facts=frozenset(), references=frozenset()): + calls.append((ref.resource_id, facts)) + assert set(related[ref.resource_id]) == references + return ArmResourceObservation( + ref.resource_id, ref.resource_type, "eastus", ref.name, {}, related[ref.resource_id], + ) + + return SimpleNamespace(identity=SimpleNamespace(name="fixture", version="1"), read=read), calls + + +def _observations() -> dict: + return { + "instance": ArmResourceObservation( + INSTANCE_ID, "Microsoft.IoTOperations/instances", "eastus", "external-instance", {}, + {"extendedLocation": LOCATION_ID}, + ), + "customLocation": ArmResourceObservation( + LOCATION_ID, "Microsoft.ExtendedLocation/customLocations", "eastus", "external-location", {}, + {"customLocations.hostResourceId": CLUSTER_ID}, + ), + "cluster": ArmResourceObservation( + CLUSTER_ID, "Microsoft.Kubernetes/connectedClusters", "eastus", "arc-first", {}, + ), + } + + +def _invoke(argv) -> int: + with patch.object(sys, "argv", ["siteops", *argv]): + with pytest.raises(SystemExit) as stopped: + main() + return stopped.value.code + + +@pytest.mark.parametrize("manifest", ["aio-install", "aio-upgrade"]) +def test_map_inputs_write_the_documented_site_maps(workspace, manifest): + contract = load_contract(workspace / "manifests" / manifest / "manifest.yaml") + fields = {field.name: field for field in contract.fields} + for name, parameter in MAPS.items(): + field = fields[name] + assert (field.type, field.required, field.site_path) == ("stringMap", False, ("parameters", parameter)) + assert fields["certManagerExtensionSettings"].when.input == "enableCertManager" + assert fields["certManagerExtensionSettings"].when.equals is True + assert fields["aioExtensionSettings"].when is None + assert fields["secretStoreExtensionSettings"].when is None + + +def test_install_example_stays_minimal_and_settings_follow_their_extensions(workspace): + contract = load_contract(workspace / "manifests" / "aio-install" / "manifest.yaml") + assert contract.example()["values"] == {"cluster": None} + site = contract.resolve(inline=[ + *MANUAL, "enableCertManager=false", "secretStoreExtensionSettings=rotationPollIntervalInSeconds=240", + ]) + assert site.properties["deployOptions"]["enableSecretSync"] is False + assert site.parameters["secretStoreConfigurationOverrides"] == {"rotationPollIntervalInSeconds": "240"} + assert "certManagerConfigurationOverrides" not in site.parameters + with pytest.raises(GuidedInputError, match="'certManagerExtensionSettings' was supplied while its condition is inactive"): + contract.resolve(inline=[*MANUAL, "enableCertManager=false", "certManagerExtensionSettings=k=v"]) + defaults = contract.resolve(inline=MANUAL) + assert not set(MAPS.values()) & set(defaults.parameters) + + +@pytest.mark.parametrize("release", ["2607", "2608", "2609"]) +def test_typed_install_settings_match_a_site_and_keep_bundled_cert_manager_values( + workspace, tmp_path, monkeypatch, release, +): + bundled = _release(workspace, release)["certManagerConfigurationOverrides"] + replaced = sorted(bundled)[0] + contract = load_contract(workspace / "manifests" / "aio-install" / "manifest.yaml") + answers = tmp_path / "answers.yaml" + answers.write_text(yaml.safe_dump({ + "apiVersion": "siteops.inputs/v1", "kind": "SiteInputValues", + "values": { + "aioRelease": release, + "aioExtensionSettings": {"observability.metrics.exportIntervalSeconds": "60"}, + "certManagerExtensionSettings": {replaced: "site"}, + }, + }), encoding="utf-8") + site = contract.resolve(values_file=answers, inline=[ + *MANUAL, "secretStoreExtensionSettings=rotationPollIntervalInSeconds=240", + ]) + expected = { + "aioConfigurationOverrides": {"observability.metrics.exportIntervalSeconds": "60"}, + "secretStoreConfigurationOverrides": {"rotationPollIntervalInSeconds": "240"}, + "certManagerConfigurationOverrides": {replaced: "site"}, + } + assert {name: site.parameters[name] for name in expected} == expected + plan = _plan(workspace, tmp_path, "aio-install", monkeypatch, sites=[site]) + assert resolve_plan_value(_parameters(plan, "aio-instance")["aioConfigurationOverrides"], {}) == ( + expected["aioConfigurationOverrides"] + ) + enablement = _parameters(plan, "aio-enablement") + assert resolve_plan_value(enablement["certManagerConfigurationOverrides"], {}) == {**bundled, replaced: "site"} + assert resolve_plan_value(enablement["secretStoreConfigurationOverrides"], {}) == ( + expected["secretStoreConfigurationOverrides"] + ) + + +def test_upgrade_contract_requires_the_release_and_the_cert_manager_answer(workspace, tmp_path): + contract = load_contract(workspace / "manifests" / "aio-upgrade" / "manifest.yaml") + assert contract.example()["values"] == {"instance": None, "aioRelease": None, "enableCertManager": None} + rows = {row["name"]: row for row in contract.describe()["inputs"]} + for name in ("aioRelease", "enableCertManager"): + assert rows[name]["status"] == "required" and "default" not in rows[name] + assert rows["enableCertManager"]["description"].startswith( + "Include the existing cert-manager Arc extension in this upgrade?" + ) + assert all(not field.resource.requires for field in contract.fields if field.resource is not None) + example = tmp_path / "example.yaml" + example.write_text(yaml.safe_dump(contract.example()), encoding="utf-8") + with pytest.raises(GuidedInputError) as missing: + contract.bind(values_file=example) + assert str(missing.value) == ( + "Missing required input 'instance'. Supply `--input instance=`." + ) + instance = f"instance={INSTANCE_ID}" + with pytest.raises(GuidedInputError) as missing: + contract.bind(inline=[instance, "aioRelease=2609"]) + assert str(missing.value) == ( + "Missing required input 'enableCertManager'. Supply `--input enableCertManager=`." + ) + with pytest.raises(GuidedInputError) as missing: + contract.bind(inline=[instance, "enableCertManager=true"]) + assert str(missing.value) == "Missing required input 'aioRelease'. Supply `--input aioRelease=`." + with pytest.raises(GuidedInputError, match="'certManagerExtensionSettings' was supplied while its condition is inactive"): + contract.bind(inline=[instance, "aioRelease=2609", "enableCertManager=false", "certManagerExtensionSettings=k=v"]) + excluded = contract.build_site( + contract.bind(inline=[instance, "aioRelease=2609", "enableCertManager=false"]), _observations(), + ) + assert excluded.properties == {"aioRelease": "2609", "deployOptions": {"enableCertManager": False}} + assert (excluded.name, excluded.parameters) == ( + "arc-first", {"aioInstanceName": "external-instance", "clusterName": "arc-first"}, + ) + + +@pytest.mark.parametrize("release", ["2607", "2608", "2609"]) +def test_typed_upgrade_applies_settings_over_release_values_at_the_actual_instance( + workspace, tmp_path, monkeypatch, release, +): + bundled = _release(workspace, release)["certManagerConfigurationOverrides"] + replaced = sorted(bundled)[0] + contract = load_contract(workspace / "manifests" / "aio-upgrade" / "manifest.yaml") + site = contract.build_site(contract.bind(inline=[ + f"instance={INSTANCE_ID}", f"aioRelease={release}", "enableCertManager=true", + "aioExtensionSettings=observability.metrics.exportIntervalSeconds=60", + "secretStoreExtensionSettings=rotationPollIntervalInSeconds=240", + f"certManagerExtensionSettings={replaced}=site", + ]), _observations()) + plan = _plan(workspace, tmp_path, "aio-upgrade", monkeypatch, sites=[site]) + assert [op.identity.step for op in plan.targets[0].operations] == [ + "resolve-aio", "resolve-extensions", "update-extensions", "deploy-release-resources", + ] + assert resolve_plan_value(_parameters(plan, "resolve-aio")["aioInstanceName"], {}) == "external-instance" + update = _parameters(plan, "update-extensions") + assert resolve_plan_value(update["aioConfigurationOverrides"], {}) == { + "observability.metrics.exportIntervalSeconds": "60", + } + assert resolve_plan_value(update["secretStoreConfigurationOverrides"], {}) == { + "rotationPollIntervalInSeconds": "240", + } + assert resolve_plan_value(update["certManagerConfigurationOverrides"], {}) == {**bundled, replaced: "site"} + assert resolve_plan_value(update["enableCertManager"], {}) is True + + +def test_typed_upgrade_reads_the_instance_route_without_identity_prerequisites(workspace, tmp_path, capsys): + reader, calls = _reader() + with ( + patch("siteops.cli.new_arm_reader", return_value=reader), + patch("siteops.orchestrator.TemplateCompilationSession", return_value=_session(tmp_path)), + patch.object(subprocess, "Popen", MagicMock(side_effect=AssertionError("No real tools"))), + ): + assert _invoke([ + "-w", str(workspace), "plan", "aio-upgrade", "--input", f"instance={INSTANCE_ID}", + "--input", "aioRelease=2609", "--input", "enableCertManager=false", "--output", "json", + ]) == 0 + document = json.loads(capsys.readouterr().out) + assert [target["name"] for target in document["plan"]["targets"]] == ["arc-first"] + assert document["plan"]["manifest"]["targetSelection"] == "explicit-site" + assert calls == [(INSTANCE_ID, frozenset()), (LOCATION_ID, frozenset()), (CLUSTER_ID, frozenset())] + + +def test_configured_and_site_file_upgrade_routes_read_no_contract(workspace, tmp_path, capsys): + site_file = tmp_path / "plant.yaml" + site_file.write_text(yaml.safe_dump({ + "apiVersion": "siteops/v1", "kind": "Site", "name": "plant", + "subscription": "00000000-0000-0000-0000-000000000001", "resourceGroup": "rg-existing", + "location": "eastus", "properties": {"aioRelease": "2609", "deployOptions": {"enableCertManager": True}}, + "parameters": {"aioConfigurationOverrides": {"example.setting": "site"}}, + }), encoding="utf-8") + with ( + patch("siteops.cli.load_contract", side_effect=AssertionError("No contract read")), + patch("siteops.cli.new_arm_reader", side_effect=AssertionError("No Azure read")), + ): + for selection in (["-l", "name=munich-dev"], ["--site-file", str(site_file)]): + assert _invoke([ + "-w", str(workspace), "plan", "aio-upgrade", "--describe", *selection, "--output", "json", + ]) == 0 + document = json.loads(capsys.readouterr().out) + assert len(document["plan"]["targets"]) == 1 + + +def test_inherited_and_overlaid_site_maps_reach_install_and_upgrade(workspace, tmp_path, monkeypatch): + project = tmp_path / "project" + (project / "sites").mkdir(parents=True) + (project / "sites.local").mkdir() + (project / "sites" / "fleet.yaml").write_text(yaml.safe_dump({ + "apiVersion": "siteops/v1", "kind": "SiteTemplate", "name": "fleet", + "properties": {"aioRelease": "2609", "deployOptions": { + "enableGlobalSite": False, "enableEdgeSite": False, "enableSecretSync": False, + "enableCertManager": True, + }}, + "parameters": { + "aioConfigurationOverrides": {"shared": "fleet", "replaced": "fleet"}, + "secretStoreConfigurationOverrides": {"rotationPollIntervalInSeconds": "120"}, + "certManagerConfigurationOverrides": {"trust-manager.secretTargets.enabled": "true"}, + }, + }), encoding="utf-8") + (project / "sites" / "plant.yaml").write_text(yaml.safe_dump({ + "apiVersion": "siteops/v1", "kind": "Site", "name": "plant", "inherits": "fleet.yaml", + "subscription": "00000000-0000-0000-0000-000000000001", "resourceGroup": "rg-existing", + "location": "eastus", "parameters": { + "clusterName": "existing-arc", "aioConfigurationOverrides": {"replaced": "site", "added": "site"}, + }, + }), encoding="utf-8") + (project / "sites.local" / "plant.yaml").write_text(yaml.safe_dump({ + "parameters": {"secretStoreConfigurationOverrides": {"rotationPollIntervalInSeconds": "240"}}, + }), encoding="utf-8") + aio = {"shared": "fleet", "replaced": "site", "added": "site"} + cert_manager = {**_release(workspace, "2609")["certManagerConfigurationOverrides"], + "trust-manager.secretTargets.enabled": "true"} + secret_store = {"rotationPollIntervalInSeconds": "240"} + install = _plan(workspace, tmp_path, "aio-install", monkeypatch, selector="name=plant", site_config_root=project) + assert resolve_plan_value(_parameters(install, "aio-instance")["aioConfigurationOverrides"], {}) == aio + enablement = _parameters(install, "aio-enablement") + assert resolve_plan_value(enablement["certManagerConfigurationOverrides"], {}) == cert_manager + assert resolve_plan_value(enablement["secretStoreConfigurationOverrides"], {}) == secret_store + upgrade = _plan(workspace, tmp_path, "aio-upgrade", monkeypatch, selector="name=plant", site_config_root=project) + update = _parameters(upgrade, "update-extensions") + assert resolve_plan_value(update["aioConfigurationOverrides"], {}) == aio + assert resolve_plan_value(update["certManagerConfigurationOverrides"], {}) == cert_manager + assert resolve_plan_value(update["secretStoreConfigurationOverrides"], {}) == secret_store + + +_ROOT = Path(__file__).resolve().parents[2] +_DOCUMENTS = ( + "docs/guided-inputs.md", + "workspaces/iot-operations/manifests/aio-install/README.md", + "workspaces/iot-operations/manifests/aio-upgrade/README.md", +) + + +@pytest.mark.parametrize("document", ["docs/guided-inputs.md", "workspaces/iot-operations/manifests/aio-upgrade/README.md"]) +def test_documented_upgrade_commands_resolve_through_the_cli(workspace, tmp_path, monkeypatch, capsys, document): + for name in ("CI", "GITHUB_ACTIONS", "TF_BUILD"): + monkeypatch.delenv(name, raising=False) + monkeypatch.setenv("SITEOPS_REDACT_OUTPUT", "0") + monkeypatch.setattr(sys.stdin, "isatty", lambda: True) + monkeypatch.setattr(sys.stderr, "isatty", lambda: True) + monkeypatch.setattr(sys.stdin, "readline", lambda: "yes\n") + commands = [ + line for line in (_ROOT / document).read_text(encoding="utf-8").splitlines() + if line.startswith("siteops ") and " aio-upgrade " in line and "" in line + ] + assert commands + # The documented command leaves the cert-manager decision to the operator. + assert all('"enableCertManager="' in line for line in commands) + for line, answer in ((line, answer) for line in commands for answer in ("true", "false")): + reader, calls = _reader() + line = line.replace("", answer) + with ( + patch("siteops.cli.open_command_context", side_effect=lambda **kwargs: nullcontext( + CommandContext(workspace, workspace), + )), + patch("siteops.cli.new_arm_reader", return_value=reader), + patch("siteops.orchestrator.TemplateCompilationSession", return_value=_session(tmp_path)), + patch.object(subprocess, "Popen", MagicMock(side_effect=AssertionError("No real tools"))), + patch.object(Orchestrator, "execute_plan", return_value=RunResult.from_sites((), elapsed=0)) as execute, + ): + assert _invoke(shlex.split(line.replace("", INSTANCE_ID))[1:]) == 0 + plan = execute.call_args.args[0].plan + assert [op.identity.step for op in plan.targets[0].operations][1:3] == [ + "resolve-extensions", "update-extensions", + ] + update = next(op for op in plan.targets[0].operations if op.identity.step == "update-extensions") + enabled = {entry.key.value: entry.value for entry in update.details.parameters.entries + if isinstance(entry.key, LiteralValue)}["enableCertManager"] + assert resolve_plan_value(enabled, {}) is (answer == "true") + assert [resource for resource, _ in calls] == [INSTANCE_ID, LOCATION_ID, CLUSTER_ID] + capsys.readouterr() + + +def test_only_upgrade_is_documented_to_keep_existing_settings(workspace): + # Installation writes extension settings without reading the cluster's, while upgrade starts from them. + install_templates = [workspace / "templates" / "aio" / "enablement.bicep", + *sorted((workspace / "templates" / "aio" / "modules").glob("instance-*.bicep"))] + for template in install_templates: + assert not re.search(r"KubernetesConfiguration/extensions@[^']*' existing", template.read_text(encoding="utf-8")) + upgrade = (workspace / "templates" / "aio" / "upgrade" / "update-extensions.bicep").read_text(encoding="utf-8") + assert "union(secretStore.configurationSettings, secretStoreConfigurationOverrides)" in upgrade + for document in ("docs/guided-inputs.md", "workspaces/iot-operations/manifests/aio-install/README.md"): + text = (_ROOT / document).read_text(encoding="utf-8") + prose = " ".join(text.split()) + assert "removes a key that the cluster already has" not in prose + assert "never clears a setting the cluster already has" not in prose + assert "without reading the" in prose + assert "aio-upgrade/README.md#change-extension-settings" in text + + +def test_upgrade_guide_maps_each_release_to_its_version_and_forbids_downgrades(workspace): + readme = (workspace / "manifests" / "aio-upgrade" / "README.md").read_text(encoding="utf-8") + for release in sorted((workspace / "parameters" / "aio-releases").glob("*.yaml")): + version = str(yaml.safe_load(release.read_text(encoding="utf-8"))["aioVersion"]) + assert f"| {release.stem} | {version} | {'.'.join(version.split('.')[:2])} |" in readme + prose = " ".join(readme.split()) + assert "--query properties.version" in readme + assert "Never select an AIO release older than the one the instance runs" in prose + assert "returns to the release value on upgrade unless you supply it again" in prose + assert "../aio-install/README.md#settings-scale-kit-derives" in readme + + +def _answer_examples(document: str) -> list[dict]: + text = (_ROOT / document).read_text(encoding="utf-8") + blocks = re.findall(r"```yaml\n(.*?)```", text, re.DOTALL) + return [yaml.safe_load(block) for block in blocks if "kind: SiteInputValues" in block] + + +@pytest.mark.parametrize("document", _DOCUMENTS) +def test_documented_answer_files_with_settings_bind_to_their_contract(workspace, tmp_path, document): + examples = [example for example in _answer_examples(document) if set(MAPS) & set(example["values"])] + assert examples + for number, example in enumerate(examples): + manifest = "aio-upgrade" if "instance" in example["values"] else "aio-install" + if manifest == "aio-upgrade": + assert example["values"]["enableCertManager"] == "" + example["values"]["enableCertManager"] = True + values = { + name: (INSTANCE_ID if name == "instance" else CLUSTER_ID.replace("rg-first", "rg-second")) + if name in {"instance", "cluster"} else value + for name, value in example["values"].items() + } + path = tmp_path / f"answers-{number}.yaml" + path.write_text(yaml.safe_dump({**example, "values": values}), encoding="utf-8") + bound = load_contract(workspace / "manifests" / manifest / "manifest.yaml").bind(values_file=path) + for name in set(MAPS) & set(values): + assert dict(bound.active_values[name]) == values[name] + assert all(isinstance(value, str) for value in values[name].values()) diff --git a/workspaces/iot-operations/manifests/aio-install/README.md b/workspaces/iot-operations/manifests/aio-install/README.md index cb386ba..45f280d 100644 --- a/workspaces/iot-operations/manifests/aio-install/README.md +++ b/workspaces/iot-operations/manifests/aio-install/README.md @@ -30,12 +30,51 @@ those defaults. Keep an existing explicit Site name when reusing configuration. IDs produced by earlier operations are supplied to their consuming steps, rather than entered by the operator. -To set Arc extension configuration, add override maps under the Site's -`parameters`. `aioConfigurationOverrides` applies to AIO, -`certManagerConfigurationOverrides` to cert-manager, and -`secretStoreConfigurationOverrides` to the secret store. Site values take -precedence over release configuration and template defaults. Use strings for -setting values. These settings are visible in Azure and are not for secrets: +Optional `properties.deployOptions` settings include `enableEdgeSite`, +`enableSecretSync` and `enableCertManager`. Secret Sync requires OIDC and +workload identity on the existing cluster. Host bootstrap flags do not enable +those features through this manifest. An Edge Site scoped to the subscription +needs a Site at subscription level, not just a flag on a Site at resource +group level. +The guided resource ID route checks the reported identity prerequisites +before writes when Secret Sync is enabled. Configured fleet Sites do not +repeat that guided read, so confirm the prerequisites for each selected +cluster before enabling Secret Sync across a fleet. + +### Arc extension settings + +Three maps of string keys and string values set Arc extension configuration. +Each key replaces the release or template value of that key. These settings +are visible in Azure, so never put secrets in them. + +| Typed input | Site parameter | Arc extension | +|---|---|---| +| `aioExtensionSettings` | `aioConfigurationOverrides` | AIO | +| `secretStoreExtensionSettings` | `secretStoreConfigurationOverrides` | Secret store, with Secret Sync enabled or disabled | +| `certManagerExtensionSettings` | `certManagerConfigurationOverrides` | cert-manager, when `enableCertManager` is true | + +On the guided route, add one `--input NAME=KEY=VALUE` to the deploy command +for each setting, for example +`--input "aioExtensionSettings=observability.metrics.exportIntervalSeconds=60"`. + +An answer file holds the same maps. Quote keys and values, because YAML +reads an unquoted `on` as true and `1.10` as 1.1: + +```yaml +apiVersion: siteops.inputs/v1 +kind: SiteInputValues +values: + cluster: "" + aioExtensionSettings: + "observability.metrics.exportIntervalSeconds": "60" +``` + +Inline entries replace answer file entries key by key. A key that contains +`=` is supported only in the answer file. See +[guided inputs](../../../../docs/guided-inputs.md#set-arc-extension-settings) +for the complete grammar. + +For a configured Site, set the Site parameters directly: ```yaml parameters: @@ -48,21 +87,67 @@ parameters: ``` Replace the placeholders with settings supported by each extension. -Typed inputs do not accept these maps. For the guided route, -[save a Site](../../../../docs/guided-inputs.md#keep-a-site-for-later) -with `siteops inputs ... --save-site` into the configured inventory, add the -override maps to that file, then deploy with `-l name=`. +Inherited Sites and `sites.local` overlays merge these maps key by key, so a +child Site adds or replaces keys but cannot remove one. An omitted or empty +typed map adds no settings. + +Installation writes each extension's settings from the template defaults, +the AIO release and the Site, without reading the settings the cluster +already has. Running installation again on an existing installation can +therefore replace or drop settings that you changed since. To change the +settings of an existing installation, run +[aio-upgrade](../aio-upgrade/README.md#change-extension-settings) at its +current AIO release, which keeps the existing settings and removes no key. + +### Settings Scale Kit derives + +This installation derives some AIO extension settings from other Site +values. A map key replaces any of them: + +| AIO extension settings | Derived from | +|---|---| +| `trustSource`, `trustBundleSettings.issuer.name`, `trustBundleSettings.issuer.kind`, `trustBundleSettings.configMap.name`, `trustBundleSettings.configMap.key` | `parameters.trustConfig` | +| `dataFlows.values.wasmGraphController.mqttBroker.caCertConfigMapRef`, `dataFlows.values.wasmGraphController.mqttBroker.caCertFileName` | `parameters.trustConfig`, for AIO releases 2608 and 2609 | +| `connectors.values.mqttBroker.address`, `connectors.values.mqttBroker.serviceAccountTokenAudience`, `dataFlows.values.tinyKube.mqttBroker.hostName`, `dataFlows.values.tinyKube.mqttBroker.port`, `dataFlows.values.tinyKube.mqttBroker.authentication.serviceAccountTokenAudience`, `schemaRegistry.values.mqttBroker.host`, `schemaRegistry.values.mqttBroker.serviceAccountTokenAudience` | The cluster namespace and the default broker listener | +| `connectors.values.securityPki.applicationUri` | The cluster, for AIO 2603 and later | +| `connectors.values.securityPki.subjectName` | The cluster, for AIO 2605 and later | +| `observability.metrics.enabled`, `observability.metrics.openTelemetryCollectorAddress` | The instance template's observability defaults | + +The curated installation keeps the trust, WebAssembly graph trust and broker +settings consistent with each other. Replacing those keys through a map is +outside that recipe. Use the dedicated settings instead: + +- `parameters.trustConfig` for a trust bundle you manage +- `parameters.brokerConfig` for broker sizing and logging +- dataflow endpoints in [resource sets](../../resource-sets/README.md) for + MQTT destinations outside the cluster +- your own manifests and templates for a materially different internal + topology + +For example, select an existing issuer and trust bundle: -Optional `properties.deployOptions` settings include `enableEdgeSite`, -`enableSecretSync` and `enableCertManager`. Secret Sync requires OIDC and -workload identity on the existing cluster. Host bootstrap flags do not enable -those features through this manifest. An Edge Site scoped to the subscription -needs a Site at subscription level, not just a flag on a Site at resource -group level. -The guided resource ID route checks the reported identity prerequisites -before writes when Secret Sync is enabled. Configured fleet Sites do not -repeat that guided read, so confirm the prerequisites for each selected -cluster before enabling Secret Sync across a fleet. +```yaml +parameters: + trustConfig: + source: CustomerManaged + settings: + issuerName: "" + issuerKind: ClusterIssuer + configMapName: "" + configMapKey: "" +``` + +`trustConfig` is a Site parameter, not a typed input. To use it on the guided +route, save the Site as the next paragraph describes. Replacing the OPC UA +certificate and observability settings above is ordinary use, for example to +match values that an OPC UA server already trusts or to send metrics to your +own collector. + +To keep the guided answers for later, including settings and parameters +outside the typed inputs, +[save a Site](../../../../docs/guided-inputs.md#keep-a-site-for-later) +with `siteops inputs ... --save-site` into the configured inventory, edit +that file, then deploy with `-l name=`. ## Review and deploy diff --git a/workspaces/iot-operations/manifests/aio-install/entry.yaml b/workspaces/iot-operations/manifests/aio-install/entry.yaml index 0f8ef02..673cd5f 100644 --- a/workspaces/iot-operations/manifests/aio-install/entry.yaml +++ b/workspaces/iot-operations/manifests/aio-install/entry.yaml @@ -57,6 +57,24 @@ inputs: requirement: optional description: Broker settings have inherited or template defaults. The typed route starts with the Medium memory profile. Review capacity and trust requirements. source: templates/aio/instance.bicep + - field: parameters.aioConfigurationOverrides + type: mapping + requirement: optional + sensitivity: non-sensitive + description: AIO Arc extension settings as string keys and values, also set by the typed aioExtensionSettings input. Each key replaces the release or template value. The values are visible in Azure, so never include secrets. + source: templates/aio/instance.bicep + - field: parameters.secretStoreConfigurationOverrides + type: mapping + requirement: optional + sensitivity: non-sensitive + description: Secret store Arc extension settings as string keys and values, also set by the typed secretStoreExtensionSettings input. They apply with Secret Sync enabled or disabled. Never include secrets. + source: templates/aio/enablement.bicep + - field: parameters.certManagerConfigurationOverrides + type: mapping + requirement: conditional + sensitivity: non-sensitive + description: cert-manager Arc extension settings when enableCertManager is true, also set by the typed certManagerExtensionSettings input. Each key replaces the release or template value. Never include secrets. + source: templates/aio/enablement.bicep - field: properties.deployOptions type: mapping requirement: conditional diff --git a/workspaces/iot-operations/manifests/aio-install/inputs.yaml b/workspaces/iot-operations/manifests/aio-install/inputs.yaml index 9e2d3fb..9cab35b 100644 --- a/workspaces/iot-operations/manifests/aio-install/inputs.yaml +++ b/workspaces/iot-operations/manifests/aio-install/inputs.yaml @@ -94,3 +94,19 @@ inputs: description: AIO broker memory profile. Review cluster capacity before selecting another profile. sitePath: parameters.brokerConfig.memoryProfile default: Medium + - name: aioExtensionSettings + type: stringMap + required: false + description: AIO Arc extension configuration settings. Each key replaces the release or template value. + sitePath: parameters.aioConfigurationOverrides + - name: secretStoreExtensionSettings + type: stringMap + required: false + description: Secret store Arc extension configuration settings, with Secret Sync enabled or disabled. Each key replaces the release or template value. + sitePath: parameters.secretStoreConfigurationOverrides + - name: certManagerExtensionSettings + type: stringMap + required: false + when: {input: enableCertManager, equals: true} + description: cert-manager Arc extension configuration settings when this installation manages cert-manager. Each key replaces the release or template value. + sitePath: parameters.certManagerConfigurationOverrides diff --git a/workspaces/iot-operations/manifests/aio-upgrade/README.md b/workspaces/iot-operations/manifests/aio-upgrade/README.md index 9049ebe..b4fb07d 100644 --- a/workspaces/iot-operations/manifests/aio-upgrade/README.md +++ b/workspaces/iot-operations/manifests/aio-upgrade/README.md @@ -1,29 +1,148 @@ # Upgrade Azure IoT Operations -Upgrade an existing AIO installation using the Site's selected -`properties.aioRelease`. The manifest updates the Arc extensions for AIO and -the resources that the selected AIO release explicitly requires. +Upgrade an existing AIO installation to a selected AIO release. The manifest +updates the Arc extensions for AIO and the resources that the selected AIO +release explicitly requires. Select the target with the Site's +`properties.aioRelease`, or with typed inputs for one existing instance. The instance ARM resource has no writable version property. This operation does not perform general migrations of brokers, dataflow profiles or other instance children. +## Choose a supported upgrade path + +Microsoft supports upgrading an instance to any patch of the same minor +version, or to the next minor version. To downgrade, or to move between a +preview and a generally available version in either direction, uninstall +and install the desired version instead. AIO does not upgrade live, so +expect downtime. See +[Upgrade Azure IoT Operations](https://learn.microsoft.com/azure/iot-operations/manage-iot-ops/howto-upgrade). + +Each AIO release in this workspace installs one AIO version: + +| AIO release | AIO version | Minor version | +|---|---|---| +| 2512 | 1.2.154 | 1.2 | +| 2602 | 1.2.189 | 1.2 | +| 2603 | 1.3.38 | 1.3 | +| 2604 | 1.3.70 | 1.3 | +| 2605 | 1.3.105 | 1.3 | +| 2606 | 1.3.137 | 1.3 | +| 2607 | 1.4.41 | 1.4 | +| 2608 | 1.4.73 | 1.4 | +| 2609 | 1.4.112 | 1.4 | + +Before you choose a target, find the version the instance runs. The +instance's Overview page in the Azure portal shows it, and so does +`az iot ops show --name --resource-group --query properties.version`. +The AIO Arc extension's version, on the cluster's Extensions page or from +`az k8s-extension list --cluster-name --resource-group --cluster-type connectedClusters --output table`, +matches the AIO version column exactly. + +Move one minor version at a time. From a 1.2 release, upgrade to a 1.3 +release before a 1.4 release. Never select an AIO release older than the one +the instance runs, because Microsoft does not support a downgrade. +`aio-upgrade` applies the AIO release you select without checking the path, +so choose a supported transition from the instance's current release. + +The MQTT broker upgrades through rolling updates, which need a backend +redundancy factor of 2 or more. Microsoft does not support broker upgrades +with a factor of 1. Scale Kit installs with 2 by default, so keep +`parameters.brokerConfig.backendRedundancyFactor` at 2 or more. + +Scale Kit's live tests have exercised selected adjacent upgrades: 2604 to +2605, 2606 to 2607 and 2608 to 2609. These tests support the content in +this workspace. They do not certify another cluster or upgrade path, +which Microsoft's guidance governs. + ## Prepare the upgrade -Confirm the configured Site identifies the existing instance and cluster. -Choose a supported transition and update `properties.aioRelease` on the Site -or its inherited configuration. See the -[release reference](../../../../docs/aio-releases.md#supported-upgrade-paths) -for supported transitions and AIO release selection. +For a configured Site, confirm it identifies the existing instance and +cluster. Update `properties.aioRelease` on the Site or its inherited +configuration. See [AIO releases](../../../../docs/aio-releases.md) for +AIO release selection. + +When you saved the installation's Site with `siteops inputs ... --save-site`, +change `properties.aioRelease` in that file. Run this manifest with +`--site-file FILE`, or with `-l name=` when the file is in your +configured Site inventory. Typed answers never change a saved Site. Prefer +the saved Site for an installation you configured, because an instance ID +cannot recover a custom Site name or earlier choices. The AIO release pin also selects API versions for workload resources. Avoid applying resource sets between changing the pin and completing the upgrade. The extension update preserves existing configuration, release train and identity settings, while applying the changes that the AIO release owns. -Site extension override maps merge over the existing settings, so they can add -or change keys but cannot remove them. See the -[install Site settings](../aio-install/README.md#configure-the-site) for the maps. +## Upgrade one instance with typed inputs + +For an existing instance without a saved Site, supply its resource ID, the +target AIO release and your cert-manager answer: + +```text +siteops deploy aio-upgrade --source "official@" --input "instance=" --input aioRelease=2609 --input "enableCertManager=" +``` + +Site Ops reads the instance, its custom location and that location's Arc +cluster with your Azure CLI identity. The related resources must stay in the +instance's subscription and resource group. The upgrade uses the actual +instance name. The Site name defaults to the cluster's name in lowercase. +Supply `siteName` to choose another, and the optional `environment` and +`country` inputs to label the Site. Run `siteops inputs aio-upgrade` with the +same `--source` to list every input. + +`aioRelease` has no default, so the upgrade never selects a target for you. +`enableCertManager` has no default either. It answers the question "Include +the existing cert-manager Arc extension in this upgrade?" Answer `true` when +Scale Kit installed cert-manager and manages it, as an installation with the +default `enableCertManager` does, and the upgrade updates its version and +settings. Answer `false` when another owner manages cert-manager, and the +upgrade neither reads nor updates that extension. The instance cannot tell +Site Ops which applies. + +The extension settings inputs from +[installation](../aio-install/README.md#arc-extension-settings) also apply +here. `certManagerExtensionSettings` needs `enableCertManager=true`. For +example, this answer file points AIO metrics at an OpenTelemetry collector +that you run separately: + +```yaml +apiVersion: siteops.inputs/v1 +kind: SiteInputValues +values: + instance: "" + aioRelease: "2609" + enableCertManager: + aioExtensionSettings: + "observability.metrics.enabled": "true" + "observability.metrics.openTelemetryCollectorAddress": "aio-otel-collector.azure-iot-operations.svc.cluster.local:4317" +``` + +Replace each placeholder, then supply the file with `--input-file` instead +of the inline answers. A setting you made at installation for a key that the +AIO release also sets, such as the cert-manager settings of releases 2607 to +2609, returns to the release value on upgrade unless you supply it again. +[Settings Scale Kit derives](../aio-install/README.md#settings-scale-kit-derives) +lists the AIO settings that the installation computes. Deploy shows +the read resource IDs and the plan, then asks for confirmation. Add `--yes` +only for explicitly authorized unattended execution. + +## Change extension settings + +Each extension update starts from the settings already on the cluster, so +settings applied at installation stay in place and the same Site applies +them again. A setting from the AIO release replaces the cluster's value on +each upgrade unless the Site sets that key. Removing a key from the Site +never removes it from the cluster. Running installation again can overwrite +settings that you manage, because installation writes the settings without +reading the cluster's. + +To change only settings, run `aio-upgrade` with the instance's current AIO +release and the new settings. It applies the same extension versions with +those settings and deploys the release's required resources again. Scale +Kit's live tests do not yet cover this route, so review the plan and check +extension provisioning afterward. Keep the current release for a settings +change rather than selecting a newer one. ## Review and deploy diff --git a/workspaces/iot-operations/manifests/aio-upgrade/entry.yaml b/workspaces/iot-operations/manifests/aio-upgrade/entry.yaml index c290dd9..048eaa0 100644 --- a/workspaces/iot-operations/manifests/aio-upgrade/entry.yaml +++ b/workspaces/iot-operations/manifests/aio-upgrade/entry.yaml @@ -5,13 +5,53 @@ category: core tags: [aio, upgrade] documentation: [README.md] outcome: Upgrade the AIO extensions and the resources that the selected AIO release explicitly requires. +inputs: + - field: properties.aioRelease + type: string + requirement: required + sensitivity: non-sensitive + description: The target AIO release. The typed aioRelease input has no default, so the upgrade never selects a target. Choose a supported transition from the instance's current release. + source: sites/base-site.yaml + - field: properties.deployOptions.enableCertManager + type: boolean + requirement: required + sensitivity: non-sensitive + description: Include the existing cert-manager Arc extension in this upgrade? True updates its version and settings. False leaves it to its separate owner, with no read or write. The typed input has no default. + source: templates/aio/upgrade/update-extensions.bicep + - field: parameters.aioInstanceName + type: string + requirement: optional + sensitivity: non-sensitive + description: The existing instance name. The typed instance input derives it from the resource ID. + defaultBehavior: The shared fallback derives the name from the Site name. + source: parameters/common/common.yaml + - field: parameters.aioConfigurationOverrides + type: mapping + requirement: optional + sensitivity: non-sensitive + description: AIO Arc extension settings, also set by the typed aioExtensionSettings input. Each key replaces the existing or release value, and no key is removed. Never include secrets. + source: templates/aio/upgrade/update-extensions.bicep + - field: parameters.secretStoreConfigurationOverrides + type: mapping + requirement: optional + sensitivity: non-sensitive + description: Secret store Arc extension settings, also set by the typed secretStoreExtensionSettings input. Each key replaces the existing or release value. Never include secrets. + source: templates/aio/upgrade/update-extensions.bicep + - field: parameters.certManagerConfigurationOverrides + type: mapping + requirement: conditional + sensitivity: non-sensitive + description: cert-manager Arc extension settings when enableCertManager is true, also set by the typed certManagerExtensionSettings input. Each key replaces the existing or release value. Never include secrets. + source: templates/aio/upgrade/update-extensions.bicep prerequisites: - - An existing AIO installation and a supported upgrade transition. - - A configured Site with the intended AIO release and permission for the selected updates. + - An existing AIO installation and a supported upgrade transition, any patch of the same minor version or the next minor version. + - A configured or saved Site with the intended AIO release, or the existing instance ID with typed answers, and permission for the selected updates. + - A broker backend redundancy factor of 2 or more, the default, so the broker can upgrade. effects: - Updates extensions and the resources the AIO release requires while preserving supported existing configuration. + - With enableCertManager false, leaves the cert-manager extension unread and unchanged. - Does not perform general migrations of instance child resources. - Avoid resource set deployments between changing the AIO release pin and completing the upgrade. removal: - There is no general rollback. Review partial outcomes and recovery guidance before retrying. -coverage: Operational synopsis. Follow the guide for AIO release selection, supported paths and workload observations after the upgrade. +coverage: Operational synopsis. Follow the guide for AIO release selection, supported paths, typed inputs and workload observations after the upgrade. diff --git a/workspaces/iot-operations/manifests/aio-upgrade/inputs.yaml b/workspaces/iot-operations/manifests/aio-upgrade/inputs.yaml new file mode 100644 index 0000000..006d97f --- /dev/null +++ b/workspaces/iot-operations/manifests/aio-upgrade/inputs.yaml @@ -0,0 +1,93 @@ +apiVersion: siteops.inputs/v1 +kind: SiteInputContract +nameFromResource: cluster +inputs: + - name: instance + type: azureResourceId + description: Existing AIO instance resource ID. Site Ops reads the instance, its custom location and that location's Arc cluster. + resource: + type: Microsoft.IoTOperations/instances + apiVersion: "2026-07-01" + derive: + subscription: subscription + resourceGroup: resourceGroup + location: location + name: aioInstanceName + - name: siteName + type: string + description: Optional Site name. The default is the associated cluster's name in lowercase. + sitePath: name + format: dnsLabel + maxLength: 59 + - name: subscription + type: string + description: Subscription derived from the existing instance. + sitePath: subscription + - name: resourceGroup + type: string + description: Resource group derived from the existing instance. + sitePath: resourceGroup + - name: location + type: string + description: Region read from the existing instance. + sitePath: location + - name: aioInstanceName + type: string + description: Actual instance name derived from its resource ID. + sitePath: parameters.aioInstanceName + - name: clusterName + type: string + description: Cluster name derived through the instance's custom location. + sitePath: parameters.clusterName + - name: customLocation + type: azureResourceId + description: Custom location referenced by the existing instance. + fromResource: {input: instance, field: extendedLocation} + resource: + type: Microsoft.ExtendedLocation/customLocations + apiVersion: "2021-08-31-preview" + - name: cluster + type: azureResourceId + description: Existing Arc cluster referenced by the custom location. + fromResource: {input: customLocation, field: customLocations.hostResourceId} + resource: + type: Microsoft.Kubernetes/connectedClusters + apiVersion: "2024-07-15-preview" + derive: + name: clusterName + - name: aioRelease + type: string + description: Target AIO release, such as 2609, never older than the instance runs. The aio-upgrade guide lists the AIO version of each release. + sitePath: properties.aioRelease + - name: enableCertManager + type: boolean + description: Include the existing cert-manager Arc extension in this upgrade? True updates its version and settings. False leaves it to its separate owner, with no read or write. + sitePath: properties.deployOptions.enableCertManager + - name: aioExtensionSettings + type: stringMap + required: false + description: AIO Arc extension configuration settings. Each key replaces the existing or release value. + sitePath: parameters.aioConfigurationOverrides + - name: secretStoreExtensionSettings + type: stringMap + required: false + description: Secret store Arc extension configuration settings. Each key replaces the existing or release value. + sitePath: parameters.secretStoreConfigurationOverrides + - name: certManagerExtensionSettings + type: stringMap + required: false + when: {input: enableCertManager, equals: true} + description: cert-manager Arc extension configuration settings. Each key replaces the existing or release value. + sitePath: parameters.certManagerConfigurationOverrides + - name: environment + type: string + required: false + format: nonEmpty + description: Optional environment label for the Site. + sitePath: labels.environment + - name: country + type: string + required: false + format: nonEmpty + description: Optional country label for the Site. + sitePath: labels.country diff --git a/workspaces/iot-operations/siteops-index.inputs.json b/workspaces/iot-operations/siteops-index.inputs.json index d76c187..16236c6 100644 --- a/workspaces/iot-operations/siteops-index.inputs.json +++ b/workspaces/iot-operations/siteops-index.inputs.json @@ -27,7 +27,7 @@ "templates/host-bootstrap/aksee/_partial.yaml", "templates/host-ops/aksee-upgrade/_partial.yaml" ], - "indexDigest": "aa7c3c088e72964ffca84084b2f5f781cfd6eab80ae456bbf4ea3339726ec121", + "indexDigest": "7aff412a9b74b83276ae8de8b896b6d91f64bb6b78f34291f405e1c16cf1405f", "inputs": [ { "digests": { @@ -135,9 +135,9 @@ }, { "digests": { - "git-blob-sha1": "0f8ef02c8ef00c477abab4830bef344330f057fc", - "git-blob-sha1-crlf": "67e5b7269a9058b5e9503da229c86047f70fb52c", - "sha256": "0ac0d0b497c55b98cde6034c029e10a3ea0eae884024d682ca2c4ab4ac4ece46" + "git-blob-sha1": "673cd5f71d882e95f863560e2ed95da4ee285ed8", + "git-blob-sha1-crlf": "b5bc334d2e7eec73d2f7949c53441062d1a006a7", + "sha256": "e268d9255999b70eef87670408032d9ceaef55e4be6f53ffe18a5ffd63a55d62" }, "path": "manifests/aio-install/entry.yaml" }, @@ -167,9 +167,9 @@ }, { "digests": { - "git-blob-sha1": "c290dd9d76a3f1d6f0e29e224460ac4026830261", - "git-blob-sha1-crlf": "4768e52a5228cc9de702e3f9b84ea2b4ff00acac", - "sha256": "eddd13b9e03f42c068c643239bddbd7931897bb0ecb77743ee7d337780fb9dfd" + "git-blob-sha1": "048eaa02d12ba75cb59185995bd66fc6d4a4680c", + "git-blob-sha1-crlf": "97e696360f3c2507716ca1b909bd7d5fa44f2cc0", + "sha256": "8ea5c879b6f4acf2067b8a6b8fd86a86db01dbced45292e1583a416d9d86b7f5" }, "path": "manifests/aio-upgrade/entry.yaml" }, diff --git a/workspaces/iot-operations/siteops-index.json b/workspaces/iot-operations/siteops-index.json index 48c030d..a0007c7 100644 --- a/workspaces/iot-operations/siteops-index.json +++ b/workspaces/iot-operations/siteops-index.json @@ -172,6 +172,27 @@ "sensitivity": "unknown", "type": "object" }, + { + "description": "AIO Arc extension settings as string keys and values, also set by the typed aioExtensionSettings input. Each key replaces the release or template value. The values are visible in Azure, so never include secrets.", + "field": "parameters.aioConfigurationOverrides", + "requirement": "optional", + "sensitivity": "non-sensitive", + "type": "mapping" + }, + { + "description": "Secret store Arc extension settings as string keys and values, also set by the typed secretStoreExtensionSettings input. They apply with Secret Sync enabled or disabled. Never include secrets.", + "field": "parameters.secretStoreConfigurationOverrides", + "requirement": "optional", + "sensitivity": "non-sensitive", + "type": "mapping" + }, + { + "description": "cert-manager Arc extension settings when enableCertManager is true, also set by the typed certManagerExtensionSettings input. Each key replaces the release or template value. Never include secrets.", + "field": "parameters.certManagerConfigurationOverrides", + "requirement": "conditional", + "sensitivity": "non-sensitive", + "type": "mapping" + }, { "description": "Review cert-manager, Edge Site and optional Secret Sync settings. The typed defaults leave Secret Sync disabled. Guided enablement requires an observed cluster reporting OIDC and workload identity before writes. Host flags do not add those features to an existing cluster.", "field": "properties.deployOptions", @@ -257,19 +278,66 @@ { "guidance": { "category": "core", - "coverage": "Operational synopsis. Follow the guide for AIO release selection, supported paths and workload observations after the upgrade.", + "coverage": "Operational synopsis. Follow the guide for AIO release selection, supported paths, typed inputs and workload observations after the upgrade.", "documentation": [ "manifests/aio-upgrade/README.md" ], "effects": [ "Updates extensions and the resources the AIO release requires while preserving supported existing configuration.", + "With enableCertManager false, leaves the cert-manager extension unread and unchanged.", "Does not perform general migrations of instance child resources.", "Avoid resource set deployments between changing the AIO release pin and completing the upgrade." ], + "inputs": [ + { + "description": "The target AIO release. The typed aioRelease input has no default, so the upgrade never selects a target. Choose a supported transition from the instance's current release.", + "field": "properties.aioRelease", + "requirement": "required", + "sensitivity": "non-sensitive", + "type": "string" + }, + { + "description": "Include the existing cert-manager Arc extension in this upgrade? True updates its version and settings. False leaves it to its separate owner, with no read or write. The typed input has no default.", + "field": "properties.deployOptions.enableCertManager", + "requirement": "required", + "sensitivity": "non-sensitive", + "type": "boolean" + }, + { + "defaultBehavior": "The shared fallback derives the name from the Site name.", + "description": "The existing instance name. The typed instance input derives it from the resource ID.", + "field": "parameters.aioInstanceName", + "requirement": "optional", + "sensitivity": "non-sensitive", + "type": "string" + }, + { + "description": "AIO Arc extension settings, also set by the typed aioExtensionSettings input. Each key replaces the existing or release value, and no key is removed. Never include secrets.", + "field": "parameters.aioConfigurationOverrides", + "requirement": "optional", + "sensitivity": "non-sensitive", + "type": "mapping" + }, + { + "description": "Secret store Arc extension settings, also set by the typed secretStoreExtensionSettings input. Each key replaces the existing or release value. Never include secrets.", + "field": "parameters.secretStoreConfigurationOverrides", + "requirement": "optional", + "sensitivity": "non-sensitive", + "type": "mapping" + }, + { + "description": "cert-manager Arc extension settings when enableCertManager is true, also set by the typed certManagerExtensionSettings input. Each key replaces the existing or release value. Never include secrets.", + "field": "parameters.certManagerConfigurationOverrides", + "requirement": "conditional", + "sensitivity": "non-sensitive", + "type": "mapping" + } + ], "outcome": "Upgrade the AIO extensions and the resources that the selected AIO release explicitly requires.", "prerequisites": [ - "An existing AIO installation and a supported upgrade transition.", - "A configured Site with the intended AIO release and permission for the selected updates." + "An existing AIO installation and a supported upgrade transition, any patch of the same minor version or the next minor version.", + "A configured or saved Site with the intended AIO release, or the existing instance ID with typed answers, and permission for the selected updates.", + "A broker backend redundancy factor of 2 or more, the default, so the broker can upgrade." ], "removal": [ "There is no general rollback. Review partial outcomes and recovery guidance before retrying."