feat: end-to-end TFC migration pipeline with guided init and import - #16
Draft
hllvc wants to merge 37 commits into
Draft
feat: end-to-end TFC migration pipeline with guided init and import#16hllvc wants to merge 37 commits into
hllvc wants to merge 37 commits into
Conversation
- pull workspace state from the TFC API instead of per-workspace init
- write migration-summary.{md,json}; replace workspace.tmpl with summary.tmpl
- map TFC vcs settings to SG VCSTriggers (push, PR speculative, file filters)
- add SGDefault*/workspaceOverrides for vcs auth, source kind and triggers
Validate generated payloads against schema/sg-payload.schema.json via yajsv.
- sg-migrate.sh + Dockerfile run the full pipeline inside a container - scripts/migrate.sh: init/apply/enrich/convert/validate/import/triggers/all - merge TFC variable sets, parallel convert/import, register VCS triggers - move convert_hcl_to_json.sh into scripts/
sg-cli exits 0 even when individual workflows fail, so the importer now parses its output. Workflows rejected because their pinned Terraform version is above SG's managed ceiling (1.5.7, last MPL/FOSS release) are re-imported with SGDefaultTerraformVersion, the payload is patched in place, and each case is logged to export/terraform-version-fallbacks.log with a printed notice. Any other per-workflow failure now fails the run. Also: sg_retry captured the wrong exit status (always 0), definitive HTTP 4xx responses are no longer retried, the trigger pass skips workflows that were never created, and failure output no longer echoes the API token.
'sg-migrate.sh completion bash|zsh' prints a completion script for the current shell session; 'init' prints the source line for the user's shell. Runs natively like 'clean'.
Running sg-migrate.sh without a command used to start the whole pipeline; it now prints the help menu, locally, without touching Docker. Usage and hints show the name the user invoked (./sg-migrate.sh) instead of the in-container script path.
Without TFE_TOKEN or a terraform login credentials file, apply used to warn and then fail deep inside terraform with "Invalid provider configuration". Both the host entrypoint and migrate.sh now stop with a clear message before anything runs. Add an explicit provider "tfe" block so tfHostname also applies to the provider, not just the state export.
An expired 'terraform login' session (or a bad TFE_TOKEN) used to fail deep inside terraform. Resolve the token the tfe provider will use, in its own precedence, and check it against /api/v2/account/details first; stop with a clear message on 401/403 or when the host is unreachable.
Every workflow was hardcoded to shared runners unless overridden per workspace. Add SGDefaultRunnerConstraints (default shared) so all workflows can be put behind a private runner group in one place; workspaceOverrides[name].RunnerConstraints still wins per workspace. Validated: type is shared|private, and private requires names.
Move the TFC/SG API helpers out of migrate.sh into scripts/lib/ and add the building blocks the upcoming init wizard, preflight and checklist need: interactive prompt helpers (tty or scripted answers), cached tfvars reads, paginated TFC listing (orgs/projects/workspaces) and SG lookups for integrations, runner groups, workflows and secrets. The enrich script now shares the same token resolution as the provider. Also stop turning curl connection failures into "000000" status codes.
'init' now walks through the configuration instead of copying the example file: it lists the TFC organisations, projects and workspaces the token can see, the SG VCS and cloud connectors and runner groups, and derives the source kind and repo prefix from the chosen connector. Every step falls back to free text when a token is missing or an API call fails, and non-interactive runs keep the template behaviour. The generated terraform.tfvars keeps the example's order and comments; re-running the wizard prefills the current values and keeps a .bak.
Verify up front what used to fail minutes later inside terraform or as API 400s: TFC token and org, that the workspace selection matches anything, SG token and org, every connector / secret / runner group referenced in terraform.tfvars (defaults and workspaceOverrides), SGDefaultSourceConfigDestKind and SGDefaultTerraformVersion format, the exportPath vs export-dir mismatch, and override keys that match no workspace. Runs automatically for apply, import and all; also available as 'preflight'; --skip-preflight bypasses it.
'all' now records each phase's inputs in .sg/state.json and skips phases whose inputs have not changed, so a failed run resumes instead of re-running terraform apply. Import records per payload file which workflows landed and which failed; unchanged, fully imported files are skipped and files with failures are re-imported (sg-cli updates existing workflows in place). --fresh discards the state. --project and --workspace restrict every phase to a subset; apply maps --workspace to the workspacenames variable.
apply now prints a condensed migration summary (workflows per project, sensitive variables skipped, unpinned Terraform versions, non-remote execution modes, renames, failed state exports) instead of leaving it in a file nobody opens. import shows a per-workflow table before the confirmation prompt: create/update, Terraform version with the 1.5.7 fallback marked, runner, triggers, variable and secret counts. 'import --dry-run' stops after the table without touching anything.
Map the API messages users actually hit (unknown connector, missing runner group, repository not reachable, bad approvers, invalid names or VCS kind, rejected token, Terraform ceiling) to a one-line hint naming the terraform.tfvars field to fix. Shown once per distinct hint, both for direct API calls and for sg-cli's per-workflow failures; unknown messages still show the raw response.
After import (or via 'checklist'), write export/post-import-checklist.md
listing everything the migration could not do by itself: secrets to
fill in, workflows that failed to import, Terraform version fallbacks
to verify, VCS triggers that failed, state that could not be exported,
non-remote execution modes and renamed workflows, with deep links into
the SG UI (SG_UI_URL). For every sensitive variable TFC would not
expose, create an SG secret with the value CHANGE_ME, reference it from
the workflow (env var or IaC input, ${secret::<name>}) and patch the
payload to match; --no-secret-stubs opts out. Trigger registration
results are now recorded in the run state.
The integrations list has no ResourceType; the kind (GITHUB_COM, AWS_RBAC, ...) is Settings.kind. Map it from there, make the type filters null-safe, and only treat msg/data as the item list when it is actually an array.
_w_csv_json emitted "[]" twice when the list was empty (grep -v on empty input trips the fallback under pipefail), so the generated file was not valid HCL and preflight reported every field as missing. Validate the file right after writing it, report an unparsable tfvars as such in preflight, remember the SG org / API host chosen in the wizard so a new shell does not need SG_ORG, pretty-print objects in the generated file and separate menu values from their descriptions.
The hint used $SHELL, which is the login shell and was wrong for a bash login shell running zsh, so the bash script got sourced into zsh and broke on the first Tab. sg-migrate.sh now detects the parent shell and 'completion' with no argument prints the matching script; each script also hands off to the other shell's version when sourced by mistake.
The per-workflow plan passed whatever sg_list_workflows returned to jq --argjson and crashed when the response was not a plain array. List helpers now always yield an array (also unwrapping data.Workflows) and the plan guards the value. The wizard no longer asks for the profile name (not required), the repo URL prefix, approvers or the fallback Terraform version — they take sensible defaults and are edited in terraform.tfvars. sg-migrate.sh says when it runs in Docker.
When terraform.tfvars is missing, all runs the wizard and then asks whether to continue right away or edit the file first; only the non-interactive template copy still stops, since it holds placeholders.
Wrap argument parsing and dispatch in main so bash reads the whole script up front. The repo is bind-mounted into the container, and a file edited during a long run was read half-old, half-new, ending in "unexpected EOF while looking for matching quote" after the import had already succeeded.
TFC_* / TFE_* variables (TFC_WORKSPACE_NAME, the TFC_AWS_* dynamic credential settings, ...) only mean something inside Terraform Cloud. A new ignoreVarPatterns input (default ["^TFC_", "^TFE_"]) drops matching terraform and env variables from the payloads, from workspaces and from variable sets, lists them in the migration summary and keeps them out of the sensitive-variable list so no secret stubs are created for them. The init wizard asks whether to strip them. Also fix the secrets link in the checklist (orgs/<org>?tab=secrets).
The "^(AWS|AZURE|GCP)_" filter also matched AZURE_DEVOPS, so an Azure DevOps VCS connector could be chosen as the cloud connector and only schema validation caught it. Both pickers now use the exact kind lists (VCS also accepts AZURE_DEVOPS_SP and GITLAB_OAUTH_SSH and maps them to the source kind), are sorted by name, and preflight rejects a DeploymentPlatformConfig kind outside the schema enum. The validator no longer prints each failure twice.
Presets: SGTerraformVersionSource (carry|preset) plus nullable SGDefaultTerraformVersion and SGDefaultRunnerConstraints. Keys left out of the payload are filled from the org's execution preset by the SG API at import. init offers the preset with the current value shown, preflight reports it, the plan marks 'preset (...)' cells, the checklist explains what changed, and a pin rejected above the 1.5.7 ceiling falls back to the preset when the fallback is null. Flow: numbered phases with timings, live progress lines for terraform and the parallel phases, plain-language preflight with connector/kind/prefix consistency checks, readable wizard review, per-file convert/validate lines, a condensed post-import checklist with group links, and a neutral message when the import is declined. Fixes: init kept the previous repo URL prefix when the connector switched provider (the prefix now comes from the TFC repositories); enrich re-ran on every 'all' because convert changed its recorded hash; quotes in connector names broke the generated tfvars; a RunnerConstraints override with names failed HCL type unification.
hllvc
changed the title
sg_api_raw does the request and returns the body on any status, with SG_HTTP_CODE set in the caller's shell; _sg_api keeps the logging and the body-on-2xx-only behaviour on top of it.
…ty iacInputData.data
sg-cli 2.2.1 round-trips every bulk entry through sg-sdk-go v1.1.0's
Workflow struct, whose iacInputData.data map is tagged omitempty, so a
workspace with no Terraform variables reaches the API without the key
and is rejected with "VCSConfig.iacInputData.data: This field is
required."
import_bulk now splits each payload file: workflows with variables go
through sg-cli as before, the ones without are created straight from
the payload via POST .../wfs/ (PATCH when the name exists) followed by
the same state upload sg-cli does. The direct path prints sg-cli's
"Failed to create <name>: <code>: <body>" line on rejection, so the
ceiling fallback, error hints and checklist cover both paths.
Workaround only; TODO(sg-cli) markers show what to remove once sg-cli
ships with sg-sdk-go >= v1.5.7, where the field became a pointer and
"data": {} is sent as given.
hllvc
force-pushed
the
feat/improvements
branch
from
September 14, 2026 09:01
cbd03da to
9662748
Compare
…failed import sg-cli's state upload sends a PUT without x-ms-blob-type, which Azure Blob rejects (400 MissingRequiredHeader), and it only recognises a literal "HTTP/1.1 200 OK" as success, so on Azure-backed environments every workflow was created without its state while the run reported "state: exported for every selected workspace". import_bulk now reads sg-cli's per-workflow upload lines and re-uploads every reported failure through sg_upload_tfstate (header included, any 2xx accepted); the direct-create path uploads the same way. Each workflow ends up as state_uploaded or state_failed in the import result, a workflow without its state fails the payload file and is retried on the next import, and the checklist reports the SG-side upload next to the TFC-side export. Workaround for sg-cli; TODO(sg-cli) marks what to remove once its upload sends the header and checks the status properly.
With more than one workflow to import, probe_import imports the first selected workflow of the first payload file on its own and requires both the create and the state upload to succeed before the rest is imported in parallel. An environment problem (a store that rejects uploads, a read-only token, a wrong connector kind) then costs one workflow instead of all of them; the probe workflow is updated again with its file afterwards.
…pdate path sg-cli switches from create to update only when the API's message says "Workflow name not unique", but the API answers 409 "Workflow ID not unique", so every re-import of an existing workflow (a retry, --fresh, the fail-fast probe) failed with that 409 and no state was uploaded. import_bulk now treats a 409 / "not unique" create failure as an existing workflow: it PATCHes the payload entry via sg_update_workflow, uploads its state and drops the failure line, so re-runs are idempotent. The direct-create path does the same fallback in sg_create_workflow. Workaround for sg-cli, marked TODO(sg-cli).
…if needed Customers cloned or forked the repo per release and had no way to pick up fixes. ./sg-migrate.sh update runs on the host: it refuses a non-git checkout, a detached HEAD or dirty tracked files, then fast-forwards the current branch and rebuilds the Docker image only when the Dockerfile changed. Config and output are untracked, so a pull never touches them.
A Git-for-Windows clone with core.autocrlf turns the bash scripts into CRLF and they fail under WSL/Docker.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Turns the migrator from a hand-driven
terraform applyplus manual editing into one guided pipeline:./sg-migrate.sh allconfigures, preflights, extracts and transforms the workspaces, merges Variable Sets, converts and validates the payloads, and bulk-imports them into per-project workflow groups. The transformer gains per-project output, a migration summary, API-based state export, VCS trigger mapping and execution-preset support.Suggested review order:
transformer/terraform-cloud/locals.tf(the TFC to SG mapping, where payload shape is decided), thenscripts/migrate.sh(phase orchestration, import parsing, resume state), thenscripts/lib/(wizard, preflight, checklist). The rest is glue and docs.Changes
sg-payload.<project>.jsonper TFC project withWorkflowGroup.name = tfc-<project>;migration-summary.{md,json}; state pulled from the TFC API instead of a per-workspaceterraform init; TFC VCS settings mapped toVCSTriggers; sensitive variables skipped and reported;ignoreVarPatternsstripsTFC_*/TFE_*variables.SGDefault*inputs plus typedworkspaceOverrides.SGTerraformVersionSourceand nullableSGDefaultTerraformVersion/SGDefaultRunnerConstraintsleave keys out of the payload so the SG API fills them from the org's execution preset.sg-migrate.shrunsscripts/migrate.shin the Docker image (native fallback) withinit|preflight|apply|enrich|convert|validate|import|triggers|checklist|all|clean|completion. Phases resume via.sg/state.json, filter with--project/--workspace;import --dry-runstops at the plan.tfc-<project>groups, parses sg-cli output (it exits 0 on per-workflow failures), retries a pin above SG's 1.5.7 Terraform ceiling with the fallback or preset, maps known API errors to a hint naming the tfvars field, and writesexport/post-import-checklist.mdwithCHANGE_MEsecret stubs for sensitive variables.schema/sg-payload.schema.jsonwithscripts/validate_payload.sh; README rewritten for the orchestrated flow.Testing
./sg-migrate.sh allagainst a multi-project TFC org: wizard, preflight, one payload per project, groups created, workflows imported, checklist written.allafter a failed import: unchanged phases are skipped, only files with failures are re-imported.export/terraform-version-fallbacks.log; withSGDefaultTerraformVersion = nullthe preset applies.terraform fmt -checkon the transformer and./scripts/validate_payload.sh export/sg-payload.*.jsonpass.Risks
sg-payload.jsonbecame one file per project andWorkflowGroup.nameistfc-<project>rather than the TFC project ID.--no-create-groupsand--no-secret-stubsopt out.terraformVersionorRunnerConstraintsomitted depend on the org's execution preset being set as intended.Deployment Notes
SG_API_TOKEN,SG_ORG, optionalSG_BASE_URL, andTFE_TOKEN(preferred overterraform login); tuning knobs are in the README.--buildto rebuild); nothing is pushed to a registry.