Repository navigation
infra(runner): keep-warm, CI deploy, and scheduled canary - #113
Conversation
… schedule a canary Cold start dominates deploy latency (103s-215s): the FlutterFlow AI SDK download and a fresh pub get run on every new instance. Pinning min-instances to 1 makes that happen once per container boot instead of per request, buying a much shorter deploy for a small always-on cost. A merged runner fix previously shipped nothing: add a workflow that deploys cloud-run/** changes on main, gated on a service-account secret so pushes before setup skip instead of failing. A daily dry-run canary to automated-test-miedro catches a broken runner before users do.
|
This PR does not match any of the 1 configured review trigger rule. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| # before one-time setup complete are skipped quietly. Until | ||
| # CCC_FFAI_RUNNER_SA_KEY_JSON exists (see DEPLOYMENT.md) this job does not | ||
| # run; once it does, a merged cloud-run change deploys. | ||
| if: secrets.CCC_FFAI_RUNNER_SA_KEY_JSON != '' |
There was a problem hiding this comment.
🔴 Secret gates disable both workflows
Both jobs use secrets in job-level conditions, where GitHub Actions does not allow that context. Neither deploys nor scheduled canaries can run.
Learn more
GitHub Actions does not allow the secrets context in a job-level if. The same gate appears in the canary job. Neither workflow can be validated with these conditions, regardless of whether the secrets exist. A job-level environment variable cannot repair a job-level condition; the gate must move inside the job or use another supported context.
Example: On a push changing cloud-run/ffai-runner, the deploy workflow encounters the invalid job condition before checkout or Google Cloud authentication; the runner is not deployed.
Recommended fix: Remove both job-level secret conditions. Set a job-level environment variable from each secret, then use a step-level if: env.<name> != '' to gate the work, or gate a supported preparatory job output.
Was this helpful? React with 👍 or 👎 to provide feedback.
| verification: { | ||
| sdkConstraint: ">=3.0.0 <4.0.0", | ||
| dependencies: {}, | ||
| dependencyOverrides: {}, | ||
| sdkPackages: ["flutter_web_plugins"], | ||
| }, |
There was a problem hiding this comment.
🔴 Canary requests always fail validation
Every canary request omits verification.sources, which _normalizeVerification requires. The runner returns HTTP 400 before the canary exercises deployment.
Learn more
The canary supplies verification, so the runner parses it using _normalizeVerification. That parser requires a sources array even when customClasses is present. Without it, the runner returns a 400 error before workspace setup. Passing an empty array only avoids the 400: _verifyCustomCode then skips the compile gate entirely.
Example: With a valid key and throwaway project, the first scheduled request receives verification.sources must be an array. rather than exercising flutter analyze and the DSL validation.
Recommended fix: Include a source entry such as {fileName: 'canary_probe.dart', content: CANARY_SOURCE} in verification.sources, then test the canary against the actual runner request contract.
Was this helpful? React with 👍 or 👎 to provide feedback.
| --allow-unauthenticated \ | ||
| --memory "$MEMORY" \ | ||
| --concurrency "$CONCURRENCY" \ | ||
| --min-instances "$MIN_INSTANCES" \ |
There was a problem hiding this comment.
🟡 First deploy still pays initialization cost
On every fresh revision, --min-instances starts a container without initializing its workspace. _ensureWorkspace runs only on the first deploy request, leaving that user with the full initialization delay.
Learn more
Cloud Run minimum instances keep a container running; they do not send a deploy request during startup. The runner starts an HTTP server in main and initializes the workspace only inside _handle. Thus a minimum instance has no downloaded workspace until someone calls /deployCustomClasses. This still preserves the workspace for subsequent calls, but it does not move the first initialization to container startup.
Example: After a new runner revision rolls out, its minimum instance starts and waits. The next user deploy triggers flutterflow ai init and pays the cold workspace cost despite an idle prestarted container.
Recommended fix: If the goal is to remove initialization from user requests, warm the workspace at container startup with an appropriate credential or provision it into the image; otherwise document that minimum instances only preserve an already initialized workspace.
Was this helpful? React with 👍 or 👎 to provide feedback.
| paths: | ||
| - "cloud-run/**" |
There was a problem hiding this comment.
🔍 Deploy-script changes await another trigger
The cloud-run/** filter excludes changes to the deployment script. Its new minimum-instance setting takes effect only on a later runner-source change or manual deployment.
Was this helpful? React with 👍 or 👎 to provide feedback.
…ayload The deployed runner's _normalizeVerification hard-requires verification.sources to be an array and throws FormatException otherwise, which the handler turns into HTTP 400 before workspace init, the compile gate, or ai validate ever run. The canary omitted sources entirely, so every scheduled run was guaranteed to fail with 'verification.sources must be an array.' - a permanently-red canary that could not tell a healthy runner from a broken one, and that never actually reached the compile gate it claimed to exercise. Include the canary source as the single probe file; a non-empty sources list is also what unskips _verifyCustomCode (server.dart returns null on empty sources), so the compile gate now truly runs on every scheduled check.
Both workflows gated the whole job with `if: secrets.X != ''`. The `secrets` context is not available in `jobs.<job_id>.if` (only github, needs, vars and inputs are), so both files were invalid and the workflows could not run at all - the canary and the merge-deploy would have silently never executed, which is the same class of failure they exist to catch. The gate is now a first step that reads the secret from env and publishes `present`; every step that needs the key skips while it does not. Behaviour is unchanged: before one-time setup runs, the jobs skip quietly and log a notice. Also passes RUNNER_GIT_SHA to the service so /healthz reports which revision is live, and documents it.
What
Three pieces of runner infrastructure:
scripts/deploy_cloud_run_ffai.shnow passes--min-instances 1by default (overrideMIN_INSTANCES). A cold start — FlutterFlow AI SDK download viaai initplus a freshflutter pub get— is paid once per container boot on a warm instance instead of on every 103–215s cold deploy..github/workflows/deploy-runner.ymldeploys the runner whenevercloud-run/**changes onmain, using a service-account key secret. UntilCCC_FFAI_RUNNER_SA_KEY_JSONexists the job silently skips (no noisy failures before one-time setup). Auth via google-github-actions/auth + setup-gcloud, pinned by SHA mirroring ci.yml style.scripts/canary-deploy.mjsPOSTs adryRun: truedeploy forautomated-test-miedrodaily (07:21 UTC) and via workflow_dispatch. It fails on non-200,success:false, or a flutter_web_plugins SDK-package rejection, and reports/healthzversions without failing when that endpoint is not yet deployed. Gated onCANARY_FF_API_KEYso it skips cleanly before setup.DEPLOYMENT.mddocuments the one-time secret setup for both workflows.Why
A merged runner fix previously shipped nothing, and a broken runner was only discovered by the next user's deploy. These workflows close both gaps without touching branch protection or running on PRs.
Verified
node --test src/*.test.js-> 399 pass (baseline unchanged).node --check scripts/canary-deploy.mjsclean; canary exercised against a local mock runner for ok, missing-key, success:false, unsupported-SDK, and healthz-absent paths.bash -n scripts/deploy_cloud_run_ffai.shclean; workflow YAMLs parse.Blocked
Baking the FlutterFlow AI SDK into the image was not done. It is downloaded by
flutterflow ai initat runtime and would need anFF_API_KEYbuild-time secret plus network duringgcloud run deploy --source(which CI has neither), and the result is tied to a specific initialized workspace. min-instances already removes the per-deploy cost.