Skip to content

test: deploy a real app with the action in CI, report failures to Slack, and move v1 on release - #25

Open
wmadden-electric wants to merge 10 commits into
mainfrom
test/e2e-deploy
Open

wmadden-electric wants to merge 10 commits into
mainfrom
test/e2e-deploy

Conversation

@wmadden-electric

@wmadden-electric wmadden-electric commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

CI now deploys a real app with the action and checks it works:

- name: Deploy with the action
  id: deploy
  uses: ./
  with:
    working-directory: e2e/app
    build-command: bun run build
    environment: e2e-${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.prisma }}
- name: Check the action's outputs and the deployed service
  run: node e2e/verify.mjs
- name: Destroy the environment
  run: bun scripts/destroy.ts

What the new e2e job does

  1. Runs the action from the branch (uses: ./) on e2e/app, a Composer app with one service. It deploys to a new preview environment for each run.
  2. e2e/verify.mjs checks the outputs: outcome is succeeded, url is set and appears in urls, and report-path names a file. It then requests the URL until the service answers with its own environment name.
  3. Destroys the environment with the destroy operation of @prisma/composer/control. The prisma CLI has no destroy command, so the action's mode: destroy can't do this yet.
  4. Destroys e2e-* environments whose GitHub run has finished, which are ones an earlier run could not destroy. The run id is part of each environment's name, so the sweep never touches an environment a running job is still testing. A leftover that won't go away fails the sweep only once it is a day old.

It runs twice. pinned uses the app's lockfile (prisma 8.0.0-rc.33, Composer 0.38.0). latest moves the app to prisma@latest and the Composer version that release bundles, so a new release is deployed for real every 6 hours.

It runs on pull requests from this repository, pushes to main, the 6-hourly schedule, and manual runs. Pull requests from forks get no secrets, so it skips them. Without the PRISMA_SERVICE_TOKEN secret, the action reports skipped-no-credential and the check step fails, so the job can't pass without deploying.

Failures go to Slack

A failed or cancelled job on main, on a scheduled run, on a manual run or on a prisma-released dispatch posts to Slack through the SLACK_WEBHOOK_URL secret. The message names the failing job and links the run. This replaces the job that opened a GitHub issue, which nobody would have seen.

v1 moves on every release

.github/workflows/major-tag.yml moves the v1 tag to each v1.x.y release when it is published, or when a prerelease is promoted to a release. Until now this was a manual step, which is why v1 was still on v1.10.0 for a while after v1.11.0 was published. Each run moves v1 to the newest published v1.x.y release, whichever release started it, so it never moves v1 back and a run GitHub cancels while waiting loses nothing. It can also be run by hand. v1 already points at v1.11.0.

How I checked it

  • Ran the action's main.mjs locally on e2e/app against real Prisma Cloud. It deployed to a preview environment and set every output. The job summary listed the service, verify.mjs passed, and the destroy step removed the environment.
  • Ran v1.10.0 the same way. It fails with CLI.COMMAND_MOVED, so this job would have caught the --stage break.
  • Deployed an environment named after a finished CI run and ran the sweep. It destroyed it.

The dev tag check is removed

CI no longer checks prisma@dev. Two repositories publish prisma under that tag: prisma-cli, and the v7 branch of prisma/orm, which publishes the Prisma 7 CLI. The tag points at whichever published last, so the check has failed on every scheduled run since 2026-11-14 with nothing wrong in the action (issue #26). The check of each dev build moves to prisma-cli, which can test the exact build it publishes.

Setup needed before merging

  • Secret PRISMA_SERVICE_TOKEN: a service token for the workspace the test deploys into.
  • Variable PRISMA_WORKSPACE_ID: that workspace's id.

The app deploys as project cloud-deploy-action-e2e in region us-east-1.

Not covered

  • Deploying with the GitHub OIDC credential, which Console-connected repositories use. That would need this repository connected to a Prisma workspace.
  • npm ci and the prisma-version input. The test app installs with Bun and carries its own prisma devDependency.
  • The sweep lists environments with prisma branch list, because @prisma/composer/control has no list operation, and destroys them with its destroy operation. Both can move to the action once the CLI has a destroy command.
  • GitHub turns off scheduled runs after 60 days without a commit to this repository, without any alert.

Agent: bombadil-28, taliesin-31

🤖 Generated with Claude Code

wmadden-electric and others added 2 commits November 10, 2026 17:35
… Slack

A new e2e job runs the action itself on e2e/app: it deploys to a new preview environment, checks the outputs and that the service answers for that environment, then destroys it. It runs with the app's pinned prisma and Composer versions and again moved to prisma@latest. Failures on main and on scheduled runs now post to Slack instead of opening an issue nobody watches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: willbot <w.a.madden+machine@gmail.com>
Signed-off-by: Will Madden <madden@prisma.io>
…ncelled runs to Slack

Fixes from review: per-step timeouts and a fetch timeout so destroy always gets its time, a sweep of e2e environments earlier runs left behind (one job per matrix entry at a time so it never removes a live one), exact-version checks on the prisma@latest move, the service token only on the steps that need it, no destroy after a skipped-no-credential run, and Slack alerts for cancelled jobs with a message that names the failing job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: willbot <w.a.madden+machine@gmail.com>
Signed-off-by: Will Madden <madden@prisma.io>
@prisma-gizmo

prisma-gizmo Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

✅ Gizmo reviewed d0e19db — posted 0 inline comment(s) this pass.

Open findings: none

Change walkthrough

This pull request adds a real end-to-end deployment test to CI, routes CI failures to Slack, and automates the v1 tag movement on releases. The incremental delta under review prunes the dev npm dist-tag from the prisma-cli matrix in .github/workflows/ci.yml:70 and updates the matching sentence in README.md:172.

CI matrix (prisma-cli job). The matrix now runs 8.0.0-rc.29, 8.0.0-rc.33, and latest. Dropping dev narrows coverage to the release channel: the scheduled run still picks up every new prisma release via the latest dist-tag, while the unstable dev channel — which can break on any upstream change — no longer turns CI red. The comment above the matrix was updated to match, and a repo-wide search found no remaining references to the dev dist-tag (the 8.0.1-dev.3 strings in tests/cli.test.mjs are prerelease-version parser fixtures, unrelated to the tag). The rc.29 pin stays as the regression guard for the --stage flag removal, since it is the last CLI that accepted it.

README. The Development section's description of the matrix ("including the npm latest tag") now matches the workflow exactly; the surrounding text on the end-to-end job, the e2e-* sweep, secret requirements, and Slack notification is unchanged and consistent with what the workflows do.

Together with the rest of the PR (the e2e job that deploys e2e/app with uses: ./ in pinned and latest flavors, verifies outputs and the live service, destroys the environment, and sweeps stale ones; the Slack notification workflow; and major-tag.yml), the change keeps CI's release-channel testing focused on what the action's users actually install while still exercising each new release automatically every six hours.

@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 3dae0835-e99a-437e-bf52-bca4a0350041
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@prisma-gizmo prisma-gizmo Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New findings: 🟡 2 minor · trace

Comment thread README.md Resolved
Comment thread e2e/app/scripts/sweep.ts Resolved
@wmadden-electric
wmadden-electric changed the base branch from fix/prisma-environment-flag to main November 10, 2026 06:48
wmadden-electric and others added 2 commits November 10, 2026 17:48
Moving v1 was a manual step after each release, so a release could leave @v1 users on the previous version. The workflow never moves v1 back to an older release and can be run by hand with a release tag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: willbot <w.a.madden+machine@gmail.com>
Signed-off-by: Will Madden <madden@prisma.io>
@wmadden-electric wmadden-electric changed the title test: deploy a real app with the action in CI, and report failures to Slack test: deploy a real app with the action in CI, report failures to Slack, and move v1 on release Nov 10, 2026

@prisma-gizmo prisma-gizmo Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New findings: 🟡 2 minor · trace

Still open from previous reviews: 🟡 2 minor

Comment thread .github/workflows/major-tag.yml Resolved
Comment thread README.md Resolved
… the README

The sweep refuses to run without this run's environment name, so an empty name can no longer match every e2e environment. Runs of the tag workflow no longer interleave. The README names the right secret for Slack and says prereleases do not move v1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: willbot <w.a.madden+machine@gmail.com>
Signed-off-by: Will Madden <madden@prisma.io>

@prisma-gizmo prisma-gizmo Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New findings: 🟡 1 minor · trace

Comment thread .github/workflows/major-tag.yml Resolved
…release is promoted

A shared concurrency group could cancel a waiting main or scheduled e2e job and post a false Slack alert. The group is gone: the sweep now destroys an e2e environment only once the run named in it has finished, and fails only for leftovers older than a day. The tag workflow listens for released, which covers a promoted prerelease, and ignores prerelease tags when deciding whether v1 is ahead. The app name is defined once, the Slack job is named for its purpose, and the README has a manual removal runbook and the full list of runs that report to Slack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: willbot <w.a.madden+machine@gmail.com>
Signed-off-by: Will Madden <madden@prisma.io>

@prisma-gizmo prisma-gizmo Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New findings: 🟡 2 minor · trace

Still open from previous reviews: 🟡 1 minor

Comment thread e2e/app/scripts/sweep.ts Resolved
Comment thread e2e/app/scripts/sweep.ts Resolved
…olerate API errors and races

The tag workflow now moves each major tag to its newest published release, whichever release started the run, so a run GitHub cancels while it waits loses nothing. The sweep leaves an environment for the next run on a 403, 429 or 5xx from the GitHub API, and only counts an environment as stuck if it still exists, since another sweep may have just removed it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: willbot <w.a.madden+machine@gmail.com>
Signed-off-by: Will Madden <madden@prisma.io>

@prisma-gizmo prisma-gizmo Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New findings: 🟡 2 minor · trace

Comment thread .github/workflows/major-tag.yml Resolved
Comment thread e2e/app/scripts/sweep.ts Resolved
…ents once in the sweep

The tag workflow fetches tags before choosing and chooses only among releases whose tag exists, so a release that outlived its tag cannot stop every major tag from moving. The sweep collects old environments it failed to destroy and lists environments once afterwards, instead of once per failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: willbot <w.a.madden+machine@gmail.com>
Signed-off-by: Will Madden <madden@prisma.io>

@prisma-gizmo prisma-gizmo Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New findings: 🟡 1 minor · trace

Comment thread .github/workflows/major-tag.yml Resolved
When the last release checked had no tag, the loop exited non-zero and bash -e stopped the step silently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: willbot <w.a.madden+machine@gmail.com>
Signed-off-by: Will Madden <madden@prisma.io>

@prisma-gizmo prisma-gizmo Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New findings: none · trace

@prisma-gizmo prisma-gizmo Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No Gizmo finding is open and the head commit has been reviewed. Approving.

Two repositories publish prisma under the dev tag: prisma-cli (the CLI the action runs) and the v7 branch of prisma/orm (the Prisma 7 CLI, which has no deploy command). The tag points at whichever published last, so this check has failed on every scheduled run since 2026-11-14 without anything being wrong with the action. The check of each dev build moves to prisma-cli, which can test the exact build it publishes.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: willbot <w.a.madden+machine@gmail.com>
Signed-off-by: Will Madden <madden@prisma.io>

@prisma-gizmo prisma-gizmo Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New findings: none · trace

@prisma-gizmo prisma-gizmo Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No Gizmo finding is open and the head commit has been reviewed. Approving.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants