Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 20 additions & 3 deletions .github/workflows/notify-startup-failure.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,13 @@
# the cap is enforced when the FILE LOADS, not when a job runs)
# - a `uses: ./.github/workflows/x.yml` path that does not exist on the ref
# - malformed YAML, an unknown `on:` filter, a bad matrix expression
# - an org Actions policy that blocks the run's event, which the org audit
# log records as `workflows.actions_policy_violation`
#
# Only runs the branch started for itself count: a push, a schedule, a manual
# dispatch, a `workflow_run` or a `repository_dispatch`. GitHub gives a pull
# request run its head branch, so the branch listing also holds the release pull
# request's runs, and those say nothing about the branch.
#
# Its consequence is the worst-shaped one available: a merge to a deploy branch
# reports nothing, so the branch is believed deployed and is not. It has already
Expand Down Expand Up @@ -260,9 +267,19 @@ jobs:
# and neither sweep is asking about them: "the branch is not deployed"
# is a statement about a pipeline. See SYNTHESIZED_PREFIX in
# scripts/branch_health.py, which drops them from the other sweep.
#
# Only the events a branch starts for itself count. GitHub gives a
# pull request run its HEAD branch, so this listing also holds the
# runs of a release pull request whose head is the branch, and a CLA
# run an Actions policy refused there read as "staging is not
# deployed". The list is DEPLOY_EVENTS in scripts/branch_health.py;
# tests/test_startup_failure_sweep.py fails if the two drift. It
# filters before the newest run is picked, so a later pull request
# run can't hide a push that never started.
jq --argjson cutoff "${CUTOFF}" --arg branch "${BRANCH}" '
[ .workflow_runs[]
| select(.path | startswith("dynamic/") | not)
| select(.event | IN("push", "schedule", "workflow_dispatch", "workflow_run", "repository_dispatch"))
| select(.conclusion == "success" or .conclusion == "failure"
or .conclusion == "timed_out" or .conclusion == "startup_failure")
| { id, path, name, conclusion, created_at, head_sha, html_url,
Expand Down Expand Up @@ -339,7 +356,7 @@ jobs:
type: "section",
text: {
type: "mrkdwn",
text: ":rotating_light: *<\($server)/\($repo)|\($repo_name)>* — \($findings[0] | length) pipeline(s) *never started*, so nothing in the run could report it. The branch is not deployed."
text: ":rotating_light: *<\($server)/\($repo)|\($repo_name)>* — \($findings[0] | length) workflow run(s) *never started*: GitHub refused them before any job ran, so nothing in the run could report it."
}
}]
+ ( $findings[0] | map({
Expand All @@ -348,14 +365,14 @@ jobs:
{ type: "mrkdwn", text: "*Workflow*\n<\($server)/\($repo)/actions/runs/\(.id)|\(.path)>" },
{ type: "mrkdwn", text: "*Branch*\n<\($server)/\($repo)/tree/\(.branch)|\(.branch)>" },
{ type: "mrkdwn", text: "*Commit*\n<\($server)/\($repo)/commit/\(.head_sha)|\(.head_sha[0:8])>" },
{ type: "mrkdwn", text: "*Pushed by*\n\(.actor)" }
{ type: "mrkdwn", text: "*Started by*\n\(.actor)" }
]
}) )
+ [{
type: "context",
elements: [{
type: "mrkdwn",
text: "`startup_failure` with no jobs: GitHub rejected the run before scheduling anything — check the caller/callee `permissions:` relationship and the workflow YAML on that ref. <\($sweep)|Sweep run>"
text: "`startup_failure` with no jobs. The cause is the caller/callee `permissions:` or the workflow YAML on that ref, or an org Actions policy that blocks the event (`workflows.actions_policy_violation` in the org audit log). <\($sweep)|Sweep run>"
}]
}]
)
Expand Down
43 changes: 35 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -284,11 +284,15 @@ repeated `failed` always post.

### The one failure it cannot see: a run that never started

`notify-main-failure` is a job *inside* the pipeline, so it covers every failure where the run exists. It cannot cover `conclusion: startup_failure` — GitHub rejecting the run at load time, with **zero jobs** — because there is no job to put the alert in. The run appears in the Actions tab and nowhere else, and the branch is not deployed.
`notify-main-failure` is a job *inside* the pipeline, so it covers every failure where the run exists. It cannot cover `conclusion: startup_failure` — GitHub rejecting the run at load time, with **zero jobs** — because there is no job to put the alert in. The run appears in the Actions tab and nowhere else, so whatever it would have deployed didn't ship.

That is not a corner case. It is produced by a caller job granting narrower `permissions:` than a called workflow's jobs declare (the cap is checked when the file *loads*, not when a job runs), by a `uses: ./.github/workflows/x.yml` path that does not exist on the ref, and by malformed YAML. It has already cost auth `staging` ten hours of running the previous image across two merges, with the recovery message being the first thing the channel heard about it.
That is not a corner case. It is produced by a caller job granting narrower `permissions:` than a called workflow's jobs declare (the cap is checked when the file *loads*, not when a job runs), by a `uses: ./.github/workflows/x.yml` path that does not exist on the ref, by malformed YAML, and by an org Actions policy that blocks the run's event, which the org audit log records as `workflows.actions_policy_violation`. It has already cost one repository's `staging` ten hours of running the previous image across two merges, with the recovery message being the first thing the channel heard about it.

`notify-startup-failure.yml` watches from outside, on a schedule. It sweeps the deploy branches and alerts when the newest conclusive run of a workflow is a `startup_failure` inside the lookback window, restricted to runs where no job ran — precisely the set the in-run notify job could not have reported:
`notify-startup-failure.yml` watches from outside, on a schedule. It sweeps the deploy branches and alerts when the newest conclusive run of a workflow is a `startup_failure` inside the lookback window, restricted to runs where no job ran — precisely the set the in-run notify job could not have reported. The message names the run and both causes: the `permissions:` or YAML on that ref, or an Actions policy.

**Only runs the branch started for itself count**: a `push`, a `schedule`, a `workflow_dispatch`, a `workflow_run` or a `repository_dispatch`. GitHub gives a pull request run its head branch, so a `staging` listing also returns the release pull request's runs, CI and the CLA check included, and those say nothing about whether `staging` deployed. Both sweeps share the list, `DEPLOY_EVENTS` in `scripts/branch_health.py`, and filter before they pick each workflow's newest run, so a later pull request run can't hide a red push. A deploy started by any other event goes unwatched.

A caller:

```yaml
name: Pipeline watchdog
Expand Down Expand Up @@ -331,7 +335,7 @@ Age is measured from the current attempt's `run_started_at`, not the run's `crea

Selection logic lives in `scripts/branch_health.py` with unit tests, because it is date arithmetic plus a two-way trigger and that does not belong in a jq expression.

Alerts repeat, by design: one failing push produces about three messages at that cadence and window, and then silence. Alerting only on the *transition* into a broken state gives exactly one message per break, which was rejected after replaying it against the auth incident — it would have said nothing about the second failing push, since the run before that one had also failed to start. A stateless sweep cannot be exactly-once, so the choice is a message that can be missed or a few that cannot, and the window is what bounds the few.
Alerts repeat, by design: one failing push produces about three messages at that cadence and window, and then silence. Alerting only on the *transition* into a broken state gives exactly one message per break, which was rejected after replaying it against that incident — it would have said nothing about the second failing push, since the run before that one had also failed to start. A stateless sweep cannot be exactly-once, so the choice is a message that can be missed or a few that cannot, and the window is what bounds the few.

Two things to know when adding it. GitHub only runs `schedule` triggers from a repository's **default branch**, so a watchdog merged to `staging` and no further is inert — `workflow_dispatch` is there to prove it before it reaches `main`. And widening `lookback-minutes` on a manual dispatch replays a past incident, which is how to check it would have caught one.

Expand All @@ -357,7 +361,7 @@ Two things to know when adding it. GitHub only runs `schedule` triggers from a r

`zizmor` is advisory because pointing it at existing pipelines surfaces a backlog, and a lint job that is red on arrival gets ignored rather than fixed. Flip `zizmor-blocking: true` per repo once its backlog is triaged.

**The permission check is not redundant with either tool**, which was verified against a tree that had already broken a deploy branch: `actionlint` reported nothing, and `zizmor --persona=auditor` reported only its generic "no `permissions:` block" note, equally true of jobs that work fine. What it checks is that no called workflow declares a permission its caller lacks. GitHub caps a called workflow at the calling job's grant and enforces that cap when the workflow **file is loaded**, so a job in a shared reusable naming a scope one caller does not grant does not run with less — it rejects that caller's whole run as a `startup_failure` with **zero jobs**, which means the pipeline's own notify job cannot report it either. In `mindsdb/auth` that cost two silently undeployed merges to `staging` and ten hours of serving the previous image.
**The permission check is not redundant with either tool**, which was verified against a tree that had already broken a deploy branch: `actionlint` reported nothing, and `zizmor --persona=auditor` reported only its generic "no `permissions:` block" note, equally true of jobs that work fine. What it checks is that no called workflow declares a permission its caller lacks. GitHub caps a called workflow at the calling job's grant and enforces that cap when the workflow **file is loaded**, so a job in a shared reusable naming a scope one caller does not grant does not run with less — it rejects that caller's whole run as a `startup_failure` with **zero jobs**, which means the pipeline's own notify job cannot report it either. In one repository that cost two silently undeployed merges to `staging` and ten hours of serving the previous image.

The rule it enforces: **a job in a shared reusable declares only what all its callers grant, and inherits anything only one of them needs.** Inheriting is the only thing that composes — the PR caller grants `pull-requests: write` and the comment posts, the push callers grant `contents: read` and the same job runs without a scope it never needed. And the reason it has to be a gate rather than a convention: a pull request only exercises the PR caller, which is usually the one that *does* grant the scope, so the mistake merges green and breaks on the merge commit.

Expand All @@ -367,7 +371,7 @@ Blind spot to know about: it can only read local (`./.github/workflows/...`) cal

## The release-freeze contract

A freeze is one thing: the repo's `staging_frozen` custom property, set to `true` or `false`. A single org-level ruleset (terraform, `newprod/global/github`) blocks pushes to `staging` in every repo where it is `true`; the workflows never touch the ruleset. Four workflows care — freeze and unfreeze write it, and the two alerting paths read it to decide whether a red staging is worth interrupting anyone for.
A freeze is one thing: the repo's `staging_frozen` custom property, set to `true` or `false`. A single org-level ruleset, managed outside this repository, blocks pushes to `staging` in every repo where it is `true`; the workflows never touch the ruleset. Four workflows care — freeze and unfreeze write it, and the two alerting paths read it to decide whether a red staging is worth interrupting anyone for.

They used to carry three separate copies of that knowledge, and only two of them took the name as an input; the alerting path hardcoded it. Renaming it in one repo would therefore have moved the freeze and left the alerting reading a name that no longer existed — silent in the direction that hurts, because a reader that cannot establish the state escalates, so every ordinary mid-week staging red would have paged the channel forever and the cause would have looked like a Slack problem.

Expand Down Expand Up @@ -599,6 +603,29 @@ whether its author has signed. Four repos learned that the hard way, gating on
`github.event_name == 'pull_request'` while triggering on events that are not
`pull_request`, so their check quietly never asked anybody to sign.

### The `pull_request_target` policy

GitHub blocks `pull_request_target` in public repositories by default from 2 Dec 2026, unless an Actions event policy allows it ([Securely using pull_request_target](https://docs.github.com/en/actions/reference/security/securely-using-pull_request_target)). Every caller triggers on `pull_request_target`, so without a policy the check never starts on a public repo.

`config/cla-actions-policy.json` is that policy. It allows `pull_request_target` and `issue_comment` for `.github/workflows/cla.yml` in every repository, and names no other workflow.

- It targets the wrapper by path, not repositories by name, so a new caller is covered the day it adds `cla.yml`.
- Its events are an allow-list. `issue_comment` is on it because signing and `recheck` arrive as comments, and the policy refuses any event it doesn't list.
- Allowing this one workflow is safe because the reusable checks out no pull request code: it runs a `gh api` step and the pinned action, nothing else.

`tests/test_cla_actions_policy.py` fails the build if the target widens or an event drops.

The `integrations/github` Terraform provider has no resource for Actions policies yet, so an org admin creates the policy with the [REST API](https://docs.github.com/en/rest/actions/policies), from a checkout of this repository:

```sh
gh api -X POST orgs/mindsdb/actions/policies \
-H "X-GitHub-Api-Version: 2026-03-10" --input config/cla-actions-policy.json
```

To read it back, `gh api orgs/mindsdb/actions/policies -H "X-GitHub-Api-Version: 2026-03-10"`. To change it, `PUT` the file to `orgs/mindsdb/actions/policies/<id>`, and to remove it, `DELETE` that path. The list gives the id.

A run the policy refuses ends as `startup_failure` with zero jobs. The org audit log records it as `workflows.actions_policy_violation` with `"allowed": false`, naming the repository and the event but not the workflow file.

### Prerequisites (provisioned once, org level, scoped to the release-train repos)

- **`mindsdb-release-train` GitHub App** with `Administration`, `Contents`, and
Expand All @@ -609,8 +636,8 @@ whether its author has signed. Four repos learned that the hard way, gating on
**`secrets.RELEASE_APP_PRIVATE_KEY`** (org secret) — the private key reaches
the reusable workflows via `secrets: inherit` in the caller.
- The **`staging_frozen` org custom property** and the **`staging-freeze` org
ruleset** that targets repos where it is `true`, both defined in terraform
(`newprod/global/github/main.tf`). The workflows only set the property — they
ruleset** that targets repos where it is `true`, both managed outside this
repository. The workflows only set the property — they
never touch the ruleset or the underlying branch protection.

### Caller wrappers
Expand Down
22 changes: 22 additions & 0 deletions config/cla-actions-policy.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"name": "CLA check: allow pull_request_target and issue_comment",
"enforcement": "active",
"conditions": {
"repository_name": {
"include": ["~ALL"],
"exclude": []
},
"workflow_path": {
"include": [".github/workflows/cla.yml"],
"exclude": []
}
},
"rules": [
{
"type": "restrict_action_events",
"parameters": {
"allowed_events": ["pull_request_target", "issue_comment"]
}
}
]
}
28 changes: 27 additions & 1 deletion scripts/branch_health.py
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,20 @@
# is asking. Naming one in `workflows:` still reports it: an explicit ask wins.
SYNTHESIZED_PREFIX = "dynamic/"

# The events by which a deploy branch starts a run for itself. GitHub gives a
# pull request run its HEAD branch, so a `staging` listing also returns every run
# of the release pull request from `staging` to `main`: its CI, and the CLA
# check's `pull_request_target`. Those runs test a merge or check a contributor;
# they say nothing about whether the branch deployed. On 9 Nov 2026 an Actions
# policy refused one, and the startup sweep reported staging "not deployed" while
# the push from the same commit had deployed eight seconds earlier.
#
# An allow-list, so a deploy started by an event not named here goes unwatched.
# None was seen in the run listings of seven repositories' deploy branches. The
# startup sweep's jq in notify-startup-failure.yml carries the same list, and
# tests/test_startup_failure_sweep.py fails if the two drift.
DEPLOY_EVENTS = ("push", "schedule", "workflow_dispatch", "workflow_run", "repository_dispatch")

# The freeze workflow's per-repo wrapper keeps this name verbatim, because the
# release-PR workflow chains off it by name. That makes it a stable handle for
# "the window just opened" without this file knowing anything about the schedule,
Expand Down Expand Up @@ -124,6 +138,16 @@ def started_at(run: dict) -> datetime:
return parse_time(run.get("run_started_at") or run["created_at"])


def started_by_the_branch(run: dict) -> bool:
"""Whether the branch started this run for itself (see ``DEPLOY_EVENTS``).

Synthesized runs carry the event ``dynamic`` and pass here, because
``in_scope`` already decides them, and naming one in `workflows:` has to keep
reporting it.
"""
return run.get("event") in DEPLOY_EVENTS or (run.get("path") or "").startswith(SYNTHESIZED_PREFIX)


def newest_conclusive_per_workflow(runs: Iterable[dict]) -> list[dict]:
"""One run per workflow file: the most recent that concluded either way.

Expand Down Expand Up @@ -204,8 +228,10 @@ def select_red(

just_froze = branch == staging_branch and freeze_opened_within(freeze_runs, cutoff=cutoff)

# Filter before picking the newest run, so a later pull request run can't
# stand in for a red push and hide it.
findings = []
for run in newest_conclusive_per_workflow(runs):
for run in newest_conclusive_per_workflow(r for r in runs if started_by_the_branch(r)):
if run["conclusion"] not in RED:
continue
if not in_scope(run["path"], only=only, exclude=exclude):
Expand Down
Loading
Loading