Skip to content

fix: make the manifest validator agree with the store it validates - #84

Draft
openvoiceos-bot wants to merge 1 commit into
masterfrom
fix/t5229-store-manifest-validation
Draft

openvoiceos-bot wants to merge 1 commit into
masterfrom
fix/t5229-store-manifest-validation

Conversation

@openvoiceos-bot

@openvoiceos-bot openvoiceos-bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

🤖 Auto-generated by Claude Opus 5 (claude-opus-5) via Claude Code — NOT human-reviewed. Verify before acting.

Verified by running the commands at this head: the entry counts, the two
validators' results, the exit codes, and the four renamed file names. Reported
as "none found by these instruments", with a positive control: the history
searches for "title" and for *.astro. A model re-check is not a human
review.

What changed

commit change
7bfec9d the validator's icon and skill_id rules accept what the store itself writes; four real gaps filled; scripts/check_all_raw_jsons.py validates the whole corpus, in CI
55c5651 four manifests take the file name their skill_id gives; their contents are untouched

A reviewer has two things to weigh: the widened icon rule, and the four
renames.

The task's premise does not hold on this repository

The brief asked to rename title to name in two manifests, add name to
five, and make the validator check those fields. Measured on master:

  • all 36 manifests carry a non-empty name AND description
  • "title" has never been a key in raw_jsons/ --
    git log -S '"title"' returns nothing, and the same search for "name"
    returns a commit, so the search works
  • name and description are already REQUIRED_FIELDS, with length bounds
  • the workflow globs raw_jsons/*.json, so ovos-skill-date-time.json is not
    missed

The review this came from cites src/components/SkillCard.astro. This
repository has a React frontend (frontend/src/components/SkillCard.tsx) and no
.astro file in any branch or anywhere in its history. So no card renders a
blank title for the reason given, and there was nothing to rename to name.

What is actually broken

Every one of the 36 entries fails validate_submission, and nothing reports
it: the workflow only checks entries a pull request touches, and
validate_raw_jsons.py admits it in its docstring -- "most of them do not
pass it".

Most of those failures are the validator's, not the data's.

icon was required to be a URL. The store renders icon as a react-icons
component name
: SkillCard.tsx resolves it against react-icons and
IconSelector.tsx offers those names on the submission form. Every committed
entry holds FaMicrophone, MdWallpaper and the like. The rule rejected the
store's own convention, and would have rejected any correct new submission.

The skill_id pattern rejected underscores, but
neon_homeassistant_skill.mikejgray is a real committed entry, and a skill id
is often derived from a Python package name. Widened, rather than renaming a
third party's skill id to fit a pattern.

With the icon error out of the way, four real gaps it had been hiding:

entry gap filled with
skill-spelling no examples four spelling phrasings
skill-wallpapers empty tags wallpaper, background, pictures, reddit
skill-audio-recording empty tags recording, audio, voice memo, dictation
skill-easter-eggs empty tags easter eggs, fun, references

36 of 36 failing before, 1 of 36 after.

The remaining one is skill-homeassistant.json, which ships no examples. It
is a third-party skill, so its voice phrasings belong to its maintainer rather
than being invented here.

The four renames, commit 55c5651

validate_raw_jsons.py asks that a file be named as apply_submission.py would
write it: the skill_id, lowercased, with dots replaced by dashes. Four entries
predate that rule and failed it, so CI was red on the rule and not on the code.
Renamed, contents byte-for-byte unchanged:

old name new name
skill-audio-recording.json skill-ovos-audio-recording-openvoiceos.json
skill-easter-eggs.json skill-easter-eggs-jarbasai.json
skill-spelling.json ovos-skill-spelling-openvoiceos.json
skill-wallpapers.json skill-ovos-wallpapers-openvoiceos.json

Nothing outside scripts/ reads these names: generate_store_data.py globs the
directory and reads the contents, so the file name is not an identifier, and
git grep finds no reference to any of the four old names outside
raw_jsons/ itself. Measured: the four renamed entries pass
validate_raw_jsons.py, 4 of 4; the control, the old skill-spelling.json
name, fails with the rule's own sentence and exit 1.

The rename also closes a duplicate-submission hole for these four. The duplicate
check at apply_submission.py:254 tests whether the target file exists, and the
target name is derived from the skill_id. A second submission of the spelling
skill is refused with "Skill already exists: ovos-skill-spelling.openvoiceos";
against the old file name the check saw nothing and would have written a second
file for the same skill. The same hole stays open for the 32 entries whose file
name still differs from their skill_id, which is the follow-up below.

The guard

scripts/check_all_raw_jsons.py runs the same validator over the WHOLE
directory, so a rule that contradicts the corpus is one command away from being
visible instead of hiding behind a per-PR diff. KNOWN_FAILURES holds the
single entry above and is asserted accurate in both directions: an entry that
starts failing cannot hide in it, and an entry that starts passing must leave
it.

Verified before committing:

emptying a passing entry's tags      -> exit 1, names the entry
adding a stale name to the list      -> exit 1, "now passes; remove it"
pointing the glob at an empty dir    -> exit 1, via the stale-exemption check
the tree as committed                -> exit 0, "36 checked, 1 failing, 1 exempt"

The workflow runs it after the per-entry check.

Follow-up, not in this pull request

The guard calls validate_submission
(scripts/check_all_raw_jsons.py:45). The per-pull-request validator calls
validate_raw_jsons.check_entry, which adds two rules on top: the file name
must equal the skill_id with dots replaced by dashes, and no two entries may
share a skill_id. So the guard does not see the rule the rename commit exists
to satisfy. Measured at this head: check_entry over all 36 entries fails 32,
every one of them on the file-name rule, while validate_submission fails 1.

The consequence is bounded but real: a later pull request that touches any of
those 32 entries turns the per-entry step red for a reason unrelated to its own
change, and the duplicate check misses them. The debt is inherited from master
and this change reduces it, so it is tracked separately: either the guard calls
check_entry with the 32 listed as exemptions, or all 36 entries are renamed.

Not done

No name field was added, because every entry carries one. If the blank cards
are real on the deployed site, they do not come from this repository's
raw_jsons/ at the head measured, and the next step is to say which artifact
the cards were read from.

@coderabbitai

coderabbitai Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@JarbasAl
JarbasAl force-pushed the fix/t5229-store-manifest-validation branch from 55c5651 to b9c4d1c Compare October 2, 2026 06:25
…dates

Every one of the 36 committed entries failed validate_submission, and
nothing reported it: the workflow only checks the entries a pull request
touches, and validate_raw_jsons.py says so in its own docstring ("most of
them do not pass it").

Most of those failures were the validator's, not the data's.

icon was required to be a URL. The store renders icon as a react-icons
component name: SkillCard.tsx resolves it against react-icons and
IconSelector.tsx offers those names on the submission form, so every
committed entry holds "FaMicrophone", "MdWallpaper" and the like. The rule
rejected the convention the store itself uses, and would have rejected any
correct new submission. A react-icons name is now accepted, and a URL
still is, because entries predating the marketplace migration used one.

The skill_id pattern rejected underscores, but neon_homeassistant_skill.mikejgray
is a real committed entry and a skill id is often derived from a Python
package name. Widened rather than renaming a third party's skill id to fit
a pattern.

That left four real gaps the icon error had been hiding:

  skill-spelling         no examples        added four spelling phrasings
  skill-wallpapers       empty tags         wallpaper, background, pictures, reddit
  skill-audio-recording  empty tags         recording, audio, voice memo, dictation
  skill-easter-eggs      empty tags         easter eggs, fun, references

36 of 36 failing before, 1 of 36 after. The one left is
skill-homeassistant.json, which ships no examples: it is a third-party skill,
so the voice phrasings belong to its maintainer rather than being invented
here.

scripts/check_all_raw_jsons.py runs the same validator over the WHOLE
directory, so a rule that contradicts the corpus is one command away from
being visible instead of hiding behind a per-PR diff. Its KNOWN_FAILURES
holds that single entry and is asserted accurate in both directions: an
entry that starts failing cannot hide in it, and an entry that starts
passing must leave it.

Four entries predate the per-entry file-name rule, which requires a file's
name to equal its skill_id lowercased with dots replaced by dashes, and
failed CI on that rule rather than on their content. Renamed, content
untouched:

  skill-audio-recording.json -> skill-ovos-audio-recording-openvoiceos.json
  skill-easter-eggs.json     -> skill-easter-eggs-jarbasai.json
  skill-spelling.json        -> ovos-skill-spelling-openvoiceos.json
  skill-wallpapers.json      -> skill-ovos-wallpapers-openvoiceos.json

The workflow now runs the whole-corpus check after the per-entry check.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@JarbasAl
JarbasAl force-pushed the fix/t5229-store-manifest-validation branch from b9c4d1c to 9d3648a Compare October 2, 2026 06:50

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants