CondonFM Skills by ohadmo · Pull Request #4 · NVIDIA-BioNeMo/CodonFM

ohadmo · 2026-09-03T23:11:36Z

This repository is read-only and does not accept Pull Requests. Please open an Issue for reproducible bugs.

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

github-actions · 2026-09-03T23:11:52Z

Thanks for your interest. This repository is read-only and does not accept Pull Requests. Please open an Issue for reproducible bugs.

github-actions · 2026-09-03T23:12:41Z

Thanks for your interest. This repository is read-only and does not accept Pull Requests. Please open an Issue for reproducible bugs.

ohadmo · 2026-09-04T16:10:07Z

/nvskills-ci

ohadmo · 2026-09-04T16:11:27Z

@greptileai

greptile-apps · 2026-09-04T16:14:30Z

Confidence Score: 5/5

[Medium risk] Adds four new skill modules with supporting infrastructure.

The PR appears safe to merge because no blocking failure remains from the previously reported embed evaluation issues.

Summary

The embed evaluation artifacts were regenerated to cover all four current cases, and the benchmark now gives a consistent PASS publication verdict.

Updates the benchmark from two to four evaluation tasks, including cases 003 and 004.
Signs the current benchmark, evaluation definition, and edge-case fixture.
Aligns the benchmark verdict, publication recommendation, and displayed scoring criteria.

_{Reviews (13) · Last reviewed commit: "Attach NVSkills validation signatures" · Reviewed by Greptile}

ohadmo · 2026-09-04T17:23:52Z

/nvskills-ci

ohadmo · 2026-09-04T17:26:38Z

/nvskills-ci

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

ohadmo · 2026-09-09T22:19:29Z

@greptileai

ohadmo · 2026-09-09T22:19:32Z

/nvskills-ci

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

ohadmo · 2026-09-10T00:56:17Z

/nvskills-ci

ohadmo · 2026-09-10T05:12:23Z

/nvskills-ci

chrisknvidia · 2026-09-10T22:58:56Z

@ohadmo : I did some debugging of the failures,
Suggestion for the resource-intensive Tier 3 evaluations:
Please add the following to skills/codonfm-score/evals/config.yml

schema_version: 1

harbor:
  timeout_multiplier: 4
  sandbox:
    templates:
      claude-code: harbor-eval-claude-code-8g
      codex: harbor-eval-codex-8g

timeout_multiplier: 4 applies to both agents.
Each template applies to both the with-skill and baseline arm for that agent.
The Claude mapping is especially relevant because the latest remaining failure occurred in the Claude baseline.
This provides equal resource headroom, but a rerun is still required.

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

ohadmo · 2026-09-10T23:25:21Z

/nvskills-ci

ohadmo · 2026-09-14T21:03:01Z

/nvskills-ci

chrisknvidia · 2026-09-17T00:10:05Z

@ohadmo : I tested the two previously failing codonfm-score cases end to end using Claude Opus 5 through NVCARPS → SkillEvaluator → Harbor → Astra.

Resolved model: aws/anthropic/bedrock-claude-opus-5
Coverage: valid_full
Claude results: 4/4 scored, with zero failed attempts or policy refusals
Attempts 2–3 were correctly skipped because every case passed on attempt 1
Test MR !94
Successful Tier 3 job

https://github.com/NVIDIA-BioNeMo/CodonFM/blob/3f95e4e01604dbe7fda83d5563ef7a55d3134772/skills/codonfm-score/evals/evals.json

Could you please add the following per-skill override to skills/codonfm-score/evals/config.yml and rerun the complete PR evaluation?

schema_version: 1

harbor:
  agents:
    claude-code:
      model: aws/anthropic/bedrock-claude-opus-5

This changes only the Claude model for codonfm-score; the Codex configuration remains unchanged. The controlled test confirms that Opus 5 avoids the refusals for the two previously failing cases. The complete PR rerun will confirm it across the full dataset.

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

ohadmo · 2026-09-17T17:46:36Z

/nvskills-ci

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

ohadmo · 2026-09-17T19:58:25Z

/nvskills-ci

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

ohadmo · 2026-09-18T17:39:37Z

/nvskills-ci

ohadmo · 2026-09-23T17:52:52Z

/nvskills-ci

ohadmo · 2026-10-02T00:12:10Z

/nvskills-ci

Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>

- codonfm-embed-003: malformed sequences_edge_cases.csv (blank split, duplicate id, oversized/truncated sequence) — tests whether the agent catches known footguns the skill documents explicitly, rather than just validating a clean CSV. - codonfm-embed-004: checkpoint-choice question that requires the preprint's benchmark knowledge rather than anything derivable from the supplied source code alone — tests the skill's distilled domain knowledge, not just code-reading ability. Both target gaps identified while investigating why codonfm-embed showed flat/negative lift in Tier 3 runs: the existing two cases are fully answerable from the shipped source code alone, so a careful baseline can match a skill-equipped agent on content. These add cases where skill content should matter more than brute-force source reading. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

ohadmo · 2026-10-02T23:10:34Z

/nvskills-ci

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

ohadmo · 2026-10-07T17:37:31Z

/nvskills-ci

Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>

ohadmo added 3 commits September 3, 2026 12:07

CodonFM skills

9960b99

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

Add author metadata

87006b1

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

nvskills workflow file

db15d24

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

ohadmo requested a review from caofan September 3, 2026 23:11

ohadmo self-assigned this Sep 3, 2026

github-actions Bot closed this Sep 3, 2026

ohadmo reopened this Sep 3, 2026

github-actions Bot closed this Sep 3, 2026

ohadmo reopened this Sep 4, 2026

condonfm skills data

7dc0a0f

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

fix tier1

c6d169d

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

fix prompts due to guardrails

3f95e4e

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

reconfig harbor

360b8a5

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

run Opus5 for codonfm-setup

842a9c2

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

tier3 skill fixes

29194e8

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

ohadmo force-pushed the omosafi/skills branch from 591639c to 29194e8 Compare October 2, 2026 00:11

Attach NVSkills validation signatures

7a3c7cb

Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>

greptile-apps Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread skills/codonfm-embed/BENCHMARK.md Outdated

greptile-apps Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread skills/codonfm-embed/evals/evals.json

Improve codonfm-embed checkpoint guidance and input validation

be43117

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

Attach NVSkills validation signatures

2d22010

Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>

ohadmo merged commit 26457c3 into main Oct 7, 2026
6 checks passed

Conversation

ohadmo commented Sep 3, 2026

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

ohadmo commented Sep 4, 2026

Uh oh!

ohadmo commented Sep 4, 2026

Uh oh!

greptile-apps Bot commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

ohadmo commented Sep 4, 2026

Uh oh!

ohadmo commented Sep 4, 2026

Uh oh!

ohadmo commented Sep 9, 2026

Uh oh!

ohadmo commented Sep 9, 2026

Uh oh!

ohadmo commented Sep 10, 2026

Uh oh!

ohadmo commented Sep 10, 2026

Uh oh!

chrisknvidia commented Sep 10, 2026

Uh oh!

ohadmo commented Sep 10, 2026

Uh oh!

ohadmo commented Sep 14, 2026

Uh oh!

chrisknvidia commented Sep 17, 2026

Uh oh!

ohadmo commented Sep 17, 2026

Uh oh!

ohadmo commented Sep 17, 2026

Uh oh!

ohadmo commented Sep 18, 2026

Uh oh!

ohadmo commented Sep 23, 2026

Uh oh!

ohadmo commented Oct 2, 2026

Uh oh!

Uh oh!

ohadmo commented Oct 2, 2026

Uh oh!

Uh oh!

ohadmo commented Oct 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

greptile-apps Bot commented Sep 4, 2026 •

edited

Loading