Skip to content

Operative: refresh evaluations, screenshots and run-aligned lab copy - #3245

Open
Scott Durow (scottdurow) wants to merge 2 commits into
microsoft:mainfrom
scottdurow:evals-update
Open

Scott Durow (scottdurow) wants to merge 2 commits into
microsoft:mainfrom
scottdurow:evals-update

Conversation

@scottdurow

@scottdurow Scott Durow (scottdurow) commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

What changed

This PR updates the Operative labs to match the current Copilot Studio evaluation UI and the copy used by the latest completed automation run.

Evaluation instructions and screenshots

  • Use the agent's Evaluate tab rather than the older AgentOps path.
  • Configure Single response, Compare meaning, and a 70/100 pass score.
  • Enter the first case in the existing blank row, then use Add → Write a question for subsequent cases.
  • Match all 23 unique evaluation questions and expected responses to the automation.
  • Replace obsolete evaluation screenshots and complete the scheduling case's Expected response.

Agent, skill and workflow copy

  • Update the copy learners paste: agent instructions, skill descriptions and procedures, test prompts, names and document templates.
  • Tell learners to replace the complete instructions where the automation does so, rather than append a partial block.
  • Include the separate calendar availability check and the complete confirmed-slot booking prompt.
  • Use Taylor's sample resume for the published-agent test.
  • Clarify that the document-builder exercise creates a package for comparison, while the remaining labs install the tested reference package.

Screenshot and guidance corrections

  • Show the consolidated Work IQ tool and Claude Opus 5.5 in the relevant setup screenshots.
  • Align permission guidance with the current Work IQ flow.
  • Correct expression-editor instructions and lightning-bolt highlights.
  • Correct evaluation identity guidance: the demonstrated Dataverse case uses the Interview Agent's Maker connection.
  • Allow shortening only the generated cover-letter summary after a length rejection, never names, email addresses or other source fields.
  • Add skillbuilder to the spelling dictionary so the required ppa_skillbuilder schema name passes CI.

Validation

  • Production documentation build passed.
  • Markdown lint passed for all eleven Operative lessons.
  • Copy checks passed for 45 blocks and all 23 evaluation Q&A pairs, allowing learner-specific accounts, record numbers and meeting times.
  • All 356 linked course images exist; checks passed for the active changed screenshots and all seven newly added screenshots.
  • Visually checked 16 key published screenshots.
  • Full local spell check passed after the dictionary update.

The broader screenshot validator also checks retired images, archived metadata and untouched historical captures; those existing findings are outside the active changed-image checks above.

Scope

Course Markdown and rendered screenshots only, plus the spelling-dictionary entry. Automation code and raw screenshot files are maintained in the separate automation repository.

Update evaluation authoring, consolidate Work IQ guidance, repair screenshots, and align learner instructions and evaluation copy with the latest completed automation run. Preserve identity-safe length handling and original copy-block paragraph formatting; rerun evaluations after release instruction changes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The replacement instructions remove critical privacy and untrusted-input safeguards and required document-skill protections, while adding application-creation guidance that conflicts with the Interview Agent’s read-only permissions.

3 open issues
What changed in this PR

Refreshes the Operative course to align its hands-on labs with the current Copilot Studio interface and the latest completed automation run.

Changes:

  • Updates evaluation setup, expected responses, connection guidance, and screenshots.
  • Revises agent instructions, document-skill installation, and supervised Work IQ scheduling.
  • Corrects workflow field-editor steps and published-agent testing guidance.
File Description
docs/​operative-nextgen/​11-publish-and-monitor/​index.md Updates evaluation, release, sharing, and monitoring steps.
docs/​operative-nextgen/​10-work-iq-scheduling/​index.md Consolidates Work IQ setup and expands availability and booking prompts.
docs/​operative-nextgen/​09-human-oversight/​index.md Corrects review-message and expression-editor instructions.
docs/​operative-nextgen/​08-workflow-agents/​index.md Refines matching prompts, tool-call limits, and card editing.
docs/​operative-nextgen/​07-workflow-trigger/​index.md Documents popup expression editing and insertion.
docs/​operative-nextgen/​06-document-skill/​index.md Clarifies builder outputs and tested reference-skill installation.
docs/​operative-nextgen/​05-intake-matching-applications/​index.md Makes intake, matching, and application-check prompts more explicit.
docs/​operative-nextgen/​04-model-response-and-safety/​index.md Refreshes safety evaluations and complete instruction blocks.
docs/​operative-nextgen/​03-connected-agent/​index.md Updates specialist setup and baseline evaluation authoring.
docs/​operative-nextgen/​02-instructions-skills-dataverse-mcp/​index.md Updates evaluation setup and restricts length-error corrections.
docs/​operative-nextgen/​01-get-started/​index.md Updates initial model guidance and intake boundaries.
cspell.json Adds skillbuilder to the spelling dictionary.

Review effort: Balanced

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The consolidated Work IQ setup needs validation in the current Copilot Studio UI because the revised steps omit explicit connection selection, leaving the identity used for real calendar invitations unconfirmed.

No open issues 🎉

3 resolved since last review

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants