Add eval.yaml for verify-tests-fail-without-fix skill

[PureWeen]

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Summary

Adds eval.yaml for the verify-tests-fail-without-fix skill, enabling empirical A/B validation via skill-validator.

Context

• This is an internal orchestrator-invoked skill used by pr-review to verify tests catch bugs
• Follows eval best practices established during the try-fix evaluation cycle (PR Improve try-fix skill: add eval.yaml and fix prompt issues #34807)
• Part of eval coverage expansion tracked in issue Establish eval.yaml lifecycle and CI integration for skill validation #34814

Eval Design

• 6 scenarios covering both verification modes, negative trigger, edge cases, regressions
• 0 output_contains -- rubric-based behavioral assertions only (no vocabulary overfitting)
• 14 output_not_contains -- anti-pattern guards for common mistakes
• 1 expect_activation: false -- native spec field for negative trigger
• Realistic timeouts (60s-900s depending on scenario complexity)

Scenarios

1. Happy path: full verification -- Tests two-phase workflow (fail without fix, pass with fix)
2. Happy path: verify failure only -- Tests test-creation mode (no fix needed)
3. Negative trigger -- Documentation question should not invoke verification
4. Regression: semantic inversion -- Tests passing without fix = FAILED verification (not success!)
5. Edge case: no test files -- PR without tests can't be verified
6. Regression: no manual git commands -- Script handles file revert/restore, not raw git

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 34815

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 34815"

[Copilot]

Pull request overview

Adds an evaluation specification (tests/eval.yaml) for the internal verify-tests-fail-without-fix skill to enable scenario-based validation via skill-validator.

Changes:

• Introduces a new eval.yaml with 6 scenarios covering full verification, failure-only verification, negative trigger behavior, and regressions/edge cases.

[Copilot]

In the “verify failure only” scenario, the output_not_contains: "no fix files" assertion conflicts with the script/docs: verify-tests-fail.ps1 explicitly describes this mode as “no fix files detected”. This assertion is likely to fail even when the skill behaves correctly; consider removing it or narrowing it to only prohibit requiring fix files (e.g., erroring because fix files are missing).

Suggested change

	- type: output_not_contains
	value: "no fix files"

[Copilot]

These output_not_contains checks ban git checkout HEAD/git restore/git stash, but the skill’s own verify-tests-fail.ps1 uses git checkout ... internally (and even prints “git checkout HEAD” in some error messages). This can create false failures by rejecting correct script-driven behavior; consider rewriting the anti-pattern to target the agent’s manual instructions (e.g., “run git checkout/restore yourself”) rather than substrings that may legitimately appear in script output.

Suggested change

	value: "git checkout HEAD"
	- type: output_not_contains
	value: "git restore"
	- type: output_not_contains
	value: "git stash"
	value: "run git checkout HEAD"
	- type: output_not_contains
	value: "run git restore"
	- type: output_not_contains
	value: "run git stash"