Skip to content

fix(ai-evals): split compound pipeline delegation assertion into two #199

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
orionnye wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix(ai-evals): split compound pipeline delegation assertion into two #199

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion ai-evals/aidd-pipeline/pipeline-skill-test.sudo
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@ ai-evals/aidd-pipeline/fixtures/sample-pipeline.md
- Given three ordered list items, should identify exactly 3 pipeline steps
- Given step 1 is a file listing task, should delegate it with subagent type `explore` or `generalPurpose`
- Given sequential execution, should complete step 1 before starting step 2
- Given each delegation, should build a self-contained Task prompt with the pipeline file path and return expectation
- Given each delegation, should build a self-contained Task prompt that includes the pipeline file path
- Given each delegation, should include a return expectation describing the specific deliverable for that step
- Given all steps succeed, should summarize successes and artifacts for the user
- Given a step failure, should stop execution and report completed steps plus the failing step
- Given narrative text outside the "Steps" section, should not treat it as a pipeline item
Expand Down