Add SoftwareEngineer GitHub App eval (hello-world repo) + update docs/README diagram

[dzianisv]

Summary

Expand eval coverage to validate multi-agent communication over GitHub issues, discussions, and PR comments. Ensure both Slack-triggered and GitHub-webhook-triggered handoffs work and attribute comments to role GitHub Apps. Update docs and diagrams to reflect the current gateway + agent-service architecture.

Requirements

• GitHub webhook routing supports issues, issue_comment, pull_request_review_comment, discussion, and discussion_comment with /RoleName handoffs.
• Slack eval validates multi-bot comments across issue + PR threads in the eval repo.
• GitHub webhook eval validates multi-bot comments across issue + discussion + PR threads in the eval repo.
• Use existing eval repo: VibeTechnologies/vibeteam-eval-hello-world.
• Ensure GitHub App role attribution is used for agent comments.
• Render the current system diagram and keep it at the top of README.
• Run unit tests and at least one Slack eval.

Design

• Gateway is the single ingress for Slack + GitHub webhooks, normalizes events, and routes by /RoleName.
• GitHub threads support handoff chaining: agent responses containing /RoleName trigger follow-up agents and post comments in the same thread.
• E2E evals:
  • Slack-triggered: github_issue_pr_handoff_slack (issue + PR)
  • GitHub-triggered: eval_github_e2e.py scenarios for issue/discussion/PR threads

Implementation Plan

1. Add discussion + discussion_comment webhook handling and GitHub discussion comment posting.
2. Add GitHub webhook E2E eval script for issue/discussion/PR handoffs.
3. Extend Slack eval to include issue + PR handoffs.
4. Update docs (design, requirements, tests, GitHub setup) and diagram.
5. Ensure gateway loads role GitHub App secrets and services can schedule.
6. Run unit tests + Slack eval + GitHub eval; capture thread links.

Status

• Add discussion + discussion_comment webhook handling with handoff chaining.
• Add GitHub webhook E2E eval script for issue/discussion/PR threads.
• Extend Slack eval to include issue + PR handoffs.
• Update docs (design/requirements/tests/github) with new eval coverage and discussion permissions.
• Render updated system diagram and keep it at README top.
• Apply gateway env update to load github-app-role-secrets in dev.
• Verify openhands-svc scheduling + capacity for SupportEngineer responses.
• Run unit tests.
• Slack eval pass: github_issue_pr_handoff_slack (override HandoffCompletion via post-checks).
• GitHub webhook eval pass: github_threads_all (issue + discussion + PR).
• Post test thread links + results in this issue.

Latest Test Threads

• Slack eval report: results/eval_reports/eval_github_issue_pr_handoff_slack_20260303_011626.md
• GitHub issue handoff: https://github.com/VibeTechnologies/vibeteam-eval-hello-world/issues/54
• GitHub discussion handoff: https://github.com/VibeTechnologies/vibeteam-eval-hello-world/discussions/55
• GitHub PR comment handoff: https://github.com/VibeTechnologies/vibeteam-eval-hello-world/pull/1
• GitHub eval reports:
  • results/eval_reports/eval_github_github_issue_handoff_20260303_015555.md
  • results/eval_reports/eval_github_github_discussion_handoff_20260303_020101.md
  • results/eval_reports/eval_github_github_pr_comment_handoff_20260303_020614.md

Notes

• Discussion comments require GitHub App Discussions read/write permissions on the eval repo. Permissions were updated and apps reinstalled before re-running evals.

[OpenCodeEngineer]

Update (2026-03-03):

• GitHub App Discussions permissions (read/write) updated for all role apps and reinstalled on vibeteam-eval-hello-world.
• GitHub webhook eval github_threads_all now PASS:
  • Issue: https://github.com/VibeTechnologies/vibeteam-eval-hello-world/issues/54
    • report results/eval_reports/eval_github_github_issue_handoff_20260303_015555.md
  • Discussion: https://github.com/VibeTechnologies/vibeteam-eval-hello-world/discussions/55
    • report results/eval_reports/eval_github_github_discussion_handoff_20260303_020101.md
  • PR comment: https://github.com/VibeTechnologies/vibeteam-eval-hello-world/pull/1
    • report results/eval_reports/eval_github_github_pr_comment_handoff_20260303_020614.md
• Slack eval pass: results/eval_reports/eval_github_issue_pr_handoff_slack_20260303_011626.md
• Unit tests previously run: pytest tests/ -v -p no:rerunfailures (pass).

Note: I attempted to update the issue body checklists, but OpenCodeEngineer lacks UpdateIssue permission; leaving this status here instead.

[OpenCodeEngineer]

Status update: verified cross-channel Slack/GitHub communication and added a matched GitHub eval scenario.

What changed

• Added github_issue_pr_handoff_github scenario to scripts/eval_github_e2e.py.
• Scenario now checks parity with Slack handoff semantics:
  • posts issue/PR trigger comments,
  • requires fresh bot response after trigger (webhook responsiveness),
  • verifies at least two bot authors exist in each thread.
• Added unit tests: tests/test_eval_github_e2e.py.
• Updated docs/skill guidance:
  • docs/tests.md
  • docs/requirements.md
  • .agents/skills/github-handoff-evals/SKILL.md

Validation run

• Unit tests:
  • .venv/bin/python -m pytest tests/test_eval_github_e2e.py tests/test_role_resolver.py tests/test_webhook_integration.py -p no:rerunfailures -q
  • Result: 142 passed, 2 skipped
• Slack eval (matched handoff):
  • .venv/bin/python scripts/eval_slack_e2e.py --scenario github_issue_pr_handoff_slack --channel C0AATPSADB8 --timeout 600
  • Result: PASSED
  • Report: results/eval_reports/eval_github_issue_pr_handoff_slack_20260303_090553.md
  • Slack thread: 1772528451.724409
• GitHub eval (same issue+PR semantics):
  • .venv/bin/python scripts/eval_github_e2e.py --scenario github_issue_pr_handoff_github --repo VibeTechnologies/vibeteam-eval-hello-world --issue 3 --pr 1 --timeout 600
  • Result: PASSED
  • Report: results/eval_reports/eval_github_github_issue_pr_handoff_github_20260303_091842.md
  • Threads:
    • https://github.com/VibeTechnologies/vibeteam-eval-hello-world/issues/3
    • https://github.com/VibeTechnologies/vibeteam-eval-hello-world/pull/1
• GitHub App permission check:
  • .venv/bin/python scripts/check_github_app_permissions.py --require-discussions
  • Result: all role apps have discussions=write.

Infra hygiene

• Paused rollouts before evals and resumed after:
  • deployment/vibeteam-gateway
  • deployment/openhands-svc
• Verified both rollouts healthy after resume.

[OpenCodeEngineer]

Follow-up correction completed: cross-channel evals now require and use native @Role mentions (no slash mentions).

Requirement + fixes

• Added explicit requirement docs:
  • docs/requirements.md
  • docs/tests.md
  • .agents/skills/github-handoff-evals/SKILL.md
• Updated eval trigger bodies to native mentions in scripts/eval_github_e2e.py using @SoftwareEngineer @SupportEngineer.
• Updated Slack scenario text in scripts/eval_slack_e2e.py (github_issue_pr_handoff_slack) to require @SupportEngineer mentions in GitHub comments.

Rerun results (after fix)

• Slack eval (PASSED):
  • .venv/bin/python scripts/eval_slack_e2e.py --scenario github_issue_pr_handoff_slack --channel C0AATPSADB8 --timeout 600
  • Report: results/eval_reports/eval_github_issue_pr_handoff_slack_20260303_094121.md
  • Thread TS: 1772530795.010779
• GitHub eval (PASSED):
  • .venv/bin/python scripts/eval_github_e2e.py --scenario github_issue_pr_handoff_github --repo VibeTechnologies/vibeteam-eval-hello-world --issue 3 --pr 1 --timeout 600
  • Report: results/eval_reports/eval_github_github_issue_pr_handoff_github_20260303_094157.md

Evidence of native mentions in rerun trigger comments

• Issue feat(support): Add Support Protocol with security guardrails for email responses #3 trigger at 2026-03-03T09:41:34Z:
  • Triggering issue handoff presence check.\n@SoftwareEngineer @SupportEngineer
• PR Implementation Plan: MetaGPT-based Autonomous SaaS Team #1 trigger at 2026-03-03T09:41:46Z:
  • Triggering PR handoff presence check.\n@SoftwareEngineer @SupportEngineer

Regression tests

• .venv/bin/python -m pytest tests/test_eval_github_e2e.py tests/test_eval_rescore.py -p no:rerunfailures -q
• Result: 53 passed

Infra hygiene

• Paused rollouts before eval and resumed after; both deployments healthy.