Skip to content

chore: audit test suite for efficiency and redundancy #56

@j4th

Description

@j4th

Context

The test suite has grown to ~1200 tests across 4 tiers (unit, integration, live, quick). Full mise run check takes ~3 minutes and check:full ~4-5 minutes. As we add more providers (Spicerack, MTGGoldfish, etc.), test count and runtime keep climbing. Time for a periodic audit.

Areas to Review

1. Test runtime profiling

  • Run pytest --durations=30 to identify the slowest tests
  • Are any unit tests doing unnecessary work (e.g., redundant fixture setup, repeated cache clearing)?
  • Can slow service tests share fixtures more efficiently?

2. Redundancy between tiers

  • Do integration tests (tests/integration/) duplicate coverage already in provider unit tests?
  • Do live tests (tests/live/) overlap with integration tests beyond what's needed for real-world regression?
  • Are there unit tests that test the same code path through slightly different inputs without adding value?

3. Provider test patterns

  • Each new provider (EDHREC, Moxfield, Spicerack, MTGGoldfish) follows the same test pattern. Are we testing boilerplate (e.g., ToolError wrapping, response_format variants) redundantly across providers?
  • Could shared parametrized fixtures reduce duplication for common patterns (lifespan init, error handling, attribution lines)?

4. Fixture overhead

  • conftest.py clears all service caches via autouse fixture. Is the clearing list growing unwieldy?
  • Are HTML/JSON fixture files larger than needed? Could we trim captured fixtures to only the fields tests actually assert on?

5. Workflow test efficiency

  • Workflow tests use AsyncMock — these should be fast. Are any accidentally doing real work?
  • test_workflow_server.py tests tool registration — does this overlap with test_orchestrator_e2e.py?

6. testmon effectiveness

  • Is pytest-testmon correctly detecting affected tests, or is it re-running too many?
  • Are there test files that testmon always marks as affected due to broad imports?

Outcome

  • Identify and remove redundant tests
  • Speed up the slowest tests
  • Document any test patterns that should be standardized
  • Target: keep mise run check under 2 minutes as the suite grows

Priority

Low — not blocking anything, but good hygiene to do periodically as the project scales.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions