Skip to content

fix: decouple mocking from evals #1148

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
akshaylive merged 1 commit into from
Jan 21, 2026
Merged

fix: decouple mocking from evals #1148

akshaylive merged 1 commit into from
Jan 21, 2026

Conversation

@cristipufu
Copy link
Member

@cristipufu cristipufu commented Jan 18, 2026 •
edited by github-actions bot
Loading

Description

Decouples mocking from evaluation context.

  • Separates the mocking infrastructure from the evaluation system, enabling mocking to be used independently.
  • Mocking can now be configured without requiring a full evaluation item.

Ref: #1117

Development Package

  • Use uipath pack --nolock to get the latest dev build from this PR (requires version range).
  • Add this package as a dependency in your pyproject.toml:
[project]
dependencies = [
  # Exact version:
  "uipath==2.5.27.dev1011484088",

  # Any version from PR
  "uipath>=2.5.27.dev1011480000,<2.5.27.dev1011490000"
]

[[tool.uv.index]]
name = "testpypi"
url = "https://test.pypi.org/simple/"
publish-url = "https://test.pypi.org/legacy/"
explicit = true

[tool.uv.sources]
uipath = { index = "testpypi" }

[tool.uv]
override-dependencies = [
    "uipath>=2.5.27.dev1011480000,<2.5.27.dev1011490000",
]

@cristipufu cristipufu requested a review from akshaylive January 18, 2026 06:34
@cristipufu cristipufu self-assigned this Jan 18, 2026
@cristipufu cristipufu added the build:dev Create a dev build from the pr label Jan 18, 2026
@github-actions github-actions bot added test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository labels Jan 18, 2026
cristipufu
cristipufu commented Jan 18, 2026
View reviewed changes
@cristipufu cristipufu force-pushed the fix/decouple_mocking branch 3 times, most recently from a583ab4 to 0f86c8f Compare January 18, 2026 06:43
@cristipufu cristipufu requested a review from mjnovice January 18, 2026 06:43
@cristipufu cristipufu force-pushed the fix/decouple_mocking branch from 0f86c8f to 631956b Compare January 18, 2026 07:33
cristipufu
cristipufu commented Jan 18, 2026
View reviewed changes
akshaylive
akshaylive reviewed Jan 18, 2026
View reviewed changes
Copy link
Collaborator

@akshaylive akshaylive left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great overall. We do need to figure out the details regarding agentInfo and evaluation_criterias in a debug context.

mjnovice
mjnovice reviewed Jan 20, 2026
View reviewed changes
@cristipufu cristipufu force-pushed the fix/decouple_mocking branch 2 times, most recently from 442ee42 to f468cd6 Compare January 21, 2026 18:23
@cristipufu cristipufu force-pushed the fix/decouple_mocking branch from f468cd6 to e44e2f8 Compare January 21, 2026 18:28
akshaylive
akshaylive approved these changes Jan 21, 2026
View reviewed changes
Copy link
Collaborator

@akshaylive akshaylive left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

src/uipath/_cli/_evals/mocks/input_mocker.py
evaluation_item: EvaluationItem,
mocking_strategy: InputMockingStrategy,
input_schema: dict[str, Any],
expected_behavior: str,
Copy link
Collaborator

@akshaylive akshaylive Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make this Optional[str] and Optional[dict[str, Any]] respectively please

src/uipath/_cli/_evals/mocks/input_mocker.py
if evaluation_item.input_mocking_strategy
else None
)
model_parameters = mocking_strategy.model if mocking_strategy else None
Copy link
Collaborator

@akshaylive akshaylive Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line can be removed -- mocking_strategy cannot be None here.

@akshaylive akshaylive merged commit 5f18795 into main Jan 21, 2026
123 of 124 checks passed
@akshaylive akshaylive deleted the fix/decouple_mocking branch January 21, 2026 18:50
mjnovice added a commit that referenced this pull request Jan 21, 2026
@mjnovice @claude
refactor: remove dummy eval item from debug simulation
66ae6b9 
Now that mocking is decoupled from evals (PR #1148), update the debug
command to use MockingContext directly instead of creating a dummy
EvaluationItem. This simplifies the code and better reflects the
separation of concerns.

Changes:
- Update load_simulation_config to return MockingContext instead of EvaluationItem
- Replace EvaluationSimulationTool with ToolSimulation
- Update all tests to work with MockingContext
- Remove unnecessary evaluation_criterias field

Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mjnovice mjnovice mjnovice left review comments

@akshaylive akshaylive akshaylive approved these changes

Assignees

@cristipufu cristipufu

Labels

build:dev Create a dev build from the pr test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@cristipufu @akshaylive @mjnovice