Remove modelRouter and add model_providers concept #224

philipph-askui · 2026-01-21T07:20:44Z

[Edited] PR Description

To get a quick idea of the new API, I added an example under examples/model_providers.py that you can run
Note: You will need an anthropic API Key for that

This PR:

Removes ModelRouter, ModelRegistry, and model_store
Introduces a provider-based configuration system (AgentSettings)
Renames VisionAgent → ComputerAgent and AndroidVisionAgent → AndroidAgent
Updates docs and examples

Summary

Replaced the ModelRouter/model_store abstraction with three typed provider slots (vlm_provider, image_qa_provider, detection_provider) configured via AgentSettings
Providers own their endpoint, credentials, and model ID — validated lazily on first API call
get() and locate() are now backed by GetTool/LocateTool, which are also available to the LLM during act() — no separate model injection path
Significantly reduced codebase complexity (~4600 lines removed)
Updated all docs and examples to reflect the new API

Key Changes

AgentSettings: single configuration object with provider slots; defaults to AskUI-hosted providers reading credentials from env vars
Built-in providers: AskUIVlmProvider, AskUIImageQAProvider, AskUIDetectionProvider, AnthropicVlmProvider, AnthropicImageQAProvider, GoogleImageQAProvider, OpenAICompatibleProvider
GetTool / LocateTool: wired into the act loop as ToolWithAgentOS — LLM can call them directly during act()
Deleted: entire src/askui/model_store/ directory
Renamed: VisionAgent → ComputerAgent, AndroidVisionAgent → AndroidAgent
Docs: 03_Using-Models-and-BYOM.md fully rewritten; VisionAgent replaced with ComputerAgent across all docs

Breaking Changes

VisionAgent is removed — use ComputerAgent
AndroidVisionAgent is removed — use AndroidAgent
act_model, get_model, locate_model constructor parameters are removed — use AgentSettings(vlm_provider=..., image_qa_provider=..., detection_provider=...)
model_store factory functions are removed
String-based model selection is removed

…del store BREAKING CHANGE: Removed ModelRouter and ModelRegistry classes. Users must now use direct model injection.

…modelrouter

…raises an Error when executing from cache)

…move() methods

…nd AskUiInferenceLocateApi

mlikasam-askui · 2026-01-30T10:17:48Z

docs/01_Setup.md

+**Problem**: Error connecting to Agent OS
+
+**Solutions**:
+1. Check if Agent OS is running (look for the system tray icon)


The agent OS doesn’t have a tray icon.

mlikasam-askui · 2026-01-30T10:18:21Z

docs/01_Setup.md

+
+**Solutions**:
+1. Check if Agent OS is running (look for the system tray icon)
+2. Restart Agent OS from your applications menu


The agent OS is not listed in the application menu.

mlikasam-askui · 2026-01-30T10:18:48Z

docs/03_Using-Models-and-BYOM.md

+    custom_settings = ActSettings(
+        messages=MessageSettings(
+            max_tokens=8192,
+            temperature=0.5,
+            betas=["computer-use-2025-01-24"],
+        )
+    )
+


Which system prompt is used in this example?
Could you please remove the betas?

mlikasam-askui · 2026-01-28T09:04:30Z

docs/01_Setup.md

+
+## Python Package Installation
+
+AskUI Vision Agent requires Python 3.10 or higher.


requires-python = ">=3.10,<3.14"

mlikasam-askui · 2026-01-28T09:06:39Z

docs/01_Setup.md

+```bash
+pip install askui[anthropic]  # Anthropic Claude support
+pip install askui[openrouter]  # OpenRouter support
+pip install askui[documents]  # PDF, Excel, Word support
+```


SDK dosent support these targets

mlikasam-askui · 2026-01-30T08:39:34Z

src/askui/models/anthropic/settings.py

+        super().__init__(self.message)
+
+
+class AnthropicModelSettings(BaseSettings):


Currently unused.

mlikasam-askui · 2026-01-30T08:42:29Z

src/askui/models/askui/locate_api.py

+        self,
+        locator: str | Locator,
+        image: ImageSource,
+        locate_settings: LocateSettings,  # noqa: ARG002


Are the Locate settings only needed for LLM-based locators?

idea was to have a general settings object for all locate commands here

mlikasam-askui · 2026-01-30T09:11:19Z

src/askui/models/shared/settings.py

+    max_tokens: int = 4096
+    temperature: float = Field(default=0.5, ge=0.0, le=1.0)
+    system_prompt: GetSystemPrompt | None = None
+    timeout: float | None = None
+
+
+class LocateSettings(BaseModel):
+    """Settings for LocateModel operations (UI element location)."""
+
+    model_config = ConfigDict(arbitrary_types_allowed=True)
+
+    query_type: str | None = None
+    confidence_threshold: float = Field(default=0.8, ge=0.0, le=1.0)
+    max_detections: int = 10
+    timeout: float | None = None
+    system_prompt: LocateSystemPrompt | None = None


Currently, only the system prompt is being used.

mlikasam-askui · 2026-01-30T09:12:32Z

src/askui/models/shared/settings.py

+    confidence_threshold: float = Field(default=0.8, ge=0.0, le=1.0)
+    max_detections: int = 10
+    timeout: float | None = None
+    system_prompt: LocateSystemPrompt | None = None


The Locate system prompt should not be configurable because the expected return is currently hard-coded. Changing the system prompt would cause the Locate code to fail.

mlikasam-askui · 2026-01-30T09:58:18Z

src/askui/models/shared/settings.py

+    timeout: float | None = None
+
+
+class LocateSettings(BaseModel):


What do you think about removing the Locate and Get settings?

I think we should generally discuss how "configurable" get and locate should be. This also includes if we want to support BYOM for these commands or if that should only be possible for act

…ttings Introduces VlmProvider, ImageQAProvider, and DetectionProvider slots on AgentSettings. GetTool/LocateTool are now ToolWithAgentOS and available in the act() loop. Renames VisionAgent→ComputerAgent and AndroidVisionAgent→AndroidAgent. Removes model_store entirely.

programminx-askui

Hello @philipph-askui,

I started to review the code. General remarks:

Changes are to big to review. -> We need to test this heavly
Creating new files, instead of rename/move files -> we don't know which code was already reviewed, which code is new.

Overall it is going in the right direction.

I've reviewed only view files, so you can start working on it. A deeper review is outstanding.

programminx-askui · 2026-01-28T06:24:31Z

docs/00_Overview.md

+- Brittle selectors that break when UI changes
+- Separate tools for different platforms (web, desktop, mobile)
+- Manual scripting of every action step
+- Constant maintenance as applications evolve


Suggested change

- Constant maintenance as applications evolve

- Constant maintenance as applications evolve

- Random application behavior

- External issues like network, rights or installation issues

programminx-askui · 2026-01-28T06:27:16Z

docs/00_Overview.md

+**1. Programmatic Control**
+```python
+from askui import VisionAgent
+
+with VisionAgent() as agent:
+    agent.click("Submit button")
+    agent.type("[email protected]")
+    result = agent.get("What's the current page title?")
+```
+
+Direct, single-step commands for precise UI control. Like traditional automation, but powered by vision models that understand what elements look like, not just their DOM structure.
+
+**2. Agentic Control (Goal-based)**
+```python
+with VisionAgent() as agent:
+    agent.act(
+        "Search for flights from New York to London, "
+        "filter by direct flights, and show me the cheapest option"
+    )
+```


I would go full Agentic. And try to avoid the "Programmtic Code" and when we show the agentic first.

Suggested change

**1. Programmatic Control**

```python

from askui import VisionAgent

with VisionAgent() as agent:

agent.click("Submit button")

agent.type("[email protected]")

result = agent.get("What's the current page title?")

```

Direct, single-step commands for precise UI control. Like traditional automation, but powered by vision models that understand what elements look like, not just their DOM structure.

**2. Agentic Control (Goal-based)**

```python

with VisionAgent() as agent:

agent.act(

"Search for flights from New York to London, "

"filter by direct flights, and show me the cheapest option"

)

```

*1. Agentic Control (Goal-based)**

```python

with VisionAgent() as agent:

agent.act(

"Search for flights from New York to London, "

"filter by direct flights, and show me the cheapest option"

)

programminx-askui · 2026-01-28T06:27:49Z

docs/00_Overview.md

+
+### Key Capabilities
+
+- **Multi-Platform**: Windows, MacOS, Linux, Android


Suggested change

- **Multi-Platform**: Windows, MacOS, Linux, Android

- **Multi-Platform**: Windows, MacOS, Linux, Android, Citric & KVM

programminx-askui · 2026-01-28T06:41:38Z

docs/00_Overview.md

+
+Understand the model system, how to choose models for different tasks, and how to integrate custom models or third-party providers.
+
+### 04 - Caching


We we want to rename "caching"? In reality it's Token Efficient Rerun.

programminx-askui · 2026-01-28T06:47:41Z

docs/00_Overview.md

+This documentation is organized to take you from setup to advanced usage:
+
+### 01 - Setup
+**Topics**: Installation, Agent OS setup, environment configuration, authentication


Suggested change

**Topics**: Installation, Agent OS setup, environment configuration, authentication

**Topics**: Installation, AgentOS setup, environment configuration, authentication

programminx-askui · 2026-02-12T12:45:13Z

src/askui/model_providers/askui_image_qa_provider.py

+        Raises:
+            ValueError: If the source data exceeds the size limit.
+        """
+        import google.genai.types as genai_types


Use a try catch fo checking if the module is installed, if not raise a PackageNotInstalledException

programminx-askui · 2026-02-12T12:49:04Z

src/askui/models/askui/locate_models/aiElement_locate_model.py

@@ -0,0 +1,55 @@
+import logging


rename the file to ai_elment_locate_model

programminx-askui · 2026-02-12T12:50:32Z

src/askui/models/askui/locate_models/anthropic_locate_model.py

+    """
+
+    # Provider-specific configuration
+    DEFAULT_RESOLUTION: tuple[int, int] = (1280, 800)


If we have a DEFAULT_RESOLUTION, then the resolution should be changeable. otherwise it's the CLAUDE_IMAGE_RESOLUTION.

programminx-askui · 2026-02-12T12:52:52Z

src/askui/models/askui/locate_models/anthropic_locate_model.py

+    """
+
+    # Provider-specific configuration
+    DEFAULT_RESOLUTION: tuple[int, int] = (1280, 800)


Can we not use a namedtuple?

Resolution = namedtuple('Resolution', ['width', 'height'])

programminx-askui · 2026-02-12T12:53:26Z

src/askui/models/askui/locate_models/anthropic_locate_model.py

+            screen_width = self.DEFAULT_RESOLUTION[0]
+            screen_height = self.DEFAULT_RESOLUTION[1]


With the namedtuple, then we can set it her more devloper friendly

programminx-askui

Hello @philipph-askui ,

I started to review the code. General remarks:

Changes are to big to review. -> We need to test this heavly
Creating new files, instead of rename/move files -> we don't know which code was already reviewed, which code is new.

Overall it is going in the right direction.

I've reviewed only view files, so you can start working on it. A deeper review is outstanding.

philipph-askui added 3 commits January 21, 2026 06:57

chore: add plan and claude rules

b99056d

refactor: simplify model selection with direct model injection and mo…

7a23ca9

…del store BREAKING CHANGE: Removed ModelRouter and ModelRegistry classes. Users must now use direct model injection.

resolve merge conflict

421d043

philipph-askui changed the title ~~Chore/modelrouter~~ Remove Modelrouter Jan 21, 2026

philipph-askui and others added 12 commits January 21, 2026 08:23

Merge branch 'main' into chore/modelrouter

188ddd2

feat: remove chat related dependencies and components

5c3ade4

Merge remote-tracking branch 'origin/feat/deprecate-chat' into chore/…

dfcda56

…modelrouter

chore: remove dependency on anthropic types

00d359c

fix(agentOS): set default value of name for Display to None (else it …

339717f

…raises an Error when executing from cache)

chore: fix typecheck errors

1d5929b

fix linting errors

021c90b

fix linting errors

2598a46

feat: add model_store

f1460a9

fix linting errors

d8ce5b2

Merge branch 'main' into chore/modelrouter

d4ea0d8

fix linting errors

a0a75b1

philipph-askui changed the title ~~Remove Modelrouter~~ Remove modelRouter and add mode_store Jan 22, 2026

philipph-askui added 11 commits January 22, 2026 15:16

remove UITARS

93ea9ee

remove ModelComposition

b2c79be

update get model interface

f4c3cb2

remove UITars, modelcomposition, and model definition

6222282

Add locate_settings and locate_model parameters to click() and mouse_…

fca6234

…move() methods

Remove AskUiBaseLocateModel class and create LocateApi (Base class) a…

e3dc24a

…nd AskUiInferenceLocateApi

chore: remove GenericActModel

4ac0df4

move HF models and OpenRouter models to model_store

25a3ad8

chore: fix docs

19c8968

fix: make models serializable for telemetry

6217b11

chore: update docs

8f3a9ea

philipph-askui marked this pull request as ready for review January 26, 2026 06:53

philipph-askui requested a review from mlikasam-askui January 26, 2026 06:54

philipph-askui requested a review from programminx-askui January 26, 2026 06:54

philipph-askui changed the title ~~Remove modelRouter and add mode_store~~ Remove modelRouter and add model_store Jan 26, 2026

mlikasam-askui requested changes Jan 30, 2026

View reviewed changes

address review remakrs

0732ac8

philipph-askui changed the title ~~Remove modelRouter and add model_store~~ Remove modelRouter and add model_providers concept Feb 12, 2026

philipph-askui added 4 commits February 12, 2026 07:23

add model_providers example

5534ace

fix: default_model_id for vlm

2eb587b

fix tests

6267e29

programminx-askui requested changes Feb 12, 2026

View reviewed changes


		## Python Package Installation

		AskUI Vision Agent requires Python 3.10 or higher.

		super().__init__(self.message)


		class AnthropicModelSettings(BaseSettings):

		timeout: float \| None = None


		class LocateSettings(BaseModel):


		### Key Capabilities

		- Multi-Platform: Windows, MacOS, Linux, Android


		Understand the model system, how to choose models for different tasks, and how to integrate custom models or third-party providers.

		### 04 - Caching

	Topics: Installation, Agent OS setup, environment configuration, authentication
	Topics: Installation, AgentOS setup, environment configuration, authentication

		screen_width = self.DEFAULT_RESOLUTION[0]
		screen_height = self.DEFAULT_RESOLUTION[1]

Remove modelRouter and add model_providers concept #224

Are you sure you want to change the base?

Remove modelRouter and add model_providers concept #224

Uh oh!

Conversation

philipph-askui commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

programminx-askui left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

programminx-askui left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

philipph-askui commented Jan 21, 2026 •

edited

Loading