[Bug]: Midscene aiAct may get stuck in local interaction loops during multi-step tasks without detecting lack of progress

[omgseven]

Version

System:
    OS: macOS 26.3
    CPU: (14) arm64 Apple M4 Pro
    Memory: 127.92 MB / 24.00 GB
    Shell: 5.9 - /bin/zsh
  Browsers:
    Chrome: 145.0.7632.160
    Safari: 26.3
  npmPackages:
    @midscene/android: ^1.5.0 => 1.5.2 
    @midscene/harmony: ^1.5.0 => 1.5.2 
    @midscene/ios: ^1.5.0 => 1.5.2 
    @midscene/web: ^1.5.0 => 1.5.2

Details

In multi-step interaction flows, the agent may repeatedly enter the same intermediate state and make locally reasonable decisions that do not lead to overall task completion. As a result, it can become trapped in a loop of repeated actions without recognizing that it is no longer making progress.

This issue is not limited to cases where the UI changes before execution. It can also happen when a task involves multiple dependent steps and the agent repeatedly:

• performs an action,
• reaches an intermediate UI state,
• makes a recovery or dismissal decision,
• returns to a prior state,
• and then triggers the same intermediate state again.

In such scenarios, Midscene may continue cycling through equivalent action patterns instead of re-evaluating the broader task strategy. The agent appears unable to:

• recognize that the same state-transition pattern is repeating,
• detect that previous attempts have not advanced the task,
• or escape the local decision loop through replanning.

In addition, from the thought process, it does not appear that the agent is aware it has entered a repeated retry / no-progress cycle.

Expected behavior

• The agent should detect repeated state-transition patterns in multi-step flows.
• It should distinguish between local action success and global task progress.
• After multiple equivalent cycles, it should re-evaluate the plan instead of continuing the same interaction pattern.
• It should have mechanisms for loop detection, no-progress detection, and recovery through replanning.

Actual behavior

• The agent continues performing repeated sequences of equivalent actions.
• It may repeatedly return to the same intermediate state.
• It does not effectively detect that the task is stuck in a local loop.
• It fails to escalate from local retries to broader replanning.

Possible directions

• Add loop detection based on repeated action-state-transition patterns.
• Track progress at the task level, not only at the individual action level.
• Detect “returned to a previously seen state without net progress” conditions.
• Trigger replanning after repeated intermediate-state cycles.
• Improve thought/reflection mechanisms so the agent can explicitly recognize when it is stuck.

Reproduce link

sorry-not-to-provide

Reproduce Steps

1. Run a multi-step navigation task with intermediate confirmation or modal states.
2. Allow the task flow to re-enter the same intermediate state after an action is dismissed or retried.
3. Use a slower-reasoning model so that planning and execution may lag behind the latest UI state.
4. Observe that the agent repeatedly performs similar actions without making task-level progress.
5. Verify from the logs / thoughts that the agent does not detect or report being stuck in a loop.

[omgseven]

Maybe this is related to llm-planning. One possible reason is that the planner does not consider enough history, and the prompt may also lack sufficient loop-avoidance guidance.