[BUG][PLUGINS]: External MCP plugin session not recovered after plugin restart

[julianstephen]

🐞 Bug Summary

Plugin adapter fails if external plugin restarts.

---

🧩 Affected Component

Select the area of the project impacted:

• mcpgateway - API
• mcpgateway - UI (admin panel)
• mcpgateway.wrapper - stdio wrapper
• Federation or Transports
• CLI, Makefiles, or shell scripts
• Container setup (Docker/Podman/Compose)
• Other (external plugin)

---

🔁 Steps to Reproduce

1. Deploy and configure MCP based external plugin
2. Update plugin config, configure and enable external plugin defined in step 1
3. Start plugin manager. (Plugin manager finds external plugin, make sure plugin works, all good)
4. Restart external plugin, wait for external plugin to be 'ready' again
5. Send new request and plugin manager fails

---

🤔 Expected Behavior

What should have happened instead?
Plugin manager should've restarted session with the plugin

---

📓 Logs / Error Output

Paste any relevant stack traces or logs here.

22:52:45 - mcpgateway.plugins.framework.external.mcp.client - ERROR - Session terminated                                                                                                                            
Traceback (most recent call last):                                                                                                                                                                                  
  File "/usr/local/lib/python3.12/site-packages/mcpgateway/plugins/framework/external/mcp/client.py", line 240, in invoke_hook                                                                                      
    result = await self._session.call_tool(INVOKE_HOOK, {HOOK_TYPE: hook_type, PLUGIN_NAME: self.name, PAYLOAD: payload, CONTEXT: context})                                                                         
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                         
  File "/usr/local/lib/python3.12/site-packages/mcp/client/session.py", line 383, in call_tool                                                                                                                      
    result = await self.send_request(                                                                                                                                                                               
             ^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                               
  File "/usr/local/lib/python3.12/site-packages/mcp/shared/session.py", line 306, in send_request                                                                                                                   
    raise McpError(response_or_error.error)                                                                                                                                                                         
mcp.shared.exceptions.McpError: Session terminated                                                                                                                                                                  
22:52:45 - httpx - INFO - HTTP Request: POST http://nemocheck-plugin-service:8000/mcp "HTTP/1.1 404 Not Found"                                                                                                      
22:52:45 - mcpgateway.plugins.framework.external.mcp.client - ERROR - Session terminated                                                                                                                            
Traceback (most recent call last):                                                                                                                                                                                  
  File "/usr/local/lib/python3.12/site-packages/mcpgateway/plugins/framework/external/mcp/client.py", line 240, in invoke_hook                                                                                      
    result = await self._session.call_tool(INVOKE_HOOK, {HOOK_TYPE: hook_type, PLUGIN_NAME: self.name, PAYLOAD: payload, CONTEXT: context})
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcp/client/session.py", line 383, in call_tool
    result = await self.send_request(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcp/shared/session.py", line 306, in send_request
    raise McpError(response_or_error.error)
mcp.shared.exceptions.McpError: Session terminated

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/mcpgateway/plugins/framework/manager.py", line 356, in _execute_with_timeout
    return await asyncio.wait_for(hook_ref.hook(payload, context), timeout=self.timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcpgateway/plugins/framework/external/mcp/client.py", line 263, in invoke_hook
    raise PluginError(error=convert_exception_to_error(e, plugin_name=self.name))
mcpgateway.plugins.framework.errors.PluginError: McpError('Session terminated')

During handling of the above exception, another exception occurred:
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/mcpgateway/plugins/framework/external/mcp/client.py", line 240, in invoke_hook
    result = await self._session.call_tool(INVOKE_HOOK, {HOOK_TYPE: hook_type, PLUGIN_NAME: self.name, PAYLOAD: payload, CONTEXT: context})
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcp/client/session.py", line 383, in call_tool
    result = await self.send_request(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcp/shared/session.py", line 306, in send_request
    raise McpError(response_or_error.error)
mcp.shared.exceptions.McpError: Session terminated
22:52:45 - mcpgateway.plugins.framework.manager - ERROR - Plugin NemoCheck failed with error: McpError('Session terminated')
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/mcpgateway/plugins/framework/external/mcp/client.py", line 240, in invoke_hook
    result = await self._session.call_tool(INVOKE_HOOK, {HOOK_TYPE: hook_type, PLUGIN_NAME: self.name, PAYLOAD: payload, CONTEXT: context})
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcp/client/session.py", line 383, in call_tool
    result = await self.send_request(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcp/shared/session.py", line 306, in send_request
    raise McpError(response_or_error.error)
mcp.shared.exceptions.McpError: Session terminated

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/mcpgateway/plugins/framework/manager.py", line 356, in _execute_with_timeout
    return await asyncio.wait_for(hook_ref.hook(payload, context), timeout=self.timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcpgateway/plugins/framework/external/mcp/client.py", line 263, in invoke_hook
    raise PluginError(error=convert_exception_to_error(e, plugin_name=self.name))
mcpgateway.plugins.framework.errors.PluginError: McpError('Session terminated')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/mcpgateway/plugins/framework/external/mcp/client.py", line 240, in invoke_hook
    result = await self._session.call_tool(INVOKE_HOOK, {HOOK_TYPE: hook_type, PLUGIN_NAME: self.name, PAYLOAD: payload, CONTEXT: context})
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcp/client/session.py", line 383, in call_tool
    result = await self.send_request(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcp/shared/session.py", line 306, in send_request
    raise McpError(response_or_error.error)
mcp.shared.exceptions.McpError: Session terminated
Traceback (most recent call last):                                                                                                                                                                         [25/1843]
  File "/usr/local/lib/python3.12/site-packages/mcpgateway/plugins/framework/manager.py", line 226, in execute_plugin
    result = await self._execute_with_timeout(hook_ref, payload, local_context)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcpgateway/plugins/framework/manager.py", line 361, in _execute_with_timeout
    return await asyncio.wait_for(hook_ref.hook(payload, context), timeout=self.timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcpgateway/plugins/framework/external/mcp/client.py", line 263, in invoke_hook
    raise PluginError(error=convert_exception_to_error(e, plugin_name=self.name))
mcpgateway.plugins.framework.errors.PluginError: McpError('Session terminated')
22:52:45 - grpc._cython.cygrpc - ERROR - Unexpected [PluginError] raised by servicer method [/envoy.service.ext_proc.v3.ExternalProcessor/Process]
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/mcpgateway/plugins/framework/external/mcp/client.py", line 240, in invoke_hook
    result = await self._session.call_tool(INVOKE_HOOK, {HOOK_TYPE: hook_type, PLUGIN_NAME: self.name, PAYLOAD: payload, CONTEXT: context})
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcp/client/session.py", line 383, in call_tool
    result = await self.send_request(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcp/shared/session.py", line 306, in send_request
    raise McpError(response_or_error.error)
mcp.shared.exceptions.McpError: Session terminated

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/mcpgateway/plugins/framework/manager.py", line 356, in _execute_with_timeout
    return await asyncio.wait_for(hook_ref.hook(payload, context), timeout=self.timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcpgateway/plugins/framework/external/mcp/client.py", line 263, in invoke_hook
    raise PluginError(error=convert_exception_to_error(e, plugin_name=self.name))
mcpgateway.plugins.framework.errors.PluginError: McpError('Session terminated')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/mcpgateway/plugins/framework/external/mcp/client.py", line 240, in invoke_hook
    result = await self._session.call_tool(INVOKE_HOOK, {HOOK_TYPE: hook_type, PLUGIN_NAME: self.name, PAYLOAD: payload, CONTEXT: context})
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcp/client/session.py", line 383, in call_tool
    result = await self.send_request(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcp/shared/session.py", line 306, in send_request
    raise McpError(response_or_error.error)
mcp.shared.exceptions.McpError: Session terminated

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/server.pyx.pxi", line 689, in grpc._cython.cygrpc._handle_exceptions
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/server.pyx.pxi", line 852, in _handle_rpc
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/server.pyx.pxi", line 677, in _handle_stream_stream_rpc 
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/server.pyx.pxi", line 501, in _finish_handler_with_stream_responses
  File "/app/src/server.py", line 289, in Process
    body_resp = await getToolPreInvokeResponse(body)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/src/server.py", line 67, in getToolPreInvokeResponse
    result, _ = await manager.invoke_hook(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcpgateway/plugins/framework/manager.py", line 601, in invoke_hook
    result = await self._executor.execute(hook_refs, payload, global_context, hook_type, local_contexts, violations_as_exceptions)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcpgateway/plugins/framework/manager.py", line 176, in execute
    result = await self.execute_plugin(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcpgateway/plugins/framework/manager.py", line 226, in execute_plugin
    result = await self._execute_with_timeout(hook_ref, payload, local_context)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcpgateway/plugins/framework/manager.py", line 361, in _execute_with_timeout
    return await asyncio.wait_for(hook_ref.hook(payload, context), timeout=self.timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mcpgateway/plugins/framework/external/mcp/client.py", line 263, in invoke_hook
    raise PluginError(error=convert_exception_to_error(e, plugin_name=self.name))
mcpgateway.plugins.framework.errors.PluginError: McpError('Session terminated')

---

🧠 Environment Info

You can retrieve most of this from the /version endpoint.

Key	Value
mcp-contextforge-gateway`	==0.9.0
Runtime	3.12.12
Platform / OS	rhel 5.14.0-611.13.1.el9_7.x86_64
Container	Docker (in kind)

---

🧩 Additional Context (optional)

Add any configuration details, flags, or related issues.

[crivetimihai]

Thanks for the detailed report and logs, @julianstephen.

Confirmed — the MCP external plugin client (plugins/framework/external/mcp/client.py) has no reconnection logic. The session is created once at init (line 315-322) and invoke_hook() (line 344-392) uses it directly with no health check or retry. When the remote plugin restarts, the stale ClientSession gets HTTP 404 → McpError("Session terminated") with no recovery path.

Notably, the config model already defines reconnect_attempts and reconnect_delay fields (models.py:1084-1085) but the MCP client never uses them. The Unix socket client (external/unix/client.py:134-201) does implement full reconnection with health checks and exponential backoff — the MCP client needs feature parity.

Suggested fix approach:

1. Add a _reconnect_session() method to ExternalPlugin that tears down the old session and re-runs the HTTP connection setup (__connect_to_http_server)
2. Wrap invoke_hook() with catch-and-retry: on McpError("Session terminated"), call _reconnect_session() and retry once
3. Wire up the existing reconnect_attempts / reconnect_delay config fields

The Unix socket client is the reference implementation for the reconnection pattern.