feat(ai): instrument tool execution spans and definitions#1109
feat(ai): instrument tool execution spans and definitions#1109constantinius wants to merge 13 commits intoconstantinius/feat/ai/integration-basefrom
Conversation
Add the initial AI integration wiring that records invoke_agent spans for prompting and prompted lifecycle events, including model/provider and token usage metadata. This establishes the stack base without chat, tool, message, or embeddings instrumentation. Made-with: Cursor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: TestAgent missing HasTools interface breaks tool definitions assertions
- I added a guarded
HasToolstest stub and madeTestAgentimplement it soresolveToolDefinitions()now includes tool definitions for the asserted spans.
- I added a guarded
Or push these changes by commenting:
@cursor push 9024ee0ae4
Preview (9024ee0ae4)
diff --git a/test/Sentry/Features/AiIntegrationTest.php b/test/Sentry/Features/AiIntegrationTest.php
--- a/test/Sentry/Features/AiIntegrationTest.php
+++ b/test/Sentry/Features/AiIntegrationTest.php
@@ -8,6 +8,12 @@
}
}
+if (!interface_exists(HasTools::class)) {
+ interface HasTools
+ {
+ }
+}
+
if (!interface_exists(Tool::class)) {
interface Tool
{
@@ -290,6 +296,7 @@
use Laravel\Ai\Attributes\MaxTokens;
use Laravel\Ai\Attributes\Temperature;
use Laravel\Ai\Contracts\Agent;
+use Laravel\Ai\Contracts\HasTools;
use Laravel\Ai\Contracts\Providers\EmbeddingProvider;
use Laravel\Ai\Contracts\Providers\TextProvider;
use Laravel\Ai\Contracts\Tool;
@@ -297,7 +304,7 @@
use Laravel\Ai\Responses\QueuedAgentResponse;
use Laravel\Ai\Responses\StreamableAgentResponse;
-class TestAgent implements Agent
+class TestAgent implements Agent, HasTools
{
public function instructions(): string
{This Bugbot Autofix run was free. To enable autofix for future PRs, go to the Cursor dashboard.
| } | ||
|
|
||
| return null; | ||
| } |
There was a problem hiding this comment.
Provider prefix overmatches unrelated HTTP requests
Medium Severity
findMatchingInvocation matches any request under the provider base URL, so unrelated calls (for example tool-triggered requests to other provider endpoints) are treated as gen_ai.chat. This creates spurious chat spans, changes the hub’s current span unexpectedly, and pollutes chatSpans ordering used for step-to-span enrichment.
Additional Locations (1)
Replace inline stub definitions for Laravel AI interfaces, classes, events, attributes, and file types with the real implementations from the laravel/ai package added as a dev dependency. Drop PHP version and Laravel version skip guards from setUp() since these tests only run on PHP 8.4+. Simplify TestAgentWithConfig to use native PHP 8 attributes directly instead of the eval() workaround for older PHP versions.
3bce234 to
43fdabf
Compare
| $data['gen_ai.conversation.id'] = $conversationId; | ||
| $span->setData($data); | ||
| } | ||
| } |
There was a problem hiding this comment.
Conversation ID write to agent span is immediately overwritten
Low Severity
setConversationIdOnSpans includes the agent span ($invocation->span) in the set of spans it updates with the conversation ID. However, the caller handleAgentPromptedForTracing already adds the conversation ID to its local $data array (line 153) and then overwrites the agent span's entire data with $agentSpan->setData($data) (line 157), immediately after calling setConversationIdOnSpans (line 154). The modification setConversationIdOnSpans made to the agent span is discarded. The agent span could be excluded from the loop in setConversationIdOnSpans to avoid the wasted read-modify-write cycle.
Additional Locations (1)
43fdabf to
9d0765c
Compare
Remove laravel/ai globally in both phpunit and phpunit-legacy jobs since it requires PHP 8.4+ and only supports Laravel 12. Re-add it conditionally for the supported combination (PHP 8.4+, Laravel 12).
9d0765c to
c0a6668
Compare
The Laravel AI classes are not available when PHPStan runs on PHP 8.3 without laravel/ai installed. Add baseline ignores matching the existing pattern used for other optional packages (Sanctum, Livewire, Lumen).
Capture provider HTTP request/response events as gen_ai.chat spans under invoke_agent and enrich each chat span with response model, usage, finish reason, and conversation id mapping. This keeps tool/message/embeddings behavior out of scope for follow-up stacked branches. Made-with: Cursor
Add execute_tool span lifecycle around InvokingTool and ToolInvoked events, and propagate serialized tool definitions onto invoke_agent and chat spans. This isolates tool-level monitoring while leaving message payload capture and embeddings for later stacked branches. Made-with: Cursor
Add a HasTools stub and implement it on TestAgent so tool definition metadata is populated under the same contract checks used by AiIntegration. Made-with: Cursor
c0a6668 to
1e34147
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
|
|
||
| $responseProvider = $event->response->meta->provider ?? null; | ||
| if ($responseProvider !== null && !isset($data['gen_ai.provider.name'])) { | ||
| $data['gen_ai.provider.name'] = strtolower($responseProvider); |
There was a problem hiding this comment.
Provider name normalization inconsistency across code paths
Low Severity
The gen_ai.provider.name attribute is set with inconsistent casing. In handlePromptingAgentForTracing, $provider->name() is stored as-is, but in handleAgentPromptedForTracing, the fallback from $event->response->meta->provider is wrapped in strtolower(). If a provider's name() method returns a mixed-case string (e.g., "OpenAI"), the primary path stores it verbatim while the fallback path lowercases it, leading to inconsistent attribute values across traces for the same provider.



Add execute_tool span lifecycle around InvokingTool and ToolInvoked events, and propagate serialized tool definitions onto invoke_agent and chat spans. This isolates tool-level monitoring while leaving message payload capture and embeddings for later stacked branches.
Included
InvokingTool-> startgen_ai.execute_toolspanToolInvoked-> finishgen_ai.execute_toolspaninvoke_agentspanchatspanstracing.gen_ai_execute_tooltoggle.Made-with: Cursor
Issues