Skip to content

feat(ai): instrument tool execution spans and definitions#1109

Open
constantinius wants to merge 13 commits intoconstantinius/feat/ai/integration-basefrom
constantinius/feat/ai/tool-spans
Open

feat(ai): instrument tool execution spans and definitions#1109
constantinius wants to merge 13 commits intoconstantinius/feat/ai/integration-basefrom
constantinius/feat/ai/tool-spans

Conversation

@constantinius
Copy link
Copy Markdown

@constantinius constantinius commented Mar 16, 2026

Add execute_tool span lifecycle around InvokingTool and ToolInvoked events, and propagate serialized tool definitions onto invoke_agent and chat spans. This isolates tool-level monitoring while leaving message payload capture and embeddings for later stacked branches.

Included

  • Implements tool lifecycle handlers:
    • InvokingTool -> start gen_ai.execute_tool span
    • ToolInvoked -> finish gen_ai.execute_tool span
  • Captures tool metadata:
    • tool name/type/description
    • tool call arguments/result (when default PII is enabled)
  • Adds and propagates serialized tool definitions:
    • on invoke_agent span
    • on chat spans
  • Extends conversation-id propagation to tool spans.
  • Adds tracing.gen_ai_execute_tool toggle.

Made-with: Cursor

Issues

Add the initial AI integration wiring that records invoke_agent spans for prompting and prompted lifecycle events, including model/provider and token usage metadata. This establishes the stack base without chat, tool, message, or embeddings instrumentation.

Made-with: Cursor
@linear-code
Copy link
Copy Markdown

linear-code bot commented Mar 16, 2026

@constantinius constantinius requested a review from Litarnus March 16, 2026 14:35
Copy link
Copy Markdown

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

  • ✅ Fixed: TestAgent missing HasTools interface breaks tool definitions assertions
    • I added a guarded HasTools test stub and made TestAgent implement it so resolveToolDefinitions() now includes tool definitions for the asserted spans.

Create PR

Or push these changes by commenting:

@cursor push 9024ee0ae4
Preview (9024ee0ae4)
diff --git a/test/Sentry/Features/AiIntegrationTest.php b/test/Sentry/Features/AiIntegrationTest.php
--- a/test/Sentry/Features/AiIntegrationTest.php
+++ b/test/Sentry/Features/AiIntegrationTest.php
@@ -8,6 +8,12 @@
     }
 }
 
+if (!interface_exists(HasTools::class)) {
+    interface HasTools
+    {
+    }
+}
+
 if (!interface_exists(Tool::class)) {
     interface Tool
     {
@@ -290,6 +296,7 @@
 use Laravel\Ai\Attributes\MaxTokens;
 use Laravel\Ai\Attributes\Temperature;
 use Laravel\Ai\Contracts\Agent;
+use Laravel\Ai\Contracts\HasTools;
 use Laravel\Ai\Contracts\Providers\EmbeddingProvider;
 use Laravel\Ai\Contracts\Providers\TextProvider;
 use Laravel\Ai\Contracts\Tool;
@@ -297,7 +304,7 @@
 use Laravel\Ai\Responses\QueuedAgentResponse;
 use Laravel\Ai\Responses\StreamableAgentResponse;
 
-class TestAgent implements Agent
+class TestAgent implements Agent, HasTools
 {
     public function instructions(): string
     {

This Bugbot Autofix run was free. To enable autofix for future PRs, go to the Cursor dashboard.

@constantinius constantinius changed the base branch from constantinius/feat/ai/integration to constantinius/feat/ai/integration-base March 16, 2026 14:53
}

return null;
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Provider prefix overmatches unrelated HTTP requests

Medium Severity

findMatchingInvocation matches any request under the provider base URL, so unrelated calls (for example tool-triggered requests to other provider endpoints) are treated as gen_ai.chat. This creates spurious chat spans, changes the hub’s current span unexpectedly, and pollutes chatSpans ordering used for step-to-span enrichment.

Additional Locations (1)
Fix in Cursor Fix in Web

Replace inline stub definitions for Laravel AI interfaces, classes,
events, attributes, and file types with the real implementations from
the laravel/ai package added as a dev dependency.

Drop PHP version and Laravel version skip guards from setUp() since
these tests only run on PHP 8.4+.

Simplify TestAgentWithConfig to use native PHP 8 attributes directly
instead of the eval() workaround for older PHP versions.
@constantinius constantinius force-pushed the constantinius/feat/ai/tool-spans branch from 3bce234 to 43fdabf Compare April 2, 2026 12:06
$data['gen_ai.conversation.id'] = $conversationId;
$span->setData($data);
}
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conversation ID write to agent span is immediately overwritten

Low Severity

setConversationIdOnSpans includes the agent span ($invocation->span) in the set of spans it updates with the conversation ID. However, the caller handleAgentPromptedForTracing already adds the conversation ID to its local $data array (line 153) and then overwrites the agent span's entire data with $agentSpan->setData($data) (line 157), immediately after calling setConversationIdOnSpans (line 154). The modification setConversationIdOnSpans made to the agent span is discarded. The agent span could be excluded from the loop in setConversationIdOnSpans to avoid the wasted read-modify-write cycle.

Additional Locations (1)
Fix in Cursor Fix in Web

@constantinius constantinius force-pushed the constantinius/feat/ai/tool-spans branch from 43fdabf to 9d0765c Compare April 2, 2026 13:11
Remove laravel/ai globally in both phpunit and phpunit-legacy jobs
since it requires PHP 8.4+ and only supports Laravel 12. Re-add it
conditionally for the supported combination (PHP 8.4+, Laravel 12).
@constantinius constantinius force-pushed the constantinius/feat/ai/tool-spans branch from 9d0765c to c0a6668 Compare April 2, 2026 13:31
The Laravel AI classes are not available when PHPStan runs on PHP 8.3
without laravel/ai installed. Add baseline ignores matching the
existing pattern used for other optional packages (Sanctum, Livewire,
Lumen).
Capture provider HTTP request/response events as gen_ai.chat spans under invoke_agent and enrich each chat span with response model, usage, finish reason, and conversation id mapping. This keeps tool/message/embeddings behavior out of scope for follow-up stacked branches.

Made-with: Cursor
Add execute_tool span lifecycle around InvokingTool and ToolInvoked events, and propagate serialized tool definitions onto invoke_agent and chat spans. This isolates tool-level monitoring while leaving message payload capture and embeddings for later stacked branches.

Made-with: Cursor
Add a HasTools stub and implement it on TestAgent so tool definition metadata is populated under the same contract checks used by AiIntegration.

Made-with: Cursor
@constantinius constantinius force-pushed the constantinius/feat/ai/tool-spans branch from c0a6668 to 1e34147 Compare April 2, 2026 13:49
Copy link
Copy Markdown

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

Fix All in Cursor

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.


$responseProvider = $event->response->meta->provider ?? null;
if ($responseProvider !== null && !isset($data['gen_ai.provider.name'])) {
$data['gen_ai.provider.name'] = strtolower($responseProvider);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Provider name normalization inconsistency across code paths

Low Severity

The gen_ai.provider.name attribute is set with inconsistent casing. In handlePromptingAgentForTracing, $provider->name() is stored as-is, but in handleAgentPromptedForTracing, the fallback from $event->response->meta->provider is wrapped in strtolower(). If a provider's name() method returns a mixed-case string (e.g., "OpenAI"), the primary path stores it verbatim while the fallback path lowercases it, leading to inconsistent attribute values across traces for the same provider.

Additional Locations (1)
Fix in Cursor Fix in Web

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant