Skip to content

Use a sidecar connection per PHP thread#3770

Open
bwoebi wants to merge 6 commits intomasterfrom
bob/thread-sidecar
Open

Use a sidecar connection per PHP thread#3770
bwoebi wants to merge 6 commits intomasterfrom
bob/thread-sidecar

Conversation

@bwoebi
Copy link
Copy Markdown
Collaborator

@bwoebi bwoebi commented Apr 2, 2026

While the sidecar has been a process global for the longest time, it's also unfair with respect to limits and not compatible with the newly introduced outbox.

Having a connection per thread ensures that every thread can have up to 100 queued items. It also reduces contention on the SidecarTransport, which was indeed adding quite a bit of latency due to threads accumulating in futex_wait for very short requests under moderate load.

Moving the background sender stats to its own thread in preparation of dropping the Mutex on the SidecarTransport fully.

bwoebi and others added 4 commits March 31, 2026 20:10
Signed-off-by: Bob Weinand <[email protected]>
…ge on wordpress

Batch ack sending too

Signed-off-by: Bob Weinand <[email protected]>
The sidecar's write_shm_file() serializes buffered_integrations as
HashSet<Integration>, but the PHP-side reader in components-rs
deserialized it as HashSet<String>. This mismatch has existed since
the SHM cache was introduced in 91222ad ("feat: reduce telemetry
sent", #3316), but was latent: config_sent was set true before any
integrations appeared in SHM, and integration dedup (the only other
consumer) was silently broken with the sidecar deduplicating
server-side as a fallback.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@bwoebi bwoebi requested review from a team as code owners April 2, 2026 12:49
@bwoebi bwoebi force-pushed the bob/thread-sidecar branch from 1efe005 to e673db0 Compare April 2, 2026 12:50
@datadog-official
Copy link
Copy Markdown

datadog-official bot commented Apr 2, 2026

⚠️ Tests

Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 10 Tests failed

ext/openssl/tests/bug74796.phpt (Bug #74796: TLS encryption fails behind HTTP proxy) from php.ext.openssl.tests   View in Datadog   (Fix with Cursor)
001- string(19) "Hello from server 0"
002- NULL
003- string(19) "Hello from server 1"
004- NULL
005- string(19) "Hello from server 2"
006- NULL
007- cs.php.net
008- uk.php.net
009- us.php.net
002+ error:0A000086:SSL routines::certificate verify failed in /usr/local/src/php/ext/openssl/tests/ServerClientTestCase.inc(191) : eval()'d code on line 15
...
ext/openssl/tests/sni_server_key_cert.phpt (sni_server with separate pk and cert) from php.ext.openssl.tests   View in Datadog   (Fix with Cursor)
001- string(%d) "cs.php.net"
002- string(%d) "uk.php.net"
003- string(%d) "us.php.net"
002+ error:0A000086:SSL routines::certificate verify failed in /usr/local/src/php/ext/openssl/tests/ServerClientTestCase.inc(191) : eval()'d code on line 9
003+ 
005+ 
007+ 
009+ 
010+ Deprecated: openssl_x509_parse(): Passing null to parameter #1 ($certificate) of type OpenSSLCertificate|string is deprecated in /usr/local/src/php/ext/openssl/tests/ServerClientTestCase.inc(191) : eval()'d code on line 11
011+ 
...
ext/openssl/tests/sni_server.phpt (sni_server) from php.ext.openssl.tests   View in Datadog   (Fix with Cursor)
001- string(%d) "cs.php.net"
002- string(%d) "uk.php.net"
003- string(%d) "us.php.net"
002+ error:0A000086:SSL routines::certificate verify failed in /usr/local/src/php/ext/openssl/tests/ServerClientTestCase.inc(191) : eval()'d code on line 9
003+ 
005+ 
007+ 
009+ 
010+ Deprecated: openssl_x509_parse(): Passing null to parameter #1 ($certificate) of type OpenSSLCertificate|string is deprecated in /usr/local/src/php/ext/openssl/tests/ServerClientTestCase.inc(191) : eval()'d code on line 11
011+ 
...
View all

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 60.68% (+0.04%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 7b5db07 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!

While the sidecar has been a process global for the longest time, it's also unfair with respect to limits and not compatible with the newly introduced outbox.

Having a connection per thread ensures that every thread can have up to 100 queued items.
It also reduces contention on the SidecarTransport, which was indeed adding quite a bit of latency due to threads accumulating in futex_wait for very short requests under moderate load.

Moving the background sender stats to its own thread in preparation of dropping the Mutex on the SidecarTransport fully.

Signed-off-by: Bob Weinand <[email protected]>
@bwoebi bwoebi force-pushed the bob/thread-sidecar branch from e673db0 to 1b4b37f Compare April 2, 2026 13:24
dd_signal_data.uc = uc;

if (ddtrace_sidecar) {
if (ddtrace_sidecar_for_signal) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (ddtrace_sidecar_for_signal) {
if (atomic_load(&ddtrace_sidecar_for_signal)) {

Comment on lines 537 to 538
#endif
if (ddtrace_sidecar_active_mode == DD_SIDECAR_CONNECTION_THREAD &&
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#endif
atomic_store(&ddtrace_sidecar_for_signal, NULL);
if (ddtrace_sidecar_active_mode == DD_SIDECAR_CONNECTION_THREAD &&

Probably best to clear this here instead of later to avoid any case of use-after-free don't you think ? 🤔

Comment on lines +874 to +881
void ddtrace_sidecar_gshutdown(void) {
if (DDTRACE_G(sidecar)) {
// Drain any accumulated background-sender metrics before the transport goes away.
ddtrace_telemetry_flush_bgs_metrics_final();
ddog_sidecar_transport_drop(DDTRACE_G(sidecar));
DDTRACE_G(sidecar) = NULL;
}
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shoudn't this also clear ddtrace_sidecar_for_signal ?

@bwoebi bwoebi force-pushed the bob/batch-endpoint branch from 8ecdabc to 3503499 Compare April 2, 2026 18:36
@bwoebi bwoebi requested review from a team as code owners April 2, 2026 18:36
@bwoebi bwoebi force-pushed the bob/batch-endpoint branch 2 times, most recently from 0db877a to 83d3af1 Compare April 2, 2026 20:56
Signed-off-by: Bob Weinand <[email protected]>
@bwoebi bwoebi force-pushed the bob/thread-sidecar branch from d58b820 to 7b5db07 Compare April 2, 2026 20:59
@bwoebi bwoebi force-pushed the bob/batch-endpoint branch 2 times, most recently from 9b92ee4 to 5cf8ebb Compare April 3, 2026 16:22
Base automatically changed from bob/batch-endpoint to master April 3, 2026 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants