Skip to content

Address Semaphore leak in SQL trigger renew leases sync#1221

Open
VasuBhog wants to merge 2 commits intomainfrom
dev/vabhog/semaphoreleak
Open

Address Semaphore leak in SQL trigger renew leases sync#1221
VasuBhog wants to merge 2 commits intomainfrom
dev/vabhog/semaphoreleak

Conversation

@VasuBhog
Copy link
Copy Markdown
Contributor

@VasuBhog VasuBhog commented Apr 7, 2026

Description

Problem

When a SQL trigger's lease renewal connection breaks (transient SQL availability issue), RenewLeasesAsync can permanently leak the _rowsToProcessLock semaphore. BeginTransaction() throws after the semaphore is acquired, but Release() is not in a finally block — so it's never called. Both the change consumption loop and lease renewal loop then deadlock permanently on the next semaphore acquisition. The trigger silently stops processing with no error logs, while the host appears healthy.

Customer impact: Trigger on table ReceiptImport was dead for ~3.5 days (ICM 51000000966606). Only a host restart resolved it.

Root Cause

In RenewLeasesAsync, _rowsToProcessLock.Release() was placed after the if block rather than in a finally — meaning any exception between WaitAsync and Release (e.g., BeginTransaction throwing on a broken connection) permanently leaks the semaphore.

The same pattern existed in 4 other sites: GetTableChangesAsync (2 sites), ProcessTableChangesAsync, and ClearRowsAsync. Only ProcessChanges correctly used try/finally.

Fix

Wrap all _rowsToProcessLock acquire/release pairs in try/finally:

  • RenewLeasesAsyncprimary fix (the site that caused the customer incident)
  • GetTableChangesAsync — commit path and error-handling path
  • ProcessTableChangesAsync — success path
  • ClearRowsAsync — state reset

No behavioral changes — just ensures the semaphore is always released.

In addition, go through the checklist below and check each item as you validate it is either handled or not applicable to this change.

Code Changes

  • Unit tests are added, if possible
  • Integration tests are added if the change is modifying existing behavior of one or more of the bindings
  • New or changed code follows the C# style guidelines defined in .editorconfig
  • All changes MUST be backwards compatible and changes to the shared az_func.GlobalState table must be compatible with all prior versions of the extension
  • Use the ILogger instance to log relevant information, especially information useful for debugging or troubleshooting
  • Use async and await for all long-running operations
  • Ensure proper usage and propagation of CancellationToken
  • T-SQL is safe from SQL Injection attacks through the use of SqlParameters and proper escaping/sanitization of input

Dependencies

Documentation

  • Add samples if the change is modifying or adding functionality
  • Update relevant documentation in the docs

@VasuBhog VasuBhog marked this pull request as ready for review April 7, 2026 22:35
Comment thread src/TriggerBinding/SqlTableChangeMonitor.cs
Address review feedback: add a catch block around the outer try/finally
in RenewLeasesAsync to log exceptions thrown by BeginTransaction (or
other code outside the inner try/catch). Previously, if BeginTransaction
failed (e.g., broken connection), the exception would propagate unlogged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@VasuBhog VasuBhog requested a review from Charles-Gagnon April 15, 2026 22:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants