Address Semaphore leak in SQL trigger renew leases sync#1221
Open
Address Semaphore leak in SQL trigger renew leases sync#1221
Conversation
Charles-Gagnon
requested changes
Apr 8, 2026
Address review feedback: add a catch block around the outer try/finally in RenewLeasesAsync to log exceptions thrown by BeginTransaction (or other code outside the inner try/catch). Previously, if BeginTransaction failed (e.g., broken connection), the exception would propagate unlogged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Charles-Gagnon
approved these changes
Apr 15, 2026
chlafreniere
approved these changes
Apr 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Problem
When a SQL trigger's lease renewal connection breaks (transient SQL availability issue),
RenewLeasesAsynccan permanently leak the_rowsToProcessLocksemaphore.BeginTransaction()throws after the semaphore is acquired, butRelease()is not in afinallyblock — so it's never called. Both the change consumption loop and lease renewal loop then deadlock permanently on the next semaphore acquisition. The trigger silently stops processing with no error logs, while the host appears healthy.Customer impact: Trigger on table
ReceiptImportwas dead for ~3.5 days (ICM 51000000966606). Only a host restart resolved it.Root Cause
In
RenewLeasesAsync,_rowsToProcessLock.Release()was placed after theifblock rather than in afinally— meaning any exception betweenWaitAsyncandRelease(e.g.,BeginTransactionthrowing on a broken connection) permanently leaks the semaphore.The same pattern existed in 4 other sites:
GetTableChangesAsync(2 sites),ProcessTableChangesAsync, andClearRowsAsync. OnlyProcessChangescorrectly usedtry/finally.Fix
Wrap all
_rowsToProcessLockacquire/release pairs intry/finally:RenewLeasesAsync— primary fix (the site that caused the customer incident)GetTableChangesAsync— commit path and error-handling pathProcessTableChangesAsync— success pathClearRowsAsync— state resetNo behavioral changes — just ensures the semaphore is always released.
In addition, go through the checklist below and check each item as you validate it is either handled or not applicable to this change.
Code Changes
az_func.GlobalStatetable must be compatible with all prior versions of the extensionILoggerinstance to log relevant information, especially information useful for debugging or troubleshootingasyncandawaitfor all long-running operationsCancellationTokenDependencies
dotnet restore --force-evaluateto update the lock files and ensure that there are NO major versions updates in either src/packages.lock.json or Worker.Extensions.Sql/src/packages.lock.json. If there are, contact the dev team for instructions.Documentation