utils: Use asyncio.gather with fail-fast behaviour#511
Open
Conversation
bbhtt
force-pushed
the
bbhtt/failfast
branch
2 times, most recently
from
57ec589 to
50eee74
Compare
|
coverage: 90.747% (-0.04%) from 90.785% — bbhtt/failfast into master |
asyncio.gather does not cancel sibling tasks when one fails, so if one checker raises an unhandled exception, other tasks running I/O may keep the event loop alive indefinitely. So use asyncio_gather_failfast to detect failures and explicitly cancel+drain all pending tasks before re-raising the exception. In Python 3.11+ this is offered by TaskGroup but it is cheap to use a wrapper here. See python/cpython#75633 Fixes: #420
Contributor
Author
|
This seems correct to me but I didn't manage to reproduce the hang with 1a0e0c7 reverted and a broken manifest locally. I think it requires some running / semi-stuck network I/O and I can't get that. So it's a bit theoretical. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
asyncio.gather does not cancel sibling tasks when one fails, so if one
checker raises an unhandled exception, other tasks running I/O may keep
the event loop alive indefinitely. So use asyncio_gather_failfast to
detect failures and explicitly cancel+drain all pending tasks before
re-raising the exception.
In Python 3.11+ this is offered by TaskGroup but it is cheap to
use a wrapper here.
See python/cpython#75633
Fixes: #420