-
Notifications
You must be signed in to change notification settings - Fork 80
fix: PostgreSQL unit test hang #1083
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
83 changes: 83 additions & 0 deletions
83
...otlin/org/cloudfoundry/credhub/config/FlywayPostgresTestStripConcurrentlyConfiguration.kt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,83 @@ | ||
| package org.cloudfoundry.credhub.config | ||
|
|
||
| import org.flywaydb.core.api.ResourceProvider | ||
| import org.flywaydb.core.api.configuration.FluentConfiguration | ||
| import org.flywaydb.core.api.migration.JavaMigration | ||
| import org.flywaydb.core.api.resource.LoadableResource | ||
| import org.flywaydb.core.internal.scanner.Scanner | ||
| import org.springframework.boot.autoconfigure.flyway.FlywayConfigurationCustomizer | ||
| import org.springframework.context.annotation.Bean | ||
| import org.springframework.context.annotation.Configuration | ||
| import org.springframework.context.annotation.Profile | ||
| import org.springframework.core.Ordered | ||
| import org.springframework.core.annotation.Order | ||
| import java.io.Reader | ||
| import java.io.StringReader | ||
|
|
||
| /** | ||
| * For unit tests against PostgreSQL, strip `CONCURRENTLY` from index DDL in Flyway SQL migrations so | ||
| * migrations do not block on other connections (e.g. Flyway bookkeeping). Production migrations are | ||
| * unchanged on disk; only the in-memory script content is transformed. | ||
| * | ||
| * Every `*.sql` resource is wrapped (Flyway relative paths omit `db/migration/postgres/`); replacements are | ||
| * no-ops for scripts that do not use concurrent index DDL. | ||
| */ | ||
| @Configuration | ||
| @Profile("unit-test-postgres") | ||
| class FlywayPostgresTestStripConcurrentlyConfiguration { | ||
| @Bean | ||
| @Order(Ordered.LOWEST_PRECEDENCE) | ||
| fun stripConcurrentlyFlywayCustomizer(): FlywayConfigurationCustomizer = | ||
| FlywayConfigurationCustomizer { configuration: FluentConfiguration -> | ||
| val locations = configuration.locations | ||
| val delegateProvider: ResourceProvider = | ||
| Scanner(JavaMigration::class.java, configuration, locations) | ||
| configuration.resourceProvider( | ||
| StripConcurrentlyResourceProvider(delegateProvider), | ||
| ) | ||
| } | ||
| } | ||
|
|
||
| internal fun stripConcurrentIndexDdl(sql: String): String = | ||
| sql | ||
| .replace(CREATE_INDEX_CONCURRENTLY, "CREATE INDEX") | ||
| .replace(DROP_INDEX_CONCURRENTLY, "DROP INDEX") | ||
|
|
||
| private val CREATE_INDEX_CONCURRENTLY = Regex("(?i)CREATE\\s+INDEX\\s+CONCURRENTLY\\b") | ||
| private val DROP_INDEX_CONCURRENTLY = Regex("(?i)DROP\\s+INDEX\\s+CONCURRENTLY\\b") | ||
|
|
||
| private class StripConcurrentlyResourceProvider( | ||
| private val delegate: ResourceProvider, | ||
| ) : ResourceProvider { | ||
| override fun getResource(name: String): LoadableResource? = delegate.getResource(name)?.let { wrap(it) } | ||
|
|
||
| override fun getResources( | ||
| prefix: String, | ||
| suffixes: Array<String>, | ||
| ): Collection<LoadableResource> = delegate.getResources(prefix, suffixes).map { wrap(it) } | ||
|
|
||
| private fun wrap(resource: LoadableResource): LoadableResource = | ||
| if (resource.filename.endsWith(".sql")) { | ||
| TransformingPostgresSqlResource(resource) | ||
| } else { | ||
| resource | ||
| } | ||
| } | ||
|
|
||
| private class TransformingPostgresSqlResource( | ||
| private val delegate: LoadableResource, | ||
| ) : LoadableResource() { | ||
| override fun read(): Reader { | ||
| val original = delegate.read().use { it.readText() } | ||
| val transformed = stripConcurrentIndexDdl(original) | ||
| return StringReader(transformed) | ||
| } | ||
|
|
||
| override fun getAbsolutePath(): String = delegate.absolutePath | ||
|
|
||
| override fun getAbsolutePathOnDisk(): String = delegate.absolutePathOnDisk | ||
|
|
||
| override fun getFilename(): String = delegate.filename | ||
|
|
||
| override fun getRelativePath(): String = delegate.relativePath | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
question: curious why this needed now, is it due to changes in Flyway or recent changes to the tests?
Is this related to flyway/flyway#3508 or flyway/flyway#3682
workaround suggests applying config
flyway.postgresql.transactional.lock=falseThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see we attempt this in: https://github.com/cloudfoundry/credhub/blob/main/applications/credhub-api/src/main/java/org/cloudfoundry/credhub/config/FlywayMigrationStrategyConfiguration.java
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The unit-tests-postgres-* jobs in Coucourse have been failing frequently with timeouts in the past, too. It seems that the timeout is just more frequent (or consistent) these days. I've never bothered looked into those CI timeouts, thinking they are flakes, as rerunning usually made the jobs pass.
For the matter of local run in my dev box, I do not regularly run the tests against postgresql database unless there is something postgresql-specific that need to be looked into. So no idea when the problem started to occur there. It is possible that this was caused by some changes in other dependencies such as hikari pool or hibernate.
Not sure if our current issue is related to those old flyway issues. Sounds similar, though. And yes, we are already setting transitionalLock to false (in FlywayMigrationStrategyTestConfig for tests).
As the goal is to prevent the hang for the tests and the issue is not to be expected to occur in production, and the change only affect test run with postgresql database, I think that just removing concurrently from the create/drop index statements for test only is the way to go.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, the latest issue: flyway/flyway#3961
Seems like there's no other solutions yet.