[REVIEW] Add OpenSearch backend to cuvs-bench#2012
[REVIEW] Add OpenSearch backend to cuvs-bench#2012jrbourbeau wants to merge 5 commits intorapidsai:mainfrom
cuvs-bench#2012Conversation
Signed-off-by: James Bourbeau <jbourbeau@nvidia.com>
| print( | ||
| f"Build failed for {config.index_name}: {build_result.error_message}" | ||
| try: | ||
| backend.initialize() |
There was a problem hiding this comment.
Note to reviewers: This is a separate, but related change to the orchestrator. I noticed the initialize and cleanup methods weren't actually being called like their docstrings said they were suppose to. This is a big diff but most of it is just intending over the existing code into a try/except block. The functional change is calling initialize + cleanup
EDIT: FWIW supporting a context manager for backend setup / teardown seems reasonable. But that's also not something I wanted to handle in this PR.
There was a problem hiding this comment.
Good catch. The initialize() and cleanup() hooks in BenchmarkBackend are defined in the abstract base class with docstrings describing their intended purpose, but neither method is actually called by the BenchmarkOrchestrator in _run_sweep() or _run_trial(). This was an oversight because the only backend implemented at the time was CppGoogleBenchmarkBackend, whose lifecycle is entirely self-contained within each subprocess call to build()/search(), making initialize()/cleanup() unnecessary for it.
There was a problem hiding this comment.
EDIT: FWIW supporting a context manager for backend setup / teardown seems reasonable. But that's also not something I wanted to handle in this PR.
I Agree that context manager support is reasonable. The purpose of the pluggable backend API is precisely to enable these kinds of extensions without modifying the core orchestrator. Since initialize() and cleanup() are already part of the BenchmarkBackend interface, wiring them up in the orchestrator (whether as explicit calls or a context manager) would ensure the plugin system works end-to-end for future backends without requiring further modifications to the core
|
|
||
| [tool.pytest.ini_options] | ||
| markers = [ | ||
| "integration: tests that require a live OpenSearch node (run with '-m integration')", |
There was a problem hiding this comment.
TODO: Make this more generic language (not just opensearch)
…ch-opensearch Signed-off-by: James Bourbeau <jbourbeau@nvidia.com>
…ch-opensearch Signed-off-by: James Bourbeau <jbourbeau@nvidia.com>
cuvs-benchcuvs-bench
Would still like to refine a bit but pushing up for early feedback
cc @singhmanas1 @janakivamaraju @afourniernv
xref #1907 which does something similar but for Elastic
EDIT: Okay, this is now ready for review (cc @jnke2016)