[GenAI] Fix CI and stability for LTX-Video LoRA support (Supersedes #3309)#3362
[GenAI] Fix CI and stability for LTX-Video LoRA support (Supersedes #3309)#3362goyaladitya05 wants to merge 4 commits intoopenvinotoolkit:masterfrom
Conversation
ab57590 to
637299a
Compare
There was a problem hiding this comment.
Pull request overview
This PR fixes CI failures from #3309 by adding the missing Python binding for LTXVideoTransformer3DModel.set_adapters() and implementing a safety check to prevent segmentation faults when the AdapterController is not initialized. The changes enable LoRA adapter support for LTX-Video generation, following the same architectural pattern used in image generation models.
Changes:
- Added Python binding for
set_adapters()method on the LTXVideoTransformer3DModel - Implemented safety check in C++ to prevent crashes when calling
set_adapters()on models compiled without adapter support - Added comprehensive tests for LoRA functionality in the video generation pipeline
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/cpp/src/video_generation/models/ltx_video_transformer_3d_model.cpp | Removed blocking assertion for adapters, added adapter initialization in compile(), implemented set_adapters() with safety check |
| src/cpp/include/openvino/genai/video_generation/ltx_video_transformer_3d_model.hpp | Added lora_adapter.hpp include, declared set_adapters() method, added AdapterController member |
| src/python/py_video_generation_models.cpp | Added Python binding for set_adapters() method with proper documentation |
| tests/python_tests/test_video_generation.py | Added TestLoRAVideoGeneration class with tests for adapter functionality in constructor, generate(), and set_adapters() |
| samples/cpp/video_generation/lora_text2video.cpp | New sample demonstrating LoRA usage with video generation, following the pattern from lora_text2image.cpp |
| samples/cpp/video_generation/CMakeLists.txt | Added build configuration for lora_text2video sample, consistent with image generation samples |
Comments suppressed due to low confidence (1)
tests/python_tests/test_video_generation.py:270
- The test
test_generate_without_cfg_after_reshape_with_cfg(lines 256-270) doesn't use any LoRA adapters functionality, so it seems misplaced in theTestLoRAVideoGenerationclass. This test appears to be testing CFG (Classifier-Free Guidance) behavior, not LoRA functionality. Consider moving it to a more appropriate test class (e.g.,TestText2VideoPipelineAdvanced) or creating a separateTestCFGBehaviorclass for CFG-related tests.
def test_generate_without_cfg_after_reshape_with_cfg(self, video_generation_model):
"""Test: reshape with CFG then generate without CFG should raise error."""
pipe = ov_genai.Text2VideoPipeline(video_generation_model)
pipe.reshape(1, 9, 32, 32, 3.0)
pipe.compile("CPU")
with pytest.raises(RuntimeError, match="guidance_scale <= 1 requested, but the compiled model expects CFG"):
pipe.generate(
"test prompt",
guidance_scale=1.0,
height=32,
width=32,
num_frames=9,
num_inference_steps=2,
)
src/cpp/src/video_generation/models/ltx_video_transformer_3d_model.cpp
Outdated
Show resolved
Hide resolved
Implements LoRA (Low-Rank Adaptation) adapter support for LTX-Video transformer model, following the same pattern as image generation. Changes: - Remove blocking assertion in ltx_video_transformer_3d_model.cpp - Add adapter initialization in compile() method - Implement set_adapters() method for runtime adapter control - Add AdapterController member to manage adapter lifecycle - Create lora_text2video.cpp sample demonstrating usage - Add comprehensive tests in test_video_generation.py The implementation allows users to: - Load and apply one or more LoRA adapters - Mix multiple adapters with different alpha values - Enable/disable adapters per generation call - Use community LoRA adapters from HuggingFace Fixes openvinotoolkit#3306
- Added Python binding for LTXVideoTransformer3DModel.set_adapters() to expose the method to Python, fixing test_transformer_has_set_adapters_method - Added safety check in set_adapters() to prevent crashes when AdapterController is not initialized - Both fixes follow the same patterns used in image generation models Fixes the CI failures in TestLoRAVideoGeneration test suite.
637299a to
7c1d319
Compare
|
Thanks for the feedback. |
likholat
left a comment
There was a problem hiding this comment.
Could you please also attach two short videos (converted to GIF) showing samples generated with and without LoRA adapter? Please add them to the PR description
4270b3e to
ed27566
Compare
|
Thanks for the feedback. I've updated the PR to address the review comments.
Samples & Documentation
Unit Tests
|
ed27566 to
90c69bb
Compare
90c69bb to
6830c2e
Compare
@likholat , I've added the samples to the PR description. Please review whenever you have the time. |
This PR builds upon the work started by @Eshaan-byte in #3309. I've integrated his logic and added the fixes for the set_adapters bindings and AdapterController initialization to resolve the CI issues.
Changes:
LTXVideoTransformer3DModel.set_adapters()to resolveAttributeErrorin theTestLoRAVideoGenerationtest suite.set_adapters()to prevent segmentation faults when theAdapterControlleris not initialized (e.g., when no adapters are provided at construction).Closes #3306
Supersedes #3309
Verification Results:
without LoRA

with LoRA (Adapter: ltxv_095_cakeify_lora.safetensors)

These samples demonstrate that the LoRA adapter is successfully loaded and applied during the denoising process in the Text2VideoPipeline.
Environment:
Model: LTX-Video
Alpha: 1.5
Prompt:"A cute golden retriever puppy running in a green grassy field on a sunny day, high quality, photorealistic"
Checklist:
tests/python_tests/test_video_generation.pypasses locally.