Add LoRA adapter support for video generation (#3306)#3309
Add LoRA adapter support for video generation (#3306)#3309Eshaan-byte wants to merge 1 commit intoopenvinotoolkit:masterfrom
Conversation
Implements LoRA (Low-Rank Adaptation) adapter support for LTX-Video transformer model, following the same pattern as image generation. Changes: - Remove blocking assertion in ltx_video_transformer_3d_model.cpp - Add adapter initialization in compile() method - Implement set_adapters() method for runtime adapter control - Add AdapterController member to manage adapter lifecycle - Create lora_text2video.cpp sample demonstrating usage - Add comprehensive tests in test_video_generation.py The implementation allows users to: - Load and apply one or more LoRA adapters - Mix multiple adapters with different alpha values - Enable/disable adapters per generation call - Use community LoRA adapters from HuggingFace Fixes openvinotoolkit#3306
|
@Eshaan-byte Thanks for the contribution! |
|
Hi @Eshaan-byte, I've opened a PR against your fork (Eshaan-byte/pull/1) with the fixes for the CI failures. The issue was a missing Python binding for Once you merge my PR into your branch, the CI should pass. Please have a look. |
|
@likholat, I've implemented the fixes for the CI failures here. Since I need this base support for #3313, I’m happy to open a consolidated PR (with eshaan's and my commits) to unblock this if @Eshaan-byte is unavailable to merge the fixes I sent to his fork. |
This PR implements LoRA (Low-Rank Adaptation) adapter support for the LTX-Video transformer model, enabling users to apply community-created LoRA adapters to customize video generation. The implementation follows the same architecture pattern established for image generation LoRA support.
Changes Made:
Core Implementation:
ltx_video_transformer_3d_model.cppcompile()methodset_adapters()method for runtime adapter controlAdapterControllermember to manage adapter lifecycleSample & Tests:
lora_text2video.cppsample demonstrating:test_video_generation.py:User Benefits:
Fixes #3306
Checklist:
TestLoRAVideoGenerationclass with 3 test methods intests/python_tests/test_video_generation.py✓ Removed blocking assertion
✓ Wired up adapter support
✓ Added AdapterConfig support to Text2VideoPipeline
✓ Created sample demonstrating LoRA usage
✓ Added tests