Skip to content

Comments

Add LoRA adapter support for video generation (#3306)#3309

Open
Eshaan-byte wants to merge 1 commit intoopenvinotoolkit:masterfrom
Eshaan-byte:master
Open

Add LoRA adapter support for video generation (#3306)#3309
Eshaan-byte wants to merge 1 commit intoopenvinotoolkit:masterfrom
Eshaan-byte:master

Conversation

@Eshaan-byte
Copy link

@Eshaan-byte Eshaan-byte commented Feb 8, 2026

This PR implements LoRA (Low-Rank Adaptation) adapter support for the LTX-Video transformer model, enabling users to apply community-created LoRA adapters to customize video generation. The implementation follows the same architecture pattern established for image generation LoRA support.

Changes Made:

Core Implementation:

  • Removed blocking assertion that prevented adapter usage in ltx_video_transformer_3d_model.cpp
  • Added adapter initialization logic in the compile() method
  • Implemented set_adapters() method for runtime adapter control
  • Added AdapterController member to manage adapter lifecycle
  • Updated header file to include LoRA adapter dependencies

Sample & Tests:

  • Created lora_text2video.cpp sample demonstrating:
    • Loading multiple LoRA adapters with custom alpha values
    • Generating videos with adapters enabled
    • Generating baseline videos for comparison
  • Added comprehensive test suite in test_video_generation.py:
    • Test adapter support in pipeline constructor
    • Test adapter support in generate() method
    • Test transformer's set_adapters() method

User Benefits:

  • Load and apply one or more LoRA adapters simultaneously
  • Mix multiple adapters with different alpha (strength) values
  • Enable/disable adapters per generation call
  • Use community LoRA adapters from HuggingFace

Fixes #3306

Checklist:

  • This PR follows GenAI Contributing guidelines.
  • Tests have been updated or added to cover the new code.
    • Added TestLoRAVideoGeneration class with 3 test methods in tests/python_tests/test_video_generation.py
    • Tests cover: adapter constructor, adapter in generate(), and set_adapters() method
  • This PR fully addresses the ticket.
    • All requirements from LoRA for Video Generation #3306 completed:
      ✓ Removed blocking assertion
      ✓ Wired up adapter support
      ✓ Added AdapterConfig support to Text2VideoPipeline
      ✓ Created sample demonstrating LoRA usage
      ✓ Added tests
  • I have made corresponding changes to the documentation.
    • Documentation update to be added in follow-up (similar to image generation LoRA docs)

Implements LoRA (Low-Rank Adaptation) adapter support for LTX-Video
transformer model, following the same pattern as image generation.

Changes:
- Remove blocking assertion in ltx_video_transformer_3d_model.cpp
- Add adapter initialization in compile() method
- Implement set_adapters() method for runtime adapter control
- Add AdapterController member to manage adapter lifecycle
- Create lora_text2video.cpp sample demonstrating usage
- Add comprehensive tests in test_video_generation.py

The implementation allows users to:
- Load and apply one or more LoRA adapters
- Mix multiple adapters with different alpha values
- Enable/disable adapters per generation call
- Use community LoRA adapters from HuggingFace

Fixes openvinotoolkit#3306
@likholat
Copy link
Contributor

likholat commented Feb 9, 2026

@Eshaan-byte Thanks for the contribution!
Some tests are currently failing on this PR. Could you please take a look?

@goyaladitya05
Copy link
Contributor

goyaladitya05 commented Feb 14, 2026

Hi @Eshaan-byte, I've opened a PR against your fork (Eshaan-byte/pull/1) with the fixes for the CI failures.

The issue was a missing Python binding for set_adapters() and a missing null-check for the AdapterController. I'm currently working on #3313, which is blocked by this LoRA implementation, hence I wish to get this merged asap.

Once you merge my PR into your branch, the CI should pass. Please have a look.

@goyaladitya05
Copy link
Contributor

@likholat, I've implemented the fixes for the CI failures here. Since I need this base support for #3313, I’m happy to open a consolidated PR (with eshaan's and my commits) to unblock this if @Eshaan-byte is unavailable to merge the fixes I sent to his fork.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LoRA for Video Generation

3 participants