feature: domain specific task names. added sam-3, flux-2. Moved some runway models to replicate provider by JoeGaffney · Pull Request #46 · JoeGaffney/deferred-diffusion

JoeGaffney · 2025-11-26T21:01:33Z

No description provided.

…nd pixel size

…rt from act-two wich is only available from runway directly

Copilot

Pull request overview

This PR implements a comprehensive update to the model infrastructure, introducing new models, standardizing task naming conventions, and improving provider integrations. The changes include migration of Runway models to Replicate, addition of new segmentation and image generation models, system prompt enhancements, and various video processing improvements.

Key Changes:

Standardized Celery task naming with domain prefixes (images., videos., texts.)
Added new models: FLUX 2, SAM 2, and SAM 3 for image generation and segmentation
Migrated runway-gen-4 and runway-upscale from Runway API to Replicate
Introduced NONE system prompt option for text models with empty default behavior
Enhanced video processing with compression, better duration handling, and resolution-aware logic

Reviewed changes

Copilot reviewed 35 out of 35 changed files in this pull request and generated 17 comments.

Show a summary per file

File	Description
workers/images/tasks.py	Added FLUX 2, SAM 2/3 tasks; standardized task naming with `images.` prefix
workers/images/schemas.py	Updated model registry with FLUX 2, SAM 2/3 definitions; added task_name property
workers/images/local/flux_2.py	New FLUX 2.0 implementation with multi-image support and inpainting
workers/images/local/sam_2.py	Renamed from segment-anything-2; improved mask coloring logic
workers/images/local/sam_3.py	New SAM 3 implementation with text-based segmentation
workers/videos/tasks.py	Standardized task naming with `videos.` prefix
workers/videos/schemas.py	Updated runway model providers to replicate; added task_name property
workers/videos/context.py	Added video compression, resolution detection, duration calculation methods
workers/videos/external/runway_gen_4.py	Refactored to use Replicate API with aspect ratio mapping
workers/videos/external/runway_upscale.py	Migrated from Runway SDK to Replicate helpers
workers/videos/external/runway_act_two.py	Minor video compression integration
workers/videos/external/openai_sora_2.py	Updated duration logic from long_video() to duration_in_seconds()
workers/videos/external/minimax_hailuo_2.py	Updated duration and resolution logic
workers/videos/external/kwaivgi_kling_2.py	Updated duration logic to use duration_in_seconds()
workers/videos/external/google_veo_3.py	Refactored duration and audio settings
workers/videos/external/bytedance_seedance_1.py	Updated duration logic and added resolution parameter
workers/videos/local/ltx_video.py	Improved frame count handling with min() logic
workers/texts/tasks.py	Standardized task naming with `texts.` prefix
workers/texts/schemas.py	Added NONE system prompt; condensed optimizer prompts
workers/texts/local/qwen_2.py	Made system prompt optional with empty check
workers/texts/external/openai_gpt.py	Added conditional system prompt handling
workers/utils/utils.py	Added mp4_to_base64_decoded; updated load functions with model parameter; removed unused imports; increased max resolution to 4096
workers/common/pipeline_helpers.py	Added offload parameter to get_quantized_model; improved optimize_pipeline with conditional safety checker; added documentation
workers/tests/images/local/test_*.py	Renamed and added tests for SAM 2/3 and FLUX 2
workers/tests/images/helpers.py	Updated model routing for renamed and new models
workers/requirements.txt	Updated diffusers commit hash; added sam3 package
api/images/schemas.py	Mirrored worker schema updates for FLUX 2, SAM 2/3
api/videos/schemas.py	Mirrored worker schema updates for provider changes
api/texts/schemas.py	Mirrored worker schema updates for NONE prompt and condensed text
clients/openapi.json	Updated OpenAPI spec with new models and system prompts
clients/nuke/gizmos/dd_image.gizmo	Updated model dropdown with SAM 2/3 replacements
README.md	Updated documentation references from models/ to local/ and tasks/

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-26T21:09:18Z

workers/images/tasks.py

-    from images.local.segment_anything_2 import main
+@typed_task(name="sam-2", queue="gpu")
+def sam_2_image(request_dict):
+    from images.local.sam_2_image import main


The import statement references images.local.sam_2_image, but the actual file is named sam_2.py. This will cause an ImportError at runtime. The import should be:

from images.local.sam_2 import main

Suggested change

from images.local.sam_2_image import main

from images.local.sam_2 import main

Copilot · 2025-11-26T21:09:18Z

workers/images/local/flux_2.py

+        model_id, text_encoder=text_encoder, transformer=transformer, torch_dtype=torch.bfloat16
+    )
+
+    # allways offload as heavy models


Typo in comment: "allways" should be "always".

Suggested change

# allways offload as heavy models

# always offload as heavy models

Copilot · 2025-11-26T21:09:19Z

workers/videos/context.py


    def get_flow_shift(self) -> float:
-        return 5.0 if self.get_is_720p() else 3.0
+        return 5.0 if self.get_resolution_type() == "720p" else 3.0


[nitpick] The logic for get_flow_shift() only checks for 720p and defaults to 3.0 for both 480p and 1080p. If 1080p should have a different flow shift value than 480p, consider adding explicit handling:

def get_flow_shift(self) -> float: resolution = self.get_resolution_type() if resolution == "720p": return 5.0 elif resolution == "1080p": return 3.0 # or appropriate value for 1080p return 3.0 # 480p

Suggested change

return 5.0 if self.get_resolution_type() == "720p" else 3.0

resolution = self.get_resolution_type()

if resolution == "720p":

return 5.0

elif resolution == "1080p":

return 3.0 # or set to a different value if needed

return 3.0 # 480p

workers/videos/context.py

api/texts/schemas.py

Copilot · 2025-11-26T21:09:22Z

workers/common/pipeline_helpers.py

 from accelerate.hooks import CpuOffload
 from cachetools.keys import hashkey
-from diffusers import GGUFQuantizationConfig
+from diffusers import DiffusionPipeline, GGUFQuantizationConfig


Import of 'DiffusionPipeline' is not used.

Suggested change

from diffusers import DiffusionPipeline, GGUFQuantizationConfig

from diffusers import GGUFQuantizationConfig

Copilot · 2025-11-26T21:09:22Z

workers/images/local/sam_3.py

@@ -0,0 +1,119 @@
+import os


Import of 'os' is not used.

Suggested change

import os

Copilot · 2025-11-26T21:09:22Z

workers/images/local/sam_3.py

+import os
+import random
+
+import numpy as np


Import of 'np' is not used.

Suggested change

import numpy as np

Copilot · 2025-11-26T21:09:23Z

workers/tests/images/local/test_flux_2.py

@@ -0,0 +1,70 @@
+from typing import List, Literal


Import of 'Literal' is not used.

Suggested change

from typing import List, Literal

from typing import List

Copilot · 2025-11-26T21:09:23Z

workers/common/pipeline_helpers.py

-    if disable_safety_checker:
+        try:
+            pipe.vae.enable_slicing()  # Enable VAE slicing to reduce memory usage
+        except:


Except block directly handles BaseException.

Suggested change

except:

except Exception:

Copilot

Pull request overview

Copilot reviewed 36 out of 36 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (1)

workers/texts/external/openai_gpt.py:4

Import of 'Omit' is not used.

from openai import Omit, OpenAI

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-27T10:07:04Z

workers/videos/context.py

+        if self.width >= 1280 or self.height >= 1280:
+            return True
+        return False
+
+    def is_1080p_or_higher(self) -> bool:
+        if self.width >= 1920 or self.height >= 1920:


The logic for determining 720p or higher resolution may be incorrect. Using or means a 1280x480 video would be considered 720p+ because width >= 1280, even though the total pixel count is much less than 720p (1280x720 = 921,600 pixels). Consider using and instead: if self.width >= 1280 and self.height >= 720: to ensure both dimensions meet the threshold.

Suggested change

if self.width >= 1280 or self.height >= 1280:

return True

return False

def is_1080p_or_higher(self) -> bool:

if self.width >= 1920 or self.height >= 1920:

if self.width >= 1280 and self.height >= 720:

return True

return False

def is_1080p_or_higher(self) -> bool:

if self.width >= 1920 and self.height >= 1080:

Copilot · 2025-11-27T10:07:05Z

workers/videos/context.py

+        if self.width >= 1280 or self.height >= 1280:
+            return True
+        return False
+
+    def is_1080p_or_higher(self) -> bool:
+        if self.width >= 1920 or self.height >= 1920:


Similar to the 720p check, the logic for determining 1080p or higher resolution may be incorrect. Using or means a 1920x480 video would be considered 1080p+ even though it doesn't meet the 1080p standard (1920x1080). Consider using and instead: if self.width >= 1920 and self.height >= 1080: to ensure both dimensions meet the threshold.

Suggested change

if self.width >= 1280 or self.height >= 1280:

return True

return False

def is_1080p_or_higher(self) -> bool:

if self.width >= 1920 or self.height >= 1920:

if self.width >= 1280 and self.height >= 720:

return True

return False

def is_1080p_or_higher(self) -> bool:

if self.width >= 1920 and self.height >= 1080:

Copilot · 2025-11-27T10:07:05Z

workers/texts/external/openai_gpt.py

 from typing import Any, Dict

-from openai import OpenAI
+from openai import Omit, OpenAI


The Omit import from openai is added but never used in this file. Consider removing it to keep the imports clean.

Suggested change

from openai import Omit, OpenAI

from openai import OpenAI

Copilot · 2025-11-27T10:07:05Z

workers/videos/context.py

+        path = export_to_video(self.video_frames, output_video_path=tmp_path, fps=fps, quality=9)
+        logger.info(f"Compressed video saved at {path}")
+
+        # convert and check size once more abort if still too large


Typo in comment: "abort" should be "and abort" or the comment should read "convert and check size once more; abort if still too large".

Suggested change

# convert and check size once more abort if still too large

# convert and check size once more; abort if still too large

JoeGaffney added 8 commits November 22, 2025 19:34

wip: adding sam-3-image

c30dc8f

feature: namespaceing the celery tasks by domain

51db0ef

feature: cleaning up the videos context and models for video length a…

2a9efd7

…nd pixel size

fix: capping at schema level

ed0ecac

feature: function to compress large videos

185df78

feature: moving the most of the runway models to replicate calls. Apa…

4492fcf

…rt from act-two wich is only available from runway directly

feature: allowing none for base test system prompt

c4f9ec2

feature: added flux-2

ea1230f

JoeGaffney requested a review from Copilot November 26, 2025 21:02

Copilot started reviewing on behalf of JoeGaffney November 26, 2025 21:03 View session

Copilot finished reviewing on behalf of JoeGaffney November 26, 2025 21:07

Copilot AI reviewed Nov 26, 2025

View reviewed changes

fix: simplifying resolution approx

ae8e94e

JoeGaffney requested a review from Copilot November 27, 2025 10:01

Copilot started reviewing on behalf of JoeGaffney November 27, 2025 10:01 View session

JoeGaffney changed the title ~~2025 11 22~~ feature: domain specific task names. added sam-3, flux-2. Moved some runway models to replicate provider Nov 27, 2025

Copilot finished reviewing on behalf of JoeGaffney November 27, 2025 10:04

Copilot AI reviewed Nov 27, 2025

View reviewed changes

fix: mypy issues

c3b0f28

JoeGaffney merged commit 71b7817 into main Nov 27, 2025
1 check passed

JoeGaffney deleted the 2025-11-22 branch November 28, 2025 16:10

	from images.local.sam_2_image import main
	from images.local.sam_2 import main

	# allways offload as heavy models
	# always offload as heavy models

-        return 5.0 if self.get_resolution_type() == "720p" else 3.0
+        resolution = self.get_resolution_type()
+        if resolution == "720p":
+            return 5.0
+        elif resolution == "1080p":
+            return 3.0  # or set to a different value if needed
+        return 3.0  # 480p

	from diffusers import DiffusionPipeline, GGUFQuantizationConfig
	from diffusers import GGUFQuantizationConfig

	# convert and check size once more abort if still too large
	# convert and check size once more; abort if still too large

Conversation

JoeGaffney commented Nov 26, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants