Fix clip_skip AttributeError on Stable Diffusion pipelines with transformers>=5.6#14043
Open
Sunt-ing wants to merge 1 commit into
Open
Fix clip_skip AttributeError on Stable Diffusion pipelines with transformers>=5.6#14043Sunt-ing wants to merge 1 commit into
Sunt-ing wants to merge 1 commit into
Conversation
…formers>=5.6 transformers 5.6 flattened CLIPTextModel (huggingface/transformers#46285): embeddings/encoder/final_layer_norm became direct submodules and the text_model wrapper was removed. The clip_skip branch of encode_prompt re-applies the final LayerNorm via self.text_encoder.text_model.final_layer_norm, so any SD1.x-family pipeline call with clip_skip set raised AttributeError. Guard the access with the same hasattr check already merged for from_single_file in huggingface#13843, propagated via make fix-copies and applied to the hand-written siblings. SDXL is unaffected (uses hidden_states[-(clip_skip+2)] and a CLIPTextModelWithProjection encoder). Signed-off-by: Ting Sun <suntcrick@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Passing
clip_skipto any SD1.x-family pipeline crashes on transformers>=5.6:transformers 5.6 flattened
CLIPTextModel(huggingface/transformers#46285):embeddings/encoder/final_layer_normbecame direct submodules and thetext_modelwrapper was removed (CLIPTextModelWithProjectionstill wraps viatext_model). Theclip_skipbranch ofencode_promptre-applies the final LayerNorm by hand viaself.text_encoder.text_model.final_layer_norm(...), so it raises as soon asclip_skipis set. The defaultclip_skip=Nonepath is unaffected because it useslast_hidden_state(already normalized) and never touches.text_model. diffusers declares no upper bound on transformers and is actively migrating to 5.x, so this is a live defect rather than an out-of-range version.This reuses the exact guard already merged for the
from_single_filepath in #13843:StableDiffusionPipeline.encode_promptis the source;make fix-copiespropagates it to 39# Copied fromconsumers, and 6 hand-written siblings (alt_diffusion x2, animatediff video2video x2, i2vgen_xl, ledits_pp) are updated to match, 46 files in total. SDXL is not affected: itsencode_promptreadshidden_states[-(clip_skip + 2)]and never callsfinal_layer_norm, andtext_encoder_2is aCLIPTextModelWithProjection(still has.text_model).Reproduction (CPU, no GPU; transformers 5.12.1)
Confirmed identically on real weights (
stabilityai/sd-turbo, aCLIPTextModel):clip_skip=Nonefine,clip_skip=1raises on main and works with this PR. The crash is inencode_prompt, before the denoise loop, so it is weight-independent and CPU-reproducible.Tests
Added
test_stable_diffusion_clip_skiptoStableDiffusionPipelineFastTests: it asserts aclip_skip=1call runs and that its output differs fromclip_skip=None. The test errors on main (the AttributeError) and passes with this PR.ruff(0.9.10),check_copies, andcheck_dummiesare clean on the changed files.Before submitting
.ai/review-rules.md?Who can review?
@asomoza @DN6