Describe the bug

Seems like LoRA unloading stopped working after the commit ebfe343.

Reproduction

Tested this with a Kohya-SS style LoRA

pipe = StableDiffusionPipeline.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    safety_checker=None,
)
pipe.load_lora_weights(...)
pipe.unload_lora_weights()
pipe(...)

Logs

No response

System Info

Running on the latest version of diffusers. Bisected it to PR #4259. Looking into the cause now 👀

Who can help?

@patrickvonplaten, @sayakpaul, and @williamberman