Skip to content

Commit cd61869

Browse files
kadirnarkadirnar
authored
[Community] Support StableDiffusionCanvasPipeline (huggingface#3590)
* added StableDiffusionCanvasPipeline pipeline * Added utils codes to pipe_utils file. * make style * delete mixture.py and Text2ImageRegion class * make style * Added the codes to the readme.md file. * Moved functions from pipeline_utils to mix_canvas
1 parent 803d653 commit cd61869

3 files changed

Lines changed: 539 additions & 403 deletions

File tree

‎examples/community/README.md‎

Lines changed: 36 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1601,7 +1601,7 @@ pipe_images = mixing_pipeline(
16011601

16021602
![image_mixing_result](https://huggingface.co/datasets/TheDenk/images_mixing/resolve/main/boromir_gigachad.png)
16031603

1604-
### Stable Diffusion Mixture
1604+
### Stable Diffusion Mixture Tiling
16051605

16061606
This pipeline uses the Mixture. Refer to the [Mixture](https://arxiv.org/abs/2302.02412) paper for more details.
16071607

@@ -1672,4 +1672,38 @@ mask_image = Image.open(BytesIO(response.content)).convert("RGB")
16721672
prompt = "a mecha robot sitting on a bench"
16731673
image = pipe(prompt, image=input_image, mask_image=mask_image, strength=0.75,).images[0]
16741674
image.save('tensorrt_inpaint_mecha_robot.png')
1675-
```
1675+
```
1676+
1677+
### Stable Diffusion Mixture Canvas
1678+
1679+
This pipeline uses the Mixture. Refer to the [Mixture](https://arxiv.org/abs/2302.02412) paper for more details.
1680+
1681+
```python
1682+
from PIL import Image
1683+
from diffusers import LMSDiscreteScheduler, DiffusionPipeline
1684+
from diffusers.pipelines.pipeline_utils import Image2ImageRegion, Text2ImageRegion, preprocess_image
1685+
1686+
1687+
# Load and preprocess guide image
1688+
iic_image = preprocess_image(Image.open("input_image.png").convert("RGB"))
1689+
1690+
# Creater scheduler and model (similar to StableDiffusionPipeline)
1691+
scheduler = LMSDiscreteScheduler(beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear", num_train_timesteps=1000)
1692+
pipeline = DiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", scheduler=scheduler).to("cuda:0", custom_pipeline="mixture_canvas")
1693+
pipeline.to("cuda")
1694+
1695+
# Mixture of Diffusers generation
1696+
output = pipeline(
1697+
canvas_height=800,
1698+
canvas_width=352,
1699+
regions=[
1700+
Text2ImageRegion(0, 800, 0, 352, guidance_scale=8,
1701+
prompt=f"best quality, masterpiece, WLOP, sakimichan, art contest winner on pixiv, 8K, intricate details, wet effects, rain drops, ethereal, mysterious, futuristic, UHD, HDR, cinematic lighting, in a beautiful forest, rainy day, award winning, trending on artstation, beautiful confident cheerful young woman, wearing a futuristic sleeveless dress, ultra beautiful detailed eyes, hyper-detailed face, complex, perfect, model,  textured, chiaroscuro, professional make-up, realistic, figure in frame, "),
1702+
Image2ImageRegion(352-800, 352, 0, 352, reference_image=iic_image, strength=1.0),
1703+
],
1704+
num_inference_steps=100,
1705+
seed=5525475061,
1706+
)["images"][0]
1707+
```
1708+
![Input_Image](https://huggingface.co/datasets/kadirnar/diffusers_readme_images/resolve/main/input_image.png)
1709+
![mixture_canvas_results](https://huggingface.co/datasets/kadirnar/diffusers_readme_images/resolve/main/canvas.png)

0 commit comments

Comments
 (0)