The documentation is not available anymore as the PR was closed or merged.

Hi @anton-l, I did a quick try of this PR on DMLExecutionProvider. It works if the safetychecker is commented out as it fails for clip_input error, similar to the error on CPUExecutionProvider.

There are few places where a provider is not passed, so I had to make minor changes to make this PR work.

1. Missing provider=provider
   

   diffusers/src/diffusers/onnx_utils.py

   Line 149 in 43d5532

   model = OnnxModel.load_model(os.path.join(model_id, model_file_name))

2. Missing provider in loading_kwargs dictionary.
   

   diffusers/src/diffusers/pipeline_utils.py

   Line 260 in 43d5532

   loading_kwargs = {}

   
   I added the provider provider = kwargs.pop("provider", None) on line 168, and set the provider in loading_kwargs after the declaration statement on line 260.
   

   diffusers/src/diffusers/pipeline_utils.py

   Line 168 in 43d5532

   torch_dtype = kwargs.pop("torch_dtype", None)

If anyone wants to test this with DmlExecutionProvider on Windows (works on AMD/NVIDIA cards), here are the steps.

conda create --name sd39 python=3.9 -y && conda activate sd39
pip install transformers ftfy scipy onnxruntime
cd diffusers/ && git checkout sd-onnx-pipeline && pip install -e .
# Download nightly onnxruntime-directml from the link below
# https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/ort-nightly-directml/overview/1.13.0.dev20220908001
pip install ort_nightly_directml-1.13.0.dev20220830001-cp39-cp39-win_amd64.whl --force-reinstall

from diffusers import StableDiffusionOnnxPipeline
pipe = StableDiffusionOnnxPipeline.from_pretrained("./sd_onnx", provider="DmlExecutionProvider")
prompt = "a photo of an astronaut riding a horse on mars"
image = pipe(prompt).images[0] 
image.save("astronaut_rides_horse.png")

Very cool - feel free to merge when you want 🥳

@pcuenca would be keen to know what inference speed numbers you get on GPU.
On a T4, I'm actually getting slower inference with Onnx (about 40 seconds), instead of 12 with vanilla pytorch.

cc @anton-l could you verify this real quick?

@tdeboissiere to enable the GPU runtime, you'll need to install one more package on top:
pip install onnxruntime-gpu

Then CUDAExecutionProvider should run at expected speeds:

pipe = StableDiffusionOnnxPipeline.from_pretrained("./sd_onnx", provider="CUDAExecutionProvider")

Yep, that's what I did.

I also used this quick hack to ensure SD was executing on GPU

# in import_utils.py
_onnx_available = importlib.util.find_spec("onnxruntime") is not None
try:
    _onnxruntime_version = importlib_metadata.version("onnxruntime")
    logger.debug(f"Successfully imported onnxruntime version {_onnxruntime_version}")
except importlib_metadata.PackageNotFoundError:
    _onnxruntime_version = importlib_metadata.version("onnxruntime-gpu")
    _onnx_available = True
except Exception:
    _onnx_available = False

I've also found that installing both onnxruntime at the same time onnxruntime-gpu led to an error ('CUDAExecutionProvider' is not in available provider names.Available providers: 'CPUExecutionProvider'). Installing onnxruntime first, then onnxruntime-gpu works though

Is someone planning on working on an onnx img2img pipeline? That would be awesome

I'm also getting much slower inference speed with Onnx on a 3090.

cc @anton-l