can you have the methods like frames2gif or frames2mpeg in here as well

@DN6 what do you think?

oops, i meant #7548

I think we can add this functionality to the export functions we have. Generally the pipelines always return an np array, a torch tensor or PIL image/ PIL image list.

I don't think so. Here's an example where tensor2vid is being used when returning the output:

let's first decide what should be video input format we accept here, I think

1. list of list of images (or a list of images, in that case, we expand it as list of list of images)
2. list of list of 4d tensors (or a list of 4d tensor, in this case, we expand)
3. list of 5d tensors (or 5d tensor, we expand)
4. list of list of 4d numpy arrays (or a list of 4d array, we expand)
5. list of 5d numpy arrays (or 5d numpy array, we expand)

cc @DN6 here, anything we would add or remove?

cc @a-r-r-o-w too since you worked on a lot of video pipelines

Okay, so I think all of those are covered. supported_formats here refer to the core base format which can be used to create lists, 4d or 5d tensors, etc.

Let me provide concrete lines of code that I think ensure all the formats listed above are supported:

1. diffusers/src/diffusers/video_processor.py

   Line 58 in 7561c1e

   if isinstance(video, list) and isinstance(video[0], list) and isinstance(video[0][0], PIL.Image.Image):

   and

2. and 3.

   diffusers/src/diffusers/video_processor.py

   Line 80 in 459843a

   video = torch.cat(video, axis=0) if video[0].ndim == 5 else torch.stack(video, axis=0)

3. and 5.

   diffusers/src/diffusers/video_processor.py

   Line 64 in 459843a

   video = np.concatenate(video, axis=0) if video[0].ndim == 5 else np.stack(video, axis=0)

Thanks for looping me in. I think that list[list[image]], list[fchw_tensor] and bfchw_tensor are most common to end up with as a user, and all cases you mentioned seem to be handled well and the code makes sense to me.

A small explanation of expected inputs to video processor in the docstrings, order of tensor dims, and some more documentation, like in image processor, would be helpful for newer users imo.

Thank you, @a-r-r-o-w! Added a bit of documentation and type annotation to hopefully make it clearer. LMK your thoughts.

a lot of the work here is overlapping with preprocess though

once we make sure the inputs are accepted as one of below (and expanded)

• list of list of images (or a list of images, in that case, we expand it as list of list of images)
• list of list of 4d tensors (or a list of 4d tensor, in this case, we expand)
• list of 5d tensors (or 5d tensor, we expand)
• list of list of 4d numpy arrays (or a list of 4d array, we expand)
• list of 5d numpy arrays (or 5d numpy array, we expand)

can we try:

video = [self.image_processor.preprocess(vid) for vid in videos]

after that we check if all the tensors has same shape in the list and throw an error if not

a lot of the work here is overlapping with preprocess though

Help me understand this a bit better? From what I understand, the current preprocess_video() is first checking if we have the inputs in the accepted format and is performing the expansion if needed. And then it's passing off the video to preprocess() like you suggested:

The only overlap I see is this:

But I see this more as a safeguard rather than an overlap.

i think something like this should work (made up code, but roughly the logic)

if isinstance(video, supported_formats):
    video = [video]
if isinstance(video[0],  PIL.Image.Image)or if isinstance(video[0], ( np.ndarray, torch.tensor) and video[0].ndim ==4):
    video = [video]

video = torch.stack([self.preprocess(f) for f in video],..)
video = video.permute(0, 2, 1, 3, 4)

How about let's first add tests like this https://github.com/huggingface/diffusers/blob/main/tests/others/test_image_processor.py
once we have the test, I can help look into refactoring this function?

Done. LMK what you think. I had to add a bit of code to deal with 5D stuff. But rest has been simplified a lot IMO.

@yiyixuxu

Technically not deprecated since we didn't support it before right? I think we just raise an error here.

it was kinda supported before

But why do we want to do this? I don't think a single-frame video would get represented properly otherwise. We don't have any special treatment for that from preprocess_video either.

why do they need video-to-video pipeline if it is single-frame? i.e. an image?

I don’t know honestly. I tried to follow the original implementation (i.e., the implementation before the refactor) as faithfully as possible.

From the documentation though, it seems like it supports multi-frame videos too:
https://huggingface.co/docs/diffusers/en/api/pipelines/animatediff

So, my best guess is that it supports both — single-frame video and multi-frame video.

maybe raise an error in check_inputs when both latents and video is not None

Already there: