refactor video processor (part # 7776) - #7861
Merged
Merged
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
yiyixuxu
commented
May 5, 2024
yiyixuxu
commented
May 5, 2024
yiyixuxu
commented
May 5, 2024
sayakpaul
reviewed
May 6, 2024
sayakpaul
reviewed
May 6, 2024
Comment on lines
+41
to
+54
| def is_valid_image(image): | ||
| return isinstance(image, PIL.Image.Image) or isinstance(image, (np.ndarray, torch.Tensor)) and image.ndim in (2, 3) | ||
|
|
||
|
|
||
| def is_valid_image_imagelist(images): | ||
| # check if the image input is one of the supported formats for image and image list: | ||
| # it can be either (1) a 4d pytorch tensor or numpy array, (2) a valid image or (3) list of valid image | ||
| if isinstance(images, (np.ndarray, torch.Tensor)) and images.ndim == 4: | ||
| return True | ||
| elif is_valid_image(images): | ||
| return True | ||
| elif isinstance(images, list): | ||
| return all(is_valid_image(image) for image in images) | ||
| return False |
sayakpaul
reviewed
May 6, 2024
2 tasks
Collaborator
Agree that list of 5D is not needed. |
DN6
reviewed
May 6, 2024
| elif isinstance(video, list) and isinstance(video[0], PIL.Image.Image): | ||
| # video processor only accepts video or a list of videos or a batch of videos (5d array/tensors) as inputs, | ||
| # while we do accept a list of 5d array/tensors, we concatenate them to a single video batch | ||
| if isinstance(video, list) and isinstance(video[0], np.ndarray) and video[0].ndim == 5: |
Collaborator
There was a problem hiding this comment.
Think it's okay to not accept a list of 5D video. Perhaps just raise an error if a 5D list is passed here with a message asking to concatenate along the batch dimension?
Collaborator
Author
There was a problem hiding this comment.
I throw a warning and deprecated it just to be more safe
Member
|
Thanks a lot, Yiyi! |
sayakpaul
added a commit
that referenced
this pull request
May 10, 2024
* introduce videoprocessor. * fix quality * address yiyi's feedback * fix preprocess_video call. * video_processor -> image_processor * fix * fix more. * quality * image_processor -> video_processor * support List[List[PIL.Image.Image]] * change to video_processor. * documentation * Apply suggestions from code review * changes * remove print. * refactor video processor (part # 7776) (#7861) * update * update remove deprecate * Update src/diffusers/video_processor.py * update * Apply suggestions from code review * deprecate list of 5d for video and list of 4d for image + apply other feedbacks * up --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * add doc. * tensor2vid -> postprocess_video. * refactor preprocess with preprocess_video * set default values. * empty commit * more refactoring of prepare_latents in animatediff vid2vid * checking documentation * remove documentation for now. * fix animatediff sdxl * fix test failure [part of video processor PR] (#7905) up * remove preceed_with_frames. * doc * fix * fix * remove video input as a single-frame video. --------- Co-authored-by: YiYi Xu <yixu310@gmail.com>
sayakpaul
added a commit
that referenced
this pull request
Dec 23, 2024
* introduce videoprocessor. * fix quality * address yiyi's feedback * fix preprocess_video call. * video_processor -> image_processor * fix * fix more. * quality * image_processor -> video_processor * support List[List[PIL.Image.Image]] * change to video_processor. * documentation * Apply suggestions from code review * changes * remove print. * refactor video processor (part # 7776) (#7861) * update * update remove deprecate * Update src/diffusers/video_processor.py * update * Apply suggestions from code review * deprecate list of 5d for video and list of 4d for image + apply other feedbacks * up --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * add doc. * tensor2vid -> postprocess_video. * refactor preprocess with preprocess_video * set default values. * empty commit * more refactoring of prepare_latents in animatediff vid2vid * checking documentation * remove documentation for now. * fix animatediff sdxl * fix test failure [part of video processor PR] (#7905) up * remove preceed_with_frames. * doc * fix * fix * remove video input as a single-frame video. --------- Co-authored-by: YiYi Xu <yixu310@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
part of #7776
I refactored the
preprocessfor both image and video processora few notes:
I refactored the image processor a little bit, too, so it is more aligned with the video processor