Skip to content

Adding support for Pixtral#435

Open
sahilsuneja1 wants to merge 7 commits into
foundation-model-stack:mainfrom
sahilsuneja1:llava
Open

Adding support for Pixtral#435
sahilsuneja1 wants to merge 7 commits into
foundation-model-stack:mainfrom
sahilsuneja1:llava

Conversation

@sahilsuneja1
Copy link
Copy Markdown
Contributor

@sahilsuneja1 sahilsuneja1 commented Jun 27, 2025 •
edited
Loading

This PR adds support for pixtral vision encoder + llava vision-language connector. This works together with a mistral language model variant (arch already supported), to yield overall Pixtral model.

@sahilsuneja1
pixtral / llava initial impl
75d4fad 
Signed-off-by: Sahil Suneja <suneja@us.ibm.com>
@sahilsuneja1
pixtral/fms matching hf output
098ff5b 
Signed-off-by: Sahil Suneja <suneja@us.ibm.com>
@sahilsuneja1
hf_pretraining config + weight unfusion
eb58593 
Signed-off-by: Sahil Suneja <suneja@us.ibm.com>
@sahilsuneja1
merge main
3ef476d 
Signed-off-by: Sahil Suneja <suneja@us.ibm.com>
@sahilsuneja1
+tests
c899e09 
Signed-off-by: Sahil Suneja <suneja@us.ibm.com>
@sahilsuneja1
formatting + cleanup
579cba2 
Signed-off-by: Sahil Suneja <suneja@us.ibm.com>
@sahilsuneja1
ruff check
48bbe5f 
Signed-off-by: Sahil Suneja <suneja@us.ibm.com>
@sahilsuneja1 sahilsuneja1 changed the title [DRAFT] Adding support for Pixtral Adding support for Pixtral Jul 9, 2025
@sahilsuneja1
Copy link
Copy Markdown
Contributor Author

sahilsuneja1 commented Jul 9, 2025 •
edited
Loading

This is ready for review.
Will update fms/models/hf/utils.py to use recursive infer_model_parameters after llava_next PR #420 merges.
There seems to be a graph break in PixtralVision::forward()

        patch_embeds_list = [
            embed[..., : (size[0] // self.patch_size), : (size[1] // self.patch_size)]
            for embed, size in zip(patch_embeds, image_sizes)
        ]

Depending upon the complexity, we can wait for the fix to be made to this PR itself, or in a subsequent one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ani300 ani300 Awaiting requested review from ani300

@afrittoli afrittoli Awaiting requested review from afrittoli

@kaoutar55 kaoutar55 Awaiting requested review from kaoutar55

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@sahilsuneja1