added hf adapter classes and llama hf implementation; added model testing suite by JRosenkranz · Pull Request #2 · foundation-model-stack/foundation-model-stack

JRosenkranz · 2023-08-31T18:01:41Z

Added the huggingface adapters along with model testing.

…el testing suite

… is what meta uses; added readme example for hf

nairbv · 2023-09-06T15:14:33Z

        **kwargs,
    ) -> BaseModelOutputWithPastAndCrossAttentions:
-        output = self.model(
+        output = self.model._helper(


ah, clearer seeing your motive in context.

nairbv · 2023-09-06T15:20:11Z


 import torch
 import torch.nn as nn
+from transformers import LlamaForCausalLM


I think it's cleaner if the model code is independent, without first needing to import HF. maybe convert_hf_llama should be in a different file?

ya that is an option, not sure where we want to put it, maybe we create a separate file called model_utils that has this for multiple different models?

…sed for a type hint in llama model; now using _has_hf in conversion

nairbv · 2023-09-07T16:03:44Z

+    """
+
+    if not _has_hf:
+        raise ImportError("in order to convert huggingface weights, you need to have transformers installed")


I think it's worth separating the HF code from the main code, i.e. putting this function in a separate file from the model code.

I think this has a place here, its providing a utility to load an FMS model from a huggingface model (not related to FMS Huggingface). If we move this, we maybe should move the meta load as well. One possibility is we add some form of FMS model creation utility.

moving both is fine too

I can make the PR for moving both to different files

nairbv · 2023-09-07T16:05:17Z

+
+import pytest
+
+


is this code related to HF equivalence? if so we should put it in that dir. otherwise it could be confusing if we add any other kind of model tests in this dir.
(and if it's not HF code, maybe should be a separate PR?)
I think I've already reviewed all of the actual code in an internal copy, but I'd like to keep it mostly consolidated in a directory. I think it'll be confusing if it's interspersed with other tests and code.

this base is related to model consistency as well, it can be used with both. Note this PR: #9

This has no HF related code in it, just the non-hf and it uses the same classes.

nairbv · 2023-09-07T16:16:36Z

+from fms.utils.config import ModelConfig
+from tests.models.base import ModelBase
+
+


Same idea here as above, I think we should use a directory to make it clear that there's a suite of tests specifically testing equivalence to HF (and include that and only that in this PR).

Just looking at the file names/paths one might assume we have "test_config.py" for testing utils/config.py independently, and then separately there's a test_hf_config for testing hf config equivalence. but they're both added as part of this HF PR and I think are actually both related to HF.

test_config is only related to our fms model config, test_hf_config includes our model config as well as hf config tests i believe

I agree that HF tests in general should be in an HF folder, with things like hf_equivalence, hf_config, hf_model becoming subfolders of this hf folder, and becoming hf/equivalence, etc.

…ch generation

JRosenkranz · 2023-09-29T17:36:22Z

This is being replaced by #39

Signed-off-by: Sahil Suneja <suneja@us.ibm.com>

* adding siglip vision support Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * import fix Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * update attn_kwargs Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * ruff Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * adding tests Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * ruff format Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * test update Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * addressing review comments Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * siglip updates post review #2 Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * adding fms/models/llava_next Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * adding fms/models/llava_next Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * update attn_kwargs Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * ruff Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * adding tests for llava_next Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * rebasing on siglip branch + addressing review comments Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * rebse atop siglip #2 Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * adding hf bs fms output equivalence test Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * moving imports to make build framework happy during testing Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * pytest.mark.slow Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * addressing review comments Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * mypy Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * adding HF config loading via hf_pretrained Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * percolate unfuse_weights to constituents Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * hf config loading check Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * recursive model config mapping Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * removing loops Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * fix graph break Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * combining input_ids and inputs_embeds args Signed-off-by: Sahil Suneja <suneja@us.ibm.com> --------- Signed-off-by: Sahil Suneja <suneja@us.ibm.com> Co-authored-by: Antoni Viros <aviros@ibm.com>

* adding siglip vision support Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * import fix Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * update attn_kwargs Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * ruff Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * adding tests Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * ruff format Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * test update Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * addressing review comments Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * siglip updates post review foundation-model-stack#2 Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * adding fms/models/llava_next Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * adding fms/models/llava_next Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * update attn_kwargs Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * ruff Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * adding tests for llava_next Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * rebasing on siglip branch + addressing review comments Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * rebse atop siglip foundation-model-stack#2 Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * adding hf bs fms output equivalence test Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * moving imports to make build framework happy during testing Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * pytest.mark.slow Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * addressing review comments Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * mypy Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * adding HF config loading via hf_pretrained Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * percolate unfuse_weights to constituents Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * hf config loading check Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * recursive model config mapping Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * removing loops Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * fix graph break Signed-off-by: Sahil Suneja <suneja@us.ibm.com> * combining input_ids and inputs_embeds args Signed-off-by: Sahil Suneja <suneja@us.ibm.com> --------- Signed-off-by: Sahil Suneja <suneja@us.ibm.com> Co-authored-by: Antoni Viros <aviros@ibm.com> Signed-off-by: Niraj Kamal Karunanidhi <nirajkkamal@gmail.com>

…g_attention Feature/ring attention

…beddings Fix weight loading to be compatible with FMS RoPE

feat: added hf adapter classes and llama hf implementation; added mod…

37454c0

…el testing suite

JRosenkranz added the enhancement New feature or request label Aug 31, 2023

JRosenkranz requested a review from nairbv August 31, 2023 18:01

JRosenkranz self-assigned this Aug 31, 2023

JRosenkranz added 3 commits August 31, 2023 15:20

fix: added equivalency test; fixed weight conversions

d2ee158

fix: defaults now match llama2; hf name now using ForCausalLM as that…

eb76dea

… is what meta uses; added readme example for hf

chore: changed generator to llama_generator

c696115

nairbv reviewed Aug 31, 2023

View reviewed changes

Comment thread fms/models/llama.py Outdated

JRosenkranz added 6 commits September 5, 2023 22:30

Merge branch 'main' into add_hf_and_model_tests

67be88d

feat: switched to lm head mixin; added function to convert from hf

259c73d

test: added train loss test

a80e099

Merge branch 'main' into add_hf_and_model_tests

c48c45c

fix: manual merge from weight cleanup

a5edae3

build: added transformers 4.31.0

09e34cd

nairbv reviewed Sep 6, 2023

View reviewed changes

Comment thread fms/models/llama.py

nairbv reviewed Sep 6, 2023

View reviewed changes

JRosenkranz added 4 commits September 6, 2023 11:22

fix: fixed merge conflicts; reformatted llama code style

78b55a4

test: moved hf equivalency test to its own file; removed the import u…

2d66344

…sed for a type hint in llama model; now using _has_hf in conversion

fix: added __init__ to models/hf_equivalence

9f6b028

fix: removed self in hf equiv test

ccb8d05

nairbv reviewed Sep 7, 2023

View reviewed changes

JRosenkranz added 3 commits September 7, 2023 13:15

fix: added al params in conversion

b04d72f

fix: updated to stay in line with the model tests

987900e

fix: added padding_side=left to all tokenizers as is required for bat…

244f667

…ch generation

JRosenkranz closed this Sep 29, 2023

ani300 pushed a commit that referenced this pull request Jul 1, 2025

siglip updates post review #2

902fd14

Signed-off-by: Sahil Suneja <suneja@us.ibm.com>

Lucas-Qian6 pushed a commit to Lucas-Qian6/Heterogeneous-setup-in-foundation-model-stack that referenced this pull request Dec 17, 2025

Merge pull request foundation-model-stack#2 from AnshSG13/feature/rin…

cb3f06e

…g_attention Feature/ring attention

maxdebayser pushed a commit to maxdebayser/foundation-model-stack that referenced this pull request Mar 23, 2026

Merge pull request foundation-model-stack#2 from maxdebayser/qwen3-em…

b235994

…beddings Fix weight loading to be compatible with FMS RoPE

		from fms.utils.config import ModelConfig
		from tests.models.base import ModelBase

Conversation

JRosenkranz commented Aug 31, 2023

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nairbv Sep 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nairbv Sep 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JRosenkranz commented Sep 29, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nairbv Sep 7, 2023 •

edited

Loading

nairbv Sep 7, 2023 •

edited

Loading