this is fine for now but we'll want to revisit with #5

it could also go in llama/utils.py though? not sure if that would be preferable

We don't have a llama folder for fms, only for hf/llama. This function is not for LLaMAHF (FMS), really it is for FMS (non-HF) LLaMA. I wonder if we should address the packaging structure for this (maybe not part of this PR), but in general.

oh I meant hf/utils.py, looking at the left margin I thought that was in the llama folder. Does somewhere else in hf/llama not make sense? I guess just for keeping the initialization functions near each other? not a big deal, especially since we'll be changing that code later anyway.

I think if there was a general initializations utility for llama, that would be best. Not sure where to put it as we do not have an fms llama folder

question: is there any generic way to do this? i.e. if we write adapters for a few models, is there some way to from_fms_model(model) without first having to check which model architecture it is?

If they conform to similar underlying api, then most likely yes. For instance, if the _helper function that is being called in _adapt had same input/output params (not necessarily the same names, but just same purpose). If they all have the same purpose and usage, we could add an implementation that can do this. It would require taking a map where it would map specific model parameters to their common parameter name. For instance, if one model uses the name x and another uses the name x_in, we can make the common name input_ids.

model = HFModel.from_fms_model(fms_model, mapper={'x': 'input_ids'})
model2 = HFModel.from_fms_model(fms_model2, mapper={'x_in': 'input_ids'})