Skip to content

Add BERT QA support#451

Draft
andrea-fasoli wants to merge 2 commits into
foundation-model-stack:mainfrom
andrea-fasoli:bert_support
Draft

Add BERT QA support#451
andrea-fasoli wants to merge 2 commits into
foundation-model-stack:mainfrom
andrea-fasoli:bert_support

Conversation

@andrea-fasoli
Copy link
Copy Markdown
Collaborator

This PR adds support for BERT Question Answering (one task only).

The architecture is named bert_question_answering. This selection will build an FMS RoBERTaQuestionAnswering model using BERT config. The models appear to be almost identical, except for max position embeddings and vocab type size, which have been adjusted in the config.

PR includes:

  • add parameters to initialize config file when using hf_pretrained
  • name adapter from HF BERT QA to FMS BERT QA (lots of overlap with RoBERTa QA, the adapters could be combined)

Draft PR for now, because accuracy of both FP16 and FP8 models is not on par with expectations (F1 ~ 55% instead of >70%).

Unit tests are passing.

Signed-off-by: Andrea Fasoli <andrea.fasoli@ibm.com>
Signed-off-by: Andrea Fasoli <andrea.fasoli@ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant