An easy-to-use, scalable, and efficient multi-modal training pipeline for multi-round multi-image interleave chat experience.

• 📰 Latest News 📰
• 🚀 What is DeepSpeed-VisualChat 🚀️
• ⚓ Get Started, Tutorial, and Documentation ⚓
• 🌱 DeepSpeed-VisualChat's Roadmap 🌱
• 💬 DeepSpeed-VisualChat and DeepSpeed Community 💬
• 🙏 Acknowledgement and Citation 🙏

• [2023/10] DeepSpeed-VisualChat: Improve Your Chat Experience with Multi-Round Multi-Image Inputs

⭐ If you find our DeepSpeed and DeepSpeedExamples repositories beneficial, please give them a star on GitHub! To cite DeepSpeed-VisualChat, please cite our arxiv report:

@article{yao2023deepspeed-visualchat,
  title={{DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attention}},
  author={Zhewei Yao and Xiaoxia Wu and Conglong Li and Minjia Zhang and Heyang Qin and Olatunji Ruwase and Ammar Ahmad Awan and Samyam Rajbhandari and Yuxiong He},
  journal={arXiv preprint arXiv:2309.14327},
  year={2023}
}

Figure 1. On the left is a DeepSpeed-VisualChat model, featuring an innovative attention design. On the right is an example of DeepSpeed-VisualChat.

With increasing interest in enabling the multi-modal capabilities of large language models, DeepSpeed is proud to announce a new training pipeline, named DeepSpeed-VisualChat. This is designed for enabling a multi-round, multi-image interleave chat framework. It enhances the language model with image understanding and reasoning capabilities. Unlike the majority of open-sourced multi-modal projects, the primary focus of DeepSpeed-VisualChat is to provide a multi-round, multi-image interleave chat experience, as illustrated in Figure 1.

To improve model quality without introducing new parameters, DeepSpeed-VisualChat incorporates a new multi-modal causal attention mechanism, which is adept at better aligning visual and text features. Additionally, to overcome the scarcity of interleaved text-and-image inputs in most available open-sourced datasets, we employ various data blending techniques on existing datasets.

Thanks to the scalable, efficient, and user-friendly nature of the DeepSpeed ecosystem, we have the capability to train using a 2B visual encoder from QWen-VL (one is additionally refined from OpenClip) and a 70B language decoder from LLaMA-2. This showcases the extraordinary scalability of the DeepSpeed-VisualChat framework.

git clone https://github.com/deepspeedai/DeepSpeedExamples.git
cd DeepSpeedExamples/applications/DeepSpeed-VisualChat/
pip install -r requirements.txt

Table below summarizes where to download the datasets that we support. {data_path} denotes the --data_path argument provided in training scripts.

Please refer to

• Training
• Evaluation
• Chat
• Helper

Our future plan includes but not limited to :

• Support more models
• Demonstrate how to training larger models with higher model quality

Just like how the success of the BLOOM model was supported by both DeepSpeed Team and many open source contributors, we welcome all AI developers/practitioners/researchers to join this on-going effort for DeepSpeed-Chat. To participate:

• Show your support by leaving a star ⭐ to our DeepSpeed and DeepSpeedExamples GitHub repositories.
• Follow us on twitter to get notified about our latest news. For Chinese users, you can also follow our Chinese Zhihu account. For Japanese users, you can also follow our Japanese twitter account.
• Currently we prefer to interact with open source users mainly on GitHub so that it's easier for all users to search for related information. For bug reports, please submit a GitHub issue. For contribution, please submit a pull request (PR). For general question/discussion, please open a new discussion or join any existing discussions.
• We are open to collaborations with universities, research labs, and companies, such as working together on deep learning research, applying DeepSpeed to empower real-world AI models and applications, and so on. For such requests (and other requests unsuitable for GitHub), please directly email to deepspeed-info@microsoft.com.

We thank the following papers and open-source repositories:

[1] LLaVa, https://github.com/haotian-liu/LLaVA
[2] Otter, https://github.com/Luodian/Otter
[3] Transformers Hugging Face, https://github.com/huggingface/transformers
[4] MiniGPT4, https://github.com/Vision-CAIR/MiniGPT-4
[5] QWen-VL, https://github.com/QwenLM/Qwen-VL
[6] Sparkles, https://github.com/HYPJUDY/Sparkles
[7] Multimodal-GPT, https://github.com/open-mmlab/Multimodal-GPT