Repository navigation
Fix compiling with fullgraph=True to improve inference time. - #162
JakubCzarlinski wants to merge 2 commits into
Conversation
|
Just for more data I ran this on my 3090 and here are my results. I am using bfloat16 on wsl. Seemed to give either the same or slightly better results Before After |
|
@GoudaCouda hey there - couple things the results you show don't seem to be post warming up. The first iterations is significantly slower? What results do you get after the model has been through a couple of full generations? Still about a 3% increase in your environment when excluding the first reported value. I'll boot into Windows to see what sort of results I get there. |
|
@GoudaCouda to add to my previous comment just letting you know that previous issues in this repo have also reported very slow perf on Windows/WSL - sadly that's just the reality of ML/compiler/HPC development on windows. Briefly touched on it in #160. |
|
I've just tried your PR using 'simple.py' : with an A100 machine, got result: with a 4090 machine: Did you see some warning log like this?: |
|
@GoudaCouda Getting back to you with regards to windows: (edit: same machine as original PR message, but booted different OS. original pr message uses latest CachyOS)
Before After So, not to dunk on windows, but yes I will dunk on windows in this regard. This is a clean install of WSL on this machine, and I was prompted by a |
Yes during the first run of the model for compilation I also see those Could you mention your results before/after? To be more clear about the methodology for benchmarking:
Edit: Example of trivially modified import torch
from dia.model import Dia
seed = 42
torch.manual_seed(seed)
model = Dia.from_pretrained("nari-labs/Dia-1.6B", compute_dtype="float16")
text = "[S1] Dia is an open weights text to dialogue model. [S2] You get full control over scripts and voices. [S1] Wow. Amazing. (laughs) [S2] Try it now on Git hub or Hugging Face."
# This will be slow on the first run.
output = model.generate(text, use_torch_compile=True, verbose=True)
# Future calls will be faster.
output = model.generate(text, use_torch_compile=True, verbose=True)
output = model.generate(text, use_torch_compile=True, verbose=True)
output = model.generate(text, use_torch_compile=True, verbose=True) |
buttercrab
left a comment
There was a problem hiding this comment.
This code compiles every step and reuse them later right? In this way we will need a warmup function to pre-compile all of the steps. Can you make a warmup function?
|
@JakubCzarlinski I added a PR to fix Speed that I got after the fix: |
|
Great job on this @JakubCzarlinski, but I am closing it due to #163. |
|
SGTM |
This PR improves performance by resolving issues occurring when using
torch.compilewithfullgraph=True.Other smaller optimisations include:
Future optimisations can include chunked inference. Currently in the below code,
pred_C[0] == audio_eos_valueimplicitly requires a sync between CPU and GPU, as it output dependant control-flow. The effect of this can be redude by either:Before
After
Benchmarks done on a 4090 @ 300W, on the text in
simple.pyEdit: using FP16 for both.