Skip to content

Fix compiling with fullgraph=True to improve inference time. - #162

Closed
JakubCzarlinski wants to merge 2 commits into
nari-labs:mainfrom
JakubCzarlinski:main
Closed

JakubCzarlinski wants to merge 2 commits into
nari-labs:mainfrom
JakubCzarlinski:main

Conversation

@JakubCzarlinski

@JakubCzarlinski JakubCzarlinski commented Apr 30, 2025 •

Copy link
Copy Markdown
Contributor

This PR improves performance by resolving issues occurring when using torch.compile with fullgraph=True.

Other smaller optimisations include:

  • Allocation then filling tensors, rather than multiple allocations then merged with a concatenation that again allocates memory.
  • Removing redundant clone/copy operation.

Future optimisations can include chunked inference. Currently in the below code, pred_C[0] == audio_eos_value implicitly requires a sync between CPU and GPU, as it output dependant control-flow. The effect of this can be redude by either:

  • Checking only every $n$ steps, where $n$ is the chunk size. Eg: $n=4$ would mean check every 4 steps.
  • Skipping the first $n$, where $n$ is the minimum number of expected inference steps. Eg: $n=86$ would mean check only after the first 1s of generated values.
if (not eos_detected and pred_C[0] == audio_eos_value) or dec_step == max_tokens - max_delay_pattern - 1:
    eos_detected = True
    eos_countdown = max_delay_pattern

Before

generate step 86: speed=176.460 tokens/s, realtime factor=2.052x
generate step 172: speed=179.140 tokens/s, realtime factor=2.083x
generate step 258: speed=178.068 tokens/s, realtime factor=2.071x
generate step 344: speed=177.850 tokens/s, realtime factor=2.068x
generate step 430: speed=177.793 tokens/s, realtime factor=2.067x
generate step 516: speed=177.898 tokens/s, realtime factor=2.069x
generate step 602: speed=177.893 tokens/s, realtime factor=2.069x

After

generate step 86: speed=196.078 tokens/s, realtime factor=2.280x
generate step 172: speed=201.137 tokens/s, realtime factor=2.339x
generate step 258: speed=200.027 tokens/s, realtime factor=2.326x
generate step 344: speed=200.779 tokens/s, realtime factor=2.335x
generate step 430: speed=200.917 tokens/s, realtime factor=2.336x
generate step 516: speed=200.252 tokens/s, realtime factor=2.329x
generate step 602: speed=199.133 tokens/s, realtime factor=2.315x

Benchmarks done on a 4090 @ 300W, on the text in simple.py

Edit: using FP16 for both.

@GoudaCouda

Copy link
Copy Markdown

Just for more data I ran this on my 3090 and here are my results. I am using bfloat16 on wsl. Seemed to give either the same or slightly better results

Before

generate step 86: speed=4.111 tokens/s, realtime factor=0.048x
generate step 172: speed=56.929 tokens/s, realtime factor=0.662x
generate step 258: speed=52.501 tokens/s, realtime factor=0.610x
generate step 344: speed=60.524 tokens/s, realtime factor=0.704x
generate step 430: speed=58.007 tokens/s, realtime factor=0.675x
generate step 516: speed=62.052 tokens/s, realtime factor=0.722x
generate step 602: speed=60.892 tokens/s, realtime factor=0.708x
generate step 688: speed=61.134 tokens/s, realtime factor=0.711x

After


generate step 86: speed=4.337 tokens/s, realtime factor=0.050x
generate step 172: speed=58.800 tokens/s, realtime factor=0.684x
generate step 258: speed=61.619 tokens/s, realtime factor=0.717x
generate step 344: speed=61.189 tokens/s, realtime factor=0.711x
generate step 430: speed=60.737 tokens/s, realtime factor=0.706x
generate step 516: speed=62.536 tokens/s, realtime factor=0.727x
generate step 602: speed=58.580 tokens/s, realtime factor=0.681x
generate step 688: speed=63.658 tokens/s, realtime factor=0.740x

@JakubCzarlinski

Copy link
Copy Markdown
Contributor Author

@GoudaCouda hey there - couple things the results you show don't seem to be post warming up. The first iterations is significantly slower? What results do you get after the model has been through a couple of full generations?

Still about a 3% increase in your environment when excluding the first reported value.

I'll boot into Windows to see what sort of results I get there.

@JakubCzarlinski

Copy link
Copy Markdown
Contributor Author

@GoudaCouda to add to my previous comment just letting you know that previous issues in this repo have also reported very slow perf on Windows/WSL - sadly that's just the reality of ML/compiler/HPC development on windows. Briefly touched on it in #160.

@tsdocode

tsdocode commented May 1, 2025 •

Copy link
Copy Markdown
Contributor

I've just tried your PR using 'simple.py' :
System:

OS: Linux
Torch version: 2.6.0

with an A100 machine, got result:

generate step 516: speed=102.529 tokens/s, realtime factor=1.192x

with a 4090 machine:

generate step 688: speed=125.472 tokens/s, realtime factor=1.459x

Did you see some warning log like this?:

skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)

@JakubCzarlinski

JakubCzarlinski commented May 1, 2025 •

Copy link
Copy Markdown
Contributor Author

@GoudaCouda Getting back to you with regards to windows: (edit: same machine as original PR message, but booted different OS. original pr message uses latest CachyOS)

Before

generate step 516: speed=111.859 tokens/s, realtime factor=1.301x
generate step 602: speed=108.119 tokens/s, realtime factor=1.257x
generate step 688: speed=105.278 tokens/s, realtime factor=1.224x

After

generate step 430: speed=172.347 tokens/s, realtime factor=2.004x
generate step 516: speed=173.663 tokens/s, realtime factor=2.019x
generate step 602: speed=172.549 tokens/s, realtime factor=2.006x

So, not to dunk on windows, but yes I will dunk on windows in this regard.


This is a clean install of WSL on this machine, and I was prompted by a torch.compile error to download a C-compiler inside of WSL. Downloaded lasted gcc and clang using pacman.

@JakubCzarlinski

JakubCzarlinski commented May 1, 2025 •

Copy link
Copy Markdown
Contributor Author

@tsdocode

Did you see some warning log like this?:

skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)

Yes during the first run of the model for compilation I also see those torch.compile logs.

Could you mention your results before/after?


To be more clear about the methodology for benchmarking:

  1. First load the model with compilation and FP16. Seed the env.
  2. Run the model once to trigger the torch compiler. At this stage the model will be very slow. Directly below I have included some of the log during this process from my testing in the WSL environment directly above, with the code from this PR. The kernel finetuning logs can be ignored, the cudagraphs logs can be ignored for now but likely resolving these will also improve performance.
...
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
generate step 344: speed=121.844 tokens/s, realtime factor=1.417x
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
skipping cudagraphs due to mutated inputs (36 instances)
...
  1. Now run the model multiple times to warm-up. Below are the logs from the same WSL env, again with the changes from this PR. Note that there are no more compilation logs.
generate step 258: speed=175.026 tokens/s, realtime factor=2.035x
generate step 344: speed=174.371 tokens/s, realtime factor=2.028x
generate step 430: speed=173.127 tokens/s, realtime factor=2.013x

Edit: Example of trivially modified simple.py for benchmarking.

import torch
from dia.model import Dia

seed = 42
torch.manual_seed(seed)


model = Dia.from_pretrained("nari-labs/Dia-1.6B", compute_dtype="float16")
text = "[S1] Dia is an open weights text to dialogue model. [S2] You get full control over scripts and voices. [S1] Wow. Amazing. (laughs) [S2] Try it now on Git hub or Hugging Face."

# This will be slow on the first run.
output = model.generate(text, use_torch_compile=True, verbose=True)


# Future calls will be faster.
output = model.generate(text, use_torch_compile=True, verbose=True)
output = model.generate(text, use_torch_compile=True, verbose=True)
output = model.generate(text, use_torch_compile=True, verbose=True)

@buttercrab buttercrab left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code compiles every step and reuse them later right? In this way we will need a warmup function to pre-compile all of the steps. Can you make a warmup function?

@tsdocode

tsdocode commented May 2, 2025 •

Copy link
Copy Markdown
Contributor

@JakubCzarlinski I added a PR to fix skipping cudagraphs due to mutated inputs (36 instances) issue here:
#163

Speed that I got after the fix:

Result:
A100:

generate step 86: speed=103.650 tokens/s, realtime factor=1.205x
generate step 172: speed=192.087 tokens/s, realtime factor=2.234x
generate step 258: speed=191.480 tokens/s, realtime factor=2.227x
generate step 344: speed=192.203 tokens/s, realtime factor=2.235x
generate step 430: speed=192.052 tokens/s, realtime factor=2.233x

4090:
generate step 86: speed=78.049 tokens/s, realtime factor=0.908x
generate step 172: speed=197.427 tokens/s, realtime factor=2.296x
generate step 258: speed=197.933 tokens/s, realtime factor=2.302x
generate step 344: speed=197.997 tokens/s, realtime factor=2.302x
generate step 430: speed=197.914 tokens/s, realtime factor=2.301x
generate step 516: speed=197.846 tokens/s, realtime factor=2.301x
generate step 602: speed=197.875 tokens/s, realtime factor=2.301x
generate step 688: speed=197.797 tokens/s, realtime factor=2.300x

@buttercrab

Copy link
Copy Markdown
Contributor

Great job on this @JakubCzarlinski, but I am closing it due to #163.

@buttercrab buttercrab closed this May 2, 2025
@JakubCzarlinski

Copy link
Copy Markdown
Contributor Author

SGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants