Skip to content

Quant tensorrt engine don't achieve advantage in inference speed over fp16 on A100 #3

Closed
Closed
Quant tensorrt engine don't achieve advantage in inference speed over fp16 on A100#3
Assignees
jingyu-ml

Description

@bigmover
bigmover
opened on May 11, 2024

I replace the unet.trt10.0.1.6.post12.dev1.engine built from Build the TRT engine for the INT8 Quantized ONNX UNet to engine/unet.trt10.0.1.plan . AND rerun the textdemo(python demo_txt2img.py "enchanted winter forest, soft diffuse light on a snow-filled day, serene nature scene, the forest is illuminated by the snow" --negative-prompt "normal quality, low quality, worst quality, low res, blurry, nsfw, nude" --scheduler Euler --denoising-steps 30 --seed 2946901).

Is there something I missed or what mistake I made? Any apply will be appreciated.

fp16
image
image

int8
image
image

Activity

  1. jingyu-ml commented on May 11, 2024

    @jingyu-ml
    jingyu-ml
    on May 11, 2024
    Contributor

    Hi @bigmover
    Could you please share your ONNX model (both FP16 and INT8)? Additionally, you can verify the latency benchmarks after building your engine; the latency information will be displayed at the end.

    Please run the following command on your FP16 ONNX models (remove the --int8 flag) and on your INT8 ONNX model. Then, share the log with us. Thank you!

    trtexec --onnx=./unet.onnx --shapes=sample:2x4x128x128,timestep:1,encoder_hidden_states:2x77x2048,text_embeds:2x1280,time_ids:2x6 --fp16 --int8 --builderOptimizationLevel=4 --saveEngine=unetxl.{version}.plan

    We've tested it internally and it should work. Once you share your ONNX model and the trtexec log, we can work on pinpointing the problem.

  2. bigmover commented on May 13, 2024

    @bigmover
    bigmover
    on May 13, 2024
    Author

    Hi @bigmover Could you please share your ONNX model (both FP16 and INT8)? Additionally, you can verify the latency benchmarks after building your engine; the latency information will be displayed at the end.

    Please run the following command on your FP16 ONNX models (remove the --int8 flag) and on your INT8 ONNX model. Then, share the log with us. Thank you!

    trtexec --onnx=./unet.onnx --shapes=sample:2x4x128x128,timestep:1,encoder_hidden_states:2x77x2048,text_embeds:2x1280,time_ids:2x6 --fp16 --int8 --builderOptimizationLevel=4 --saveEngine=unetxl.{version}.plan

    We've tested it internally and it should work. Once you share your ONNX model and the trtexec log, we can work on pinpointing the problem.

    Hi @jingyu-ml !
    The fp16 onnx is too large to share. But my model for quantize is the Standard Model from Huggingface(SD 1.5)(fp16 pytorch model).
    image

    The int 8 onnx after calibration
    unet.onnx.zip

    The output int8 log of

    TensorRT-10.0.1.6/targets/x86_64-linux-gnu/bin/trtexec --onnx=./sd-1.5-onnx-3/unet.onnx --shapes=sample:2x4x64x64,timestep:1,encoder_hidden_states:2x77x768 --fp16 --builderOptimizationLevel=4 --saveEngine=unet.trt10.0.1.6.post12.dev1_test.engine

    trtexec.log

    The output fp16 log of

    ../../TensorRT-10.0.1.6/targets/x86_64-linux-gnu/bin/trtexec --onnx=../../TensorRT/demo/Diffusion/onnx/unet/model.onnx --shapes=sample:2x4x64x64,timestep:1,encoder_hidden_states:2x77x768 --fp
    16 --builderOptimizationLevel=4 --saveEngine=unet.trt10.0.1.6.post12.dev1_test.engine > trtexec_fp16.log

    trtexec_fp16.log

  3. cjluo-nv
    assigned
    jingyu-ml
    on May 13, 2024

  4. jingyu-ml commented on May 15, 2024

    @jingyu-ml
    jingyu-ml
    on May 15, 2024
    Contributor

    Hi @bigmover,

    You only provided the ONNX graph, so it's not executable. However, I just tried creating the ONNX model myself. I see you're using an A100 for SD 1.5. The A100 is a very powerful GPU, but running SD 1.5 may not fully utilize its tensor cores, resulting in less optimal speedups when comparing FP16 & INT8. For a single image, the speedup is only about 1.10x.

    If you switch to GPUs like the L40s or Ada RTX 6000, you'll notice that for SD 1.5 in FP16, these GPUs perform faster than the A100, due to their VRAM and Tensor Core diff etc. However, for larger models like SDXL, the A100 is much, much faster. In a word, A100 is more suitable for bigger models and bigger batch size.

    You might want to consider using less powerful GPUs or increasing the batch size to maximize tensor core utilization, vram and communication bandwidth, etc.

    And I just tested this on an A100 and it works well, I can get 1.1+ speedup on the scale of 2 images for SD 1.5, and 1.2-1.3 speedup for A100 on SDXL, 2 images. bigger batch size means bigger utilization and more speedup, you can even try 8 batch size, 16 batch size or even bigger on A100/H100. Let me share each step I took so you can replicate the process:

    2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100 80GB PCIe          On  |   00000000:41:00.0 Off |                    0 |
| N/A   25C    P0             41W /  300W |       0MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

    Tested on A100-PCIE-80GB with CUDA 12.4.

    I downloaded the latest version of TensorRT from this location:

    TensorRT 10.0 GA for Linux x86_64 and CUDA 12.0 to 12.4 TAR Package

    Please verify your system and CUDA version, and download the corresponding TensorRT TAR from here:

    https://developer.nvidia.com/tensorrt/download/10x

    Run

    docker run --rm -it --gpus all nvcr.io/nvidia/pytorch:23.12-py3 /bin/bas

wget https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/10.0.1/tars/TensorRT-10.0.1.6.Linux.x86_64-gnu.cuda-12.4.tar.gz

git clone https://github.com/NVIDIA/TensorRT-Model-Optimizer.git

tar -xvzf TensorRT-10.0.1.6.Linux.x86_64-gnu.cuda-12.4.tar.gz

cd TensorRT-10.0.1.6

export LD_LIBRARY_PATH=$(pwd)/lib:$LD_LIBRARY_PATH

export PATH=$(pwd)/bin:$PATH

cd python

pip install tensorrt-10.0.1-cp<python_version>-*-linux_x86_64.whl

cd ../onnx_graphsurgeon

pip install onnx_graphsurgeon-0.5.0-py2.py3-none-any.whl

cd ../../

git clone https://github.com/NVIDIA/TensorRT.git

cd TensorRT-Model-Optimizer/

pip install "nvidia-modelopt[all]~=0.11.0" --extra-index-url https://pypi.nvidia.com

cd diffusers/

pip install -r requirements.txt

    Let's first focus on the performance instead of the image, so we can set the calibration parameters to random numbers.

    For SDXL, you can try:

    python quantize.py \
  --model stabilityai/stable-diffusion-xl-base-1.0 \
  --format int8 --batch-size 2 \
  --calib-size 8 --collect-method min-mean \
  --percentile 1.0 --alpha 0.8 \
  --quant-level 3.0 --n_steps 30 \
  --exp_name sdxl.test

python run_export.py --model stabilityai/stable-diffusion-xl-base-1.0 --quantized-ckpt ./unet.state_dict.sdxl.test.pt --format int8 --quant-level 3.0 --onnx-dir sdxl.test.onnx

python onnx_utils/sdxl_graphsurgeon.py --onnx-path ./sdxl.test.onnx/unet.onnx --output-onnx ./sdxl.test.onnx/unet.surged.onnx

trtexec --onnx=./sdxl.test.onnx/unet.surged.onnx --shapes=sample:4x4x128x128,timestep:1,encoder_hidden_states:4x77x2048,text_embeds:4x1280,time_ids:4x6 --fp16 --int8 --builderOptimizationLevel=3 --saveEngine=unetxl.int8.plan > sdxl.int8.log

    For SD1.5, you can try

    python quantize.py \
  --model runwayml/stable-diffusion-v1-5 \
  --format int8 --batch-size 2 \
  --calib-size 8 --collect-method global_min --alpha 0.7 \
  --quant-level 3.0 --n_steps 30 \
  --exp_name sd1.5.test

python run_export.py --model runwayml/stable-diffusion-v1-5 --quantized-ckpt ./unet.state_dict.sd1.5.test.pt --format int8 --quant-level 3.0 --onnx-dir sd1.5.test.onnx

python onnx_utils/sdxl_graphsurgeon.py --onnx-path ./sd1.5.test.onnx/unet.onnx --output-onnx ./sd1.5.test.onnx/unet.surged.onnx

trtexec --onnx=./sd1.5.test.onnx/unet.surged.onnx --shapes=sample:4x4x64x64,timestep:1,encoder_hidden_states:4x77x768 --fp16 --int8 --builderOptimizationLevel=3 --saveEngine=unet.int8.plan > sd.int8.log

  5. jingyu-ml commented on May 15, 2024

    @jingyu-ml
    jingyu-ml
    on May 15, 2024
    Contributor

    @bigmover
    Also please run this on both your INT8 and FP16 models and share the profiling file with us, and we will involve the performance team.

    nsys profile -t cuda,nvtx,cublas,cublas-verbose,cusparse,cusparse-verbose,cudnn --force-overwrite=true --output=int8.profiling trtexec --loadEngine=int8.plan --avgRuns=10

  6. bigmover commented on May 16, 2024

    @bigmover
    bigmover
    on May 16, 2024
    Author

    Hi @bigmover,

    You only provided the ONNX graph, so it's not executable. However, I just tried creating the ONNX model myself. I see you're using an A100 for SD 1.5. The A100 is a very powerful GPU, but running SD 1.5 may not fully utilize its tensor cores, resulting in less optimal speedups when comparing FP16 & INT8. For a single image, the speedup is only about 1.10x.

    If you switch to GPUs like the L40s or Ada RTX 6000, you'll notice that for SD 1.5 in FP16, these GPUs perform faster than the A100, due to their VRAM and Tensor Core diff etc. However, for larger models like SDXL, the A100 is much, much faster. In a word, A100 is more suitable for bigger models and bigger batch size.

    You might want to consider using less powerful GPUs or increasing the batch size to maximize tensor core utilization, vram and communication bandwidth, etc.

    And I just tested this on an A100 and it works well, I can get 1.1+ speedup on the scale of 2 images for SD 1.5, and 1.2-1.3 speedup for A100 on SDXL, 2 images. bigger batch size means bigger utilization and more speedup, you can even try 8 batch size, 16 batch size or even bigger on A100/H100. Let me share each step I took so you can replicate the process:

    2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100 80GB PCIe          On  |   00000000:41:00.0 Off |                    0 |
| N/A   25C    P0             41W /  300W |       0MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

    Tested on A100-PCIE-80GB with CUDA 12.4.

    I downloaded the latest version of TensorRT from this location:

    TensorRT 10.0 GA for Linux x86_64 and CUDA 12.0 to 12.4 TAR Package

    Please verify your system and CUDA version, and download the corresponding TensorRT TAR from here:

    https://developer.nvidia.com/tensorrt/download/10x

    Run

    docker run --rm -it --gpus all nvcr.io/nvidia/pytorch:23.12-py3 /bin/bas

wget https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/10.0.1/tars/TensorRT-10.0.1.6.Linux.x86_64-gnu.cuda-12.4.tar.gz

git clone https://github.com/NVIDIA/TensorRT-Model-Optimizer.git

tar -xvzf TensorRT-10.0.1.6.Linux.x86_64-gnu.cuda-12.4.tar.gz

cd TensorRT-10.0.1.6

export LD_LIBRARY_PATH=$(pwd)/lib:$LD_LIBRARY_PATH

export PATH=$(pwd)/bin:$PATH

cd python

pip install tensorrt-10.0.1-cp<python_version>-*-linux_x86_64.whl

cd ../onnx_graphsurgeon

pip install onnx_graphsurgeon-0.5.0-py2.py3-none-any.whl

cd ../../

git clone https://github.com/NVIDIA/TensorRT.git

cd TensorRT-Model-Optimizer/

pip install "nvidia-modelopt[all]~=0.11.0" --extra-index-url https://pypi.nvidia.com

cd diffusers/

pip install -r requirements.txt

    Let's first focus on the performance instead of the image, so we can set the calibration parameters to random numbers.

    For SDXL, you can try:

    python quantize.py \
  --model stabilityai/stable-diffusion-xl-base-1.0 \
  --format int8 --batch-size 2 \
  --calib-size 8 --collect-method min-mean \
  --percentile 1.0 --alpha 0.8 \
  --quant-level 3.0 --n_steps 30 \
  --exp_name sdxl.test

python run_export.py --model stabilityai/stable-diffusion-xl-base-1.0 --quantized-ckpt ./unet.state_dict.sdxl.test.pt --format int8 --quant-level 3.0 --onnx-dir sdxl.test.onnx

python onnx_utils/sdxl_graphsurgeon.py --onnx-path ./sdxl.test.onnx/unet.onnx --output-onnx ./sdxl.test.onnx/unet.surged.onnx

trtexec --onnx=./sdxl.test.onnx/unet.surged.onnx --shapes=sample:4x4x128x128,timestep:1,encoder_hidden_states:4x77x2048,text_embeds:4x1280,time_ids:4x6 --fp16 --int8 --builderOptimizationLevel=3 --saveEngine=unetxl.int8.plan > sdxl.int8.log

    For SD1.5, you can try

    python quantize.py \
  --model runwayml/stable-diffusion-v1-5 \
  --format int8 --batch-size 2 \
  --calib-size 8 --collect-method global_min --alpha 0.7 \
  --quant-level 3.0 --n_steps 30 \
  --exp_name sd1.5.test

python run_export.py --model runwayml/stable-diffusion-v1-5 --quantized-ckpt ./unet.state_dict.sd1.5.test.pt --format int8 --quant-level 3.0 --onnx-dir sd1.5.test.onnx

python onnx_utils/sdxl_graphsurgeon.py --onnx-path ./sd1.5.test.onnx/unet.onnx --output-onnx ./sd1.5.test.onnx/unet.surged.onnx

trtexec --onnx=./sd1.5.test.onnx/unet.surged.onnx --shapes=sample:4x4x64x64,timestep:1,encoder_hidden_states:4x77x768 --fp16 --int8 --builderOptimizationLevel=3 --saveEngine=unet.int8.plan > sd.int8.log

    HI @jingyu-ml Thank you very much for your prompt response!
    I test again with batch 2 and gained ~15% up in inference speed on A100 with SD 1.5! But actually there is hardly any acceleration effect when I switch to batch=1. However I gained ~20% over inference speed on A10 which SD 1.5 of batch 1.
    So as you said that the A100 has sufficient computation, int8 offers no advantage over fp16. I saw int8 achieve Nearly 2x Faster(about 1.72x) on NVIDIA RTX 6000 Ada GPU https://developer.nvidia.com/blog/tensorrt-accelerates-stable-diffusion-nearly-2x-faster-with-8-bit-post-training-quantization/. So the performance data on the A100 GPU mentioned above(hardly any effect with batch 1 on A100) is acceptable ?

    Besides dynamic engine is supported when quant?

  7. jingyu-ml commented on May 16, 2024

    @jingyu-ml
    jingyu-ml
    on May 16, 2024
    Contributor

    HI @jingyu-ml Thank you very much for your prompt response! I test again with batch 2 and gained ~15% up in inference speed on A100 with SD 1.5! But actually there is hardly any acceleration effect when I switch to batch=1. However I gained ~20% over inference speed on A10 which SD 1.5 of batch 1. So as you said that the A100 has sufficient computation, int8 offers no advantage over fp16. I saw int8 achieve Nearly 2x Faster(about 1.72x) on NVIDIA RTX 6000 Ada GPU https://developer.nvidia.com/blog/tensorrt-accelerates-stable-diffusion-nearly-2x-faster-with-8-bit-post-training-quantization/. So the performance data on the A100 GPU mentioned above(hardly any effect with batch 1 on A100) is acceptable ?

    @bigmover
    Yes, I would say so. For SD1.5 BS=1 (one image) our internal benchmarks can have 1.10x speedup on A100 80G. Since the A100 is not deigned for such small models like SD 1.5. If you still want to run small models on the A100, as we discussed, try using the maximum batch size that doesn't cause an OOTM issue. This might help you achieve 100% GPU utilization for FP16 and better speedup using INT8.

    Besides dynamic engine is supported when quant?

    I'm not quite sure I understood your question. Were you referring to dynamic quantization, or something else?

  8. bigmover commented on May 16, 2024

    @bigmover
    bigmover
    on May 16, 2024
    Author

    @jingyu-ml

    I'm not quite sure I understood your question. Were you referring to dynamic quantization, or something else?

    I mean whether I can calibrated pt--> build dynamic onnx -> build dynamic engine?

    For SD1.5 BS=1 (one image) our internal benchmarks can have 1.10x speedup on A100 80G

    It's tested on TRT 10 or TRT 9.3.? I can gain 10% with BS=1 only in quant mode =2. The int8 engine use input(fp16-> int8) * weight (int8) --> out(int8 ->fp16) to softmax, right? How many times faster is int8 * int8 compared to that speed of tensor core of fp16?

  9. jingyu-ml commented on May 25, 2024

    @jingyu-ml
    jingyu-ml
    on May 25, 2024
    Contributor

    I mean whether I can calibrated pt--> build dynamic onnx -> build dynamic engine?

    What does "dynamic" mean in this context? Are you referring to dynamic batch size?

    It's tested on TRT 10 or TRT 9.3.? I can gain 10% with BS=1 only in quant mode =2. The int8 engine use input(fp16-> int8) * weight (int8) --> out(int8 ->fp16) to softmax, right? How many times faster is int8 * int8 compared to that speed of tensor core of fp16?

    It has been tested on both. The int8 engine used here only executes Gemm and Conv operations in int8, while other parts remain in fp16. The int8 * int8 Gemm and Conv operations are 1.9 times faster than those in fp16.

  10. cjluo-nv commented on May 28, 2024

    @cjluo-nv
    cjluo-nv
    on May 28, 2024
    Collaborator

    @bigmover has @jingyu-ml resolved this issue?

  11. bigmover commented on May 29, 2024

    @bigmover
    bigmover
    on May 29, 2024
    Author

    @bigmover has @jingyu-ml resolved this issue?

    I understand the issue at hand. And I will close the issue. Thanks for your support!

  12. bigmover
    closed this as completedon May 29, 2024
  13. alexeldeib
    added a commit that references this issue on Apr 26, 2026

    quanty: meta-init + per-rank sharded safetensors load

    f14e6d0
  14. hychiang-git
    added 4 commits that reference this issue on Jun 8, 2026

    [OMNIML-4944] peft: smoke test for LoRA factor save/load round-trip

    d51e8a2

    [OMNIML-5051] peft: smoke test for LoRA factor save/load round-trip

    033a6ae

    [OMNIML-5051] peft: per-expert weight quant in MoE LoRA svdquant init

    48d4376

    [OMNIML-5051] peft: compose test for per-expert quant + MoE LoRA roun…

    44a12ea
  15. sychen52
    added a commit that references this issue on Oct 2, 2026

    Fix vLLM fake-quant serving of FP8 checkpoints, MLA KV presets, CUDA …

    b1b66cb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions