Skip to content

perf(opsd): benchmark shared prefill with CUDA Graph decode - #1010

Open
nathon-lee wants to merge 3 commits into
deepspeedai:masterfrom
nathon-lee:perf/opsd-shared-prefill-cuda-graph-benchmark
Open

perf(opsd): benchmark shared prefill with CUDA Graph decode#1010
nathon-lee wants to merge 3 commits into
deepspeedai:masterfrom
nathon-lee:perf/opsd-shared-prefill-cuda-graph-benchmark

Conversation

@nathon-lee

Copy link
Copy Markdown
Contributor

Summary

Extend the OPSD HybridEngine benchmark with:

  • Prefill/decode forward timing
  • Native CUDA Graph decode benchmarking
  • Combined shared-prefill and CUDA Graph benchmarking
  • JSON mode metadata and documentation

This accompanies deepspeedai/DeepSpeed#8350 and the shared-prefill CUDA Graph follow-up.

Validation

  • Shared-prefill CUDA Graph smoke benchmark: passed
  • Four-mode OPT-6.7B benchmark completed: eager, shared, graph, shared + graph
  • Combined result: 10.04% lower total latency and 11.16% higher throughput
  • Commit is signed off

Thank you for reviewing!

Signed-off-by: nathon-lee <leejianwoo@gmail.com>
Signed-off-by: nathon-lee <leejianwoo@gmail.com>
Signed-off-by: nathon-lee <leejianwoo@gmail.com>
@nathon-lee
nathon-lee requested a review from tjruwase as a code owner August 29, 2026 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant