Checksum utilities, particularly sha1sum and sha256sum, slow compared to GNU coreutils

I tested the common checksumming utilities, and they seem significantly slower than the GNU ones. Note that the GNU coreutils tests were run on a bare metal Ubuntu 24.04 machine, and the uutils tests on a Ubuntu 26.04 VM running on the same HW, so some differences are expected, but this level would suggest significant optimization opportunities.

Results for GNU coreutils 9.4:

File: /tmp/checksum-bench-zi1p2vm0 (104,857,600 bytes (100.00 MiB))
Runs per utility: 5
Fastest: cksum (24.257 ms)

Utility         Best time      Ratio
---------- -------------- ----------
cksum           24.257 ms      1.00x
sha1sum        138.245 ms      5.70x
b2sum          176.584 ms      7.28x
md5sum         188.270 ms      7.76x
sha512sum      201.897 ms      8.32x
sha256sum      281.013 ms     11.58x

Running the same with uutils 8.0:

File: /tmp/checksum-bench-371wmgzt (104,857,600 bytes (100.00 MiB))
Runs per utility: 5
Fastest: cksum (33.813 ms)

Utility         Best time      Ratio
---------- -------------- ----------
cksum           33.813 ms      1.00x
b2sum          126.431 ms      3.74x
md5sum         237.955 ms      7.04x
sha512sum      301.289 ms      8.91x
sha1sum        482.523 ms     14.27x
sha256sum      963.138 ms     28.48x

One can see that uutils b2sum is actually faster than the coreutils one, most utilities are somewhat slower, but sha1sum and sha256sum in particular are a lot slower for the uutils versions.

what is the file size that you are using to test?
any args are provided? thanks

Do we use our logic for checksum instead of relying on external crates?
There is nothing we can do for external crates.

what is the file size that you are using to test? any args are provided? thanks

As mentioned in the output of the test wrapper, 100 MiB. No further command-line arguments given to the utilities except the path to the 100MiB test file.

Do we use our logic for checksum instead of relying on external crates? There is nothing we can do for external crates.

Maybe we could gain some performance if there are some SIMD options we are not leveraging at the moment.
I'll give it a look

Were the utilities compiled from source? There are significant checksum performance improvements when building with RUSTFLAGS="-C target-cpu=x86-64-v3", as discussed in #12003.

It is not a solution at here. There are many repositories doing same things for GNU coreutils.

@jabl

Ubuntu 26.04 now uses cksum from uutils.

@jabl Are you using the Ubuntu package build for these benchmarks? If so, is it the amd64v3 variant?

Ubuntu 26.04 now uses cksum from uutils.

@jabl Are you using the Ubuntu package build for these benchmarks? If so, is it the amd64v3 variant?

Yes, this was done with an out-of-the-box Ubuntu 26.04 install, with the uutils provided by Ubuntu. Quite sure it was just the standard amd64, nothing special was done to enable a amd64v3 variant.

@jabl

@jabl If you are running on compatible hardware you can upgrade to amd64v3 packages in Ubuntu 26.04:
https://documentation.ubuntu.com/release-notes/26.04/summary-for-lts-users/#architecture-variants-and-amd64v3

ubuntu 24.04:

time ./coreutils-0.8.0-i686-unknown-linux-gnu/coreutils sha256sum random_file.bin
f07842e25aaae475cb66c3cc27738e69679030d724e0bea1d5d578057fb9dd8e random_file.bin

real 0m0.644s
user 0m0.561s
sys 0m0.082s
ubuntu@geode:~/sha256test$ time sha256sum random_file.bin
f07842e25aaae475cb66c3cc27738e69679030d724e0bea1d5d578057fb9dd8e random_file.bin

real 0m0.646s
user 0m0.567s
sys 0m0.079s

Time almost the same. I suggest you to compare speeds on the same machine. How can you compare results between bare metal and VM?

Very well. Running everything on the same Ubuntu 26.04 VM. With out of the box uutils 8.0:

File: /tmp/checksum-bench-7qx_d31w (104,857,600 bytes (100.00 MiB))
Runs per utility: 5
Fastest: cksum (34.374 ms)

Utility         Best time      Ratio
---------- -------------- ----------
cksum           34.374 ms      1.00x
b2sum          129.912 ms      3.78x
md5sum         247.262 ms      7.19x
sha512sum      314.170 ms      9.14x
sha1sum        500.403 ms     14.56x
sha256sum      1.000433 s     29.10x

With GNU coreutils 9.7:

File: /tmp/checksum-bench-7hsgctfl (104,857,600 bytes (100.00 MiB))
Runs per utility: 5
Fastest: gnucksum (25.461 ms)

Utility           Best time      Ratio
------------ -------------- ----------
gnucksum          25.461 ms      1.00x
gnusha1sum       142.138 ms      5.58x
gnub2sum         168.350 ms      6.61x
gnumd5sum        184.193 ms      7.23x
gnusha512sum     196.741 ms      7.73x
gnusha256sum     283.434 ms     11.13x

Results pretty similar to my original message where I opened this bug.

gnu should be v3 binary too

@jabl

@jabl Are you using amd64 or amd64v3 packages, as detailed in #12251 (comment)?

Also, could you please provide the output of lscpu on your machine?

feat: add OpenSSL backend for checksum utilities

@Devel08

@Devel08 I enabled dynamic openssl on AUR/uutils-coreutils-git. Would you bench md5sum on both build?

Summary
  md5sum-openssl /lib/chromium/chromium ran
    1.16 ± 0.00 times faster than md5sum-pure-rust /lib/chromium/chromium

@oech3

@oech3 between GNU and uutils implementation?

openssl vs no-openssl

Alright

@oech3

@oech3

[Devel@archlinux Desktop]$ hyperfine -i  "/home/Devel/Desktop/uutils-coreutils-git/pkg/uutils-coreutils-git/usr/bin/md5sum /home/Devel/Desktop/archlinux-2026.02.01-x86_64.iso"   "/home/Devel/Desktop/uutils-without-openssl/pkg/uutils-coreutils-git/usr/bin/md5sum /home/Devel/Desktop/archlinux-2026.02.01-x86_64.iso"
Benchmark 1: /home/Devel/Desktop/uutils-coreutils-git/pkg/uutils-coreutils-git/usr/bin/md5sum /home/Devel/Desktop/archlinux-2026.02.01-x86_64.iso
  Time (mean ± σ):      2.706 s ±  0.111 s    [User: 2.475 s, System: 0.190 s]
  Range (min … max):    2.586 s …  2.932 s    10 runs

Benchmark 2: /home/Devel/Desktop/uutils-without-openssl/pkg/uutils-coreutils-git/usr/bin/md5sum /home/Devel/Desktop/archlinux-2026.02.01-x86_64.iso
  Time (mean ± σ):      3.347 s ±  0.130 s    [User: 3.058 s, System: 0.200 s]
  Range (min … max):    3.172 s …  3.545 s    10 runs

Summary
  /home/Devel/Desktop/uutils-coreutils-git/pkg/uutils-coreutils-git/usr/bin/md5sum /home/Devel/Desktop/archlinux-2026.02.01-x86_64.iso ran
    1.24 ± 0.07 times faster than /home/Devel/Desktop/uutils-without-openssl/pkg/uutils-coreutils-git/usr/bin/md5sum /home/Devel/Desktop/archlinux-2026.02.01-x86_64.iso

uutils-coreutils-git is the latest commit, uutils-without-openssl is e553b6abf949509f549d9a0e5a4135f75cef178f
my cpu is Intel i3-8130U (4) @ 3.400GHz

Looks fine!

The size of the file is 1.4GB

It might be able to back to pure-Rust md5 by RustCrypto/hashes#917 which was submitted by author of fast-md5.

@oech3

@oech3 If you want to track this, please open a new issue for this topic.

Navigation Menu

Repository navigation

Checksum utilities, particularly sha1sum and sha256sum, slow compared to GNU coreutils #12251

6 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Checksum utilities, particularly sha1sum and sha256sum, slow compared to GNU coreutils #12251

Description

Activity

sylvestre commented on May 12, 2026

oech3 commented on May 12, 2026

jabl commented on May 12, 2026

RenjiSann commented on May 12, 2026

xtqqczze commented on May 14, 2026

oech3 commented on May 14, 2026

xtqqczze commented on May 14, 2026

jabl commented on May 14, 2026

xtqqczze commented on May 14, 2026

scx1332 commented on May 17, 2026

jabl commented on May 18, 2026

oech3 commented on May 18, 2026

xtqqczze commented on May 18, 2026

6 remaining items

oech3 commented on Jun 29, 2026

oech3 commented on Jun 29, 2026

Devel08 commented on Jun 29, 2026

oech3 commented on Jun 29, 2026

Devel08 commented on Jun 29, 2026

Devel08 commented on Jun 29, 2026

oech3 commented on Jun 29, 2026

Devel08 commented on Jun 29, 2026

oech3 commented on Sep 17, 2026

xtqqczze commented on Sep 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions