Slower decompression with bigger blocks

[tansy]

Slower decompression with bigger blocks

I noticed that with bigger blocks decompression does not scale very well.
I couldn't prove it until I did the test, and the result confirmed that suspicion.
Used silesia corpus increased in size with some regular data up to full 256 MiB, so blocks would fit perfectly, in power of 2 multiple.

Why is decompression so much slower with bigger block size? Up to almost two times slower with one frame.
For comparison, single frame file (below) does basically the same, regardless of number of threads used.

All measures were done in tmpfs, essentially in ramdisk and i/o slowdown was involved.
I couldn't fit more threads on the screen but you get te picture.

$ cat silesia.tar dicom* | head -c $((256*(1<<20))) > silesia.tar256
$ A=silesia.tar256; for b in 8 16 32 64 128 256; do zstd-mt -9 -b$b -c $A > $A.zst-b$b; done

$ for f in silesia.tar256.zst-*;		$ for f in silesia.tar256.zst-*;
  do CMD="zstd-mt -T1 -t $f";			  do CMD="zstd-mt -T2 -t $f";
  echo $ $CMD; time eval $CMD;			  echo $ $CMD; time eval $CMD;
  echo; done					  echo; done

$ zstd-mt -T1 -t silesia.tar256.zst-b008	$ zstd-mt -T2 -t silesia.tar256.zst-b008

real    0m1.584s				real    0m0.910s
user    0m1.513s				user    0m1.595s
sys     0m0.070s				sys     0m0.152s

$ zstd-mt -T1 -t silesia.tar256.zst-b016	$ zstd-mt -T2 -t silesia.tar256.zst-b016

real    0m1.652s				real    0m0.986s
user    0m1.584s				user    0m1.673s
sys     0m0.060s				sys     0m0.208s

$ zstd-mt -T1 -t silesia.tar256.zst-b032	$ zstd-mt -T2 -t silesia.tar256.zst-b032

real    0m1.651s				real    0m1.220s
user    0m1.584s				user    0m1.743s
sys     0m0.067s				sys     0m0.417s

$ zstd-mt -T1 -t silesia.tar256.zst-b064	$ zstd-mt -T2 -t silesia.tar256.zst-b064

real    0m1.660s				real    0m1.324s
user    0m1.596s				user    0m1.921s
sys     0m0.061s				sys     0m0.546s

$ zstd-mt -T1 -t silesia.tar256.zst-b128	$ zstd-mt -T2 -t silesia.tar256.zst-b128

real    0m1.665s				real    0m1.860s
user    0m1.597s				user    0m2.245s
sys     0m0.068s				sys     0m0.944s

$ zstd-mt -T1 -t silesia.tar256.zst-b256	$ zstd-mt -T2 -t silesia.tar256.zst-b256

real    0m1.676s				real    0m2.693s
user    0m1.603s				user    0m1.848s
sys     0m0.070s				sys     0m0.844s

$ zstd -9 -c $A > $A.zst-9
$ time zstd-mt -T1 -t silesia.tar256.zst-9	$ time zstd-mt -T2 -t silesia.tar256.zst-9

real    0m1.906s				real    0m1.958s
user    0m1.834s				user    0m1.894s
sys     0m0.071s				sys     0m0.059s