splitjob performance tests

Quick links to results for different compression programs:
Graphs with results
gzip with 50 MB block size on machines with few cores
Different compression programs with 10 MB block size on machines with many cores

Description of the tests

The input data file linux-3.15.6.tar (the Linux kernel source) with the size 571576320 bytes (546 MB) is being compressed in parallel by splitting up the file in blocks of 50 MB for a possibility of up to 11 simultaneous jobs or 10 MB for a possibility of up to 55 simultaneous jobs.

Each command is run 3 times and the best time of these 3 times is used in the result. The file is read from a cached local file system and output is sent to /dev/null so the time measured should only be the time needed for compression. The times was measured with the time command from the bash shell. In real life usage you might have to use some program like bfr to get best compression performance and avoid waiting for input or output.

The block sizes 50 MB and 10 MB are only chosen to make it possible to run enough parallel jobs on this file. Depending on the compression method a small block size might affect the resulting compressed size as described in this test with different block sizes and different compression programs.

Graphs with all data

gzip with 50 MB block size on machines with few cores

Below are tables with all data and comments.

Different compression programs with 10 MB block size on machines with many cores

Below are tables with all data and comments. Logarithmic scales have been used to make it easier to see were the performance graphs for 5 networked machines crosses the performance graph of a single machine.

gzip with 50 MB block size on machines with few cores

With gzip and 50 MB block size the resulting compressed data becomes 121485376 bytes (115 MB) compared to 121474801 bytes (115 MB) for gzip without splitjob.

VIA Nano U3300 @1200MHz (single core)

Command	jobs	time	seconds	Comment
gzip < linux-3.15.6.tar > /dev/null	NA	1m17.595s	77.595	Compression without splitjob
splitjob -b 50M -j 1 gzip < linux-3.15.6.tar > /dev/null	1	1m20.783s	80.783	Splitjob adds a slight overhead but gives no speedup on a single core machine.
splitjob -b 50M -j 2 gzip < linux-3.15.6.tar > /dev/null	2	1m23.084s	83.084
splitjob -b 50M -j 3 gzip < linux-3.15.6.tar > /dev/null	3	1m37.089s	97.089
splitjob -b 50M -j 4 gzip < linux-3.15.6.tar > /dev/null	4	1m32.139s	92.139

Intel Pentium Dual CPU T3200 @2.00 GHz (dual core, no hyperthreading)

Command	jobs	time	seconds	Comment
gzip < linux-3.15.6.tar > /dev/null	NA	0m35.625s	35.625	Compression without splitjob
splitjob -b 50M -j 1 gzip < linux-3.15.6.tar > /dev/null	1	0m37.623s	37.623	Splitjob adds a slight overhead.
splitjob -b 50M -j 2 gzip < linux-3.15.6.tar > /dev/null	2	0m21.751s	21.751	An extra core is useful.
splitjob -b 50M -j 3 gzip < linux-3.15.6.tar > /dev/null	3	0m20.092s	20.092	Not much point in adding more jobs than cores.
splitjob -b 50M -j 4 gzip < linux-3.15.6.tar > /dev/null	4	0m20.793s	20.793
splitjob -b 50M -j 5 gzip < linux-3.15.6.tar > /dev/null	5	0m21.054s	21.054
splitjob -b 50M -j 6 gzip < linux-3.15.6.tar > /dev/null	6	0m20.691s	20.691
splitjob -b 50M -j 7 gzip < linux-3.15.6.tar > /dev/null	7	0m20.801s	20.801
splitjob -b 50M -j 8 gzip < linux-3.15.6.tar > /dev/null	8	0m20.778s	20.778
splitjob -b 50M -j 9 gzip < linux-3.15.6.tar > /dev/null	9	0m20.857s	20.857
splitjob -b 50M -j 10 gzip < linux-3.15.6.tar > /dev/null	10	0m21.666s	21.666

Intel Core i3 4130 @3.40 GHz (dual core + hyperthreading)

Command	jobs	time	seconds	Comment
gzip < linux-3.15.6.tar > /dev/null	NA	0m18.631s	18.631	Compression without splitjob
splitjob -b 50M -j 1 gzip < linux-3.15.6.tar > /dev/null	1	0m19.147s	19.147	Splitjob adds a slight overhead.
splitjob -b 50M -j 2 gzip < linux-3.15.6.tar > /dev/null	2	0m11.525s	11.525	An extra core is useful.
splitjob -b 50M -j 3 gzip < linux-3.15.6.tar > /dev/null	3	0m8.677s	8.677	Hyperthreading is not as good as real cores, but at least it is to some use.
splitjob -b 50M -j 4 gzip < linux-3.15.6.tar > /dev/null	4	0m7.298s	7.298
splitjob -b 50M -j 5 gzip < linux-3.15.6.tar > /dev/null	5	0m7.304s	7.304	Not much point in adding more jobs than cores.
splitjob -b 50M -j 6 gzip < linux-3.15.6.tar > /dev/null	6	0m6.956s	6.956
splitjob -b 50M -j 7 gzip < linux-3.15.6.tar > /dev/null	7	0m6.982s	6.982
splitjob -b 50M -j 8 gzip < linux-3.15.6.tar > /dev/null	8	0m6.699s	6.699
splitjob -b 50M -j 9 gzip < linux-3.15.6.tar > /dev/null	9	0m7.002s	7.002
splitjob -b 50M -j 10 gzip < linux-3.15.6.tar > /dev/null	10	0m7.034s	7.034

gzip with 10 MB block size on machines with many cores

With gzip and 10 MB block size the resulting compressed data becomes 121521501 bytes (115 MB) compared to 121474801 bytes (115 MB) for gzip without splitjob.

dual Xeon X5492 @3.40GHz (8 cores total, no hyperthreading)

Command	jobs	time	seconds	Comment
gzip < linux-3.15.6.tar > /dev/null	NA	0m20.733s	20.733	Compression without splitjob
splitjob -b 10M -j 1 gzip < linux-3.15.6.tar > /dev/null	1	0m21.642s	21.642	Splitjob adds a slight overhead.
splitjob -b 10M -j 2 gzip < linux-3.15.6.tar > /dev/null	2	0m11.061s	11.061	Extra cores are useful.
splitjob -b 10M -j 3 gzip < linux-3.15.6.tar > /dev/null	3	0m7.688s	7.688
splitjob -b 10M -j 4 gzip < linux-3.15.6.tar > /dev/null	4	0m5.823s	5.823
splitjob -b 10M -j 5 gzip < linux-3.15.6.tar > /dev/null	5	0m4.985s	4.985
splitjob -b 10M -j 6 gzip < linux-3.15.6.tar > /dev/null	6	0m4.321s	4.321
splitjob -b 10M -j 7 gzip < linux-3.15.6.tar > /dev/null	7	0m3.689s	3.689
splitjob -b 10M -j 8 gzip < linux-3.15.6.tar > /dev/null	8	0m3.387s	3.387
splitjob -b 10M -j 9 gzip < linux-3.15.6.tar > /dev/null	9	0m3.250s	3.250	Not much point in adding more jobs than cores.
splitjob -b 10M -j 10 gzip < linux-3.15.6.tar > /dev/null	10	0m3.212s	3.212
splitjob -b 10M -j 11 gzip < linux-3.15.6.tar > /dev/null	11	0m3.156s	3.156
splitjob -b 10M -j 12 gzip < linux-3.15.6.tar > /dev/null	12	0m3.190s	3.190

5 machines with dual Xeon X5492 @3.40GHz (40 cores total, no hyperthreading)

In this test gzip is used on localhost as well as 4 other machines (h2, h3, h4 and h5). The other machines will be accessed with ssh, when more than 1 command is given to splitjob the commands are called in round robin order.

Command	jobs	time	seconds	Comment
gzip < linux-3.15.6.tar > /dev/null	NA	0m20.733s	20.733	Compression without splitjob
splitjob -b 10M -j 1 gzip "ssh -x h2 gzip" "ssh -x h3 gzip" "ssh -x h4 gzip" "ssh -x h5 gzip" < linux-3.15.6.tar > /dev/null	1	0m30.380s	30.380	Splitjob adds a slight overhead and those ssh logins adds even more overhead.
splitjob -b 10M -j 2 gzip "ssh -x h2 gzip" "ssh -x h3 gzip" "ssh -x h4 gzip" "ssh -x h5 gzip" < linux-3.15.6.tar > /dev/null	2	0m15.452s	15.452	Extra cores are useful, but the 1 Gb/s network gives a bottleneck. 80% of the data is compressed on remote machines, that is 0.8*571576320=436 MB. 436 MB / 4.3 s is 100 MB/s which is about what you would expect being capable of sending out to remote machines on a 1 Gb/s network. With gzip we got better performance by not using any remote machines. For this particular case, we would probably get the best performance with a command like: splitjob -b 10M -j 12 gzip gzip "ssh -x h2 gzip" The above command would place 8 jobs on localhost and 4 jobs on a remote machine and would probably take about 2.2 seconds. However, I did not realize that the network was the bottleneck while I still was at the test machines so I haven't run this test.
splitjob -b 10M -j 3 gzip "ssh -x h2 gzip" "ssh -x h3 gzip" "ssh -x h4 gzip" "ssh -x h5 gzip" < linux-3.15.6.tar > /dev/null	3	0m10.696s	10.696
splitjob -b 10M -j 4 gzip "ssh -x h2 gzip" "ssh -x h3 gzip" "ssh -x h4 gzip" "ssh -x h5 gzip" < linux-3.15.6.tar > /dev/null	4	0m8.068s	8.068
splitjob -b 10M -j 5 gzip "ssh -x h2 gzip" "ssh -x h3 gzip" "ssh -x h4 gzip" "ssh -x h5 gzip" < linux-3.15.6.tar > /dev/null	5	0m7.075s	7.075
splitjob -b 10M -j 6 gzip "ssh -x h2 gzip" "ssh -x h3 gzip" "ssh -x h4 gzip" "ssh -x h5 gzip" < linux-3.15.6.tar > /dev/null	6	0m6.050s	6.050
splitjob -b 10M -j 7 gzip "ssh -x h2 gzip" "ssh -x h3 gzip" "ssh -x h4 gzip" "ssh -x h5 gzip" < linux-3.15.6.tar > /dev/null	7	0m5.258s	5.258
splitjob -b 10M -j 8 gzip "ssh -x h2 gzip" "ssh -x h3 gzip" "ssh -x h4 gzip" "ssh -x h5 gzip" < linux-3.15.6.tar > /dev/null	8	0m4.853s	4.853
splitjob -b 10M -j 9 gzip "ssh -x h2 gzip" "ssh -x h3 gzip" "ssh -x h4 gzip" "ssh -x h5 gzip" < linux-3.15.6.tar > /dev/null	9	0m4.550s	4.550
splitjob -b 10M -j 10 gzip "ssh -x h2 gzip" "ssh -x h3 gzip" "ssh -x h4 gzip" "ssh -x h5 gzip" < linux-3.15.6.tar > /dev/null	10	0m4.503s	4.503
splitjob -b 10M -j 11 gzip "ssh -x h2 gzip" "ssh -x h3 gzip" "ssh -x h4 gzip" "ssh -x h5 gzip" < linux-3.15.6.tar > /dev/null	11	0m4.307s	4.307
splitjob -b 10M -j 12 gzip "ssh -x h2 gzip" "ssh -x h3 gzip" "ssh -x h4 gzip" "ssh -x h5 gzip" < linux-3.15.6.tar > /dev/null	12	0m4.282s	4.282
splitjob -b 10M -j 13 gzip "ssh -x h2 gzip" "ssh -x h3 gzip" "ssh -x h4 gzip" "ssh -x h5 gzip" < linux-3.15.6.tar > /dev/null	13	0m4.233s	4.233
splitjob -b 10M -j 14 gzip "ssh -x h2 gzip" "ssh -x h3 gzip" "ssh -x h4 gzip" "ssh -x h5 gzip" < linux-3.15.6.tar > /dev/null	14	0m4.237s	4.237
splitjob -b 10M -j 15 gzip "ssh -x h2 gzip" "ssh -x h3 gzip" "ssh -x h4 gzip" "ssh -x h5 gzip" < linux-3.15.6.tar > /dev/null	15	0m4.248s	4.248

bzip2 with 10 MB block size on machines with many cores

With bzip2 and 10 MB block size the resulting compressed data becomes 95272415 bytes (90 MB) compared to 95116256 bytes (90 MB) for bzip2 without splitjob.

dual Xeon X5492 @3.40GHz (8 cores total, no hyperthreading)

Command	jobs	time	seconds	Comment
bzip2 < linux-3.15.6.tar > /dev/null	NA	0m58.884s	58.884	Compression without splitjob
splitjob -b 10M -j 1 bzip2 < linux-3.15.6.tar > /dev/null	1	1m0.379s	60.379	Splitjob adds a slight overhead.
splitjob -b 10M -j 2 bzip2 < linux-3.15.6.tar > /dev/null	2	0m30.932s	30.932	Extra cores are useful.
splitjob -b 10M -j 3 bzip2 < linux-3.15.6.tar > /dev/null	3	0m21.707s	21.707
splitjob -b 10M -j 4 bzip2 < linux-3.15.6.tar > /dev/null	4	0m16.028s	16.028
splitjob -b 10M -j 5 bzip2 < linux-3.15.6.tar > /dev/null	5	0m14.289s	14.289
splitjob -b 10M -j 6 bzip2 < linux-3.15.6.tar > /dev/null	6	0m12.816s	12.816
splitjob -b 10M -j 7 bzip2 < linux-3.15.6.tar > /dev/null	7	0m11.798s	11.798
splitjob -b 10M -j 8 bzip2 < linux-3.15.6.tar > /dev/null	8	0m10.013s	10.013
splitjob -b 10M -j 9 bzip2 < linux-3.15.6.tar > /dev/null	9	0m9.994s	9.994	Not much point in adding more jobs than cores.
splitjob -b 10M -j 10 bzip2 < linux-3.15.6.tar > /dev/null	10	0m9.840s	9.840
splitjob -b 10M -j 11 bzip2 < linux-3.15.6.tar > /dev/null	11	0m9.632s	9.632
splitjob -b 10M -j 12 bzip2 < linux-3.15.6.tar > /dev/null	12	0m9.817s	9.817

5 machines with dual Xeon X5492 @3.40GHz (40 cores total, no hyperthreading)

In this test bzip2 is used on localhost as well as 4 other machines (h2, h3, h4 and h5).

Command	jobs	time	seconds	Comment
bzip2 < linux-3.15.6.tar > /dev/null	NA	0m58.884s	58.884	Compression without splitjob
splitjob -b 10M -j 1 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	1	1m9.524s	69.524	Splitjob adds a slight overhead and those ssh logins adds even more overhead.
splitjob -b 10M -j 2 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	2	0m35.654s	35.654	Extra cores (both local and remote) are useful. Performance scales rather linear with number of jobs.
splitjob -b 10M -j 3 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	3	0m24.710s	24.710
splitjob -b 10M -j 4 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	4	0m18.427s	18.427
splitjob -b 10M -j 5 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	5	0m15.400s	15.400
splitjob -b 10M -j 6 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	6	0m12.817s	12.817
splitjob -b 10M -j 7 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	7	0m12.025s	12.025
splitjob -b 10M -j 8 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	8	0m10.100s	10.100
splitjob -b 10M -j 9 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	9	0m9.193s	9.193
splitjob -b 10M -j 10 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	10	0m8.597s	8.597
splitjob -b 10M -j 11 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	11	0m7.616s	7.616
splitjob -b 10M -j 12 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	12	0m7.517s	7.517	With 546 MB to compress in blocks of 10 MB each we have 55 blocks to compress. 54 blocks are 10 MB, the last block is 6 MB. With 12 and more parallel jobs we start seeing quantification effects on the performance. The blocks will be compressed in parallel with 12 jobs like 12+12+12+12+7 and with 13 jobs like 13+13+13+13+3. Compressing the last 3 blocks might take as long time as compressing the last 7 blocks in parallel. However even though all blocks have the same size the time to compress each block might depend upon the contents in the block.
splitjob -b 10M -j 13 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	13	0m6.967s	6.967
splitjob -b 10M -j 14 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	14	0m6.339s	6.339
splitjob -b 10M -j 15 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	15	0m6.219s	6.219
splitjob -b 10M -j 16 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	16	0m5.688s	5.688
splitjob -b 10M -j 17 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	17	0m5.545s	5.545
splitjob -b 10M -j 18 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	18	0m5.419s	5.419
splitjob -b 10M -j 19 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	19	0m5.406s	5.406
splitjob -b 10M -j 20 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	20	0m5.391s	5.391
splitjob -b 10M -j 21 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	21	0m5.327s	5.327
splitjob -b 10M -j 22 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	22	0m5.363s	5.363
splitjob -b 10M -j 23 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	23	0m4.833s	4.833
splitjob -b 10M -j 24 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	24	0m4.844s	4.844
splitjob -b 10M -j 25 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	25	0m4.865s	4.865
splitjob -b 10M -j 26 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	26	0m4.907s	4.907
splitjob -b 10M -j 27 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	27	0m4.831s	4.831
splitjob -b 10M -j 28 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	28	0m4.903s	4.903
splitjob -b 10M -j 29 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	29	0m4.808s	4.808
splitjob -b 10M -j 30 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	30	0m4.753s	4.753
splitjob -b 10M -j 31 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	31	0m4.693s	4.693
splitjob -b 10M -j 32 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	32	0m4.747s	4.747
splitjob -b 10M -j 54 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	54	0m4.580s	4.580
splitjob -b 10M -j 55 bzip2 "ssh -x h2 bzip2" "ssh -x h3 bzip2" "ssh -x h4 bzip2" "ssh -x h5 bzip2" < linux-3.15.6.tar > /dev/null	55	0m4.573s	4.573	This is as close as we get to about 4.3 seconds which was the network bottleneck we found with gzip. However, 55 jobs on only 40 cores will not get 100% CPU each so 4.6 seconds is still rather good.

xz with 10 MB block size on machines with many cores

With xz and 10 MB block size the resulting compressed data becomes 84431992 bytes (80 MB) compared to 82223868 bytes (78 MB) for xz without splitjob.

dual Xeon X5492 @3.40GHz (8 cores total, no hyperthreading)

Command	jobs	time	seconds	Comment
xz < linux-3.15.6.tar > /dev/null	NA	4m51.192s	291.192	Compression without splitjob
splitjob -b 10M -j 1 xz < linux-3.15.6.tar > /dev/null	1	4m26.274s	266.274	Splitjob now gives a negative overhead. This might seem good, but the explanation is that the small 10 MB blocks are not enough for best xz compression, this also shows in the resulting compressed size which increases about 3%.
splitjob -b 10M -j 2 xz < linux-3.15.6.tar > /dev/null	2	2m13.392s	133.392	Extra cores are useful.
splitjob -b 10M -j 3 xz < linux-3.15.6.tar > /dev/null	3	1m30.535s	90.535
splitjob -b 10M -j 4 xz < linux-3.15.6.tar > /dev/null	4	1m9.514s	69.514
splitjob -b 10M -j 5 xz < linux-3.15.6.tar > /dev/null	5	1m0.400s	60.400
splitjob -b 10M -j 6 xz < linux-3.15.6.tar > /dev/null	6	0m51.910s	51.910
splitjob -b 10M -j 7 xz < linux-3.15.6.tar > /dev/null	7	0m47.072s	47.072
splitjob -b 10M -j 8 xz < linux-3.15.6.tar > /dev/null	8	0m41.383s	41.383
splitjob -b 10M -j 9 xz < linux-3.15.6.tar > /dev/null	9	0m41.562s	41.562	Not much point in adding more jobs than cores.
splitjob -b 10M -j 10 xz < linux-3.15.6.tar > /dev/null	10	0m43.046s	43.046

5 machines with dual Xeon X5492 @3.40GHz (40 cores total, no hyperthreading)

In this test xz is used on localhost as well as 4 other machines (h2, h3, h4 and h5).

Command	jobs	time	seconds	Comment
xz < linux-3.15.6.tar > /dev/null	NA	4m51.192s	291.192	Compression without splitjob
splitjob -b 10M -j 1 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	1	4m37.567s	277.567	Splitjob removes more overhead than the ssh logins add.
splitjob -b 10M -j 2 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	2	2m19.368s	139.368	Extra cores (both local and remote) are useful. Performance scales rather linear with number of jobs.
splitjob -b 10M -j 3 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	3	1m34.599s	94.599
splitjob -b 10M -j 4 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	4	1m12.415s	72.415
splitjob -b 10M -j 5 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	5	0m59.060s	59.060
splitjob -b 10M -j 6 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	6	0m49.108s	49.108
splitjob -b 10M -j 7 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	7	0m43.845s	43.845
splitjob -b 10M -j 8 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	8	0m38.889s	38.889
splitjob -b 10M -j 9 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	9	0m33.951s	33.951
splitjob -b 10M -j 10 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	10	0m33.564s	33.564
splitjob -b 10M -j 11 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	11	0m28.768s	28.768
splitjob -b 10M -j 12 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	12	0m28.548s	28.548	Again we are seeing the quantification effects we saw with bzip2 on multiple hosts.
splitjob -b 10M -j 13 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	13	0m27.728s	27.728
splitjob -b 10M -j 14 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	14	0m23.509s	23.509
splitjob -b 10M -j 15 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	15	0m23.237s	23.237
splitjob -b 10M -j 16 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	16	0m22.780s	22.780
splitjob -b 10M -j 17 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	17	0m22.399s	22.399
splitjob -b 10M -j 18 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	18	0m18.701s	18.701
splitjob -b 10M -j 19 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	19	0m18.612s	18.612
splitjob -b 10M -j 20 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	20	0m18.555s	18.555
splitjob -b 10M -j 21 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	21	0m18.591s	18.591
splitjob -b 10M -j 22 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	22	0m18.774s	18.774
splitjob -b 10M -j 23 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	23	0m18.849s	18.849
splitjob -b 10M -j 24 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	24	0m18.328s	18.328
splitjob -b 10M -j 25 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	25	0m17.873s	17.873
splitjob -b 10M -j 26 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	26	0m17.394s	17.394
splitjob -b 10M -j 27 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	27	0m14.524s	14.524
splitjob -b 10M -j 28 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	28	0m14.330s	14.330
splitjob -b 10M -j 29 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	29	0m14.023s	14.023
splitjob -b 10M -j 30 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	30	0m13.904s	13.904
splitjob -b 10M -j 31 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	31	0m14.049s	14.049
splitjob -b 10M -j 32 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	32	0m14.298s	14.298
splitjob -b 10M -j 33 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	33	0m14.261s	14.261
splitjob -b 10M -j 34 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	34	0m13.936s	13.936
splitjob -b 10M -j 35 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	35	0m13.915s	13.915
splitjob -b 10M -j 36 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	36	0m13.831s	13.831
splitjob -b 10M -j 37 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	37	0m13.859s	13.859
splitjob -b 10M -j 38 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	38	0m13.741s	13.741
splitjob -b 10M -j 39 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	39	0m13.902s	13.902
splitjob -b 10M -j 40 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	40	0m13.749s	13.749
splitjob -b 10M -j 41 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	41	0m13.769s	13.769
splitjob -b 10M -j 42 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	42	0m14.064s	14.064
splitjob -b 10M -j 52 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	52	0m14.272s	14.272
splitjob -b 10M -j 53 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	53	0m13.461s	13.461
splitjob -b 10M -j 54 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	54	0m10.794s	10.794
splitjob -b 10M -j 55 xz "ssh -x h2 xz" "ssh -x h3 xz" "ssh -x h4 xz" "ssh -x h5 xz" < linux-3.15.6.tar > /dev/null	55	0m10.772s	10.772	This is as good as it gets with xz and 40 cores, the network is no bottleneck. If we would have had 55 cores we might have gotten closer to 7.5 seconds as we got 14.5 seconds with 27 jobs. But 7.5*55/40 gets about 10.3 which is rather close to the 10.8 seconds we got.

Splitjob performance example comparison

With gzip without splitjob we could produce a compressed data size of 115 MB in about 20 seconds. With a network of equal machines xz with splitjob compressed the same data to 80 MB in about 10 seconds.

Back to splitjob main page