Accelerated command processing with GNU Parallel

Job Center

Even with GNU Parallel working as expected, image processing can still be sluggish. This command can help:

$ parallel mogrify -resize 50% ::: *.tif

During testing, this command cuts the processing time in half. To avoid having to list all the processed files after the triple colons, the command uses the shell, which groups all available *.tif files in a separate TIFF directory in advance. GNU then calls up mogrify -resize 50% for each file.

With this, GNU Parallel tries to run as many jobs simultaneously as possible – one per core by default. With a quad-core processor, the command resizes four photos at the same time. The additional -j (--jobs ) option specifies the number of simultaneous jobs (known as jobslots) that you want done. The following command processes two photos at a time:

$ parallel -j 2 mogrify -resize 50% ::: *.tif

You can also specify the number after -j in a parameters file that has the value set, with no line break after the number:

$ parallel -j /tmp/jobs.txt...

GNU Parallel reads in this file with each new job. This approach has the advantage that you can change the number of simultaneous jobs in the middle of processing if you have a large collection of images. GNU Parallel shows you how many and which jobs are currently running with the standard error output (STDERR ) as soon as you send it the SIGUSR1 signal (Figure 4).

Figure 4: By sending the SIGUSR1 signal, GNU Parallel shows the currently running jobs, which in this example, are two photos being resized.

For larger image collections, GNU Parallel shows how far the processing has advanced when you use the --progress option. To cancel processing, simply press Ctrl+C or send a TERM signal (e.g., with killall -TERM parallel ). In the latter case, GNU Parallel waits to complete its work first.

Transfer of Ownership

The examples so far have used three colons to separate the commands from the images. As an alternative, you can pass the parameters by using a pipe (line 1 of Listing 1) or by collecting all parameters  – the file names in this case – in a text file (line 2).

01 $ find . -name '*.tif' | parallel mogrify -resize 50%
02 $ parallel -a dateinamen.txt mogrify -resize 50%
03 $ find . -name '*.tif' | parallel -0 mogrify -resize 50%

In both cases, GNU Parallel assumes that each parameter is separated by a newline. GNU Parallel doesn't break a file name containing spaces into separate values but passes it on to Mogrify as one, as it should. However, if someone happened to break a file name entry between lines, you can use the -0 (--null ) option (line 3) to specify separating the parameter values with the null (\0 ) instead of newline (\n ) character.