moreutils completes the shell's toolbox

lckdo

When automated processes are running, there are many situations in which just one instance of a program is permitted to run. If a process takes longer to execute due to an unexpectedly large amount of data that needs to be processed, then conflicts can arise, and you can even lose data. If cron launches programs at intervals that are too short, it can lead to a never-ending loop. The first instance has not yet finished its task while the next one overwrites the incomplete database.

In these situations, programmers will normally resort to lock files. Before the actual work begins, you should have the task set up a file. Often the file is empty, and typically you would set it up under /var/lock/ . When the task finishes, it deletes the file. If the file already exists when the task tries to start, then the software doesn't even run. Sometimes the lock file also contains the ID for the process that set it up.

This kind of lock mechanism is fairly uncomplicated to implement, even in scripts. However the lckdo option makes life even easier (Listing 1). To use this option, you should use a command like nice or nohup . The options control the tool behavior precisely.

$ lckdo [Option] Lock-File Program

If a lock file already exists, then the tool will report this (Listing 2, line 3). Sometimes you won't be able to generate lock files in /var/lock/ because you may lack permission. The program also notifies you of this (Listing 2, line 5). In this case, you should use another directory, like your home directory.

01 $ lckdo yes.lock yes&
02 $ lckdo yes.lock yes
03 lckdo: lockfile `yes.lock' is already locked
04 $ lckdo /var/lock/yes.lock yes&
05 lckdo: unable to open ,/var/lock/yes.lock': Permission denied

lckdo deletes the lock file that has been generated after the program terminates. However, the lock files do not contain a process ID. Such an ID would make it easier to manually terminate a program if it has hung.

Another tool, flock , comes in the util-linux package:

sudo apt install util-linux

This program has a similar functionality to lckdo but comes with different options. lckdo 's man page therefore states that this tool will probably not be available in future versions of the moreutils package.

Parallel utils

There are a variety of tools for starting multiple programs in parallel, even with different arguments. All of these tools are named either parallel or something similar. The moreutils package contains parallel-moreutils tool. The simple syntax for the tool is not much different than that of other implementations (Listing 3, line 1).

01 $ parallel-moreutils Options -- Program with Arguments
02 $ parallel-moreutils Options Program -- Arguments
03 $ parallel-moreutils ufraw --out-path=ready/ ... -- *raw

In addition, the tool supports a second syntax variant (Listing 3, line 3), which can also be useful. There the double hyphen divides the RAW files that are defined by the pattern. By default, the software starts a process for each CPU core. Therefore this often significantly accelerates processing speeds. However, there is no linear correlation between number of cores and elapsed processing time since input and output operations constitute a bottleneck.

The option -j <number> lets you control the number of processes that start in parallel. It makes sense to keep <number> smaller than the number of processor cores. However, it is actually possible to make the number larger. Using -l <decimal number> , you can define the maximum load (this is a measure of system load) up to which the parallel-moreutils tool will start additional processes. Once the load exceeds this limit, no further processes are started until it falls below the limit.

It is possible to manually manipulate the commands executed with parallel-moreutils , which makes it universally applicable. There are two options for doing this. The -i option lets you rearrange the command. By default, you add the contents generated by an argument, such as the file names generated with *.raw , to the end of the command line. Using -i , the parallel-moreutils tool replaces each occurrence of {} in the command line with the text generated with its argument, such as the file names.

With -n <number> , you define the number of arguments that parallel-moreutils will take in one step and then pass onto the program that is its argument. You should not use this option in combination with -i .