The idea of lsof [1] is quite simple: The application identifies special system resources – regular files, directories, block and graphical devices, streams or network files (sockets) – which are opened by the kernel. Then it prints a list of these resources, along with additional information, such as the PID, UID, status, and so on. In addition, lsof discloses the paths of opened files, which allows you to find out which libraries and configuration files an application is currently using.

This information helps you detect and debug errors in running processes on the system. For example, if you encounter a problem while trying to unmount a plugged in USB hard disk, lsof shows you the blocking process that is still accessing the resource. Quitting this process makes removing the devices in an orderly fashion possible. It gets harder if processes do not respond to signals anymore or have crashed. In those cases, you may still be able identify them, but you may not be able to terminate them with kill .

However, lsof is way more than a simple tool to find open files or more specifically internally used file descriptors. It includes features that ps , netstat , and a number of other tools provide. It is exactly this diversity of functions that makes working with lsof not that easy. As with other command-line programs, the lsof's special features can be activated and deactivated with options (Table 1). It's worth taking a deeper look at lsof's options syntax.

Syntax

As indicators for options (prefixes), lsof uses both the well-known minus sign (-<option> ) and the plus sign (+<option> ). Minus signs introduce and activate options while you can combine multiple switches: -a -b can also be written as -ab , and -ac <count> works the same as -a +c <count> .

The plus sign is reserved for a more special option. With options, it is important whether you call them with a plus or minus sign prefix. If you put an option with a plus sign in front of options with minus signs, lsof usually combines them without any issues: +M -a is equivalent to +Ma . However, when in doubt, you should write the options separately or in groups with the respective prefixes.

If you don't specify, lsof combines the options with a logical OR internally (<condition1> OR <condition2> ). The -a option changes the default behavior, combining the other options with a logical AND (<condition1> AND <condition2> ). Furthermore you can negate conditions with a caret (^ ). For example, to exclude a UID use -u ^<UID> . This is roughly similar to the syntax of groups in regular expressions.

Outputs

If you start lsof without options, it lists all currently opened files of all processes (Listing 1). At the very beginning, the output shows a short info row with the column descriptions. You can sort the output by many of these columns or reduce the output with options. By default, lsof sorts the output by the first column (COMMAND ) so that all details to a process are grouped together. This corresponds with a sorting by the process ID (PID ).

$ lsof
COMMAND  PID  USER   FD  TYPE  DEVICE  SIZE/OFF    NODE  NAME
systemd    1  root  cwd   DIR     8,6      4096       2  /
systemd    1  root  rtd   DIR     8,6      4096       2  /
systemd    1  root  txt   REG     8,6   1511624  400489  /usr/lib/systemd/systemd
systemd    1  root  mem   REG     8,6     19024  413874  /usr/lib/libuuid.so.1.3.0
[...]

Under user , you find (unsurprisingly) the name of the user, but not its UID. Up next, there is the column with the file descriptors (FD ) that create channels to certain files or directories. The most important of the possible values can be found in the Table 2.

Numeric values followed by one or more alphabetical characters stand for descriptor numbers with details about file access. The first character can be either r for read access, w for write access, or u for both. The optional second character describes the type of file lockings. Here, too, the second character can be r , w , and u and also variants with capital letters. With that information, you can see whether parts of the file (small letters) or the whole file (capital letters) are access locked. You can find more about this under the section LOCKS in the lsof man page.

The next field contains a TYPE for the file descriptor. This marks the connection type, such as a local file, a pipe, a FIFO queue, a network connection, and so on. The most frequent variants are listed in the Table 3.

The number combination under DEVICE reveals which device is connected. The file size can be found under SIZE/OFF . The NAME column is of particular importance since it contains the path and name of the opened file – as long as the entry corresponds to a regular local file.

Practice

lsof prints much more information than you'll ever need in most cases. Usually it makes sense to condense the data stream to show only the relevant details by using a filter. Often grep is used for this; alternatively you can fall back to a pager like less or more :

$ lsof | less -j22p <keyword>

By using -j22 , less doesn't show the matches at the upper edge of the window any more, but instead in row 22, which is roughly equivalent to the screen center of a full-screen terminal window. This enables you to also see the context of a match. The option -p <keyword> defines the search pattern, analogous to grep.

lsof also offers options to filter the output. Using the switch -c <pattern> , you can limit the considered processes, for example. The pattern can consist of a character string (i.e., process or program name) or a regular expression – the latter has to be enclosed by slashes (/ ).

With lsof /dev/sdc1 , you can quickly find out which process(es) access a device or a partition. If you want to know which processes use lock files under /var/lock/ , you can type lsof +D /var/lock/ . If the output is still to extensive, limit it to processes that begin with sys and cup , for example:

$ lsof -c sys -c cup [...]

As mentioned above, lsof combines options that control output with logical OR by default. If you only want to see rows that match two options at the same time, activate the AND mode with lsof -a .

If you only want to investigate processes started by a certain user, for example, those started by root, use the -u option. You can pass the username (root ), as well as the UID (0 ), using the -u option. To terminate all processes by a certain user identified by lsof, use the option -t . This will print an output that can be used as an argument for kill :

$ kill $(lsof -tu <user>)

In quite a few cases, you'll need information about which files have been opened by a certain process in a certain way. For this, find out the process's PID (for instance with Ps ), and use this as an argument for -p to lsof:

$ lsof -p <PID>

This provides all of the given process's opened files, including the used libraries. To exclude the latter, filter the output by grep -v ,/lib/' if necessary.

A speciality of lsof is the so-called repeat mode, which you can activate with the -r option: In this case an output takes place in fixed intervals, by default every 15 seconds. With -r <seconds> , you can change the interval, but it has to be at least two seconds.

Networking

Since Linux treats network connections as files, lsof can handle them in similar fashion as normal file access. By calling lsof with the -i option, it lists all current network connections. As a normal user, you will just see the connections of email clients, the web browser, or a Dropbox client if you have one. As root, this looks completely different because of running processes like the printing system via CUPS, the DHCP client, the zeroconf client Avahi, and probably the Network Time Protocol daemon (ntpd).

The option -i 6 limits the lsof output to connections using the IPv6 protocol; -i 4 shows only IPv4 connections. Together with -a and a PID, you can quickly find all network connections of a certain process.

$ lsof -a -i -p <PID>

lsof can also identify processes listening on a certain port. For this, you have to use the option -i again but provide the port number preceded by a colon as an argument. For instance, lsof -i :22 lists all SSH connections.

All TCP connections can be viewed with lsof -i TCP . By adding :ESTABLISHED , you can limit the output with lsof -i -sTCP:ESTABLISHED to established connections. Analogously, this works with lsof -i UDP for UDP connections, with lsof -i -sTCP:LISTEN for open ports, or with lsof -i<IP>[:<Port>] for connections to certain remote stations and ports.

GUI

Because working with lsof may overwhelm beginners quickly, Daniele Francesconi developed a graphical interface that provides all the important lsof options and presents the extensive output clearly, at least to a certain degree. The Java tool is called Glsof [2] and consists of two components: Glsof FileMonitor [3] is the simple graphical interface from Figure 1; Glsof Queries [4] serves the administration of multiple lsof instances and is only useful in larger installations.

Figure 1: The Glsof FileMonitor provides a powerful interface for lsof.

In order to use Glsof FileMonitor, download the program archive from SourceForge and save it in the directory of your choice. When you extract it, it will create the directory filemonitor/ with all necessary files. Then you call the GUI with the command from Listing 2. To use Glsof FileMonitor, you have to configure it first. You can do this by clicking on the "hamburger" icon on the far right in the top row and then by selecting the option Preferences .

$ java -Djava.security.policy=/path/filemonitor/security-client.txt -jar /path/filemonitor/filemonitor.jar

A reusable, retroactively editable configuration can be set up with a click on the plus sign in the toolbar. First, you give it a name, and, in the following dialog, you define the appropriate options and parameters. By default, FileMonitor uses lsof with the options -F -T +c0 -o -n -P to create the output. The FileMonitor option Avoid is equivalent to the lsof counterpart -b , meaning that it ignores blocking functions.

A click on Save saves the configuration (Figure 2) that you now mark with a radio button in Preferences . Now, you can start the data collection with a mouse click on the green button on the right-hand side of the toolbar in the application window. You can filter the shown information any time by typing in text in the field at the top. In addition, there is a selection field, which enables you to restrict the filters to certain lsof columns. A click on the funnel symbol activates the filter. A click on the brush icon will be delete all the filters, and the whole display will be cleaned.

Figure 2: The configuration of Glsof FileMonitor is managed via the Preferences dialog.

Conclusion

lsof is one of Linux's real power tools and proves to be a valuable help in diagnosing various cases of problems. However, as a basic requirement, the user needs a reasonably founded knowledge about the system's internal interrelationships. Nevertheless, even less experienced users can gather information with lsof that simpler applications like Fuser cannot provide, especially when it comes to blocked resources.

That said, before using lsof, you should definitely take a look in the man page, as well as in the FAQ [5] that supplies useful information and tips.