The Apache HTTP server is the most popular web server in use on the Internet [1]. The first version appeared on the market two decades ago, in April 1995. Due to constant development efforts by the Apache Software Foundation, the server is still in use today. It is known for its modular architecture as well as its rich functionality.

However, the competition has not been asleep. For example, the market share of Nginx has been growing quite a bit in the past few years. As a result, one goal of the Apache Software Foundation is to improve performance of the Apache HTTP server to ensure that its performance compares well with Nginx. And, in fact, version 2.4 has caught up somewhat in terms of performance.

This becomes apparent when comparing the performance of consecutive versions of the Apache web server running on Debian Wheezy (2.2.22) and Debian Jessie (2.4.10). Figure 1 shows that version 2.4 processes one million requests at a rate that is about 20 seconds faster than the previous 2.2 version.

Figure 1: The Apache HTTP server in its standard configuration has faster performance on Debian Jessie than on Debian Wheezy.

On Jessie, this performance increase was achieved by activating the Multi-Processing Module (MPM) Worker in place of the MPM Event in order to use the same module. I tested direct access via localhost in a virtual box (VM) with 512MB each of RAM. I used the Apache benchmarking tool ab each time to access a text file containing a simple string: hello world .

The performance of a server when delivering web pages depends on many factors. For example, actual web applications mostly use script languages like PHP and Perl. Additionally, there are database queries via MySQL and the like. All these aspects are the real reason for slow page delivery by a web server. The Apache web server by itself usually does not cause these slowdowns.

It is a good idea to maintain different versions of your configuration by using etckeeper [3]. This will keep you from accidentally destroying your working configuration by repeated tuning activities. You should check completed configuration changes for syntactic accuracy with apache2ctl -t before executing an Apache HTTP reload/restart.

RAM, RAM, and More RAM

Every Apache HTTP process needs several megabytes of working storage. Therefore, it is essential that the server has enough working storage available to deal with a large number of requests. If there is not enough RAM, the server will begin to swap and transfer the working memory onto the hard disk. Even in these times of SSDs, the throughput of hard drives is still orders of magnitude slower than that of RAM. The web server administrator should definitely take steps to avoid swapping.

With Linux, more RAM also means that the kernel can hold a larger page cache. This makes for an incredible increase in the speed of I/O requests. The web server admin should also consider this when allocating RAM. The Apache configuration also has to be coordinated with available RAM. If there are too many processes, then available RAM will quickly be consumed. The formula in Figure 2 can be used for specifying the MPM Prefork.

Figure 2: The number of processes on an Apache server should correspond to the size of the available RAM.

Web server administrators can determine component values via the top tool (Figure 3). If you press Shift+M, the tool sorts the processes according to memory usage. The actual main memory requirements of the processes appear in the RES column. There are 16GB of RAM available altogether in the example provided here. MySQL needs 39MB. The largest Apache HTTP process needs 22MB. The web server admin can ignore the DNS software BIND; it barely uses 1MB of RAM.

Figure 3: The preinstalled Linux tool top indicates the processes running on the HTTP server.

Therefore, if the formula presented above is applied, allocating 100MB for the operating system, 50MB for MySQL, and 2GB for the page cache, then this yields a value of 644 for MaxRequestWorkers . This variable determines how many concurrent requests the server can handle. It is a good idea to build in a bit of reserve and therefore choose the value 400 for the number of workers.

As an alternative, it is possible to use assistants to calculate this for you; see the "apachebuddy.pl" box for more information.

01 root@testserver:~# wget apachebuddy.pl -O apachebuddy.pl
02 root@testserver:~# perl apachebuddy.pl
03 ########################################################################
04 # Apache Buddy v 0.3 ###################################################
05 ########################################################################
06 Gathering information...
07 We are checking the service running on port 80
08 The process listening on port 80 is /usr/sbin/apache2
09 The process running on port 80 is Apache/2.2.16 (Debian)
10 Apache has been running 7d 01h 39m 11s
11 The full path to the Apache config file is: /etc/apache2/apache2.conf
12 Apache is using prefork model
13
14 Examining your Apache configuration...
15 Apache runs as apache
16 Your max clients setting is 150
17
18 Analyzing memory use...
19 Your server has 16024 MB of memory
20 The largest apache process is using 30.59 MB of memory
21 The smallest apache process is using 15.01 MB of memory
22 The average apache process is using 20.53 MB of memory
23 Going by the average Apache process, \
   Apache can potentially use 3079.51 MB RAM (19.22 % of available RAM)
24 Going by the largest Apache process, \
   Apache can potentially use 4588.51 MB RAM (28.64 % of available RAM)
25
26 Generating reports...
27 ### GENERAL REPORT ###
28
29 Settings considered for this report:
30
31         Your server's physical RAM:             16024MB
32         Apache's MaxClients directive:          150
33         Apache MPM Model:                       prefork
34         Largest Apache process (by memory):     30.59MB
35 [ OK ]  Your MaxClients setting is within an acceptable range.
36         Max potential memory usage:             4588.5 MB
37
38         Percentage of RAM allocated to Apache   28.64 %
39
40 -----------------------------------------------------------------------
41 -----------------------------------------------------------------------

Multi-Processing Modules

Apache HTTP Version 2.4 supports three different Multi-Processing Modules on Linux. They are:

• Prefork
• Worker
• Event

Additionally, version 2.4 makes it possible for the server to load MPMs during runtime.

The MPM Prefork module, mod_mpm_prefork.so , does not make use of threads. This means that each server request will be handled by its own process. When using PHP as an Apache module, this is usually the only option for running the server because, in PHP, many third-party provider libraries are not thread safe [4].

Fast CGI and PHP-FPM make it possible to use a threaded MPM together with PHP. However, it is not clear from a performance perspective that this is a good idea. Due to the loss of performance caused by the PHP integration via Fast CGI, the advantages of using threaded MPM are partially lost [5]. The following options are relevant to MPM Prefork:

• StartServers
• MinSpareServers/MaxSpareServers
• MaxRequestWorkers
• ServerLimit
• MaxConnectionsPerChild

The StartServers option specifies the number of processes that will initially launch when the server starts. MinSpareServer and MaxSpareServers maintain an appropriate number of spare server processes based on the number of incoming requests. MaxRequestWorkers , referred to as MaxClients up to version 2.3.13, limits the maximum number of processes and as a result also the number of concurrent requests. You should also pay attention to the directive ServerLimit . This sets the upper limit for MaxRequestWorkers with the standard value being 256 .

The ServerLimit directive will have to be raised in parallel with an increase in the value for MaxRequestWorkers . MaxConnectionsPerChild sets the number of connections that may be handled by a single process. In a perfect world, this value would be 0, indicating that the server processes would handle an unlimited number of requests. However, when there are complex applications it makes sense to restart the processes after every few hundred connections. This releases the RAM area that is used by the process and prevents memory leaks.

If there are not enough configured MaxRequestWorkers , the message from Listing 2 will appear in the log file.

[Fri Jun 05 13:15:24.760818 2015] [mpm_prefork:error] \
  [pid 1649] AH00161: server reached MaxRequestWorkers setting, \
  consider raising the MaxRequestWorkers setting.

The web server administrator doesn't need to worry about using threaded MPMs when no PHP module is present. By using threaded, the server handles a request via a thread instead of a process. Because a thread uses less overhead than a process, this approach enables the server to achieve a better performance.

Two threaded MPMs are available on Apache. These are MPM Worker, or mod_mpm_worker.so , which was introduced in version 2.0, and Event mod_mpm_event.so , which was added with version 2.2 and which has been considered stable since version 2.4.

The most important configuration options for MPM Worker and Event are identical:

• ThreadsPerChild
• MinSpareThreads/MaxSpareThreads
• MaxRequestWorkers
• ServerLimit

ThreadsPerChild specifies how many threads may be created by a single process. MinSpareThreads/MaxSpareThreads function analogously to the MinSpareServers/MaxSpareServers directives referred to above.

MaxRequestWorkers is used to limit the total number of threads. The default value for ServerLimit is 16 with threaded MPMs. Multiplying ThreadsPerChild with a standard value of 25, by the ServerLimit , with a standard value of 16, gives the upper limit for the number of threads. As a result, 400 is the default setting of the threads value for MaxRequestWorkers .

Purging Modules

Apache HTTP comes with approximately 120 modules and has the capability to integrate many more third-party provider modules [6]. When operating a web server you should always keep in mind that additional modules usually consume additional RAM. For performance and security reasons, it makes sense to check activated modules (Listing 3) and deactivate unnecessary ones.

root@debian:~# apache2ctl -M
Loaded Modules:
 core_module (static)
 so_module (static)
 watchdog_module (static)
 http_module (static)
 log_config_module (static)
 logio_module (static)
 version_module (static)
 unixd_module (static)
 access_compat_module (shared)
 alias_module (shared)
 auth_basic_module (shared)
 authn_core_module (shared)
 authn_file_module (shared)
 authz_core_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 deflate_module (shared)
 dir_module (shared)
 env_module (shared)
 filter_module (shared)
 mime_module (shared)
 mpm_prefork_module (shared)
 negotiation_module (shared)
 php5_module (shared)
 setenvif_module (shared)

Debian and Ubuntu users can activate and deactivate modules very easily with the a2enmod and a2dismod commands. Apache 2.4 is relatively uncluttered in a standard configuration. Another good thing is that Apache provides a warning if you try to turn off an essential module (Listing 4). When using CentOS and RHEL, modules are usually administered via the /etc/httpd/conf.d file.

root@debian:~# a2dismod mime
WARNING: The following essential module will be disabled.
This might result in unexpected behavior and should NOT be done
unless you know exactly what you are doing!
 mime
To continue type in the phrase 'Yes, do as I say!' or retry by \
  passing '-f':

DNS Lookups and KeepAlive

The HostnameLookups directive should definitely be set to No . This has been the default setting since Apache version 1.3. When this directive is activated, Apache will start a DNS reverse lookup for each web server connection, thereby causing unnecessary delay in the connection. Alternative means for resolving a DNS would be to use a piece of log analysis software or the Apache tool logresolve .

A similar situation arises with the Allow/Deny example.org directive or with Require host example.org , which is the syntax recommended by version 2.4. These directives also cause unnecessary DNS lookups. As a result, you should directly use an IP address for the directives, for example, Allow/Deny from 192.0.2.0 and Require ip 192.0.2.100 .

If you would like to keep host name lookups for specific files or directories, you can do so as follows.

HostnameLookups off
<Files ~ "\.(cgi)$">
  HostnameLookups on
</Files>

The KeepAlive directive activates keep alive connections by default. This allows a single TCP connection to process several requests. Problems can occur with this approach when the value for KeepAliveTimeout is too high, thereby generating too many queued processes and threads.

Avoiding Unnecessary Administrative I/O

Frequently, admins use .htaccess files that control the Apache server directly from the DocumentRoot and also take care of things like IP limits, password requests, and rewrites. Here, the directive AllowOverride is put to use. The disadvantage, however, is that the web server will need to check for each request whether an .htaccess file that needs processing exists in the respective directory. Therefore, if at all possible, you should universally set the AllowOverride directive to None and then explicitly allow it only for those directories where it is used.

You can always configure an .htaccess file straight from the Apache configuration. Often, however, there is no direct access to the configuration files in a shared hosting environment even if the web applications require this capability.

The SymLinksIfOwnerMatch option from the Options directive behaves in a similar fashion. For reasons of security, it is actually better when the directive is missing from a shared hosting environment. From a performance perspective, the absence of this directive is not so good because the server then has to monitor the owner of symlinks when the server is accessed.

If you want to wring out the last bit of performance from a web server, you should probably use the DirectoryIndex directive without a wildcard, (index ). Likewise, it's a good idea to deactivate the MultiViews option and use type maps instead. Finally, the send file support should be set to EnableSendfile On even though this may at times cause problems with network mounts.

Browser Caching

The mod_expires module can tell the browser to cache static data for long periods of time. This reduces the number of requests to the Apache web server. The module is activated on Debian and Ubuntu via

a2enmod expires

The next step is to specify the file types that should be cached (Listing 5). Additionally, it is important to deliver content in a compressed format. The mod_deflate module assumes this task. A standard configuration for this module already exists on Debian (see Listing 6).

ExpiresActive on
ExpiresByType image/gif "access plus 1 months"
ExpiresByType image/jpeg "access plus 1 months"
ExpiresByType image/png "access plus 1 months
ExpiresByType application/x-font-woff "access plus 1 months"
ExpiresByType application/javascript "access plus 1 months"
ExpiresByType text/css "access plus 1 months"

AddOutputFilterByType DEFLATE text/html text/plain text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/x-javascript \
  application/javascript application/ecmascript
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/xml

Starting with Apache 2.4, the mod_deflate module only compresses files when the resulting overhead is smaller than the overhead for compressed data. Under certain circumstances, the overhead doesn't touch very small files. Listing 7 shows how you can check expiration dates and compression with the aid of the server header.

root@debian:~# wget --server-response \
  --header="accept-encoding: gzip" http://localhost/test.css
--2015-06-07 06:06:57--  http://localhost/test.css
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Date: Sun, 07 Jun 2015 04:06:57 GMT
  Server: Apache/2.4.10 (Debian)
  Last-Modified: Fri, 05 Jun 2015 13:02:45 GMT
  ETag: "aa0-517c4e8861a8a-gzip"
  Accept-Ranges: bytes
  Vary: Accept-Encoding
  Content-Encoding: gzip
  Cache-Control: max-age=2592000
  Expires: Tue, 07 Jul 2015 04:06:57 GMT
  Content-Length: 102
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: text/css
Length: 102 [text/css]
Saving to: 'test.css'

Separating Static and Dynamic Content

When individual Apache processes need a lot of RAM, such as for a complex PHP application, then it may be a good idea to have separate processes deliver the static content. It is easier when a separate Apache server with a lean configuration and minimal memory footprint takes over this task. This can then be made into a front end server that directs the PHP requests on to an additional Apache web server via mod_proxy .

Alternatively, it is possible to use your own sub-domains for static content (e.g., static.ubuntu-user.com ) or its own domain, as with i.ytimg.com for YouTube.

Monitoring

The mod_status module lets you request information about the activity and performance of the Apache server. The module is inactive by default. It can easily be activated under Debian or Ubuntu via

a2enmod status

The standard URL for accessing the server is /server-status . Typically, you will have to explicitly set the access rights.

<Location /server-status>
        SetHandler server-status
        Require ip 192.0.2.0/24
</Location>

After restarting the web server, access to the status page should be available. Hopefully, this access is limited to the LAN. (See the "Access Is Too Easy" box.)

Figure 4: The statistics are there to help with administration, but they should not be made available on the Internet.

The status page is used for manual analysis, for example, when special load peaks need to be measured. At the same time, this page provides a basis for automatic monitoring that is carried out via something like a Nagios or Icinga plugin [7][8] or via the Percona Apache monitoring template for Cacti [9] (Figure 5). In this way, a web server administrator can systematically monitor the Apache web server and analyze the data.

Figure 5: Various monitoring solutions can be connected to the Apache server. Percona is one of these solutions.

Obviously, the use of mod_status creates a certain amount of overhead. In spite of the overhead, this is a fair trade off.

Benchmarks

To assess whether tuning activities make sense, it is necessary to have concrete statistics about the web server. The Apache web server has its own benchmarking tool named ab , which works directly via HTTP/HTTPS and is applicable to the web server of your choice.

The tester configures the number of requests (-n <requests> ) and the number of concurrent requests (-c <concurrency> ). An example is provided in Figure 1 above.

A more complex benchmarking tool, which also comes from the Apache Software Foundation is the Java-based Jmeter. Additional tools include curl-loader and httperf . Care is advised when using these tools in a production environment in order not to bring a system down through an unintended denial-of-service attack.

Operators of a PHP application should look at a PHP profiler like Xhprof . This is the only way to determine how many resources PHP and the accompanying database queries require.

Conclusion

The Apache web server remains the most universal web server on the market. The Apache server continues to hold its own as a solution among its contemporaries because of optimized configuration, a solid understanding of web applications, and sufficient working memory.

In any case, before exchanging Apache for another web server, it makes sense to hunt down bottlenecks that might form as the result of a web application. When a slowdown occurs, the HTTP server is usually not the cause. Instead, the culprit is often an extensive PHP application that generates large numbers of database queries.