For those awaiting the ‘Build your own HD youtube’ series completion, I have a dedi server en route, to allow completion of this series.

“Dell Core 330 v2 - Core2Duo E4600 (2×2.4GHz) 1GB RAM 160GB HDD”

CentOS 64bit, from what I have read the cpu uses Intels EMT, which is realy only about 48bits … but neither here or there at the moment, this is coming from the guy’s at UkFast.

So with the new dev server on the way the question is … what do you want to see?

What tutorials / videos would you be of intrest to you?

Post a comment with suggestions, and just so you are aware comments are on moderation due to the couple of thousand spam comments posted every day … so don’t worry if your comment doesn’t appear right away it’s just waiting for me to review it.

Cheers

Buzz

In Part 1

I covered the use of pmap to provide an insight into the memory footprint of the apache service.

In Part 2

I covered a list modules that are typically enabled in a standard httpd/apache setup, and listed which modules I was disabling to reduce the memory foot print.

After improving the appmem script in Part 4 of the linux sysadmin script I updated part 3 of this series with the new memory foot print figures.

Summary aside, now we have reduced the memory footprint of your apache installation lets go onto actually making the most of your new found memory excess ..’potential’ max clients.

As stated in part 3 of this series, I am assuming the server you are making these configuration changes is a dedicated apache server.

However you also need to be aware of your CGI programs such as PHP, if you are to configure PHP with a maximum memory of 32MB, you must be aware that there is the potential for PHP to try to use this much memory on each allocation, this is where “optimizing PHP gor high load sites” comes into play, a furture series I plan to write, for the time being however I am only covering the configuration of apache.

Your httpd.conf should have the following entries (or similar).


# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves

StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves

StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0




[buzz@buzz_srv ~]httpd -l
Compiled in modules:
core.c
prefork.c
http_core.c
mod_so.c


As can be seen above in my installation the part of the config I am interested in is prefork.c.

Now with my theorehtical max clients in part 3 hitting 800.5, lets first go for a safer figure, say 60%



StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 480
MaxClients 480
MaxRequestsPerChild 4000



From here use the script in part 1 of the Linux sysadmin script, to monitor your active http connections, and ‘tweak’ your configuration accordingly.

Remember you must only set the MaxClients and ServerLimit to a value you can safely contain in ram, swapping out to disk will cause a major slow down.

Part 3 itself is not incorrect, it correctly takes a memory footprint for each process running, the same as VIRT in top …

However in processes such as APACHE the VIRT memory is the size of all shared libraries, as correctly shown by pmap …

So what does this mean realy?

The memory usage is infact the following VIRT + RSS, where RSS is the resident set size, the RSS is a representation of the memory in use by the PID, and VIRT is shared between the child processes.


[buzz@buzz_srv ~]# ps aux | grep httpd | grep -v ‘grep’
root 16378 0.0 0.1 148640 3024 ? Ss Nov13 0:00 /usr/sbin/httpd
apache 20088 0.0 0.1 148640 3304 ? S Nov13 0:00 /usr/sbin/httpd
apache 20101 0.0 0.1 148640 3304 ? S Nov13 0:00 /usr/sbin/httpd
apache 20756 0.0 0.1 148640 3312 ? S Nov13 0:00 /usr/sbin/httpd
apache 20759 0.0 0.1 148640 3300 ? S Nov13 0:00 /usr/sbin/httpd
apache 20790 0.0 0.1 148640 3284 ? S Nov13 0:00 /usr/sbin/httpd
apache 20792 0.0 0.1 148640 3312 ? S Nov13 0:00 /usr/sbin/httpd
apache 20798 0.0 0.1 148640 3308 ? S Nov13 0:00 /usr/sbin/httpd
apache 20804 0.0 0.1 148640 3308 ? S Nov13 0:00 /usr/sbin/httpd
apache 20886 0.0 0.1 148640 3304 ? S Nov13 0:00 /usr/sbin/httpd
apache 20906 0.0 0.1 148640 3300 ? S Nov13 0:00 /usr/sbin/httpd
apache 20907 0.0 0.1 148640 3308 ? S Nov13 0:00 /usr/sbin/httpd
apache 20912 0.0 0.1 148640 3304 ? S Nov13 0:00 /usr/sbin/httpd
apache 20915 0.0 0.1 148640 3312 ? S Nov13 0:00 /usr/sbin/httpd
apache 20959 0.0 0.1 148640 3304 ? S Nov13 0:00 /usr/sbin/httpd
apache 20969 0.0 0.1 148640 3300 ? S Nov13 0:00 /usr/sbin/httpd
apache 20994 0.0 0.1 148640 3320 ? S Nov13 0:00 /usr/sbin/httpd
apache 20995 0.0 0.1 148640 3288 ? S Nov13 0:00 /usr/sbin/httpd
apache 20996 0.0 0.1 148640 3320 ? S Nov13 0:00 /usr/sbin/httpd
apache 20997 0.0 0.1 148640 3320 ? S Nov13 0:00 /usr/sbin/httpd
apache 20999 0.0 0.1 148640 3296 ? S Nov13 0:00 /usr/sbin/httpd


As can be seen above the ‘VIRT’ does not change between the child processes, where as the RSS does dependant on what the thread is doing at that time.

So below is an improved appmem function to allow for this:


function appmem {
if [ -z "$1" ]; then
echo “Usage: sysadmin appmem app_name i.e. (sysadmin appmem apache)”;
else
RRES=(`ps aux | grep “$1″ | grep -v ‘grep’ | grep -v “$0″ | awk ‘{print $6}’`);
VRES=(`ps aux | grep “$1″ | grep -v ‘grep’ | grep -v “$0″ | awk ‘{print $5}’`);
COUNT=0;
VMEM=0;
RMEM=0;
for RSS in ${RRES[@]}
do
RMEM=$(($RSS+$RMEM));
done;
for VIRT in ${VRES[@]}
do
VMEM=$(($VIRT+$VMEM));
COUNT=$(($COUNT+1));
done;
VMEM=$(($VMEM/$COUNT));
VMEM=$(($VMEM/1024));
RMEM=$(($RMEM/1024));
echo -e “$YELLOW —– MEMORY USAGE REPORT FOR ‘$1′ —– $CLEAR”;
echo “PID Count: $COUNT”;
echo “Shared Mem usage: $VMEM MB”;
echo “Total Resident Set Size: $RMEM MB”;
echo “Mem/PID: $(($RMEM/$COUNT)) MB”;
fi
}


Example output:


----- MEMORY USAGE REPORT FOR 'httpd' -----
PID Count: 41
Shared Mem usage: 140 MB
Total Resident Set Site: 95 MB
Mem/PID: 2 MB


• LoadModule auth_basic_module modules/mod_auth_basic.so

ENABLED: User authentication for access control using HTTP basic authentication (.htaccess and .htpasswd)

• LoadModule auth_digest_module modules/mod_auth_digest.so

ENABLED: Similar to auth_basic_module but instead of using a plain text authentication scheme, it uses a cryptographic one. (MD5 etc …)

• LoadModule authn_file_module modules/mod_authn_file.so

ENABLED: Allows authentication front-ends such as auth_digest_module and auth_basic_module to authenticate users by looking up users in plain text password files. This function was previously part of auth_module and auth_digest_module.

• LoadModule authn_alias_module modules/mod_authn_alias.so

DISABLED: Allows extended authentication providers to be created within the configuration file and assigned an alias name.

• LoadModule authn_anon_module modules/mod_authn_anon.so

DISABLED: Allows anonymous user access and logs the password given. Previously known asauth_anon_module.

• LoadModule authn_dbm_module modules/mod_authn_dbm.so

DISABLED: Allows authentication front-ends such as auth_digest_module and auth_basic_module to authenticate users by looking up users in dbm password files. Previously known asauth_dbm_module.

• LoadModule authn_default_module modules/mod_authn_default.so

DISABLED: Fallback authentication module - it simply rejects any credentials supplied by the user

• LoadModule authz_host_module modules/mod_authz_host.so

ENABLED: Access control by browser hostname. Previously known asaccess_module.

• LoadModule authz_user_module modules/mod_authz_user.so

DISABLED: Provides authorization capabilities so that authenticated users can be allowed or denied access to portions of the web site. This function was previously part of auth_module.

• LoadModule authz_owner_module modules/mod_authz_owner.so

DISABLED: Authorization based on file ownership.

• LoadModule authz_groupfile_module modules/mod_authz_groupfile.so

DISABLED: Group authorization using plaintext files. This function was previously part of auth_module.

• LoadModule authz_dbm_module modules/mod_authz_dbm.so

DISABLED: Group authorization using DBM files. This function was previously part of auth_dbm_module.

• LoadModule authz_default_module modules/mod_authz_default.so

DISABLED: Fallback authorisation module - it simply rejects any authorization request.

• LoadModule ldap_module modules/mod_ldap.so

DISABLED: LDAP connection pooling and result caching services for use by other LDAP modules.

(I don’t use LDAP, if you use apache interaction with ldap keep this module, if you use LDAP within PHP you can turn this mod off!)

• LoadModule authnz_ldap_module modules/mod_authnz_ldap.so

DISABLED: Allows authentication front-ends such as auth_basic_module to authenticate users through an ldap directory. Previously known asauth_ldap_module. (same as above)

• LoadModule include_module modules/mod_include.so

DISABLED: Server-side includes.

• LoadModule log_config_module modules/mod_log_config.so

ENABLED: Configurable logging of requests and reponses.

• LoadModule logio_module modules/mod_logio.so

ENABLED: Logging of input and output bytes per request.

• LoadModule env_module modules/mod_env.so

DISABLED: Changes the environment that CGI program are run in.

• LoadModule ext_filter_module modules/mod_ext_filter.so

ENABLED: Pass the response body through an external program before delivery to the client (i’m using mods like gzip compression, disalbe this if you don’t need it).

• LoadModule mime_magic_module modules/mod_mime_magic.so

ENABLED: Determines MIME types based on file contents. (Great for directly serving download easily).

• LoadModule expires_module modules/mod_expires.so

ENABLED: Autogenerates the Expires: header according to user rules.

• LoadModule deflate_module modules/mod_deflate.so

ENABLED: Compress content prior to serving it. (If you don’t have compression configured for your websites disable this mod).

• LoadModule headers_module modules/mod_headers.so

ENABLED: More general control of HTTP headers. (You can disable this if you do not want to handle bespoke HTTP headers)

• LoadModule usertrack_module modules/mod_usertrack.so

DISABLED: Provision of cookies. (I am using PHP to handle cookies).

• LoadModule setenvif_module modules/mod_setenvif.so

ENABLED: Sets the environment for CGI programs based on properties of the request.

• LoadModule mime_module modules/mod_mime.so

ENABLED: Determines MIME types based on file names.

• LoadModule dav_module modules/mod_dav.so

DISABLED: Distributed Authoring and Versioning functionality. (This is a webserver not serving a versioning system).

• LoadModule status_module modules/mod_status.so

ENABLED: Provides information about the server’s current status via a web request. (If you don’t have this set in your httpd.conf then disable the mod, I use this for apache monitoring).

• LoadModule autoindex_module modules/mod_autoindex.so

ENADLED: Automatically picks the index file when browsing to a folder, (index.html, index.php etc ..)

• LoadModule info_module modules/mod_info.so

DISABLED: Similar to a php info call, provides running configuration information about the apache server.

• LoadModule dav_fs_module modules/mod_dav_fs.so

DISABLED: More no required D.A.V

• LoadModule vhost_alias_module modules/mod_vhost_alias.so

ENABLED: Allows for handling enormous numbers of virtual hosts without having to change the configuration each time.

• LoadModule negotiation_module modules/mod_negotiation.so

ENABLED: Provides for content negotiation between server and client. (Accept encoding etc…)

• LoadModule dir_module modules/mod_dir.so

ENABLED: Supports the use of index.html files for directory lookups.

• LoadModule actions_module modules/mod_actions.so

ENABLED: Run specific CGI programs according to the MIME content type of the object served.

• LoadModule speling_module modules/mod_speling.so

ENABLED: Attempts to correct misspelled URLs. (Ignores case etc..)

• LoadModule userdir_module modules/mod_userdir.so

ENABLED: Allow public_html userdir and use of http://servername/~username, I have disabled this as can be used to verify the presence of a valid username to be used in a brute force attack

• LoadModule alias_module modules/mod_alias.so

ENABLED: Override the DocumentRoot directive for specific URLs.

• LoadModule rewrite_module modules/mod_rewrite.so

ENABLED: Allows the use of the Rewriteengine to provide SEO friends URL’s amoungst other things.

• LoadModule proxy_module modules/mod_proxy.so

ENABLED: Lets your web server be a proxy. Typically it needs additional modules for specific protocols.

• LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
• LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
• LoadModule proxy_http_module modules/mod_proxy_http.so
• LoadModule proxy_connect_module modules/mod_proxy_connect.so

ENABLED: As above, I have need to have apache act as a proxy, in the future

• LoadModule cache_module modules/mod_cache.so

ENABLED: Implements an RFC 2616 compliant HTTP content cache that can be used to cache either local or proxied content.

• LoadModule suexec_module modules/mod_suexec.so

ENABLED: Allows CGI scripts to run as a user other than wwwrun.

• LoadModule disk_cache_module modules/mod_disk_cache.so
• LoadModule file_cache_module modules/mod_file_cache.so
• LoadModule mem_cache_module modules/mod_mem_cache.so

ENABLED: More cache-ing functions (reduces need for disk read time).

• LoadModule cgi_module modules/mod_cgi.so

ENABLED: Run CGI programs

• LoadModule version_module modules/mod_version.so

ENABLED: Version dependent configuration (Can be handy when writing configs to have this enabled).

Source’s used:

Module information can be found HERE: http://www-uxsup.csx.cam.ac.uk/~jw35/courses/apache/html/a2617.htm


----- MEMORY USAGE REPORT FOR 'apache' -----
PID Count: 32
Mem usage: 4594 MB
Mem/PID: 143 MB
For more information run: pmap -x 12345


So 143mb down from 189mb, saving 46mb/thread.

Not much of a saving right? … well actually wrong.

Take for example a webserver with 2048MB Avail system ram, lets allocate 15% of that for the OS use.

So we have 1741MB to play with, with the original setup of 189mb 9.21 concurrent connections could be active an any one given time.

now however 12.71 concurrent connections can be in memory at any given time (28% increase).

Still a long way to go to bring the memory foot print down to a minimal level, but as you can see spending 5 minutes disabling mod’s in the config yields a quick win 28% increase in capacity.

UPDATE

Following the improvements to the appmem script in part 4 here are the new figures.

With all mods enabled:


----- MEMORY USAGE REPORT FOR 'httpd' -----
PID Count: 37
Shared Mem usage: 176 MB
Total Resident Set Size: 112 MB
Mem/PID: 3 MB


Results attained running “ab -n 1000 -c 100 http://xxx.xxx.xxx.xxx/”.

Results with disabled mods (listed above):


----- MEMORY USAGE REPORT FOR 'httpd' -----
PID Count: 41
Shared Mem usage: 140 MB
Total Resident Set Size: 95 MB
Mem/PID: 2 MB


Shared mem down 36MB (20.45% improvement), RSS/PID down 1mb (33% improvement)

Allowing for 1741MB availble ram, means 1601mb (1741MB -140MB shared) is available for apache threads, allowing for 800.5 concurrent connections with the improved config and by comparison

1565 MB Avail for threads (1741MB - 176MB), allowing 521.66 concurrent connections.

So with the improvements outlined above in theory the Apache server can now handle ~280 more concurrent connections.

So why would you want to reduce the memory footprint?

Simple really so you can have more concurrent connections to the apache server without having to constantly keep adding more memory to your web server.

This ‘tutorial’ does assume the following:

• You are running apache on a server dedicated to doing so.
• You have a basic understanding of how to install and configure apache.
• You have a basic understanding of bash scripting.
• You have the pmap package installed on your server.

As was introduced in part-3 the function appmem will report the memory usage of all current running threads of an application name, report the average memory usage, and an example pmap command using the last PID in the list.

This is where this entry will pick up, once pmap is run you will see the following example output:


----- MEMORY USAGE REPORT FOR 'apache' -----
PID Count: 6
Mem usage: 1135 MB
Mem/PID: 189 MB
For more information run: pmap -x 25883
------
[buzz@buzz_srv .sysadmin]# pmap -x 25883
25883: /usr/sbin/httpd
Address Kbytes RSS Anon Locked Mode Mapping
00002b777eac8000 308 - - - r-x– httpd
00002b777eb15000 100 - - - rw-s- zero (deleted)
00002b777eb2e000 88 - - - rw-s- zero (deleted)
00002b777ed14000 16 - - - rw— httpd
00002b777ed18000 12 - - - rw— [ anon ]
00002b777ed1b000 104 - - - r-x– ld-2.5.so
00002b777ed35000 4 - - - rw— [ anon ]
00002b777ed3f000 4 - - - rw— [ anon ]
00002b777ef35000 4 - - - r—- ld-2.5.so
00002b777ef36000 4 - - - rw— ld-2.5.so
00002b777ef37000 520 - - - r-x– libm-2.5.so
00002b777efb9000 2044 - - - —– libm-2.5.so
00002b777f1b8000 4 - - - r—- libm-2.5.so
00002b777f1b9000 4 - - - rw— libm-2.5.so
00002b777f1ba000 108 - - - r-x– libpcre.so.0.0.1
00002b777f1d5000 2048 - - - —– libpcre.so.0.0.1
00002b777f3d5000 4 - - - rw— libpcre.so.0.0.1
00002b777f3d6000 84 - - - r-x– libselinux.so.1
00002b777f3eb000 2048 - - - —– libselinux.so.1
00002b777f5eb000 8 - - - rw— libselinux.so.1
00002b777f5ed000 8 - - - rw— [ anon ]
00002b777f5ef000 112 - - - r-x– libaprutil-1.so.0.2.7
00002b777f60b000 2044 - - - —– libaprutil-1.so.0.2.7
00002b777f80a000 8 - - - rw— libaprutil-1.so.0.2.7
00002b777f80c000 36 - - - r-x– libcrypt-2.5.so
00002b777f815000 2044 - - - —– libcrypt-2.5.so
00002b777fa14000 4 - - - r—- libcrypt-2.5.so
00002b777fa15000 4 - - - rw— libcrypt-2.5.so
00002b777fa16000 184 - - - rw— [ anon ]
00002b777fa44000 224 - - - r-x– libldap-2.3.so.0.2.15
00002b777fa7c000 2048 - - - —– libldap-2.3.so.0.2.15
00002b777fc7c000 8 - - - rw— libldap-2.3.so.0.2.15
00002b777fc7e000 4 - - - rw— [ anon ]
00002b777fc7f000 52 - - - r-x– liblber-2.3.so.0.2.15
00002b777fc8c000 2048 - - - —– liblber-2.3.so.0.2.15
00002b777fe8c000 4 - - - rw— liblber-2.3.so.0.2.15
00002b777fe8d000 964 - - - r-x– libdb-4.3.so
00002b777ff7e000 2044 - - - —– libdb-4.3.so
00002b778017d000 20 - - - rw— libdb-4.3.so
00002b7780182000 128 - - - r-x– libexpat.so.0.5.0
00002b77801a2000 2044 - - - —– libexpat.so.0.5.0
00002b77803a1000 12 - - - rw— libexpat.so.0.5.0
00002b77803a4000 4 - - - rw— [ anon ]
00002b77803a5000 152 - - - r-x– libapr-1.so.0.2.7
00002b77803cb000 2048 - - - —– libapr-1.so.0.2.7
00002b77805cb000 4 - - - rw— libapr-1.so.0.2.7
00002b77805cc000 84 - - - r-x– libpthread-2.5.so
00002b77805e1000 2044 - - - —– libpthread-2.5.so
00002b77807e0000 4 - - - r—- libpthread-2.5.so
00002b77807e1000 4 - - - rw— libpthread-2.5.so
00002b77807e2000 16 - - - rw— [ anon ]
00002b77807e6000 8 - - - r-x– libdl-2.5.so
00002b77807e8000 2048 - - - —– libdl-2.5.so
00002b77809e8000 4 - - - r—- libdl-2.5.so
00002b77809e9000 4 - - - rw— libdl-2.5.so
00002b77809ea000 4 - - - rw— [ anon ]
00002b77809eb000 1320 - - - r-x– libc-2.5.so
00002b7780b35000 2044 - - - —– libc-2.5.so
00002b7780d34000 16 - - - r—- libc-2.5.so
00002b7780d38000 4 - - - rw— libc-2.5.so
00002b7780d39000 20 - - - rw— [ anon ]
00002b7780d3e000 236 - - - r-x– libsepol.so.1
00002b7780d79000 2048 - - - —– libsepol.so.1
00002b7780f79000 4 - - - rw— libsepol.so.1
00002b7780f7a000 40 - - - rw— [ anon ]
00002b7780f84000 12 - - - r-x– libuuid.so.1.2
00002b7780f87000 2048 - - - —– libuuid.so.1.2
00002b7781187000 4 - - - rw— libuuid.so.1.2
00002b7781188000 4 - - - rw— [ anon ]
00002b7781189000 128 - - - r-x– libpq.so.4.1
00002b77811a9000 2048 - - - —– libpq.so.4.1
00002b77813a9000 8 - - - rw— libpq.so.4.1
00002b77813ab000 352 - - - r-x– libsqlite3.so.0.8.6
00002b7781403000 2048 - - - —– libsqlite3.so.0.8.6
00002b7781603000 8 - - - rw— libsqlite3.so.0.8.6
00002b7781605000 68 - - - r-x– libresolv-2.5.so
00002b7781616000 2048 - - - —– libresolv-2.5.so
00002b7781816000 4 - - - r—- libresolv-2.5.so
00002b7781817000 4 - - - rw— libresolv-2.5.so
00002b7781818000 12 - - - rw— [ anon ]
00002b778181b000 96 - - - r-x– libsasl2.so.2.0.22
00002b7781833000 2048 - - - —– libsasl2.so.2.0.22
00002b7781a33000 4 - - - rw— libsasl2.so.2.0.22
00002b7781a34000 268 - - - r-x– libssl.so.0.9.8b
00002b7781a77000 2048 - - - —– libssl.so.0.9.8b
00002b7781c77000 24 - - - rw— libssl.so.0.9.8b
00002b7781c7d000 1172 - - - r-x– libcrypto.so.0.9.8b
00002b7781da2000 2048 - - - —– libcrypto.so.0.9.8b
00002b7781fa2000 124 - - - rw— libcrypto.so.0.9.8b
00002b7781fc1000 20 - - - rw— [ anon ]
00002b7781fc6000 572 - - - r-x– libkrb5.so.3.3
00002b7782055000 2048 - - - —– libkrb5.so.3.3
00002b7782255000 16 - - - rw— libkrb5.so.3.3
00002b7782259000 84 - - - r-x– libnsl-2.5.so
00002b778226e000 2044 - - - —– libnsl-2.5.so
00002b778246d000 4 - - - r—- libnsl-2.5.so
00002b778246e000 4 - - - rw— libnsl-2.5.so
00002b778246f000 12 - - - rw— [ anon ]
00002b7782472000 176 - - - r-x– libgssapi_krb5.so.2.2
00002b778249e000 2048 - - - —– libgssapi_krb5.so.2.2
00002b778269e000 8 - - - rw— libgssapi_krb5.so.2.2
00002b77826a0000 8 - - - r-x– libcom_err.so.2.1
00002b77826a2000 2044 - - - —– libcom_err.so.2.1
00002b77828a1000 4 - - - rw— libcom_err.so.2.1
00002b77828a2000 144 - - - r-x– libk5crypto.so.3.1
00002b77828c6000 2044 - - - —– libk5crypto.so.3.1
00002b7782ac5000 8 - - - rw— libk5crypto.so.3.1
00002b7782ac7000 4 - - - rw— [ anon ]
00002b7782ac8000 80 - - - r-x– libz.so.1.2.3
00002b7782adc000 2044 - - - —– libz.so.1.2.3
00002b7782cdb000 4 - - - rw— libz.so.1.2.3
00002b7782cdc000 32 - - - r-x– libkrb5support.so.0.1
00002b7782ce4000 2044 - - - —– libkrb5support.so.0.1
00002b7782ee3000 4 - - - rw— libkrb5support.so.0.1
00002b7782ee4000 8 - - - r-x– libkeyutils-1.2.so
00002b7782ee6000 2044 - - - —– libkeyutils-1.2.so
00002b77830e5000 4 - - - rw— libkeyutils-1.2.so
00002b77830e6000 12 - - - rw— [ anon ]
00002b77830e9000 8 - - - r-x– mod_auth_basic.so
00002b77830eb000 2044 - - - —– mod_auth_basic.so
00002b77832ea000 8 - - - rw— mod_auth_basic.so
00002b77832ec000 24 - - - r-x– mod_auth_digest.so
00002b77832f2000 2044 - - - —– mod_auth_digest.so
00002b77834f1000 8 - - - rw— mod_auth_digest.so
00002b77834f3000 8 - - - r-x– mod_authn_file.so
00002b77834f5000 2044 - - - —– mod_authn_file.so
00002b77836f4000 8 - - - rw— mod_authn_file.so
00002b77836f6000 8 - - - r-x– mod_authn_alias.so
00002b77836f8000 2044 - - - —– mod_authn_alias.so
00002b77838f7000 8 - - - rw— mod_authn_alias.so
00002b77838f9000 4 - - - r-x– mod_authn_anon.so
00002b77838fa000 2048 - - - —– mod_authn_anon.so
00002b7783afa000 8 - - - rw— mod_authn_anon.so
00002b7783afc000 4 - - - r-x– mod_authn_dbm.so
00002b7783afd000 2048 - - - —– mod_authn_dbm.so
00002b7783cfd000 8 - - - rw— mod_authn_dbm.so
00002b7783cff000 4 - - - r-x– mod_authn_default.so
00002b7783d00000 2044 - - - —– mod_authn_default.so
00002b7783eff000 8 - - - rw— mod_authn_default.so
00002b7783f01000 8 - - - r-x– mod_authz_host.so
00002b7783f03000 2044 - - - —– mod_authz_host.so
00002b7784102000 8 - - - rw— mod_authz_host.so
00002b7784104000 4 - - - r-x– mod_authz_user.so
00002b7784105000 2044 - - - —– mod_authz_user.so
00002b7784304000 8 - - - rw— mod_authz_user.so
00002b7784306000 8 - - - r-x– mod_authz_owner.so
00002b7784308000 2044 - - - —– mod_authz_owner.so
00002b7784507000 8 - - - rw— mod_authz_owner.so
00002b7784509000 8 - - - r-x– mod_authz_groupfile.so
00002b778450b000 2044 - - - —– mod_authz_groupfile.so
00002b778470a000 8 - - - rw— mod_authz_groupfile.so
00002b778470c000 8 - - - r-x– mod_authz_dbm.so
00002b778470e000 2044 - - - —– mod_authz_dbm.so
00002b778490d000 8 - - - rw— mod_authz_dbm.so
00002b778490f000 4 - - - r-x– mod_authz_default.so
00002b7784910000 2044 - - - —– mod_authz_default.so
00002b7784b0f000 8 - - - rw— mod_authz_default.so
00002b7784b11000 44 - - - r-x– mod_ldap.so
00002b7784b1c000 2044 - - - —– mod_ldap.so
00002b7784d1b000 8 - - - rw— mod_ldap.so
00002b7784d1d000 24 - - - r-x– mod_authnz_ldap.so
00002b7784d23000 2044 - - - —– mod_authnz_ldap.so
00002b7784f22000 8 - - - rw— mod_authnz_ldap.so
00002b7784f24000 36 - - - r-x– mod_include.so
00002b7784f2d000 2048 - - - —– mod_include.so
00002b778512d000 8 - - - rw— mod_include.so
00002b778512f000 20 - - - r-x– mod_log_config.so
00002b7785134000 2044 - - - —– mod_log_config.so
00002b7785333000 8 - - - rw— mod_log_config.so
00002b7785335000 8 - - - r-x– mod_logio.so
00002b7785337000 2044 - - - —– mod_logio.so
00002b7785536000 8 - - - rw— mod_logio.so
00002b7785538000 4 - - - r-x– mod_env.so
00002b7785539000 2048 - - - —– mod_env.so
00002b7785739000 8 - - - rw— mod_env.so
00002b778573b000 16 - - - r-x– mod_ext_filter.so
00002b778573f000 2044 - - - —– mod_ext_filter.so
00002b778593e000 8 - - - rw— mod_ext_filter.so
00002b7785940000 20 - - - r-x– mod_mime_magic.so
00002b7785945000 2044 - - - —– mod_mime_magic.so
00002b7785b44000 8 - - - rw— mod_mime_magic.so
00002b7785b46000 8 - - - r-x– mod_expires.so
00002b7785b48000 2044 - - - —– mod_expires.so
00002b7785d47000 8 - - - rw— mod_expires.so
00002b7785d49000 16 - - - r-x– mod_deflate.so
00002b7785d4d000 2044 - - - —– mod_deflate.so
00002b7785f4c000 8 - - - rw— mod_deflate.so
00002b7785f4e000 12 - - - r-x– mod_headers.so
00002b7785f51000 2044 - - - —– mod_headers.so
00002b7786150000 8 - - - rw— mod_headers.so
00002b7786152000 8 - - - r-x– mod_usertrack.so
00002b7786154000 2048 - - - —– mod_usertrack.so
00002b7786354000 8 - - - rw— mod_usertrack.so
00002b7786356000 8 - - - r-x– mod_setenvif.so
00002b7786358000 2048 - - - —– mod_setenvif.so
00002b7786558000 8 - - - rw— mod_setenvif.so
00002b778655a000 16 - - - r-x– mod_mime.so
00002b778655e000 2044 - - - —– mod_mime.so
00002b778675d000 8 - - - rw— mod_mime.so
00002b778675f000 84 - - - r-x– mod_dav.so
00002b7786774000 2044 - - - —– mod_dav.so
00002b7786973000 8 - - - rw— mod_dav.so
00002b7786975000 16 - - - r-x– mod_status.so
00002b7786979000 2044 - - - —– mod_status.so
00002b7786b78000 8 - - - rw— mod_status.so
00002b7786b7a000 32 - - - r-x– mod_autoindex.so
00002b7786b82000 2044 - - - —– mod_autoindex.so
00002b7786d81000 8 - - - rw— mod_autoindex.so
00002b7786d83000 16 - - - r-x– mod_info.so
00002b7786d87000 2044 - - - —– mod_info.so
00002b7786f86000 8 - - - rw— mod_info.so
00002b7786f88000 44 - - - r-x– mod_dav_fs.so
00002b7786f93000 2044 - - - —– mod_dav_fs.so
00002b7787192000 8 - - - rw— mod_dav_fs.so
00002b7787194000 8 - - - r-x– mod_vhost_alias.so
00002b7787196000 2044 - - - —– mod_vhost_alias.so
00002b7787395000 8 - - - rw— mod_vhost_alias.so
00002b7787397000 28 - - - r-x– mod_negotiation.so
00002b778739e000 2044 - - - —– mod_negotiation.so
00002b778759d000 8 - - - rw— mod_negotiation.so
00002b778759f000 8 - - - r-x– mod_dir.so
00002b77875a1000 2044 - - - —– mod_dir.so
00002b77877a0000 8 - - - rw— mod_dir.so
00002b77877a2000 8 - - - r-x– mod_actions.so
00002b77877a4000 2044 - - - —– mod_actions.so
00002b77879a3000 8 - - - rw— mod_actions.so
00002b77879a5000 8 - - - r-x– mod_speling.so
00002b77879a7000 2048 - - - —– mod_speling.so
00002b7787ba7000 8 - - - rw— mod_speling.so
00002b7787ba9000 8 - - - r-x– mod_userdir.so
00002b7787bab000 2044 - - - —– mod_userdir.so
00002b7787daa000 8 - - - rw— mod_userdir.so
00002b7787dac000 12 - - - r-x– mod_alias.so
00002b7787daf000 2044 - - - —– mod_alias.so
00002b7787fae000 8 - - - rw— mod_alias.so
00002b7787fb0000 56 - - - r-x– mod_rewrite.so
00002b7787fbe000 2044 - - - —– mod_rewrite.so
00002b77881bd000 8 - - - rw— mod_rewrite.so
00002b77881bf000 64 - - - r-x– mod_proxy.so
00002b77881cf000 2044 - - - —– mod_proxy.so
00002b77883ce000 8 - - - rw— mod_proxy.so
00002b77883d0000 20 - - - r-x– mod_proxy_balancer.so
00002b77883d5000 2044 - - - —– mod_proxy_balancer.so
00002b77885d4000 8 - - - rw— mod_proxy_balancer.so
00002b77885d6000 28 - - - r-x– mod_proxy_ftp.so
00002b77885dd000 2044 - - - —– mod_proxy_ftp.so
00002b77887dc000 8 - - - rw— mod_proxy_ftp.so
00002b77887de000 24 - - - r-x– mod_proxy_http.so
00002b77887e4000 2044 - - - —– mod_proxy_http.so
00002b77889e3000 8 - - - rw— mod_proxy_http.so
00002b77889e5000 8 - - - r-x– mod_proxy_connect.so
00002b77889e7000 2044 - - - —– mod_proxy_connect.so
00002b7788be6000 8 - - - rw— mod_proxy_connect.so
00002b7788be8000 28 - - - r-x– mod_cache.so
00002b7788bef000 2044 - - - —– mod_cache.so
00002b7788dee000 8 - - - rw— mod_cache.so
00002b7788df0000 4 - - - r-x– mod_suexec.so
00002b7788df1000 2048 - - - —– mod_suexec.so
00002b7788ff1000 8 - - - rw— mod_suexec.so
00002b7788ff3000 16 - - - r-x– mod_disk_cache.so
00002b7788ff7000 2048 - - - —– mod_disk_cache.so
00002b77891f7000 8 - - - rw— mod_disk_cache.so
00002b77891f9000 8 - - - r-x– mod_file_cache.so
00002b77891fb000 2044 - - - —– mod_file_cache.so
00002b77893fa000 8 - - - rw— mod_file_cache.so
00002b77893fc000 24 - - - r-x– mod_mem_cache.so
00002b7789402000 2044 - - - —– mod_mem_cache.so
00002b7789601000 8 - - - rw— mod_mem_cache.so
00002b7789603000 20 - - - r-x– mod_cgi.so
00002b7789608000 2048 - - - —– mod_cgi.so
00002b7789808000 8 - - - rw— mod_cgi.so
00002b778980a000 8 - - - r-x– mod_version.so
00002b778980c000 2044 - - - —– mod_version.so
00002b7789a0b000 8 - - - rw— mod_version.so
00002b7789a0d000 28 - - - r-x– mod_proxy_ajp.so
00002b7789a14000 2044 - - - —– mod_proxy_ajp.so
00002b7789c13000 8 - - - rw— mod_proxy_ajp.so
00002b7789c15000 16 - - - r-x– mod_suphp.so
00002b7789c19000 2044 - - - —– mod_suphp.so
00002b7789e18000 4 - - - rw— mod_suphp.so
00002b7789e22000 40 - - - r-x– libnss_files-2.5.so
00002b7789e2c000 2044 - - - —– libnss_files-2.5.so
00002b778a02b000 4 - - - r—- libnss_files-2.5.so
00002b778a02c000 4 - - - rw— libnss_files-2.5.so
00002b778a02d000 16 - - - r-x– libnss_dns-2.5.so
00002b778a031000 2044 - - - —– libnss_dns-2.5.so
00002b778a230000 4 - - - r—- libnss_dns-2.5.so
00002b778a231000 4 - - - rw— libnss_dns-2.5.so
00002b778b815000 1476 - - - rw— [ anon ]
00007fff2bfcd000 84 - - - rw— [ stack ]
ffffffffff600000 8192 - - - —– [ anon ]
—————- —— —— —— ——
total kB 193636 - - -

—–


As can be seen above via the script and pmap output, the majority of apache’s memory footprint is taken up by dynamicaly loaded libraries or ‘mods’.

Take for example mod_authz_ldap, this module itself on each thread spawn (new connection) is taking up ~2mb of physical memory space, now this may not seem like much until you are serving 100’s of concurrent connections, at which point this becomes a very real problem.

mod_authz_ldap I have used as an example because it is an authentication module for use with the Lightweight Directory Access Protocol, something not commonly used in a generic LAMP setup.

By going through your httpd.conf and disabling modules you know you do not require (such as the example above), you can substantially reduce the memory footprint of apache, and therefor increase the number of concurrent connections apache can serve with the available memory.

In part 2 I will provide a new ouput of the script and pmap post optimization … stay tuned

So if you have found anything usefull on this blog please vote for me here

“David Busby”

http://www.firebrandtraining.co.uk/itsuperheroes/

In part 3, I am going to cover a bash function that will allow you to profile the memory usage of any application by name.

By adding the function below into your script you can execute a command such as: sysadmin appmem apache


function appmem {
if [ -z "$1" ]; then
echo “Usage: sysadmin appmem app_name i.e. (sysadmin appmem apache)”;
else
if [ -x '/usr/bin/pmap' ]; then
APID=(`ps aux | grep “$1″ | grep -v ‘grep’ | grep -v “$0″ | awk ‘{print $2}’`);
COUNT=0;
AMEM=0
for PID in ${APID[@]}
do
TMP=$((`pmap -x $PID | grep “total” | awk ‘{print $3}’`));
AMEM=$(($AMEM+$TMP));
COUNT=$(($COUNT+1));
done
AMEM=$(($AMEM/1024));
echo -e “$YELLOW —– MEMORY USAGE REPORT FOR ‘$1′ —– $CLEAR”;
echo “PID Count: $COUNT”;
echo “Mem usage: $AMEM MB”;
echo “Mem/PID: $(($AMEM/$COUNT)) MB”;
echo -e “$RED”
echo -e “For more information run: pmap -x $PID $CLEAR”;
else
echo ‘Could not execute /usr/bin/pmap … aborting’;
exit;
fi
fi
}

Sample output:


—– MEMORY USAGE REPORT FOR ‘apache’ —–
PID Count: 6
Mem usage: 1134 MB
Mem/PID: 189 MB

For more information run: pmap -x 123456


You can of course replace ‘apache’ with the application or daemon name you want to profile the memory usage of.

This script does require that pmap is installed, if the script can not find it, it will abort.

As always any problems, post a comment.

UPDATE: Apparently I need to point out that if you haven’t read PART 2! then the colored output will not work … That’s why this entry is titled part 3, it does assume a degree of competence on your part in realizing part’s 1 and 2 may just be required reading …

NOTE: The above provides a complete memory footprint of the indvidual PID, the same as VIRT in top.

VIRT — Virtual Image (kb)
* The total amount of virtual memory used by the task. It includes all code, data and shared libraries plus pages that have been swapped out.
* VIRT = SWAP + RES

However as of mySQL 5.x or higher you can issue a statement in the my.cnf file calling init_connect.

This will trigger a series of defined commands / queries every time a non super user connects (So if you are using root to connect to your mySQL database, stop reading now and slap yourself HARD).

i.e.

[mysqld]
init_connect=’SET collation_connection = utf8_general_ci’
init_connect=’SET NAMES utf8′
default-character-set=utf8
character-set-server=utf8
collation-server=utf8_general_ci

Restart mySQL and check the mysqld.log has not returned any errors (Or your event viewer if you are using windows).

Every client connection will now default to utf-8 encoding and not latin-1, removing the need to add a SET NAMES call on every connection.

This will work for PHP, C++, ruby etc… as the client encoding is now handeled server side, rather that waiting on the client to issue a SET NAMES command.

Onto the point, at the moment I am maintaining 2 different installations of eclipse … one for PHP, C++.

Wouldn’t it be great if I could have both of these in one happy installation? … Yeh it would however getting all the dependencies is an utter nightmare … unless you have a program do it for you.

Long story short: http://ondemand.yoxos.com/geteclipse/start

Customize your eclipse before you download it.

(Thanks Austin!)

In this part I will be covering the concepts of secured shared hosting, and why you as a shared hosting provider should be taking steps to ensure this is how you deploy your hosting environments.

Let’s first take a typical L.A.M.P setup:

PHP Compiled from source as apache module.
mySQL installed from RPM or update package (yum / up2date).
HTTPD installed as RPM or update package (yum / up2date).

Please note at the time of writing if you yum / apt-get / up2date install your PHP package you will have varying results when attempting to compile and install suPHP, as such grab the source code from php.net, and follow this series.

As a shared hosting provider lets say you have 5 clients all hosted from the one server, each client using vsftpd is chrooted() into their home directory, and their ssh access disabled, supposedly secure enough.

Unfortunatly not so, due to the L.A.M.P configuration the ‘apache’ user needs a minimum of read and execute permissions over all the PHP files on the system, why is this a problem?

This is a problem largely due to human nature of the client, your ‘joe bloggs’ client doesn’t care about the technical aspects of web hosting or websites, they just want an easy pretty interface to get their corner of the internet online, downloading something like drupal or joomla.

Now this isn’t a dig at open source CMS, this is an insight into human nature, look at the changelog for any open CMS and you will see ’security fixes’, unfortunatly all ‘joe bloggs’ cares about is that their website is working, and this is wher things take a turn for the worse.

Joe Bloggs never updates his open CMS platform, meaning any vulnerabilities patched in subsequent releases are still exploitable on his website, worst case scenario that this is an XSSI (Cross Server Script Includes) vulnerbility.

An attacker finds this website and idetifies the security hole, using XSSI to install a PHP interactive shell, giving the attacker SSH like access to the hosting environment, most people at this point think so the attacker has compromise one site … so what we can restore that site from backups and it’s only one site that’s affected, the other 4 users either do not use open CMS or are up to date with all the security patches.

Well that’s where you would be wrong, with the hosting setup outlined above the SSH like PHP shell is now running as the apache user, meaning the attacker can go anywhere and read anything apache can, and with the hosting setup oulined above that mean reading things like datbase connection files, suddenly all the clients on the hosting environment have their websites compromised as the attacker gains mySQL access and starts changing content on thewebsites, despite the fact that the other 4 sites themselves were never exploited.

One clients error just became a cascading exploit on your hosting platform, now make that a more realistic platform say 30 clients on the box, some are online shops, the issue just became a whole lot bigger there is lost revenue due to downtime of the shop sites, and worse still the attacker now has access to any customer details those shops were storing! but it’s not Joe Bloggs that’s accountable it’s YOU as the hosting provider, you can take steps to prevent one exploited site becoming 30, and this web series will tell you host to do it.

coming in part 2:

an introduction to suPHP
compiling php as a cgi binary, and why you need to do so