Saiweb

Sounds simple enough, right?

Use a cache to serve pages faster, well yes that is true but people often do not realize the fundamentals of caching and how if not done properly it can lead to a detriment in performance.

The first thing you need to realize that by caching your content is no longer dynamic, … (short pause while we wait for the outrage in the back to die down).

The whole point behind your cache is that it will be used instead of processing all your code, why this is beneficial?

You have to remember that PHP is an interpreted language, meaning it takes the following I/O flow:

Apache -> mod_php -> Script -> Interpreter -> Bytecode -> Execution -> Output Buffer

Now there are two types of caching to consider, the first is completion output caching, this also yields the best performance, the second is opcode caching, this caches the byte code generated by the interpreter thus removing that step from the chain of execution.

With me so far? Ok take a deep breath because here we go …

Output caching

This option often yields the best performance, but at the cost of removing the dynamic element from your web app.
But this can be summed up in a single line: What good is dynamic content if you can serve all of 5% of your audience at a given time?

Another turn of phrase is “The slashdot effect”, there are many options for output caching, and you should ideally provide gziped and plain cache files to your end user, for instance on this blog I use WP Super Cache, and can high recommend it, as new content is posted the relevant caches are regenerated, if you are writing your own WebApp check for the “Accept-Encoding:gzip” header being sent via the users browser.

For end user transparency couple this with some mod_rewrite voodoo

1: If gzip is supported
2: and the cache file exists
3: Redirect visitor to compressed cached file

You “chain of execution” is now

Apache -> readfile

To serve non gziped content:

Now to clarify a point you should not be caching images,css,js etc, we’re only covering dynamic content here, and the above are only examples to get you started, you should write rules to exclude certain content specific to your needs.

And before going of at any more of a tangent, here are some figures for you!

ab -c 100 -n 500 -g ./saiweb-nocache-nogzip.bpl http://www.saiweb.co.uk/

• No caching
• No Gzip

Server Hostname: www.saiweb.co.uk
Server Port: 80

Document Path: /
Document Length: 109086 bytes

Concurrency Level: 100
Time taken for tests: 123.304 seconds
Complete requests: 500
Failed requests: 0
Write errors: 0
Total transferred: 54831652 bytes
HTML transferred: 54692607 bytes
Requests per second: 4.06 [#/sec] (mean)
Time per request: 24660.828 [ms] (mean)
Time per request: 246.608 [ms] (mean, across all concurrent requests)
Transfer rate: 434.26 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 57 423 225.5 374 1837
Processing: 2331 20460 16701.2 17232 115192
Waiting: 270 1835 4155.8 576 38549
Total: 2656 20882 16648.1 17692 115421

Percentage of the requests served within a certain time (ms)
50% 17692
66% 20700
75% 24063
80% 25770
90% 35157
95% 53328
98% 82957
99% 101497
100% 115421 (longest request)

As can be seen as the number of requests grew the response time began to increase sharply and the overall performace of the site degrade, bare in mind these benchmarks are being made on my home DSL for the time being.

ab -c 100 -n 500 -g ./saiweb-cached.bpl http://www.saiweb.co.uk/

Server Hostname: www.saiweb.co.uk
Server Port: 80

Document Path: /
Document Length: 109086 bytes

Concurrency Level: 100
Time taken for tests: 79.212 seconds
Complete requests: 500
Failed requests: 0
Write errors: 0
Total transferred: 54889292 bytes
HTML transferred: 54705058 bytes
Requests per second: 6.31 [#/sec] (mean)
Time per request: 15842.342 [ms] (mean)
Time per request: 158.423 [ms] (mean, across all concurrent requests)
Transfer rate: 676.70 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 56 314 112.5 322 1341
Processing: 2545 14721 5116.7 14296 36677
Waiting: 216 1283 2228.2 351 13776
Total: 2647 15035 5108.9 14624 36897

Percentage of the requests served within a certain time (ms)
50% 14624
66% 16675
75% 18058
80% 19093
90% 21608
95% 23489
98% 27684
99% 29972
100% 36897 (longest request)

A much more consistent line here, however as you can clearly see response times are roughly equal this is due to my DSL connection, so lets run these tests from somewhere with a little more bandwidth say the webserver itself using a loop back connection.

ab -c 100 -n 500 -g ./saiweb-cached.bpl http://www.saiweb.co.uk/

Server Hostname: www.saiweb.co.uk
Server Port: 80

Document Path: /
Document Length: 109086 bytes

Concurrency Level: 100
Time taken for tests: 0.262199 seconds
Complete requests: 500
Failed requests: 0
Write errors: 0
Total transferred: 54945406 bytes
HTML transferred: 54761172 bytes
Requests per second: 1906.95 [#/sec] (mean)
Time per request: 52.440 [ms] (mean)
Time per request: 0.524 [ms] (mean, across all concurrent requests)
Transfer rate: 204642.27 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 2.6 0 9
Processing: 4 45 10.3 49 58
Waiting: 1 38 9.9 41 50
Total: 9 47 9.5 50 64

Percentage of the requests served within a certain time (ms)
50% 50
66% 51
75% 52
80% 52
90% 54
95% 56
98% 59
99% 61
100% 64 (longest request)

In this case the response times rise and then plateau, no after which no further degradation occurs.

ab -c 100 -n 500 -g ./saiweb-nocache.bpl http://www.saiweb.co.uk/

Server Hostname: www.saiweb.co.uk
Server Port: 80

Document Path: /
Document Length: 109086 bytes

Concurrency Level: 100
Time taken for tests: 8.919565 seconds
Complete requests: 500
Failed requests: 0
Write errors: 0
Total transferred: 54680788 bytes
HTML transferred: 54543000 bytes
Requests per second: 56.06 [#/sec] (mean)
Time per request: 1783.913 [ms] (mean)
Time per request: 17.839 [ms] (mean, across all concurrent requests)
Transfer rate: 5986.73 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 14 30.7 0 85
Processing: 246 1556 714.3 1365 6735
Waiting: 241 1539 707.8 1360 6731
Total: 250 1571 708.0 1368 6735

Percentage of the requests served within a certain time (ms)
50% 1368
66% 1451
75% 1550
80% 1700
90% 2658
95% 3121
98% 3491
99% 3638
100% 6735 (longest request)

Oh dear of dear lets cut to the hard facts shall we?

We’ve gone from serving 1906.95 requests a second to 56.06

• a 97.1% decrease in performance when removing caching
• or a 3401.1% increase in performance when implementing caching

We’ve gone from a response time of ~50ms to ~2000ms

• a 97.5% decrease in performance when removing caching
• or a 4000% increase in performance when caching is on

Then there is the CPU an memory overheads to consider, in this case a more prolonged test is required to gain the relevant sar data,
now let me tell you that intentionally trying to get a test like this to run over a 10 minute period with the correct caching on is a lot harder than it sounds, the tests infact were completing far too quickly …

The problem I face is to make ab perform a long enough timed duration of results cached, I know for a fact uncached the server will fail under the load, so I have no way at present of grabbing this reliably,

what I can tell you is that this command: ab -c 300 -n 1000000 -g ./saiweb-cached.bpl http://www.saiweb.co.uk/

caused a load average of 2.96, 1.9,0.93 cache, and got as high as 21 before I killed it uncached.

Now I am going to bring this post to an end as it is getting quiet long, I plan to cover the following in a 2nd part.

1. Opcode caching
2. CPU & Memory usage, Cached vs. UNcached

Before you read any further note, I will not be including the original hack file, simply due to peoples stupidity in putting this on a production environment to play with, if you use the code you do so at your own risk, and by reading this blog entry / using the code provided you agree to accept all liability upon yourself for your own actions. Don’t be an idiot.

Around 10 days ago I came across this seemingly innocuous little file.

What I am going to cover in this entry is dissecting the ‘payload’ and not so much the web app in question or methods used to compromise it,

Whereas I will not at this time provide the original file, I will provide you with the md5 and sha1 hashes of the file so you can check it’s not lurking on your systems:

md5: 9ee3e6523d154114460d320477a8665a
sha1: 9c64fecea5620d70a716bbd74f6e89612a4a79c7

The bit we are interested in is the last line of the file:

Were you to run this line you would get

Confused yet? now I can appreciate the thinking behind packing a payload to avoid detection, but in this case the payload is packed 12 times, and no before you ask I did not manually run each returned statement to find this out.

Enter Python-Fu:

Copy the first payload lines into a file named payload.raw, take the above code and copy it into a file named dissect.py.

When dissect.py is run you will get the following output:

As such you may want to run it using the following command:

And what you will find after unpacking 12 times in total, the “payload” is the r57shell, this script is an information gathering tool and pseudo shell, meaning it will run any command on the host server that php can, providing in most cases ssh esq access to the exploited host, allowing you to do pretty much anything you wanted at this point, some of the features also include /etc/passwd /etc/shadow dumping, aswell as searching for a tirade of common file *.sql* admin* etc, it’s a one stop script for information gathering on a LAMP/WAMP based host.

Defense: modify php.ini to disable eval(), exec, shell_exec and all none essential functions.

And of course, ensure your web apps are patched and up to date as well as the host they are running on.

For security newer distros of RHEL and their derivatives an mounting /tmp with the noexec option.

Now if you have ever had to clean up a compromised web app you can see why this makes a lot of sense, and if not here’s a quick example.

Yours/Clients web app becomes compromised, running kernel has a buffer overflow that can lead to privilege escalation, attack writes out their code and compiles in /tmp, then runs said app from /tmp creating a pseudo root level shell, aka you’ve just been root kitted.

However there are legitimate reasons for using /tmp to compile, well I say legitimate, what I in fact mean is things like pecl, which you use to install extensions like APC require this …

workaround:

Failing that:

service httpd stop

DO NOT ALLOW ANY WEBAPP ACCESS WHILE NOEXEC IS IN USE!

DO NOT REMOVE THE NOEXEC OPTION IN /ETC/FSTAB PERMANENTLY YOU WILL REGRET DOING SO

Redhat bug 537535

Take for instance this code saved as test.php.png

Low and behold this will render out the entire post array! and will interpret the php itself, now lets be clear here the proper use of selinux and directory structures to prevent UGC from being allowed to be access directly and / or run arbitrary code would of prevented this, however as is often the case the setup is such that the preventative conditions could not / are not deployed.

At any rate this bug comes courtesy of the apache AddHandler directive,

The statement above seems to ‘loose’ match the .php extension meaning a file simply only contain .php anywhere in it’s filename to be interpreted as PHP.

The suggested work around for this is as follows:

Note this does not effect the AddType directive, after testing on the same version using:

Is not effected by this ‘bug’.

Because a picture is worth over 9000 internets … apparently

UPDATE: AKA “hayabusayuri” link … seriously who plays everquest? … maybe all that time playing everquest finally made the guy snap … and PHP & windows … never a good combination … infact Windows and internet is a bad combination

LINKY

Screencap incase it is removed:

(Thanks to the guys who forwarded me this)

Just as a warning and as a poke to say WHY are you not running PHP 5.x yet.

Parsing “” and it seems some multibyte chars to html_entity_decode() in PHP 4.3.10 will cause it to crash, returning random memory contents.

In my case some contents in memory from other sites running on the box were returned.

If you’ve seen the new twitter feed to the right you may of seem some ramblings about ‘cura’.

Cura is a PHP class I have authored in Co-operation with Psycle Interactive (The company I now work for, so be sure to thank them for allowing me to publish this write up!)

So what does it do?
Cura sets several call back objects in your PHP application that re-directs all session data to a mySQL database.

But why do I need that?
The average 1 server end user can stop reading here, as I can tell you now that Cura is not for you.

If however you are a business fielding multiple web servers then read on.

By passing all your PHP sessions to a database you remove the work around requirements for a load balanced solution.

i.e. web1 web2

1) Shopper arrives at web1 and logs in.
2) Shopper adds item to cart, which is logged against their session.
3) web1 is subjected to a search engine index.
4) web2 is now serving the shopper, shoppers basket is now logged out as their session id has changed …

There are numerous work around methods for this, such as having a single shared mount point for the PHP session files, the use of cookies etc …

The problem is in a high availability solution that a single mount point is just that, it’s singular and therefor a single point of failure.

Then there is the use of cookies, which is fine until you start to store a lot of data during your users session, at which point on each server change you are reliant on the cookie data being transmitted back to the server each time, raising the question what is the point of adding a load balanced solution if the user experience becomes degraded due to it’s deployment?

So secret option number 3 is to use a database, you can remove the single point of failure by having a mySQL cluster, and you haven’t got to worry too much about how much data you are storing.

Because everything is in a database whenever your web application is run (web1, web2) it will read the data from one source, allowing persistent sessions across your whole platform without the need for single mount points or session replication.

The source files are available from: http://svn.saiweb.co.uk/branches/cura-php/trunk/

To deploy this solution simply add the following lines to any file that calls session_start();

Ensure that you have created a ‘sessions’ table as per the provided sessions.sql file in your database.

I will be adding simplified support for wordpress and joomla shortly these will become available from: http://svn.saiweb.co.uk/branches/cura-php/trunk/

The problem that most people face when setting up a UTF-8 database in mySQL is that without calling ‘SET NAMES’ in the mySQL client prior to issuing any queries (PHP, C++ etc …) that the client connection will actually in most cases default to  latin-1.

However as of mySQL 5.x or higher you can issue a statement in the my.cnf file calling init_connect.

This will trigger a series of defined commands / queries every time a non super user connects (So if you are using root to connect to your mySQL database, stop reading now and slap yourself HARD).

i.e.

UPDATE 04/09/09

my mySQL version 5.0.45 x64 only picks up the last entry of init_connect

Use this example in this case:

Restart mySQL and check the mysqld.log has not returned any errors (Or your event viewer if you are using windows).

Every client connection will now default to utf-8 encoding and not latin-1, removing the need to add a SET NAMES call on every connection.

This will work for PHP, C++, ruby etc… as the client encoding is now handeled server side, rather that waiting on the client to issue a SET NAMES command.

UPDATE 30/03/09: Added “skip-character-set-client-handshake” this ignores the clients request to set the connection charset, this info courtesy of “wardo” http://word.wardosworld.com/?p=164

UPDATE 10/09/09

Been having some issues with this working the workaround is to add this config as a single line:

/usr/bin/ld: skipping incompatible /usr/lib/libcom_err.so when searching for -lcom_err

his one has been bugging me for a couple of hours now, when trying to compile PHP on a 64bit OS …

Simple put it’s a missing symlink, and the config script is trying to “failover” to the version is can find which is 32 bit …

ln -sf /lib64/libcom_err.so.2 /lib64/libcom_err.so

Et voila fixed!

This is another _old_ proof of concept I had several years ago, you can infact use PHP to scan ports, bare in mind the legality of this is still somewhat _hazy_ therefore if you must portscan I recomend you only do so on Systems you operate.

Disclaimer: This tutorial is provided for informational purposes only.

UPDATE: Project file now available from http://svn.saiweb.co.uk/branches/port_scanning/trunk/port_scanning.php

Sample output:

NOTE: The current timeout is 0.5s per socket meaning you have a potential runtime of (($endport – $start_port) * 0.5) seconds. Make sure this does not excced your max execution time, or in the construct add:

This will increased the max execution time with a 5 second buffer.

Please also note in most cases of “shared” hosting you will not be able to crate socketed connections, they will either be blocked by the hosting providers firewall, or disabled at the php runtime, therfor not giving an accurate result.

Again please note this is a proof of concept, you may freely distribute the code under the MIT licence