Saiweb

Part 1, what is varnish?

The varnish cache project is one you really need to get familiar with if you manage any high volume websites, it can mean the difference between a self destructing web app that buckles under it’s own load, and an apparently seamless web app serving 1000′s of concurrent connections per second with relative ease.

How does it work?

Varnish acts as a proxy server, in that when a use sends a GET request varnish will lookup in its internal database for a cached version and if it can not find one it will pass the request to the “back end” or in this case an apache server, varnish will then cache the response for subsequent accesses.

Now you may ask yourself why do you need this? this boils down to what you are trying to achieve with your web application, if your application is heavily reliant on dynamic content and regularly gets some 400 concurrent users for example, lets assume the following:

1. 400 concurrent unique users
2. Average page render time is 0.85s

The Math

Based on this if you were to place varnish in front of your application with a 60second ttl (time to live, length of time varnish will hold an object in cache):

1. Varnish ttl 60 seconds
2. 400/0.85 = 470.59/second
3. 28235.29/minute
4. Factor of reduction to “back end”: x28235.29

So in the example above simply by caching a page for as little as 60 seconds, the requests/minute as reduced from 28235.29 to 1, now even reducing the cache times to 10 seconds in this example would give a x4705.88 reduction.

How is this reduction a good thing, well time on cpu for one, varnish when configured correctly is very very fast, and even with an out of the box configuration it’s still going to be much faster than your dynamic web application.

Summary

So here ends a brief introduction to varnish and why you realy want to start using it, in the following parts we will cover

• Configuration overview
• brief overview of each sub section based on the 2.1 syntax
• Advanced configuration
• Load balancing
• Failover handling
• Raising cache hitrate
• Pros and cons of each setup
• Benchmarks

Tweet

This is one of those things I find my jaw dropping at, whilst punching myself for not knowing about it sooner.
It’s true as much as I live in the cli & ssh to do my job I find sometimes I require a VNC connection (i.e. the plethora of system-config-* stuff in RH)

Now however there is an alternative (so long as your client machine has x11 installed)

That’s it simple as that, now use a cli command to launch your normal gui tool i..e

And x11 will launch on the machine you are working from, now don’t think the gui is running form your machine it’s not!

your machine is now acting as a thin client simply interacting over SSH, with the gui tool running from the server itself!

And there is where the awesomeness lies, esp if like me you run OSX whilst managing *nix servers.

*grin*

Tweet

Namely a bug to do with iptables rate limiting,

works!

-j REJECT also produces the same.

Simply increasing the “hitcount” causes this error, the only work around I have come up with is decreasing the –seconds arg, to yield more hits/sec, still bloody annoying!

Tweet

ESP Ghostscript 815.02: Unrecoverable error, exit code 255

I got this issue today whilst running CentOS 5.4 x64 post investigation of images not being scaled when processing a specific PDF, the solution unfortunately is to build ghostscript and imagemagick from the latest sources.

Unpack, configure, make && make install

To fix compatibility with pear imagick

Tweet

Most of the time when I review our log watches each morning I become enraged at the number of automated attacks,

But ever so occasional I find one that frankly intrigues me.

Today is just such an occasion where I have had multiple Brute force login attempts, the ingenious part is this attack has been designed to bypass tools such as fail2ban, blockhosts etc, and this is how

1. Attack is launched from
2. has PTR set for
3. Failed login attempts record due to reverse lookup
4. There is no A record, attacker maintains their own nameservers for the
5. fail2ban notes failed logins, attempts to resolve to an IP but fails, due to missing A record
6. Attacker can continue brute force attempts unhindered by being banned

I am still reading into how to counter this and will update this post as I figure out how to work around it, it’s a very sneaky and frankly quiet clever method of working around most automated blacklisting/banning tools.

Update 1:
One method I am trialing is the “log target” feature of iptables, in an attempt to match login failure times to the iptables log, I’ll post back with results.

Outputs

Update 2: Defeating the hack

Now granted this would be a lot worse had the attacking IP been dynamic, fortunatly in this case it’s not

Ip’s have been masked to prevent anyone complaining or threatening legal action (again) for inferring you should block their ip / network range … and me firing off the obligatory “Well if you policed your own network I wouldn’t have to post this no would I” email,

Maybe I am just being Cynical in my “old” age …

Any how as you may have guess I’m black holing the ip with the 390 connection entries.

Thanks

Being as I spoke to a load of people during the course of this I realy can not remember who contributed what to this solution, so I’ll just have to thank you all let me know if you want a crediting link.

Tweet

This is something I have wanted to get working for some time now, and thanks to James P for passing me a note that as of OpenSSH 4.4 you can infact add command line args for the Subsystem configuration, which when combined with the (I assume new) logging functionality of the sftp-service allows you to finally log what is occuring during an sftp session.

Note: Requires OpenSSH >= 4.4

Replace the susbsystem line in your /etc/ssh/sshd_config with

Add the following to /etc/syslog.conf

Restart the sshd and syslog services, try an sftp upload and review the logs @ /var/log/sftpd.log
Tweet

Ever wanted / needed HTTPD or another service to run with a raised thread priority?

Well you have a couple of options, add additional lines to the /etc/init.d script to change the nice level by adding additional lines on startup, or if you only need to do this on a temporary basis without restarting the service but need every thread to have a raised priority you can use a bash script

You can renice between -20 and +20, depending on your requirements you can use this script in a cron job to raise/lower priorities, change httpd for whatever service you want to change the thread priority for.

Ever needed to check files were being accessed / written to?

For this one you’re going to need the inotify-tools package, specifically the inotifywait binary.

An example usage is to ensure that caching is working correctly and that cache files are being used in place of processing PHP files, simply change “/folder/to/watch” to be your cache folder, and refresh a few pages.

All being well you’ll get an output similar to the following:

As can be seen the re-write rules are redirecting users to the cached files/folders, in the example above I have used my wp-supercache folder.

Ever needed to quickly get the memory usage of all threads for a service?

You have two options for this a single line

or a bash function you can place in your ~/.bashrc

You can then call this (after logging back in again to load the .bashrc up) using

replacing for instance with httpd will give you an output similar to the following:

This shows memory in kilobytes, command, process id, you can see here I currently have 3mb/pid for each httpd process (due to my optimizations, I highly recommend you read parts 1-3)

Dump mysql data and compress on the fly

Self explanatory that one, pipes the output from mysqldump through bzip2 (which has better compression over gzip) and dumps it out to a file, if you _realy_ need a gziped file just replace bzip2 with gzip in the line above.

Ever needed a selection of passwords generated?

For this one you can use the secpwgen

Plant this in your ~/.basrc for a callable function that will genrate a selection of 10 secure passwords, handy when you’re fed up of 1337′ifying everything

example output:

Modify the “-Aadhs” args to your taste.

This concludes Volume 1 and a very long post, please contribute your one liners / helper scripts via the comments.

Cheers

buzzTweet

I’ve no idea to this day why my bash script would not work with a CSV export from mysql by simply using mysql -e “SQL COMMAND HERE”.

So I had to come up with a workaround quickly.

This lead to using expect, scripting in this method can be used for numerous purposes, I am currently in the process of writing a few test scripts using tcl and this package for pop,imap,smtp testing.

Pretty simple realy once you have the hang of it, you tell it what to expect and what to reply with, there are more advanced methods going on from here, including conditional sends based on response.

I’ll be covering those soon.Tweet

Strangely I’ve had some people reporting issues with being prompted for a username and password when accessing files on svn.saiweb.co.uk

it would appear in mod_dav_svn-1.4.2-4.el5_3.1 that this directive: AuthzSVNNoAuthWhenAnonymousAllowed

now defaults to OFF, well that was a p.i.t.a trying to track down, having never seen that directive in ANY of the documentation …

Anyway pass this on to other facing the same issue.

Tweet

The default install of VI is very basic, and being as I spend a lot of my time in there I find syntax highlighting invaluable, to get this however you will need the vim-enhanced package.

So run the following to install this package and setup an alias for vi.

And you’re done:

Tweet