Saiweb

The mock tool can be a wonderful thing, allowing you to produce rpm packages for any rpm based system (assuming your have the written .cfg for it).

What I did find a little lacking on the documentation side was the SCM integration (read: Source Control Management), git/svn etc …

In short so long as your rpm spec file is in your SCM (and it should be), moc will build your rpm from your sources in scm, which can be used for.

1. bleeding edge builds for testing
2. builds from “stable tags”

Yes yes yes … obvious I know …

So with no futher ado here is the syntax:

1. scm-enable – turns on the use of scm
2. scm-option – set an option for the scm in use

The above worked for me, you will need to adjust it acordingly, i.e. if your spec file is not named identically to that of your git project: –scm-option spec=’specfile_name.spec’

This will tie me over untill I get chance to play with my monkey farmTweet

Initially this took about ~7hours to diagnose and fix, with what I have learned about the inner workings of gluster and the tools I am providing opensource this should cut resolution time down to ~5minutes.

Firs you must meet the following conditions:

1. You are running gluster >= 3.0 <= 3.2 (May also work on 2.x I have not tested, and will not work with future versions if gluster change their use of xattrs)
2. You are running a replicated volume (Again I have not tested distributed volumes, in theory remove, re-add and rebalance will fix these)
3. You have a “good” copy of you data (This is essential this assume you have at least 1 brick with a good copy of the file system

Restrain and restore the “bad” brick

1. Shutdown all services that are using the mounted filesystem (i.e. httpd / nginx / *ftpd)
2. Unmount all the file systems on the node (glusterfs / nfs / etc …)
3. Grab a copy of stripxattr.py make sure you READ the README for installation requirements and usage
4. Run stripxattr.py against the backing filesystem on the “bad” node ONLY NOT AGAINST A GLUSTER MOUNT
5. From the “good” node, not rsync the data: rsync -gioprtv –progress /path/to/filesystem root@:/path/to
6. From the “good” node, trigger an “auto heal” this will re-populate the xattr data (this must be done on a glusterfs mount not nfs/cifs/etc…)
7. Download listxattr.py once the self heal has completed see the README file for a “quick and dirty” consistency check
8. All being well you have now resolved a split-brain and can return your node to service

Current known gluster issues

1. NFS is much (48x in tests) faster for small files i.e. php webapps, but does not support distributed locking meaning: all nodes can write to the same file at the same time, this is what cause our original split brain

So what is the resolution int his case?

Selective use, use glusterfs for filesystems that you need distributed locking, often in large production deploys php files will not change often, in this case NFS is perfect.

If you are still writing php sessions to a file system then STOP IT and use a database! (Better yet use memcache).

Tweet

They say necessity is the mother of invention, if this is true then surely the mother of all fuck ups is shoddy customer service, say an isp that will randomly shut down a port because it has high bandwidth usage without asking the customer about it first, and flat out refusing to do anything for 24hrs …

In one of the worst customer service experiences I’ve ever had the miss fortune to have been a part of all access was closed to our in office version control systems due to “high usage”, now this is a pretty essential service as you might imagine, how then to restore access, when the restrictions are beyond your control? (And I mean EVERY inbound port was dead …)

Fortunately it would seem outbound SSH was not affected, so after much vocal drawing and re-drawing many times over on the whiteboard I had a cunning plan …

Using 3 linux devices I would create the following.

1. A device through which using host entries / dns changes the version control would be available whilst not actually running on the box itself.

2. An in house device which would be the device on which the tunnels are created in the first place.

3. The device(s) on which the version control systems reside.

Gateway device

On the gateway device sshd_config needs to be updated with:

And sshd reloaded.

Also if you are using a local firewall (i.e. iptables) you will need to setup the appropriate rules as if the service were running natively on the device

Pivot Device

Generate rsa ssh keys and deploy your id_rsa.pub to the gateway device, (update sshd_config to enable RSA Auth if required)

The tunnel.

Now you only really need to use root if the port you need to gateway is a privileged port (<1024).

Here 10.0.0.1 is the internal address of the device the connection should "pivot" onto.

Once the tunnel was in place the services could be reached via the gateway device, this was essentially a "poor mans" NAT in a time of need, I really do not suggest this for long term use.

Tweet

See if hosts are up using ping in range 60 -> 200

Note: for OSX use “ping -c 1 -t 1″

Chaining “UP” hosts for a quick (syn) port scan

Recover from a bad mysql password set (Update mysql.users set password=’Iforgotawherestatemenlulz’)

Assumes for every user there is an @localhost host, grabs the in memory password hash and resets

If you’ve run FLUSH PRIVILEGES; however you == b0ned.

Quick substitute and run

Command1:

Opps that’s OSX synatx

Command2:

et voila corrected syntax.

Shortcuts

!! – Execute last command
!ping – Execute last ping command, can be used to !any command just be careful.
ctrl+r – reverse search, just start typing the cmd for it to search your history, hit tab to complete
ctrl+a – jump to beginning of line
ctrl+e – jump to end of the line

cURL FU

curl -I -L blahblah.tld – Run a HEAD and follow redirects (very handy for quicklooking @ bit.ly short URLS before hitting them in a browser).

python FU

python -m SimpleHTTPServer – serves the current `pwd` as a browseable directory (Very cool but VERY insecure)
python -m cProfile script.py – generate trace stats for a script execution (Very handy for finding excessive loops)

DNS Fu

Wikipedia over DNS:

host -t txt fu.wp.dg.cx

fu.wp.dg.cx descriptive text “Fu may refer to: Fu (Technology, especially computer related) (used as a suffix) – relating to a person – Possessing superior skills in an art\; relating to an artifact – representing an expression of high art. code-fu, Perl-fu, C-fu, etc, Fu (literature),” ” a Chinese genre of rhymed prose, Fu (kana), a symbol in Japanese syllabaries, Fu County, in Shaanxi, China, Fu Foundation… http://a.vu/w:Fu”

Useful on _some_ public wifi connections if you just want to look something up quick (dns is not always re-written).

Get all MX servers for a domain:

dig google.co.uk MX

; <<>> DiG 9.6.0-APPLE-P2 <<>> google.co.uk MX
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64165
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 4, ADDITIONAL: 4

;; QUESTION SECTION:
;google.co.uk. IN MX

;; ANSWER SECTION:
google.co.uk. 10800 IN MX 10 google.com.s9a1.psmtp.com.
google.co.uk. 10800 IN MX 10 google.com.s9a2.psmtp.com.
google.co.uk. 10800 IN MX 10 google.com.s9b1.psmtp.com.
google.co.uk. 10800 IN MX 10 google.com.s9b2.psmtp.com.

;; AUTHORITY SECTION:
google.co.uk. 59925 IN NS ns2.google.com.
google.co.uk. 59925 IN NS ns3.google.com.
google.co.uk. 59925 IN NS ns4.google.com.
google.co.uk. 59925 IN NS ns1.google.com.

;; ADDITIONAL SECTION:
ns1.google.com. 158334 IN A 216.239.32.10
ns2.google.com. 158334 IN A 216.239.34.10
ns3.google.com. 158741 IN A 216.239.36.10
ns4.google.com. 158334 IN A 216.239.38.10

;; Query time: 68 msec
;; SERVER:
;; WHEN: Mon Sep 26 16:41:26 2011
;; MSG SIZE rcvd: 310

mySQL FU

in one line, take a database, in stream replace content and stream into another db.

mysqldump original_db | sed ‘s/content_or_regex_to_replace/content_or_backref_replacement/g’ | mysql destination_db
Tweet

Ok ok … as some of the people work with are aware, I did this months ago fro one project, ment to blog and document it then in fact I have a draft post last modified 06/05/2011 covering full spam score reduction, and half finished instructions on setting up a mail relay … so in the interim of finishing that post I’m going to cover improving user experience through proper php configuration.

Out of the box, php will use sendmail, and it will do so as follows.

1. mail() forks sendmail process
2. sendmail attempts to send email to destination server
3. sendmail returns on send complete

This sets the delivery mode to background, sendmail will return to php near instantly and send the email in the background by placing in into a queue.

TL;DR

Put the above in your php.ini to not hang around to sendmail, and hav it return instantly.
Tweet

If you haven’t tried boxgrinder then you are missing out, it makes it extremely easy to script the generation of a virtual machine for output to Rackspace (Well not yet), ec2, vmware, virtualbox, KVM etc.

In this post I will cover the basic generation of a LAMP (Linux Apache MySQL PHP) stack CentOS appliance, nothing to complicated I assure you, and no magic like auto deployment spin up etc … that’s for later … no skipping ahead!

First of all you’re going to need boxgrinder I recommend downloading the Meta appliance, as it has all the tools you need already.

Now I am covering the following.

1. basic use of boxgrinder-build on the meta appliance
2. creation of centos lampstack basic
3. deploying the image to KVM

I’m going to have to assume that you are capable of downloading and starting up the meta appliance yourself, and focus more on the stack setup.

Grinding your VM

Ok so you are going to need a YAML file defining the CentOS lamp stack, save this on your meta appliance as CentOS-lamp.yaml

On your Meta appliance run.

This process will take a while, so go and get a coffee, this will produce ./build/appliances/x86_64/centos/5/CentOS-lamp/CentOS-lamp-sda.raw once complete, if you run into issues the -d flag is “debug” paste your log output int the comments and I will do my best to diagnose and fix your issue.

Deploying to KVM

boxgrinder has SFTP support for pushing to remote servers, you can use this if you like to automate the “push” of the image to your KVM server, at the moment automated deployment to KVM is not support but may be coming soon.

Assuming you have placed you image in /var/lib/libvirt/images/

Post startup

this is a VERY basic setup I have not covered any of the post install options in this post (but I will in future posts), so.

This will set your services to automatically start at startup, and start them.Tweet

If you tie in your web application to automatically PURGE content when you modify it, thus keeping the content “fresh” while using Varnish you may notice if you made the jump from 2.x to 3.x that your PURGE VCL is no longer working, I refer you to: https://www.varnish-software.com/blog/bans-and-purges-varnish-30

In short replace your usual

with

Substituting “~ purge” with your ACL name, the above implement wild card purging aswell, if you do not want this and only want PURGE for the exact passed URL replace

“req.url ~ “+req.url

with

“req.url == “+req.url

Tweet

Pre-req reading: Part 1

In this part we will cover setting up a backend. A backend is your application server, whether this be apache / nginx / iis (IIS – Is Inherently Stupid) you are telling varnish where it should sends it’s requests to.

Very basic configuration

For a quick start that’s it really you tell varnish a backend and the port to connect to it on … just make sure you use it in vcl_recv, but you’re not here for simple and quick start are you? lets add the following.

• timeout settings
• probe settings

Timeout settings

Your timeout settings deinf how long varnish should wait for a response from your backend

• connect_timeout wait 50ms for a tcp connection to take place
• first_byte_timeout wait 2s for the first byte of data to be sent from the backend
• between_bytes_timeout wait 2s if there is a pause mid data stream

Timeouts are a basic way of determining if a backend is down / miss behaving if you have multiple backends if timeouts occur then the backend is marked as sick and the other backends will be used.

probe settings – Trust me I’m a doctor

• url the URL to to query this must return a 200 OK response, you could use a php script to return a 500 on say a mySQL outage
• timeout how long to wait for a 200 OK response from the URL
• window keep the result of the last 5 probes in memory
• threshold how many of the window total must be OK for the backend to be “healthy”
• interval how often to run the probe

And that about wraps up this post.Tweet

Go ahead and run

You will get

As an attacker looking to hit a web app, one of the first things you’re going to want to look into is what version of web server is running, in this case you can see this blog in fact runs apache … but there is not much else to go on here is there.

That’s intentional, and by manual configuration changes I have put in place, this is not the case of a default LAMP install, take for instance, this snippet from another website,

This already has given me a wealth of information to go on and begin prepping an attack, I now know the site is running php version 5.2.14 Apache version 2.2.16 and that the underlying OS is Debian.

See the dilemma? your default roll outs are just declaring their running versions to anyone willing to listen, so lets make it a little more stealthy.

First and foremost if you are using php, edit your php.ini and set the following:

Now head into your httpd.conf and set the following

and

With these 3 simple steps all the headers will now return is Server: Apache this is the first step to shielding your app, I’ll be covering further steps as time allows.

Tweet

Following reader feedback please see below for an updated version of Volume 1

Ever wanted / needed HTTPD or another service to run with a raised thread priority?

Well you have a couple of options, add additional lines to the /etc/init.d script to change the nice level by adding additional lines on startup, or if you only need to do this on a temporary basis without restarting the service but need every thread to have a raised priority you can use a bash script

Much cleaner script here again thanks to Matthew Ife.

You can renice between -20 and +20, depending on your requirements you can use this script in a cron job to raise/lower priorities, change httpd for whatever service you want to change the thread priority for.

Ever needed to check files were being accessed / written to?

For this one you’re going to need the inotify-tools package, specifically the inotifywait binary.

An example usage is to ensure that caching is working correctly and that cache files are being used in place of processing PHP files, simply change “/folder/to/watch” to be your cache folder, and refresh a few pages.

All being well you’ll get an output similar to the following:

Alternatively you can use the following approach contributed by Matthew Ife:

This will persist for the duration of your ssh session and relevant log entries will appear in /var/log/audit/audit.log, admittedly with far more useful information than inotifywait, and does not require you to install additional packages.

As can be seen the re-write rules are redirecting users to the cached files/folders, in the example above I have used my wp-supercache folder.

Ever needed to quickly get the memory usage of all threads for a service?

You have two options for this a single line

or a bash function you can place in your ~/.bashrc

You can then call this (after logging back in again to load the .bashrc up) using

replacing for instance with httpd will give you an output similar to the following:

This shows memory in kilobytes, command, process id, you can see here I currently have 3mb/pid for each httpd process (due to my optimizations, I highly recommend you read parts 1-3)

Dump mysql data and compress on the fly

Self explanatory that one, pipes the output from mysqldump through bzip2 (which has better compression over gzip) and dumps it out to a file, if you _realy_ need a gziped file just replace bzip2 with gzip in the line above.

Ever needed a selection of passwords generated?

Using a slightly modified line originally provided by Matthew Ife,

Plant this in your ~/.basrc for a callable function that will generate a selection of 10 secure passwords, handy when you’re fed up of 1337′ifying everything

example output:

If you want runtime variable length you could change to cut -1-$1 and then call pwgen 15 for example.

Check mySQL myISAM fragmentation

Gives you a very quick overview of make up of your myISAM tables and their fragmentation (Data free vs data length).

Tweet