Saiweb

PHP mail() not working?

getting “sh: -t: command not found” when testing using the cli?
what you have is a missing devel package!!!!

In my case sendmail-devel was missing, you’d think the configure script would alert on this but alas no, devel pack installed and one recompile later and the issue is solved.

Title for a cheesy sysadmin novel I know.

But over the last month or so I have been plagued by a mySQL server that was reporting a full root partition, when it wasn’t full …

Causing me some headaches

Allow me to explain:

df -h
Filesystem                     Size    Used    Avail   Use%   Mounted On
/dev/sda1                     20G   17G      1.8G    91%    /   
 
...

Looks simple enough right? I just need to free up some space?

Afraid not.

du -mcs /
2264 /
2264 Total

Just for some clarification this small partition is in use for the operating system only, the mysql instance itself is infact mounted on a much larger partition using the same method as detailed in mysql moving /var/lib/mysql and error121

So here’s the problem, df and therfor all the monitoring systems are reporting the disk as full where as du clearly shows it is not …

Leaving me in the position of if I can not find where the disk usage is with du I have no way of freeing the disk space and bringing the service back online …

Or do I?

After talking with Matthew Ife of ukfast he suggests there must be a an unclosed file IO (aka a file descriptor) that is using up the diskspace, these descriptors do not show up using du

After some searching around I find the command lsof this command will list the open file descriptors for a process including their current size …

psa ux | grep mysqld
mysql     8131  2.8  1.5 304088 63668 ?        Sl   09:37   0:36 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --socket=/var/lib/mysql/mysql.sock
 
lsof -p 8131
 
...
mysqld  27878 mysql    3w   REG                8,2 14930490713   3290408 /var/log/mysql-slow.log.1 (deleted)
...

As you can see above the open file descriptor flagged as (deleted) was increasing in size until the diskspace ran out, for the time being I have since disabled mysql slow query logging whilst I sort out the log rolling as described in Mysql slow query log rotation

In order to get pyinotify working on CentOS 5 x64 a few workarounds need to take place.

(Thanks to Matthew Ife at ukFast for help with this)

First off you are going to need the python-ctypes RPM, available from DAG here: python-ctypes-1.0.0-2.el5.rf.x86_64.rpm

Once installed you are going to need the Fedora 8 python-inotify SOURCE rpm available from here: python-inotify-0.8.0-3.r.fc8.src.rpm

The easiest way I found to extract the required packages was using the following:

mkdir ./python-inotify
cd ./python-inotify
wget ftp://ftp.pbone.net/mirror/archive.fedoraproject.org/fedora/linux/updates/8/SRPMS.newkey/python-inotify-0.8.0-3.r.fc8.src.rpm
rpm2cpio ./python-inotify-0.8.0-3.r.fc8.src.rpm | cpio -idv
tar -zxvf ./pyinotify-0.8.0r.tar.gz
cd ./pyinotify
./setup.py install

I ment to note this down yesterday but everything is going ten to the dozen at the moment.

basically I have now authored a nagios addon for monitoring master-master replication between two servers, this carries out 4 stages of checks

1. Validates all required data is passed by servers
2. Slave IO is running on both servers
3. Seconds_Behind_Master check, args can be passed to vary warn and critical thresholds
4. (slave) Master_Log_File == (master) File

The 5th check was a comparison on the binlog positions themselves, comparing (slave) Read_Master_Log_Pos and (master) Position

Here in lies the problem, which took a while to track down, the problem is that no matter what I tried the slave was ALWAYS behind the master position … but why?

The reason is why I designed the High Availability solution in the first place … Very high traffic level, in the region of 20,800 transactions per second.

Why was this the problem? the two queries run to gather the data are done sequentially per server, using the python time library I was able to find that there is a 0.02s interval between gathering datasets (20 milliseconds) … in that time 416 transactions had take place.

i.e.

time: binlog pos

Slave A

0.000: 100

Master B

0.020: 516

This unfortunately has now lead to some 32 lines of code being commented out, as I can see no way to reliably use the binlog positions for monitoring the replication in this situation, if any delay occurs anywhere at any point during the dataset collection i.e. network latency, delay in query processing due to traffic peak on one server … etc. the collected samples will always be different

The only way I ever see this working is if you can validate that the datasets came from the same exact point in time down to the nanosecond, this however is again not possible, on the network the servers currently reside there is a 0.13 millisecond ping response time this works out to 13,000 nanoseconds (0.00013 * 10^9)

If anyone has any theories on how to overcome this please let me know.

NOTE: At present due to the programming of this addon being done during working hours the nagios addons are not for public release at this time, this may be subject to change in the future should my employers allow their release.

I meant to write this up over a week ago now, basically the need arose for one of my Python scripts to use HTTP Basic authentication when grabbing the output from a URL.

An example script can be seen below:

Quick description, the script will attempt to connect to a URL and read the data supplied by the webserver, if a HTTP 401 error is returned (Authentication required) the script will then go on to attempt to authenticate using the credentials supplied.

Printing out to the console at each point.

Subversion: http://svn.saiweb.co.uk/branches/python/urllib2_httpbasic_auth.py

Highlighted source (at the time of writing)

#!/usr/bin/env python
"""
    Author: David Busby (http://saiweb.co.uk)
    Program: Python HTTP Basic Auth Exa
    Copyright: David Busby 2009. All rights reserved.
    License: http://creativecommons.org/licenses/by-sa/2.0/uk/
"""
 
import urllib2, base64
 
""" URL List """
urls = {
               0:{"url":"www.saiweb.co.uk/some/fictional/auth/area","user":"someuser","pass":"somepass"}
}
 
def main():
   ulen = len(urls)
   for i in range(0,ulen):
       url = "http://%s" % (urls[i]["url"])
       req = urllib2.Request(url)
       try:
           res = urllib2.urlopen(req)
           headers = res.info().headers
           data = res.read()
       except IOError, e:
            if hasattr(e, 'reason'):
                err = "%s ERROR(%s)" % (urls[i]["url"],e.reason)
                print err
            elif hasattr(e, 'code'):
                if e.code != 401:
                    err = "%s ERROR(%s)" % (urls[i]["url"],e.code)
                    print err
                #401 = auth required error
                elif e.code == 401:
                    base64string = base64.encodestring('%s:%s' % (urls[i]["user"], urls[i]["pass"]))[:-1]
                    authheader =  "Basic %s" % base64string
                    req.add_header("Authorization", authheader)
                    try:
                        res = urllib2.urlopen(req)
                        headers = res.info().headers
                        data = res.read()
                    except IOError, e:
                        if hasattr(e, 'reason'):
                            err = "%s:%s@%s ERROR(%s)" % (urls[i]["user"],urls[i]["pass"],urls[i]["url"],e.reason)
                            print err
                        elif hasattr(e, 'code'):
                            err = "%s:%s@%s ERROR(%s)" % (urls[i]["user"],urls[i]["pass"],urls[i]["url"],e.code)
                            print err
                    else:
                        err = "%s query complete" % (urls[i]["url"])
                        print err
       else:
            err = "%s query complete" % (urls[i]["url"])
            print err
 
if __name__ == "__main__":
    main()

NOTES: This script does not check the authentication type, it always assumes it is HTTP BASIC, HTTP DIGEST for example is not compatible with this script, though there is no reason why you can not get the Auth type form the headers returned by the server and write in a Digest auth method.

As most have no doubt noticed the plugin is delayed … _again_

Too many projects, too few hours in the day.

Froomi – All in one HTTP Server, Search Engine, Pseudo Streaming Media server
(Might add transcoding)

This project is up on google code: http://code.google.com/p/froomi/

So far the HTTP module is basically complete and already has GZIP compression support for browsers that send the appropriate header to indicate they support it.

I will being working on the Pseudo streaming component shortly, to recap this is in an Alpha state at the moment, I will be putting the server live on http://www.froomi.com shortly once the Pseudo streaming component is complete.

Flowplayer for wordpress, is still in development the code base has been completely re-written as per the previews you can now see the meta boxes in action, along with the license detection, I think I am going to cut a couple of features and just get the next release out the door (yes RTMP will still be in this release!).

Contributions, some people have shown an interest in contributing to these projects, contributions are very welcome and will carry the appropriate credits in the source code and license, I would go so far as to encourage contributions from developers as this will actually speed up releases for these projects.

So feel free if to contact me.

Contributor Requirements:

• Familiarity with the programming language used in the project you are contributing to (PHP,Python,etc..)
• Working knowledge of creating diff patches (not required bug a huge bonus)
• Ability to document your changes in English, sorry this is a requirements
• Skype / MSN / Ventrilo client

o

Because a picture is worth over 9000 internets … apparently

UPDATE: AKA “hayabusayuri” link … seriously who plays everquest? … maybe all that time playing everquest finally made the guy snap … and PHP & windows … never a good combination … infact Windows and internet is a bad combination

LINKY

Screencap incase it is removed:

(Thanks to the guys who forwarded me this)

Tonight I will be pushing to the development SVN a beta preview of the 2.1 release.

Here are some of the changes:

Poor tags, we barely knew them…

GOODBYE! inpost tags, (sort of), configuration will no longer be handled using the inpost tags, the old tag structure will be retired in favor of an anchor to place the player in your content [FLOWPLAYER], configuration of the player will now be handled by an admin menu box.

That’s quite a list you’ve got there…

(basic) Playlists support has been added, this is configurable from the admin menu for the post

Dude, where’s your config file?

The saiweb_wpfp.conf file has now been removed *gasp*, now reliant in internal wordpress *magic* for the storing of the plugin config.

Your media is a great big canvas, and you should throw all the paint you can on it

Fixed a bug with the canvas colour settings

Is that a logo in your pocket, or are you just pleased to see me?

The commercial version of flowplayer will now only be used if a license key is specified, the free version will now be used if no key is specified which has a reduced logo branding.

I Once Was Blind, But Now I See

Player embed causing issues with some navigation menus, this should be resolved with the wmode setting.

Details of how to get the preview version and install it along with screen casts of the new menus (time allowing) will be added to this post once everything is committed to subversion.

UPDATE: 15/04/2009 Got my hands on flowplayer 3.1 code is around 60% finished, went for a complete re-write.

After taking another look at Python I am quickly coming to love it, as an “exercise” in re-learning python I decided to write a very simple command line “tweeter” this uses the Twitter API to update your twitter status, extending from the “update twitter in a single line”

You can grab a copy of the script from here: http://svn.saiweb.co.uk/branches/python/tweet.py

Example usage:

 
 
./tweet.py -u username -p password -t your tweet goes here

If you want to parse the JSON data normally returned after submitting a new tweet simply add the -j flag.

If you are prompted for a username and password when running this script the username and password supplied using the -u and -p flags was incorrect.

An example of getting the current page / post ID, identifying whether the current item is a page or a post, and then appending the results to the content.

All from within a plugin.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

<?PHP
/*
Plugin Name: Get Page / Post ID using a plugin by D.Busby Saiweb.co.uk
Plugin URI: http://saiweb.co.uk
Description: Identifies the current page/post and appends text to the content
Version: 0.1
Author: David Busby
Author URI: http://saiweb.co.uk
*/
 
//WP hooks start
add_filter('the_content', 'post_page');
//WP hooks end
 
function post_page($content){
    global $post; //wordpress post global object
    if($post->post_type == 'page'){
        $content .= '<br /> This item is a page and the ID is: '.$post->ID;
    } elseif($post->post_type == 'post') {
        $content .= '<br /> This item is a post and the ID is: '.$post->ID;
    }
    return $content;
}
?>

Install the above as a plugin i.e. in wp-content/plugins/test/test.php

Head over to your admin menu and enable the plugin, now each page and post will identify itself as a page or post, and provide it’s ID.

There is a lot available in the $post object for a list add

ob_start();
var_dump($post);
$content .= ob_get_contents();
ob_end_clean();