Saiweb

Following on from the python bindings post I found myself with a real problem,

the netsnmp bindings I could not for the life of me get to take the redhat cluste MIB files, so what did that leave me with, walking the entire parent cluster OID, manually matching the returned OID’s to their MIB names based on the value returned as I couldn’t find a decent mib browser or script to convert them …

At any rate here is a subset of OID’s for polling the redhat cluster service using snmp, please note that are more OID’s but these vary on your cluster config.

Python code:

Following on from Linux – Generating file manifests and then checking them I was always getting the same questions …

How long left on the manifest Buzz ?
How long left on the verification Buzz ?

And I HATE having to turn around an say … I don’t know …

The problem with the usual command line method is that it give no indication of progress, and by extension no indication that it was infact running and not ‘hung’ …

As such I have now added the ‘manifest’ command set to the Sysadmin toolset

The manifest command take two data types, the first is a folder path from which to build the file manifest from, the manifest itself is also compatible with the “md5sum –check” function.

The second is the path to the manifest itself, in this case the manifest command will verify each file against it’s entry in the manifest:

At each point the command give you an indication of it’s current status, however this does come at a small cost, the script has no concept of the size of you console and as such will always render out the same number of character meaning if you console is not wide enough it will not render correctly, in the videos I have the console on a high resolution monitor as can be seen each video itself is 900 pixels wide.

This process is CPU intensive (20-45% on one core of a intel core2duo 2.8GHZ) and uses around 140KB of memory.

I had a major issue facing the iconv functionality of the sysadmin toolset namely due to rushed coding.

When loading a file to be re-encoded the entire file was loaded into the buffer, encoded as whole and written out to the new file, this of course meant the  memory usage was roughly double the size of the file to be converted plus any overheads to do with the encoding itself.

Today I had need to convert a 1.3GB sql file, needles to say the script was crashing out with a memory error.

As such I have now completely re-written the function it now processes the file in 1kb ‘chunks’, moving the load to the CPU, this process is now very cpu intensive the the memory overhead is minimal (during test processed the 1.3GB file using 113kb of memory!!!).

Also I have now added BOM (Byte order mark) detection:

This issue has come about whilst having to migrate a positively huge number of files, and have to check the integrity of the transfer.

Build the manifest

• -type f : This flag tells find to only return files
• -print0: This flag tells find to null terminate strings, this allows us to take files with spaces
• –null: This flag tells xargs to accept null terminated strings
• NOTE: PUT THE MANIFEST OUTSIDE THE FOLDER YOU ARE INDEXING!

Checking the manifest

The above will return all failed checks, if you want a simple count (maybe for automated reporting) just add | wc -l

FAQ

How big is the manifest?

This depends entirely on the length of your filepaths, taking UTF-8 as an encoding example each char is 8bits or 1byte, each manifest line consists of the md5hash, a space and the filepath as the filepath length varies there is no exact way to estimate the filesize of the manifest.

However each line is always 32 + 1 + len(path) bytes.

The more sub directories you have the larger the manifest size will be.

How long does the manifest take to build?

This depends on the number of files you have to index, along with any other factors such as network shares, in test runs 2819 files indexed in 1.493 seconds.

PHP mail() not working?

getting “sh: -t: command not found” when testing using the cli?
what you have is a missing devel package!!!!

In my case sendmail-devel was missing, you’d think the configure script would alert on this but alas no, devel pack installed and one recompile later and the issue is solved.

As it turns out twitter account can be updated in a single line, this makes writing “bots” just that little bit easier.

This also returns JSON should you want to parse the reply data.

i.e.

At the moment I am looking at hooking this into Nagios, from there the feed will be passed into a ’service status page’.

But in theory this single line could be used for any purpose.

I should of really written about this ages ago.

SSH Keys allow you to log into a server without the need for passwords by providing a public, private keypair for authentication, you can of course choose to specify a password for the authentication for an added level of security (Allowing you to have one unified login for you servers).

For the general user I would suggest the use of a password for securing the key further, you can forgo this in the case of secured automated processes however. (i.e. server to server backup via scp).

From the client machine:

Generate the key

Follow the prompts to enter your password (or just hit enter for no password).

Copy the key

You must now copy the key to the server you wish to log in to.

Now log into the target server.

Now exit the shell on the target server, and re-login.

If you are prompted to enter a password this should be the password you entered when generating the key, if you did not specify a password you should now be logged into the target server without being prompted for a password.

This process works for both linux and MAC OSX, when generating keys as the client.

NOTE: If you regenerate the key for whatever reason this will replace the olde key pair, and you will need to go through the procedure of copying to the target server again.

How to write a bash ‘hello world’ script in 60 seconds, admitedly it could of been faster … damn typos

Also the first line you can add as an alias, if your going to be writing a lot of bash scripts.

Or you can copy paste and have it done in about 5 seconds

The reason for the echo and awk is when trying to do echo “#!$BPATH” > script.sh my shell wouldn’t cooperate so all the awk does is take out the space .

Liam (http://goodingsweb.com) has shared his findings on tweaking ffmpeg output with h.264, he has taken a 174MB flv file down to just 24MB (86% reduction in file size).

There is no noticeable difference in quality either, you can follow the forum thread here: http://flowplayer.org/forum/7/12671.

Click the video for a preview of Liam’s settings:

For those not in the know Folding@Home is a piece of software that runs in the background of your desktop, server, heck even your PS3.

I originaly started out back in 99/2000 with the united devices cancer research client, their website of UD.com however seems to have long since slipped into web history, no doubt due to their nature of charging for CPU time on ‘thier’ grid, which was made of donor machines … Folding @ Home however is Open Source and not run by some shady business but by a variety of labs and Educational bodies (http://folding.stanford.edu/English/About).

Ok so what is this ‘folding’ all about?

When protein chains combine in your body to form more complex chains, and eventualy cells the process of combination is called folding, and problems during the ‘folding’ stage can lead to Cancer, Alzheimers, Parkinson’s disease etc …

The problem faced when looking at protein folding is the shear number of possible ‘folds’ for each different type of protein, of course this is where computer power comes in, but even that has it’s limits this is where distributed computing helps.

Rather than a large super computer which is limited in budget, size and power distributed computing takes place by assigning a small work load to a ‘volunteer machine’, what this has lead to is a virtual super computer larger than any other, driven by software, and at the time of writing claims some 260,000 Active CPU’s

So please install this small piece of software on your machine by visiting the downloads page here: http://folding.stanford.edu/English/Download

Once you have the client running please join the Saiweb team 156680

Cheers

Buzz

NOTE:

I will be dedicated several machines to this, they will appear in the team members list prefixed buzz_ , one core from our Dedi server has also been dedicated to running F@H.