Saiweb

This issue has come about whilst having to migrate a positively huge number of files, and have to check the integrity of the transfer.

Build the manifest

• -type f : This flag tells find to only return files
• -print0: This flag tells find to null terminate strings, this allows us to take files with spaces
• –null: This flag tells xargs to accept null terminated strings
• NOTE: PUT THE MANIFEST OUTSIDE THE FOLDER YOU ARE INDEXING!

Checking the manifest

The above will return all failed checks, if you want a simple count (maybe for automated reporting) just add | wc -l

FAQ

How big is the manifest?

This depends entirely on the length of your filepaths, taking UTF-8 as an encoding example each char is 8bits or 1byte, each manifest line consists of the md5hash, a space and the filepath as the filepath length varies there is no exact way to estimate the filesize of the manifest.

However each line is always 32 + 1 + len(path) bytes.

The more sub directories you have the larger the manifest size will be.

How long does the manifest take to build?

This depends on the number of files you have to index, along with any other factors such as network shares, in test runs 2819 files indexed in 1.493 seconds.

In on of those “oh ffs” moments I found myself writing a BASH script to quickly dump all database on a mySQL server.

This script gets a list of all databases, dumps them out with UTF8 encoding, and gzip compresses the SQL file into the given “DEST” folder.

If you want to skip over certain databases i.e. “mysql”

Change this line:

To:

Or for multiple exclusions

I may re-write this in Python, if I get time.

How to write a bash ‘hello world’ script in 60 seconds, admitedly it could of been faster … damn typos

Also the first line you can add as an alias, if your going to be writing a lot of bash scripts.

Or you can copy paste and have it done in about 5 seconds

The reason for the echo and awk is when trying to do echo “#!$BPATH” > script.sh my shell wouldn’t cooperate so all the awk does is take out the space .

In part 4, I am going to cover more of an improvement than anything else to part 3

Part 3 itself is not incorrect, it correctly takes a memory footprint for each process running, the same as VIRT in top …

However in processes such as APACHE the VIRT memory is the size of all shared libraries, as correctly shown by pmap …

So what does this mean realy?

The memory usage is infact the following VIRT + RSS, where RSS is the resident set size, the RSS is a representation of the memory in use by the PID, and VIRT is shared between the child processes.

As can be seen above the ‘VIRT’ does not change between the child processes, where as the RSS does dependant on what the thread is doing at that time.

So below is an improved appmem function to allow for this:

Example output:

PART 3 IS INACCURATE, THE BELOW SCRIPT IS FOR REFERENCE ONLY, IT HAS BEEN REPLACED IN PART 4

In part 3, I am going to cover a bash function that will allow you to profile the memory usage of any application by name.

By adding the function below into your script you can execute a command such as: sysadmin appmem apache

Sample output:

You can of course replace ‘apache’ with the application or daemon name you want to profile the memory usage of.

This script does require that pmap is installed, if the script can not find it, it will abort.

As always any problems, post a comment.

UPDATE: Apparently I need to point out that if you haven’t read PART 2! then the colored output will not work … That’s why this entry is titled part 3, it does assume a degree of competence on your part in realizing part’s 1 and 2 may just be required reading …

NOTE: The above provides a complete memory footprint of the indvidual PID, the same as VIRT in top.

VIRT — Virtual Image (kb)
* The total amount of virtual memory used by the task. It includes all code, data and shared libraries plus pages that have been swapped out.
* VIRT = SWAP + RES

Part 2 has finally arrived …. don’t all cheer at once now …

In part two I will cover how to run an IP range scan using bash script, and if the host can be pinged retrieve the MAC address of the connected host.

Now bare in mind this script was written to run from a MAC running OSX Leopard.

To make this work on your Linux distro replace -t in the ping command with -W and check the awk entry for the arp output, a display of (no) means that no ARP entries could be found for the host, and change the IP range to that of your network.

That’s it for this part, dump this is a file and chmod +x as useual and run with ./script.sh ipscan.

Prompted by the following remarks today …

Kerm: “;) there is always an abbreviation in the CLI as all sysadmins are lazy feckers”

Kerm: “Someone might think you actually do work occasionally, god forbid!”

Sysadmins are NOT inherently lazy, we just know how to save time, and are quite adept at doing so …ok?

You cheeky sods!

So let me clear up one instance in which I take a lot of information, and make it quickly and easily accessible using a “Lazy feckers” abbreviation …

Be warned this is a very jaded write up, read on at your own peril.

Right then, onto the point of this post, the sysadmin script part 1, this is going to cover how to check how many connections to a specific port you have on your server.

Trust me this becomes very useful when you have exhausted all other options when trying to figure out why your web server is running like a dog with no legs …

After running the above on your SSH session you will see lines, and lines … and yet more lines of network connection information, especially if you just run this on a busy server.

Example (colours added):

tcp 0 0 ***.***.***.***:25 ***.***.***.***:32794 ESTABLISHED

Key:

PROTOCOL Tx Rx LOCALHOST:PORT FOREIGN_HOST:PORT CONNECTION STATE

From this information it’s pretty easy to spot this is an inbound SMTP connection.

(If you can’t see why, don’t worry it’s ok maybe it’s genetic)

Now this may be handy, but other than taking all this information and dumping it into a spreadsheet (god knows you love those spreadsheets !!! ), how are you going to figure out how many connections are occurring from that external host?

How infact are you going to be able to easily see how many total connections to that port you have ?!?!

Bash script, now for some history, Bash is the Bourne Again Shell, or as I like to think of it, it is the verb for what I will do to your head if you ask me what BASH / SSH / Shell is again …

Now create a directory:

Note the prefixing dot, this will create a “hidden” directory in your home directory (~), the reason for this is so you don’t have system admin script sat in your home directoy, as if you are like me, all sorts of crap moves in an out of that directory on a daily basis, and the last thing you want to do is to have to rummage through backups trying to find “that script you wrote to diagnose connection problems a year ago“.

The point is these scripts will become part of your workflow, once written they will rarely need updating, and should never be called directly, (I mean we’re lazy right? WTH do we want to be typing the full script path for? … oh yeh it saves time!).

In this case:

You can of course call your script whatever you want, and use any text editor you want, if you don’t like / know vi …

Ok so the above code is provided with two functions usage and portcon.

MAKE SURE YOU REPLACE “ABC.DEF.HIJ.KLM” WITH YOUR LOCAL IP ADDRESS

CHMOD this file to allow execution.

Now edit your bashrc file.

And add the following:

alias buzz=’~/.sysadmin/buzz.sh’

Now exit (logout) your SSH session and log back in (or SU root > SU your_user for testing).

Now run the portcon check …

(Yes before you ask ***.***.***.*** does display the correct IP address, I have purposely removed them for security).

So, I have taken something that would of resulted in netstat output > spreadsheet to formulas > at a estimate 30mins a time analysis to something that now takes less than 5 seconds to type, and get the relevant output, for roughly the same initial effort (30 mins scripting time).

You could argue you can keep a spreadsheet pre-setup with the right formulas / pivot tables and just dump the data each time, well yes you could but that’s no where near as quick as this …

And no trying to convince me it is as quick and better than the script above, for

1. You have to wait for excel to open the spreadsheet
2. You have to copy paste the data
3. You have to wait for excel to process the formulas

If you have a machine that can do that in time equal to or less than the time it takes the script above to output the data, the only thing I have to say is, stop spending such a budget on desktops and get a better server.

Final Thoughts:

This write up is in jest, and is intended to be read as such, the code and methods provided above are factual. etc …