Keeping track of resources with Monit

Posted by John
on Sunday, 08 June 2008

Another handy tool i've grown to like is Monit, which makes keeping tabs on memory-hungry processes much easier.

Basically once configured it helps keep an eye on particular process (e.g PostFix, PostGreSQL, Apache or your disk space). If one is starting to use too much memory, your running low on disk space or it's just no longer responding it can automatically restart it for you and alert you in the process; making system administration that little bit easier.

Installing

Install with Aptitude using,

sudo aptitude install Monit

After which it will tell you it's won't start until it's been configured,

Starting daemon monitor: -e monit won't be started/stopped
  unless it it's configured
  -e  please configure monit and then edit /etc/default/monit
  -e  and set the "startup" variable to 1 in order to allow
  -e  monit to start

Configuring

First make a backup copy of it's initial config, in case you need to rollback:

sudo cp /etc/monit/monitrc /etc/monit/monitrc.orig

Now edit it with:

sudo nano /etc/monit/monitrc
  • uncomment set daemon 120, so monit will run every two minutes.
  • uncomment set alert and enter the email address you want to use for email alerts.
  • uncomment include /etc/monit.d/* so it will read in all config files within that directory.
  • save and create the monit.d directory,

    sudo mkdir /etc/monit.d

By using a wildcard for that last config change you can build up separate .monitrc files for each service you want to monitor, making future modifications easier.

For example, monitor disk usage using:

sudo nano /etc/monit.d/system.monitrc

check system mysite.com
  if loadavg (1min) > 4 then alert
  if loadavg (5min) > 2 then alert
  if memory usage > 85% then alert

check device rootfs with path /dev/sda1
  if space usage > 85% then alert

Starting

After making your changes, check the syntax you used is correct with,

sudo monit -t

...if you uncommented the last line include /etc/monit.d/* and you've got no config files in there you'll get an error stating it can't find any config files there. don't worry too much about that, no need running it if you haven't got anything for it to do.

Now edit,

sudo nano /etc/default/monit

And change startup=0 to 1 to enable it, save the file and run,

sudo /etc/init.d/monit start

To start the service, stop it with,

sudo /etc/init.d/monit stop

...based partly on an example from Monit for Peace of Mind & Monitoring Ubuntu Services with Monit. More info at the official Monit site.

Plus a good example pastie here,

Linux mail, mongrel and others

Posted by John
on Sunday, 17 February 2008

Software Install Commands

sudo aptitude install postfix
sudo aptitude install sendmail

sudo aptitude remove postfix
sudo aptitude remove sendmail

install telnet (not installed by default)

sudo aptitude install telnet

Mail Commands

show pending mail in queue

mailq

access mail system

mail

pending mails stored in /var/spool

Sending Mail Manually with Telnet

telnet localhost 25

ehlo localhost
mail from: root@localhost
rcpt to: fmaster@localhost
data
Subject: hello
some content
.
quit

...the dot marks the end of the email

SendMail Commands

sudo /etc/init.d/sendmail start
sudo /etc/init.d/sendmail stop
sudo /etc/init.d/sendmail restart

config files saved in /etc/mail/config.cf

Mongrel Cluster Commands

mongrel_rails cluster::start
mongrel_rails cluster::stop
mongrel_rails cluster::restart

MySQL login

login to server

mysql -u root -p

Shutdown Server Gracefully

shutdown now and reboot

shutdown -r now

Login as Another

su anotheruser

Show Pending Jobs

jobs

Mongrel Cluster Restart

mongrel_rails cluster::restart -C /var/www/apps/[myapp]/config/mongrel_cluster.yml

replace [myapp] with your apps dir

CRONTAB + Mongrel - fire on restart

add above line to /home/deploy/start.sh

crontab -e

then add...

@reboot (sh /home/deploy/start.sh) > /dev/null 2>&1

save and should fire + restart on reboot

Kill Processes

find if firefox is running...

ps aux | grep firefox

kill firefox...

killall firefox

or specifically...

kill 19919  <- processid

NGINX

sudo /etc/init.d/nginx start
sudo /etc/init.d/nginx stop
sudo /etc/init.d/nginx restart

Using ZFS with Leopard

Posted by John
on Monday, 21 January 2008

With the inception of Sun OS 11 has come the much hoped 'dream' file system, the Zettabyte File System.

This wonder offers the ability to self-heal files, repair file corruption transparently, resize partitions on-the-fly, hot-swappable, RAID built-in and dynamic encryption. A real swiss army knife for file storage.

It also protected the files of billions when the Joyent service StrongSpace went down (now back up), thankfully data was kept under ZFS so it was all safely protected and restored without loss.

Apple are saying they'll have the ZFS file-system available with a future update (maybe 10.5.2) but if you're not scared of delving into the terminal then try installing it for yourself.

Remember this is all Alpha so not fully finalized but workable to use.

Installing the Binaries

first off, get these from...

once downloaded, go into the Terminal and navigate to the unzipped directory, you should see a 'build' directory in there.

Now to add them to your O/S, run...

sudo cp build/Release/zfs /usr/sbin/zfs
sudo cp build/Release/zpool /usr/sbin/zpool
sudo cp build/Release/libzfs.dylib /usr/lib/libzfs.dylib

Now we need to destroy any existing builds of ZFS on your machine, if existing...

sudo /bin/rm -rf /System/Library/Filesystems/zfs.fs /System/Library/Extensions/zfs.kext
sudo cp -R build/Release/zfs.fs /System/Library/Filesystems/zfs.fs
sudo cp -R build/Release/zfs.kext /System/Library/Extensions/zfs.kext

Great, now navigate to the copied files locations and make sure they have ROOT WHEEL access via...

ls -la [dir / filename]
ls -la zpool

You should see in there a column for 'ROOT WHEEL', if it's the case with all six areas your set.

Reboot and you'll be ready to start.

Building a ZFS Memory Stick

Now for this experiment I'll convert my memory stick over to using the ZFS file system.

So first off, fire up Terminal again and type...

diskutil list

This will list all disks available to the system, so in this case my 8gb stick has it's disk called : /dev/disk1

So I partition it with ZFS doing...

diskutil partitiondisk /dev/disk1 GPTFormat ZFS %noformat% 100%

Then create a simple spool for it..

zpool create stick /dev/disk1s2

...note the previous command will list the disks within this ZFS drive, one of them will be our 8GB partition, hence /dev/disk1s2.

...and stick will be the name of our new ZFS disk drive.

Next up we'll need to upgrade the ZFS spool, as by default the current build will create using version 6 for backwards compatibility of the ZFS structure so we'll upgrade it to 8 doing...

zpool upgrade

Excellent, now before you start dragging and dropping into the new stick drive on your desktop let's set some permissions so you don't have to authorise every file action.

So in the desktop, select the stick drive and press Option+J to see it's disk options.

On the bottom of the popup you'll see 'Permissions', click the padlock icon to unlock these, type in your system password and set the System and Wheel to both have R/W access.

Excellent, now you can drag & drop, delete files from the ZFS disk as if it was a normal disk drive.

Ejecting the ZFS Stick

Before you pull your USB stick out of your computer and cause a panic error, make sure to run this command to eject the drive.

sudo zpool export -f stick

Replace 'stick' with the name of your ZTS spool drive.

Deleting Trash from my Stick

There's bug in the current build so clearing your trash on a ZFS drive isn't exactly easy, so back in Terminal...

cd /volumes/stick

And now find the .Trashes directory with..

ls -la

Now delete the .Trashes directory to clear your trash,

rm -rf .Trashes

Bingo, job done.

Closing Points

Because of how the way our simple ZFS drive is structured, it performs lazy writes much like a usb drive in Windows; the user gets told the file has been commited but really it's waiting around to action.

By forcing a zpool export you are essentially telling the ZFS spool to finish any pending operations and eject the drive safely.

Some time in the future this'll be a much simpler process but for now it's pretty decent having this file system available to keep your files safe.

Looking forward to the final release.

Subversion or SourceSafe ?

Posted by John
on Saturday, 04 August 2007

In the avenue of Software Version Control and Agile Technologies there are two competing technologies, open-source Subversion SVN and Microsoft SourceSafe 2005 (now Team System).

But which is which and what benefits does each offer?

Subversion

Subversion in it’s essence is open-source so is perfectly suited for LAMP (Linux, Apache, MySQL, PHP) projects. It is uniquely flexible and simple to apply to any medium type or operating system, with easy integration to Windows via the TortoiseSVN software add-on.

It allows you to import, commit, export and differentiate successive code releases down to their line numbers really easily.

Also being in the public domain, the source-code is frequently updated and bugs fixed rapidly due to the larger exposure of it’s code-base.

Microsoft SourceSafe

Very similar to Subversion, it has been designed as a source-code cataloging and version control tool primarily for .NET / Visual Studio projects; and thus it’s flexibility ends there. It does however allow you to set aside a server as your primary project Source center and drill down to key changes in code, much like SVN.

People however have complained about a long-running database corruption issue within but this should be fixed.

So Which ?

Well the tight integration into the VS.NET IDE environment is a definite plus and on top of this SourceSafe gives you a database and Vault repository in which it keeps track of code changes; however the O/S inflexibility and more importantly the new VS.NET Team System pricing does put it out of scope for small businesses and independent startups.

It really is like my dad says ‘horses for courses’, SS maybe good in a Microsoft-Only environment, but SVN maybe better for independent startups and projects where flexibility and O/S independence are vital.

Really down to you, I know this isn’t that detailed a guide but should give you some idea as to which to choose.

What do I use ?

In reality I employ both,

For my Ruby on Rails projects I use a mixture of Capistrano, Deprec capped to a SubVersion repository on the Linux server. Works well as my development machine for that is an Apple Mac.

Workwise, I employ Subversion + TortoiseSVN to keep catalogued and protected all my web script files and xhtml micro-sites, with the repositories kept on a protected remote server.

However all my VS.NET applications are kept under SourceSafe control, hooked up to a remote server in another office; used by other developers.

Works quite well, the quick-install nature of SVN would allow me to get up-to-speed from a stolen laptop really fast (if another machine existed on standby). VS.NET would take longer but then that comes with the territory, but could be faster if you had a disk-image on standby with a carbon-copy of your development system setup.

Your choice.

John.