Problems running check_memory.pl under Groundworks/Nagios

This is the error I'm getting

Can't locate Nagios/Plugin.pm in @INC ...

Looks like the script can't find the Plugin module form Nagios for Perl so we have to install this using this command (as root):

# perl -MCPAN -e 'install Nagios::Plugin' (as root)

Update: If this command fails try to delete ~/.cpan directory and retry the command

Now you should be able to login as the nagios user and run the check_memory.pl command...

$ ./check_mem.pl

CHECK_MEM OK - 4664M free | free=4890664960b;;

BAM!!

Here is the link for the check_memory.pl script

check_memory.pl

Journey with Thinking Sphinx and Crond

This is my journey through time :)

Environment:
Ruby => 1.8.7-72
Rails => 2.3.2
Thinking-sphinx => 1.1.3


I had a server running thinking-sphinx through crontab but noticed that this was not working so I started doing some debugging to see what was causing the non running thinking-sphinx:index task...

First I Changed crontab to have full path like so: (crontab -e)

*/10 * * * * cd /rails/current && RAILS_ENV=production /usr/local/bin/rake thinking_sphinx:index >> /dev/null 2>&1

This did nothing... Then I changed the output to go to a file to see if this would give me something

*/10 * * * * cd /rails/current && RAILS_ENV=production /usr/local/bin/rake thinking_sphinx:index >> /tmp/sphinx.output 2>&1

This gave me a timestamp ??? Running that exact command from the shell gives me a full reindex without any issues

Then I figured maybe I needed a trace so I moved the cron task to /etc/cron.d/thinking-sphinx.ct and put this in the file

*/10 * * * * cd /rails/current && RAILS_ENV=production /usr/local/bin/rake thinking_sphinx:index --trace >> /tmp/sphinx.output 2>&1

AND THERE IT WAS !!

!!! The bundled mysql.rb driver has been removed from Rails 2.2. Please install the mysql gem and try again: gem install mysql.

Damn I should have know. So its a PATH issue when running rake under cron... Here is the latest /etc/cron.d/thinking-sphinx.ct I have that is now WORKING!!

# ------------- minute (0 - 59)
# | ----------- hour (0 - 23)
# | | --------- day of month (1 - 31)
# | | | ------- month (1 - 12)
# | | | | ----- day of week (0 - 6) (Sunday=0)
# | | | | |
# * * * * * command to be executed

PATH=/usr/local/mysql/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/sbin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
SHELL=/bin/bash
RAILS_ENV="production"
LD_LIBRARY_PATH=/usr/local/mysql/lib:${LD_LIBRARY_PATH}


# re-index production sphinx every 15 minutes
*/10 * * * * root cd /rails/current && /usr/local/bin/rake thinking_sphinx:index >> /dev/null 2>&1

There are two important settings there setting the PATH with ruby/rakes install path and MySQLs install path. But also the LD_LIBRARY_PATH so that rails can find the MySQL library. I have that set in /etc/profile.d/mysql.sh but cron does not load the profile.

Rotation Catalina.out in Tomcat 5/6

By using swallowOutput="true" and configuring a default context we will send all logging to the default log that we have timestamp rolling enabled for. 

Engine

Logger className="org.apache.catalina.logger.FileLogger"               prefix="catalina_log." suffix=".txt" timestamp="true"
Host

Logger className="org.apache.catalina.logger.FileLogger" directory="logs" prefix="localhost_log." suffix=".txt" timestamp="true" DefaultContext reloadable="false" crossContext="true" swallowOutput="true"

Eclipse MemoryAnalyzer Exit Code=-1

I was looking for a nice tool to read java heap dumps and found that regular JHat was not a great way to go with the 1.6GiB dump files I have. A friend pointed me to a Eclipse project Memory Analyzer (MAT) http://www.eclipse.org/mat/ which is the old SAP analyzer...

Starting it up I got the Exit Code=-1 error and found that this tool is currently not working with java 1.6.x on a mac. If you want to force the tool to use 1.5 open a terminal window and follow these instructions:

freddy@svn:~/mat/MemoryAnalyzer.app/Contents$ pwd

/Users/freddy/mat/MemoryAnalyzer.app/Contents

freddy@svn:~/mat/MemoryAnalyzer.app/Contents$ vi Info.plist

Uncomment this line:

... eworks/JavaVM.framework/Versions/1.5.0/Commands/java

This will force the use of Java 1.5!

uninitialized constant CGI::Session (NameError)

After upgrading one project to Rails 2.3.2 I got an error on startup uninitialized constant CGI::Session (NameError)

Looking into the source I found that I was loading this class CGI::Session::ActiveRecordStore::Session

After a little hunting on Google I was directed to the Rails release notes and found this section:

script/server has been switched to use Rack, which means it supports any Rack compatible server. script/server will also pick up a rackup configuration file if one exists. By default, it will look for a config.ru file, but you can override this with the -c switch.

CGI::Session::ActiveRecordStore has been replaced by ActiveRecord::SessionStore.

Done!

Monitoring thins with monit...

Short story:

Download and install the latest code from http://mmonit.org/monit/download/ (configure, make, make install.. done! )

edit /etc/monitrc

# ========================================================
# Monit global settings:
# ========================================================
set daemon 60
set logfile syslog
set mailserver localhost
set mail-format { from: monit@cfandersen.com }
# ========================================================
# Monit http server settings:
# ========================================================
set httpd port 2812
allow localhost

INCLUDE "/etc/monit/thins.monitrc"

/etc/monit/thins.monitrc

check process thin-8000 with pidfile /app/shared/pids/thin.8000.pid
group rails
start program = "/usr/local/bin/thin -C /app/current/config/thin_cluster.yml start --only 8000"
stop program = "/usr/local/bin/thin -C /app/current/config/thin_cluster.yml stop --only 8000"
if failed host localhost port 8000 protocol HTTP
request "/token/monit" with timeout 60 seconds then restart
if cpu is greater than 60% for 2 cycles then alert
if cpu > 80% for 5 cycles then restart
if totalmem > 100.0 MB for 5 cycles then restart
if loadavg(5min) greater than 10 for 8 cycles then stop
if 5 restarts within 5 cycles then timeout
depends on thin_bin

check file thin_bin with path /usr/local/bin/thin
group rails
if failed checksum then unmonitor
if failed permission 755 then unmonitor
if failed uid root then unmonitor
if failed gid root then unmonitor

Thats the setup now the ISSUE!

Monit would NOT start thin, and the error I was looking at in the thin log was:

/usr/local/lib/ruby/gems/1.8/gems/rails-2.1.0/lib/rails_generator/lookup.rb:35:in `expand_path': couldn't find HOME environment -- expanding `~' (ArgumentError)
from /usr/local/lib/ruby/gems/1.8/gems/rails-2.1.0/lib/rails_generator/lookup.rb:35:in `user_home'
from /usr/local/lib/ruby/gems/1.8/gems/rails-2.1.0/lib/rails_generator/lookup.rb:114:in `use_component_sources!'
from /usr/local/lib/ruby/gems/1.8/gems/rails-2.1.0/lib/rails_generator/lookup.rb:55:in `included'
from /usr/local/lib/ruby/gems/1.8/gems/rails-2.1.0/lib/rails_generator.rb:38:in `include'
from /usr/local/lib/ruby/gems/1.8/gems/rails-2.1.0/lib/rails_generator.rb:38:in `send'
from /usr/local/lib/ruby/gems/1.8/gems/rails-2.1.0/lib/rails_generator.rb:38
from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
... 26 levels...
from /usr/local/lib/ruby/gems/1.8/gems/thin-1.0.0/lib/thin/runner.rb:139:in `run!'
from /usr/local/lib/ruby/gems/1.8/gems/thin-1.0.0/bin/thin:6
from /usr/local/bin/thin:20:in `load'
from /usr/local/bin/thin:20

I did not want to have a "startup" script hanging around SO I added this in my /usr/local/bin/thin file:

ENV["HOME"] = "/tmp"

Problem solved!

Mac(OSX) Jruby and Rails 2.2 -- Part 2

Running webrick isn't doing what we want... How about Tomcat? Or maybe Glassfish? Well I have heard that glassfish is VERY EASY, lets checkit out...

Step 1:
~$ sudo jruby -S gem install glassfish
Successfully installed glassfish-0.9.0-universal-java
1 gem installed
Installing ri documentation for glassfish-0.9.0-universal-java...
Installing RDoc documentation for glassfish-0.9.0-universal-java...

DONE... no I'm serious... THAT WAS IT!!!

Lets start the server:

~$ jruby -S glassfish_rails myapp
Nov 17, 2008 7:21:23 PM com.sun.enterprise.glassfish.bootstrap.ASMain main
INFO: Launching GlassFish on Apache Felix OSGi platform
Nov 17, 2008 7:21:23 PM com.sun.enterprise.glassfish.bootstrap.ASMainOSGi findDerbyClient
INFO: Cannot find javadb client jar file, jdbc driver not available
...

Now I can access the rails app at http://localhost:3000/

NICE!!

Here is some more information about the glassfish gem:

-c, --contextroot PATH: change the context root (default: '/')
-p, --port PORT: change server port (default: 3000)
-e, --environment ENV: change rails environment (default: development)
-n --runtimes NUMBER: Number of JRuby runtimes to crete initially
--runtimes-min NUMBER: Minimum JRuby runtimes to crete
--runtimes-max NUMBER: Maximum number of JRuby runtimes to crete
APPLICATION_PATH (optional): Path to the application to be run (default:
current).


This is almost too easy for a development environment...

THIS IS NOT A BENCHMARK BUT even so here is the info...

My laptop with a test app I have and seed data installed thin 1.0 and Glassfish...

With threading:
============
Requests per second: Thin 10 [#/sec]
Requests per second: Glassfish 14 [#/sec]

Without threading:
==============
Requests per second: Thin 3 [#/sec]
Requests per second: Glassfish 2.6 [#/sec]

:) Not what I was hoping for ...