Passenger tuning for rails application

Guys, it has been a long over due post. I have finished tuning passenger long back but caught up with Diwali celebration.

Anyways generally I prefer simple and easily understandable configuration. So following configuration is as per my best knowledge and google findings. ;)

I have verified my configuration using ab - Apache HTTP server benchmarking tool. Also tried other tools such as Jmeter, httperf. I have also used passenger-memory-stats for finding rails instance size and passenger-status for finding number of request which are pending in global queue.

I have started passenger tuning using passenger nginx user guide. You will also find passenger-apache guide.

As I am using nginx, following configuration is with respect to nginx but same will applies for apache except some extra directives and syntax.

After digging into passenger tuning, I realized that passenger well configured for production. But still Based on nginx-passenger user guide I have collected following list of directive, which we can configure according to our needs.

Directive	Default Value	Nginx Block	Use
passenger_max_pool_size <integer>	6	http	Maximum instances on server
passenger_pool_idle_time <integer>	300 sec	http	instance idle time
passenger_max_instances_per_app <integer>	0	http	Maximum instances allowed to single app
passenger_min_instances <integer>	1	http, server, location, if	minimum number of application instances which are always active
passenger_pre_start <url>	-	http	Pre start application
passenger_use_global_queue <on|off>	on	http, server, location, if	Turns the use of global queuing on or off
passenger_ignore_client_abort <on|off>	off	http, server, location, if	Ignore client aborts
rails_framework_spawner_idle_time <integer>	1800 sec	http, server, location, if	FrameworkSpawner server idle time
rails_app_spawner_idle_time <integer>	600 sec	http, server, location, if	ApplicationSpawner server idle time
passenger_log_level <integer>	0 (Can be 0, 1, 2, 3)	http	for how much information Passenger should write into error.log
passenger_debug_log_file <filename>	error.log	http	allow to specify the file that debugging and error messages should be written
passenger_pass_header <header name>		http, server, location, if	Used to pass headers

Almost all of these directive has default value which is good enough for an applications. But there are some directive which very important to configure depending on our application environment.

1. passenger_max_pool_size :
            The maximum number of Ruby on Rails or Rack application instances that may be simultaneously active. A larger number results in higher memory usage, but improved ability to handle concurrent HTTP clients.
            The value should be at least equal to the number of CPUs (or CPU cores) that you have. If your system has 2 GB of RAM, then we recommend a value of 30. If your system is a Virtual Private Server (VPS) and has about 256 MB RAM, and is also running other services such as MySQL, then we recommend a value of 2.
            I recomond you to find your application instance size using passenger-memory-stats and configure value of passenger_max_pool_size.
            e.g.  If your application instance size is 300 MB and RAM size 2 GB then value of passenger_max_pool_size should be 4 because 1200 MB(4*300) will be used by application instance and remaning for other process.

2. passenger_max_instances_per_app :
            The maximum number of application instances that may be simultaneously active for a single application. This helps to make sure that a single application will not occupy all available slots in the application pool.
            This value must be less than passenger_max_pool_size. A value of 0 means that there is no limit placed on the number of instances a single application may use, i.e. only the global limit of passenger_max_pool_size will be enforced.

3. passenger_pre_start :
            By default, Phusion Passenger does not start any application instances until said web application is first accessed. The result is that the first visitor of said web application might experience a small delay as Phusion Passenger is starting the web application on demand. If that is undesirable, then this directive can be used to pre-started application instances during Nginx startup.
            This directive accepts the URL of the web application you want to pre-start. It may be specified any number of times.

passenger_pre_start http://foo.com/; 
passenger_pre_start http://bar.com:3500/; 
passenger_pre_start http://myblog.com/store;

Let's get into live scenarios

1. A server with multiple rails applications (Development and test server)
            I have a server with 16 GB of RAM and 8 core cpu, on which redmine(bug tracking system), test rails app and other developers app instances are running. Means you can say mutiple apps are running.
            According to passenger-memory-stats redmine is taking 300 MB and other rails app 500 MB. So I have kept following configuration

passenger_max_pool_size 16;
# arround 8 GB for rails application and remaning for other process
passenger_pool_idle_time 150;
# reduced idle time as multiple app are running
passenger_max_instances_per_app 8; 
# since single app should not use whole pool size, but simuteniouly should handle mutiple requests

2. A server with single rails application (Production server)
            On my production enviroment we have 8 GB of RAM and 4 core cpu, on which only single rails application is running.
            As my rails application instance size is 500 MB, I have kept following configuration on production

passenger_max_pool_size 10;
# arround 5 GB for rails application and remaning for other process
passenger_pool_idle_time 600;
# Increased as it is production
passenger_min_instances 2;
# atlease two instances should in memory at any time
passenger_pre_start http://myprodapp.com/;
# pre start my instance at the time of nginx start instade on first request

For testing whether required instances are forked or not use ab (Apache benchmark) as described above.

Reference : http://www.alfajango.com/blog/performance-tuning-for-phusion-passenger-an-introduction/

Simple nginx tuning for rails application on production server

Recently I was working on tuning my production environment for better performance. In production server mainly three things makes a difference to the web application,

1. REE-GC Tuning
2. Nginx Tuning
3. Passenger Tuning

As I have already done REE-GC tuning in previous articles, so lets discuss on Nginx tuning. Passenger tuning will be get covered in subsequent articles.

In Nginx there are many ways to boost your application performance, but I would like to share few important tweaks.

To validate performance improvement, you can use page-speed or Yslow

Before doing these tweaks on my rails application, yslow and page-speed score was around 25-35/100. After doing these tweaks it increased to 90-96/100

To get 100/100 we have to write a good code-design not best code-design ;). In our application the performance dip is due to not using CSS sprite.

A. Enable Gzip :

Gzip helps to compress and decompress the data on-the-fly between browser and server. Nginx has HttpGzipModule for same.

Add following lines into to http block nginx.conf

gzip  on;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript;
gzip_vary on;
gzip_disable     "MSIE [1-6]\.";
#gzip_proxied any;
#gzip_buffers 128 128k;
#gzip_min_length  1100;

Depending on your need you can set the above commented directives.

B. Enable caching :

Setting an expiry date or a maximum age in the HTTP headers for static resources instructs the browser to load previously downloaded resources from local disk rather than over the network.

Add following lines into to server block nginx.conf

   location ~* \.(ico|css|js|gif|jpe?g|png|swf)(\?[0-9]+)?$ {
         expires max;
         break;
   }

C. Minify JavaScript :

Compacting JavaScript code can save many bytes of data and speed up downloading, parsing, and execution time.

For this you can use any third party tools or gems like Jammit,asset_packager to minify your JavaScript files. Rails 3 has built in support.

Sample nginx.conf :

Apart from this you can use multiple assets server for static resources, which will be covered in subsequent articles.

Install Redis server on Ubuntu maunally as service

This post is all about installing redis server manually on Ubuntu system and then setup it as a service.

For this post I have originally referred a blog post of denofubiquity.

By default Ubuntu provides redis server installation by

sudo apt-get install redis-server

And you done with the installation of redis server.

But if you want to install latest version of redis server, then you can go through the rest of post.

wget http://redis.googlecode.com/files/redis-2.2.12.tar.gz
# You can specify a link of redis-server which you want to install
tar -zxf redis-2.2.12.tar.gz
cd redis-2.2.12
make
sudo make install

Now setup a redis.conf. I have copied a redis.conf from default installation of redis-server which I have used. If you want to install redis-server for your production machine you may refer a redis.conf from denofubiquity's post

wget https://raw.github.com/gist/1164482/77e4ecf14ffac42b0e987e7ffe16cb757d734ff9/redis.conf
sudo mkdir /etc/redis
sudo mv redis.conf /etc/redis/redis.conf

Copy redis-server startup scripts.

wget https://raw.github.com/gist/1164503/d1bc2cc6782b943d9b28aa93fc7038f4ae5a905f/redis-server
sudo mv redis-server /etc/init.d/redis-server
sudo chmod +x /etc/init.d/redis-server

sudo useradd redis
sudo mkdir -p /var/lib/redis
sudo mkdir -p /var/log/redis
sudo chown redis.redis /var/lib/redis
sudo chown redis.redis /var/log/redis

Mark redis server as startup services.

sudo update-rc.d redis-server defaults

Start your redis server.

sudo /etc/init.d/redis-server start

Note :

You can customise your redis configuration as per your requirements.
Like from Redis 2.2, it supports unix socket, so you can use it if required.
- Specify socket in redis.conf using this line
unixsocket /tmp/redis.sock
- Tell Redis.new() to read it as required. like
Redis.new(:path => "/tmp/redis.sock")

Also you can customise memory configuration in your redis.conf.

Reference :

http://www.denofubiquity.com/nosql/412

Nginx rewrite non-www url to www

One of my project facing issue with non-www url. At Browser side we are calling multiple services from different domain and same time we are sharing same cookies for calling services from our website. But it was giving authentication failure for our services.

Issue:

We have embedded zimbra collaboration suite into our rails application. Once user logged into our application, he will automatically gets Single signed into zimbra collaboration suite. We ware showing zimbra UI in separate iframe on web page.

Zimbra has nice feature to extend their own functionality using zimlet.

Zimlet is simply JavaScript code which runs on client side(On Browser).

Using the power of zimlet we want to access the services of our own application, where we ware sharing cookies which already got created for portal on browser side when user signed in.

Every thing was working fine on production. But some time some users were facing issue of authentication failure for zimlet functionality, even though they have signed in into rails application.

After digging into issue, found that some users accessing the application using without www. i.e "https://abcrails.com", and zimlet was trying to access a rails application using "https://www.abcrails.com" where authentication is failing. As cookies ware created for "https://abcrails.com" and not for "https://www.abcrails.com"

For this I have only one solution depending on my requirements, is force non-www url to www.

So following nginx configuration is for:

1. Force http reuest to https

2. Force non-www url to www

Replace DelayedJob with Resque

Background task in Ruby on Rails

One of our task of migrating background task from Delayed_job server to the Resque server. But I have kept this task with low priority in my task list queue.

But since last week, our IT team is facing an issue with Delayed_job server. Our delayed_job server is getting crashed 20 times in 24 hours.

I havn't done the root cause analysis. but I have decided to take this an opportunity to replace delayed_job to resque server on high priority.

Is is not mean that Resque is better than Delayed_job. They have their own features. The working functionality of both server is same.

As Resque is best fitting in our requirements I have choose Resque in place of Delayed_job.

Also our IT team don't have any way or clue get the list of background tasks which are completed, which are pending and which are currently in process.

Why Resque server:

Well following info is sufficient from page https://github.com/defunkt/resque

Resque vs DelayedJob

How does Resque compare to DelayedJob, and why would you choose one over the other?

• Resque supports multiple queues
• DelayedJob supports finer grained priorities
• Resque workers are resilient to memory leaks / bloat
• DelayedJob workers are extremely simple and easy to modify
• Resque requires Redis
• DelayedJob requires ActiveRecord
• Resque can only place JSONable Ruby objects on a queue as arguments
• DelayedJob can place any Ruby object on its queue as arguments
• Resque includes a Sinatra app for monitoring what's going on
• DelayedJob can be queried from within your Rails app if you want to add an interface

If you're doing Rails development, you already have a database and ActiveRecord. DelayedJob is super easy to setup and works great. GitHub used it for many months to process almost 200 million jobs.

Choose Resque if:

• You need multiple queues
• You don't care / dislike numeric priorities
• You don't need to persist every Ruby object ever
• You have potentially huge queues
• You want to see what's going on
• You expect a lot of failure / chaos
• You can setup Redis
• You're not running short on RAM

Choose DelayedJob if:

• You like numeric priorities
• You're not doing a gigantic amount of jobs each day
• Your queue stays small and nimble
• There is not a lot failure / chaos
• You want to easily throw anything on the queue
• You don't want to setup Redis

In no way is Resque a "better" DelayedJob, so make sure you pick the tool that's best for your app.

Action:

This task is done for, quick solution to the production issue. So you might also use it, if you have same issue.

This task I have done on one of our project which is running on rails 2.3.5 with REE 1.8.7. But it will work same for rails 3 also.

Before diving into migration, I highly encourage you to checkout railscasts 171(delayed_job) and RAILSCASTS-271(Resque).

1. Install Redis server

It is dependency for resque server. Like Delayed_jobs uses database(delayed_jobs table), Resque uses Redis server.

I am using ubuntu so,

> sudo apt-get install redis-server

If you want to install latest version of the Redis you can install manually.

2. Install resque gem

If you are using bundler mention it in Gemfile or in your config/environment.rb depending on your application.

3. Load Resque server tasks

Add a file called lib/tasks/resque.rake

# loads all rake task of resque
require 'resque/tasks'

# following statement is required only if your background task needs rails enviroment else skip it
task "resque:setup" => :environment

4. Replace your Delayed_job code with Resque

A.

Find a places in code where you or your developers has been used delayed_jobs.

For this you can simply find a line "send_later" or "Delayed::Job.enqueue" in your code.

Use your development IDE or use grep tool from command line

> cd your_project

> grep -rn "send_later" .

and

> grep -rn "Delayed::Job.enqueue" .

B.

From the above result you will able to find location where delayed_job is used.

So if your are using SomeModel.send_later(function_name, parameter1, parameter2, ..., parameter n)

Replace a line like this

#SomeModel.send_later(function_name, parameter1, parameter2, ..., parameter n)
Resque.enqueue(NameOfModelOrClass, parameter1, parameter2, ..., parameter n)

Name of NameOfModelOrClass can be define from the following step

Either use one of the following

1. Modify respective model

Add following line into the model,

@queue = :name_of_queue

Name of the queue can be any thing whatever you want. You can also logically separate background task by using queue name.

Mofify function_name in such way,

def self.function_name(parameter1, parameter2, ..., parameter n)
{
	# Defination of a function
}

to,

#def self.function_name(parameter1, parameter2, ..., parameter n)
def self.perform(parameter1, parameter2, ..., parameter n)
{
	# Defination of a function
}

2. Separate code from model which I have preferred.

Define class in lib folder(any location you want)

Class FileImportWorker
{
	@queue = :name_of_the_queue

	def self.perform()
	{
		# cut the defination of the function_name and paste here
	}

}

Pass this class name to Resque.enqueue method.

C.

if you are using Delayed::Job.enqueue(ClassOfJob.new(parameter1, parameter2, ..., parameter n))

#Delayed::Job.enqueue(ClassOfJob.new(parameter1, parameter2, ..., parameter n))
Resque.enqueue(ClassOfJob, parameter1, parameter2, ..., parameter n)

e.g

class NewsletterJob < Struct.new(:text, :emails)
    def perform
      emails.each { |e| NewsletterMailer.deliver_text_to_email(text, e) }
    end    
  end 

Delayed::Job.enqueue NewsletterJob.new('lorem ipsum...', Customers.find(:all).collect(&:email))

to,

class NewsletterJob
    @queue = :name_of_the_queue

    def self.perform(text, emails)
      emails.each { |e| NewsletterMailer.deliver_text_to_email(text, e) }
    end    
  end 

Resque.enqueue(NewsletterJob, 'lorem ipsum...', Customers.find(:all).collect(&:email))

5. Now start your Resque server,

You can start server using a following command.

rake resque:work QUEUE=* COUNT=1

QUEUE=* : specify on which queue to work. * for all Queue

COUNT=1 : Number of worker process

6. Access to queue

Now you can run your code and can have look of pending and in-progress tasks. For this run the following command,

resque-web

It will start resque web interface on port 5678

http://localhost:5678

Bonus Point:

1. Mount Reqsue web interface server into your application

Instead of accessing resque web interface using localhost:5478, you can mount web interface into your application.

Add the following line into your config/routes.rb

mount Resque::Server, :at => "/resque"

Now, access via http://your_server/resque

Also, you can provide basic authentication for it,

add a file called config/initializers/resque_auth.rb with following code,

Resque::Server.use(Rack::Auth::Basic) do |user, password|
  password == "secret"
end

2. You can club background tasks (which we have defined in lib folder) into single folder

As done in rails cast you can add a folder app/workers, and keep your all tasks file in it.

or

Which I done by adding folder lib/workers

Only you need to load this folder into your application,

In rails 2,

Add file config/initializers/load_workers.rb with following code.

Dir["#{RAILS_ROOT}/lib/workers/*.rb"].each { |f| require(f) }

In rails 3,

add following line in application.rb,

config.autoload_paths += %W(#{config.root}/lib/workers)

Troubleshooting:

1. Resque database connection issue

We are using postgresql in our portal but found that when I pushed my code on testing server where we are using same configuration which we have for our production ( We have configured Nginx + Passenger + REE for same), resque was not working.

After looking into resque-web interface found following error,

PGError: server closed the connection unexpectedly

Which I solved by adding following code into /lib/tasks/resque.rake

task "resque:setup" => :environment do
  ENV['QUEUE'] = '*'

  Resque.after_fork do |job|
    ActiveRecord::Base.establish_connection
  end

end

desc "Alias for resque:work (To run workers on Heroku)"
task "jobs:work" => "resque:work"

You can find this solution in following URL.

RAILSCASTS-271(Resque)

http://stackoverflow.com/questions/2611747/rails-resque-workers-fail-with-pgerror-server-closed-the-connection-unexpectedly

I didn't find time to do root cause analysis for this issue, but I thing this is due to passenger as it was working well on my development machine in production mode.

2. Sequence issue

Exact error I forgot but I have faced a sequence related error. Solution was to not to pass a complex object in call Resque.enqueue, like ActiveRecord, File data etc.

3. undefined method `******' for #Hash:000

This is due to, if your are storing complex object(ActiveRecord) in resque. Resque uses Redis server which is key-value store, which returns object in hash.

So tackle this simple pass object-id as parameter and in background processing code extract the original object again using ID.

Convert string to a class/model

This post is related to how to convert string to class ( model ). I came across this situation while migrating one of my organization's project on rails 3.

Many times we came across a situation where we need to convert a string to respective class or model. So that we can call methods on it. Generally to handle this situation new rails developers use case statement like this,

case class_name_string
when 'Account'
object = Account.find(id)

when 'Contact'
object = Contact.find(id)

when 'Opportunity'
object = Opportunity.find(id)

when 'Campaign'
object = Campaign.find(id)

when 'Model1'
object = Model1.find(id)

when 'Model2'
object = Model2.find(id)
:
:
:
:
when 'Modeln'
object = Modeln.find(id]) 
end

This can be actually done pretty easily by using constantize in only one line

class_name_string.constantize.find(id)

Touch pad is not working on Ubuntu 11.04

Following is blog post with respect to the problem which I faced on my dell xps laptop with Ubuntu 11.04. My mouse pad(Touch pad) was not working after login screen.

After some googling I found forum articles which describing same problem. Those peoples have different version of Ubuntu with different manufactures of laptop.

But I didn't found an exact cause of this issue. One of the person from the forum was saying, this is might be due to use of mobile broadband network.

Might be this is the cause because, I am also using mobile broadband on my laptop where issue is occurring. But at the same time in my company I am using Ubuntu 11.04 on my development machine with wired Internet but I never faced this issue.

Anyways following is the solution,
Execute following command on terminal

gconftool-2 --set --type boolean /desktop/gnome/peripherals/touchpad/touchpad_enabled true

Reference:

https://help.ubuntu.com/community/SynapticsTouchpad
http://ubuntuforums.org/showthread.php?t=1658103

Configure Rack-bug for rails 3

While migrating one of my rails application on rails 3, I thought of using rack-bug simultaneously for improving performance.

Following are the steps which I used for configuring rack-bug for rails 3 (3.0.6).


1. Install rack-bug (branch rails3) as plugin
   cd vendor/plugins
   git clone -b rails3 https://github.com/brynary/rack-bug.git


If you want to you it as gem then add following line into Gemfile
gem 'rack-bug', :git => 'https://github.com/brynary/rack-bug.git', :branch => 'rails3'


2. Replace the code from file actionview_extension.rb
which is avilable in vendor/plugins/rack-bug/lib/rack/bug/panels/templates_panel/ as specified in bug of rack_bug repository


if defined?(ActionView) && defined?(ActionView::Template)

  ActionView::Template.class_eval do
    def render_with_rack_bug(*args, &block)
      Rack::Bug::TemplatesPanel.record(virtual_path) do
        render_without_rack_bug(*args, &block)
      end
    end
    alias_method_chain :render, :rack_bug
  end
end

If you are using gem override the specified file in some way


3. Add following lines into your config.ru
   require 'rack/bug'
   use Rack::Bug, :secret_key => "someverylongandveryhardtoguesspreferablyrandomstring"
   run myapp


4. Start your server and access the URL http://your_app/__rack_bug__/bookmarklet.html
   and enter the password.

Ruby Version Manager with Gemset - ree installation

Recently I had given a session on Ruby version manager(RVM) with gemset, explained some of the important and useful bonus points which makes our life easier.


The reason behind this session being the need for our team to work on multiple projects and manage them efficiently, which run on different version of Rails. We are evaluating different version of ruby, like Ruby Enterprise edition, Ruby 1.9.2, Ruby 1.8.7.


I have posted my presentation on slideshare. You can download from there.

Ruby version manager

View more presentations from Rahul Chaudhari

Now, I would like to discuss a problem which my colleagues has faced while installing Ruby enterprise edition under RVM.


A. While they using a command,

rvm install ree

They have got following error,


ERROR: Error running './installer -a /home/usera/.rvm/rubies/ree-1.8.7-2011.03 --dont-install-useful-gems ', please read /home/usera/.rvm/log/ree-1.8.7-2011.03/install.log
ERROR: There has been an error while trying to run the ree installer. Halting the installation.


So I have investigated about this error and I found that, some of the dependecy are met for ree installation.
This dependency I found using, installing ree manually under rvm,
Go to ree source folder, where ree source code is located

cd ~/.rvm/src/ree-1.8.7-2010.02

Then copy the command from above error, which is

./installer -a /home/usera/.rvm/rubies/ree-1.8.7-2011.03  --dont-install-useful-gems

After running above command, it will shows why ree installation failed.
On my colleague's machine, it is shows package libreadline5-dev is not installed

So,

sudo apt-get install libreadline5-dev

Then install the dependeny that ree asked, and then install ree again

rvm install ree

B. Also one of my colleague faced following issue, while starting server using ruby script/server


/home/usera/.rvm/gems/ree-1.8.7-2011.03@livia_portal/gems/activesupport-2.3.5/lib/active_support/dependencies.rb:55: uninitialized constant ActiveSupport:ependencies::Mutex (NameError)
from /home/usera/.rvm/gems/ree-1.8.7-2011.03@livia_portal/gems/activesupport-2.3.5/lib/active_support.rb:56:in `require'
from /home/usera/.rvm/gems/ree-1.8.7-2011.03@livia_portal/gems/activesupport-2.3.5/lib/active_support.rb:56
from /home/usera/.rvm/gems/ree-1.8.7-2011.03@livia_portal/gems/rails-2.3.5/lib/commands/server.rb:1:in `require'
from /home/usera/.rvm/gems/ree-1.8.7-2011.03@livia_portal/gems/rails-2.3.5/lib/commands/server.rb:1
from script/server:3:in `require'
from script/server:3


After investing I found that, latest version of gem is conflicting with old versions of rails (Rails-2.3.5 and gem 1.6.2).


So we have added following line into config/boot.rb

require 'thread'

Which resolved the above issue.

Zimbra authentication preauth in Ruby

In one of our project, we have integrated Zimbra Collabration Server. It’s an email and calendar server plus much more; think about it like a next-generation Microsoft Exchange server. In addition to email and calendar, it provides document storage and editing, instant messaging, and simplified administrative controls all in an award winning webmail user interface built with the latest AJAX web technology. ZCS also provides mobility and syncs to desktop client applications; the server is deployed on commodity Linux and Mac server hardware.

Previosuly it was part of Yahoo, now it is part of Vmware.

Zimbra provides two types of authentication mode for single sign on,
1. URL based
2. SOAP based

The concept behind this is same is to generate pre authentication token called preauth.

As we have integrated into ruby on rails application, I have wrote a code for generate preauth.
Following is link for same.

http://wiki.zimbra.com/wiki/Preauth#Sample_Ruby_code_for_computing_the_preauth_value

Uninstall all the gems

Many time a rails developer have an situation where they want to uninstall all gems which are installed.
Here we can take an advantage of pipelining output from one command to other command.

gem clean

This command will remove multiple versions of gems and will keep only higher version of gem.

gem list --no-version | xargs gem uninstall

Here we are using pipelining, where command first will list all gems with no version, and second command will take it for consume.

Extra :

If you need to exclude some gem from being uninstall, use following command,

gem list --no-version | grep -v "rake" | xargs rvmsudo gem uninstall

Monitor Your Rails/Passenger App with Munin

This article is originally copied from Steve Schwartz blog post.

I have only modified in sense of accessing graphs using nginx as sub directory.  

Munin is a great tool to monitor resources on your server, showing graphs over time, so that you can analyze resource trends to find what’s killing your server before it causes major problems. It is also very configurable and can be made to profile and graph just about anything via plugins. And with a couple tricks, you can get it to monitor your Phusion Passenger application with ease.
Update: If you have RVM installed on your server and need Munin to work with RVM’s Passenger installation, follow all of the instructions below, and then make the changes described in Monitor Passenger with Munin when using RVM(http://tech.tjscreations.com/web/monitor-passenger-with-munin-when-using-rvm/).*
These changes will still use your server’s non-RVM-installed default Ruby to run the Munin plugin, which will in turn use your RVM-installed Ruby to run the passenger-status and passenger-memory-stats commands.
If you already have Munin installed and working, and just want to know how to get the Passenger Plugins working, you can skip directly to Install Munin Passenger Plugins, Configure Munin for Passenger Stats, and be sure to read the Gotcha.

Install Munin and Munin-node

The first step is to install Munin. If you have your server running Ubuntu, this is pretty easy. One you’ve SSH’d into your server, enter:

sudo apt-get install munin munin-node -y

If you’re running another flavor of Linux, see the Munin’s Linux install instructions. For Mac OSX, see Mac install instructions.

Install Munin Passenger Plugins

The next step is to install the Passenger plugins. The first is passenger_status:

wget http://gist.github.com/20319.txt

Modify passenger-status path in file 20319.txt, as per your path of passenger-status.
Tip: Use sudo find / -name passenger-status

sudo mv 20319.txt /usr/share/munin/plugins/passenger_status
sudo chmod a+x /usr/share/munin/plugins/passenger_status
sudo ln -s /usr/share/munin/plugins/passenger_status /etc/munin/plugins/passenger_status

The second plugin is passenger_memory_stats:

wget http://gist.github.com/21391.txt

Modify passenger_memory_stats path in file 21391.txt, as per your path of passenger_memory_stats.
Tip: Use sudo find / -name passenger_memory_stats

sudo mv 21391.txt /usr/share/munin/plugins/passenger_memory_stats
sudo chmod a+x /usr/share/munin/plugins/passenger_memory_stats
sudo ln -s /usr/share/munin/plugins/passenger_memory_stats /etc/munin/plugins/passenger_memory_stats

No go ahead and restart Munin-node (this is the process that runs Munin at regular intervals):

sudo /etc/init.d/munin-node restart

Configure Munin

Now here are where those couple of tricks come in to get Munin playing nicely with your Rails application. First we want to tell Munin where to store the html and graph images that you can access through the browser.

sudo nano /etc/munin/munin.conf

And change the following lines:

htmldir /path/to/your/rails/public/munin
[yoursite.com]

Note that the htmldir can really be any directory, but make sure it’s persistent (i.e. if you’re using Capistrano to keep revisions of your app on the server, make sure the munin directory is in the shared directory outside of your rails app root directory).
Also note that the [yoursite.com] part is only a descriptive name, so it really doesn’t matter what you call it. If you have a more complex application that runs from multiple directories or multiple servers, you can group Munin stats, and in this case, you actually have to put some thought into this line. But that’s outside the scope of this article, so you can read up on customizing Munin Master on your own time if you’d like.
Now if you haven’t already, you need to actually create that directory you just told Munin about:

cd /path/to/your/rails/public
mkdir -p munin
sudo chown munin:munin munin

Configure Munin For Passenger Stats

And finally, you need to allow the munin user to run the passenger-status and passenger-memory-stats commands without a password, since they both require sudo powers to run properly.

sudo visudo

And at the bottom of the file add:

munin   ALL=(ALL) NOPASSWD:/usr/bin/passenger-status, /usr/bin/passenger-memory-stats

Gotcha

At this point, Munin is suppose to start doing it’s stuff and all is happy in the ruby-munin marriage. For me, however, this was not the case. After combing the Munin error logs and digging through the Munin documentation and code more than I care to admit, I realized that Munin-node needs to preface the passenger stat commands with ruby. So, here’s how we fix that:

sudo nano /etc/munin/plugin-conf.d/munin-node

And then add this to the bottom of that file:

[passenger_*]
user munin
command ruby %c

Final Munin Restart

Now we’ll give Munin-node one more restart and we’re in business.

sudo /etc/init.d/munin-node restart

After waiting a few minutes, you should start to see .html files and graphs and whatnot in the /path/to/your/rails/shared/directory/munin directory. If not, you want to check out the Munin-node error logs to see what’s going on. On Ubuntu, this is found at /var/log/munin/munin-node.log.

Access Munin

http(s)://your-rails-app/munin/index.html 

Access munin as a part of rails aplication using nginx

You can access munin as part of you rails application using nginx's location directive
Add following lines into your rails application server tag of nginx,

location /munin {
  root path_to_your_munin_folder;
  index index.html index.htm;
}

Reference

http://www.alfajango.com/blog/how-to-monitor-your-railspassenger-app-with-munin#install-munin-passenger-plugins

REE garbage collector performance tuning

Ruby’s garbage collector tries to adapt memory usage to the amount of memory used by the program by dynamically growing or shrinking the allocated heap as it sees fit. For long running server applications, this approach isn’t always the most efficient one. The performance very much depends on the ratio heap_size / program_size. It behaves somewhat erratic: adding code can actually make your program run faster.
With REE, one can tune the garbage collector’s behavior for better server performance. It is possible to specify the initial heap size to start with. The heap size will never drop below the initial size. By carefully selecting the initial heap size one can decrease startup time and increase throughput of server applications.
Garbage collector behavior is controlled through the following environment variables. These environment variables must be set prior to invoking the Ruby interpreter.

• RUBY_HEAP_MIN_SLOTS
  This specifies the initial number of heap slots. The default is 10000.

• RUBY_HEAP_SLOTS_INCREMENT
  The number of additional heap slots to allocate when Ruby needs to allocate new heap slots for the first time. The default is 10000.
  For example, suppose that the default GC settings are in effect, and 10000 Ruby objects exist on the heap (= 10000 used heap slots). When the program creates another object, Ruby will allocate a new heap with 10000 heap slots in it. There are now 20000 heap slots in total, of which 10001 are used and 9999 are unused.

• RUBY_HEAP_SLOTS_GROWTH_FACTOR
  Multiplicator used for calculating the number of new heaps slots to allocate next time Ruby needs new heap slots. The default is 1.8.
  Take the program in the last example. Suppose that the program creates 10000 more objects. Upon creating the 10000th object, Ruby needs to allocate another heap. This heap will have 10000 * 1.8 = 18000 heap slots. There are now 20000 + 18000 = 38000 heap slots in total, of which 20001 are used and 17999 are unused.
  The next time Ruby needs to allocate a new heap, that heap will have 18000 * 1.8 = 32400 heap slots.

• RUBY_GC_MALLOC_LIMIT
  The amount of C data structures which can be allocated without triggering a garbage collection. If this is set too low, then the garbage collector will be started even if there are empty heap slots available. The default value is 8000000.

• RUBY_HEAP_FREE_MIN
  The number of heap slots that should be available after a garbage collector run. If fewer heap slots are available, then Ruby will allocate a new heap according to the RUBY_HEAP_SLOTS_INCREMENT and RUBY_HEAP_SLOTS_GROWTH_FACTOR parameters. The default value is 4096.

GC configuration With respect to my portal

As mentioned into coffeepowdered.net(reference link), we have used scrap plugin.
It's given following statistic

Number of objects : 2181349 (1735589 AST nodes, 79.56%)
Heap slot size : 40
GC cycles so far : 415
Number of heaps : 9
Total size of objects: 85208.95 KB
Total size of heaps : 119579.90 KB (34370.95 KB = 28.74% unused)
Leading free slots : 389261 (15205.51 KB = 12.72%)
Trailing free slots : 2 (0.08 KB = 0.00%)
Number of contiguous groups of 16 slots: 38143 (19.94%)
Number of terminal objects: 6617 (0.22%)

As, you see in the statistic, we required 9 heaps slot to feet 2181349 objects. To reduce this to feet into single heap we have added following configuration.
1. Created a file with any name. e.g

sudo vi /opt/ruby-enterprise-1.8.7-2010.02/bin/ruby_with_env

2. Added following line in it

#!/bin/bash
export RUBY_HEAP_MIN_SLOTS=2500000
export RUBY_HEAP_SLOTS_INCREMENT=200000
export RUBY_HEAP_SLOTS_GROWTH_FACTOR=1
export RUBY_GC_MALLOC_LIMIT=40000000
export RUBY_HEAP_FREE_MIN=25000
exec "/opt/ruby-enterprise-1.8.7-2010.02/bin/ruby" "$@" 

3. Given execute permission to it

sudo chmod +x /opt/ruby-enterprise-1.8.7-2010.02/bin/ruby_with_env

4. As we using passenger, so we have modified nginx.conf to use above configuration,

passenger_ruby /opt/ruby-enterprise-1.8.7-2010.02/bin/ruby;

with

passenger_ruby /opt/ruby-enterprise-1.8.7-2010.02/bin/ruby_with_env;

After doing this configuration, we got following statistic,

Number of objects : 2093655 (1698545 AST nodes, 81.13%)
Heap slot size : 40
GC cycles so far : 14
Number of heaps : 1
Total size of objects: 81783.40 KB
Total size of heaps : 97656.29 KB (15872.89 KB = 16.25% unused)
Leading free slots : 406346 (15872.89 KB = 16.25%)
Trailing free slots : 0 (0.00 KB = 0.00%)
Number of contiguous groups of 16 slots: 25396 (16.25%)
Number of terminal objects: 6993 (0.28%)

As you can see two important variables are come down drastically.
GC cycles so far : 14
Number of heaps : 1

Reference

http://www.coffeepowered.net/2009/06/13/fine-tuning-your-garbage-collector
http://www.mikeperham.com/2009/05/25/memory-hungry-ruby-daemons/

Monitor rails instances of passenger in Nginx

A month back, we have faced an issue of memory out of space which leads the server crash. We have solved this issue with help of following script. This script is sending an kill signal  rails instance which is taking a more memory. Due to this crash will be get notified at client side.

This script is use full for monitoring rails instance of passenger. As other available tools are not able to monitor rails instance which is maintained by passenger.

It will monitor rails instance such that, it will kill rails instance which is taking more than 500MB and if rails instance has processed 200 requests.

After killing rails instance, passenger will automatically fork another rails instance if required.

The reason of restarting instance after certain requests, to keep memory available for other rails instances. Since I have found an articles which are saying "Rails expands the Ruby process so much that additional memory allocation grows much larger than we actually need, due to the exponential growth factor. And since MRI never gives back unused memory"

I have saw passenger has PassengerMaxRequests and PassengerMaxMemory (Not sure) for Apache server but not available for nginx.

My script will do the same thing for nginx :).

Open any file,
e.g

vi monitor_rails_instance

and paste following code in it.

 1 #!/bin/sh
 2 
 3 while true; do
 4 
 5   passenger-memory-stats | grep Rails:\ /home  | awk ' { if($2 > 500) print "kill -9 " $1}' | bash
 6   # above line will kill all the rails instances which are using memory more than 500MB
 7 
 8   passenger-status | grep Processed:  | awk ' { if($7 > 200) print "kill -6 " $3}' | bash
 9   # above line will abort all rails instance who have processed 200 request. 
10 
11   sleep 2
12 done

Then give execute permission to the file

sudo chmod +x monitor_rails_instance

And then run this script as super user

sudo ./monitor_rails_instance

Note:

If you want to run this script as background process on server

sudo nohup ./monitor_rails_instance

Streaming replication in postgresql 9.1

Following document will tell you how to steup streaming replication for postgresql server 9.1.

We are using it for failover and high avilability of database for rails application.

Note : If you need an installation procedure for postgresql server 9.1, please refer this link. 

A. Set up primary ( master ) server as per postgresql server installation document

1. Open postgres.conf from data folder and made changes as per this,

listen_addresses = '10.50.4.91'  # IP address        

wal_level = hot_standby  # Option are ( host_standby, warm standby etc...)
#Need to study the other option and their implications

max_wal_senders = 5

wal_keep_segments = 32
# Need to explore
# Follwing commands are used earlier than 9.1 where replication works via xlog and    
# archive-restore command.
#archive_mode    = on
#archive_command = 'cp %p /path_to/archive/%f'

2. Open pg_hba.conf and add authentication for self machine and for stand by machine

host    all             all             10.50.4.91/22           trust
host    replication     postgres        10.50.4.58/22           trust
Note : Replication signifies the slave will replicate for all DB on master

B. Set up stand-by machine(for e.g here 10.50.4.58) as per postgres installation document till step 10.1 (Don't do 10.2 as we are copying data folder from primary)

C. On Primary log in as postgres
Start the postgresql server and then do the following

psql -Upostgres -h10.50.4.91 -c "SELECT pg_start_backup('label', true)"
rsync -a /usr/local/pgsql/data/ 10.50.4.58:/usr/local/pgsql/data/ --exclude postmaster.pid
psql -Upostgres -h10.50.4.91 -c "SELECT pg_stop_backup()"

D. Step 3 will create a data folder inside /usr/local/pgsql

on standby
1. Edit postgresql.conf of standby
a. Change listen_address as per standby
listen_addresses = '10.50.4.58'

b. comment out the rest of part in postgresql.conf (which we have done on primary server)

E. Enable read-only queries on the standby server. But if wal_level is archive on the primary, leave hot_standby unchanged (i.e., off).

hot_standby = on  #In postgres.conf
   
F. Edit nano /usr/local/pgsql/data/pg_hba.conf for your connection if required

for general setting

modify last two line(as we copied from primary)

host    all             all             10.50.4.58/22           trust
#host    replication     postgres        10.50.4.58/22           trust   

G. Add /usr/local/pgsql/data/recovery.conf on standby with following content

standby_mode          = 'on'
primary_conninfo      = 'host=10.50.4.91 port=5432 user=postgres'
trigger_file = '/tmp/trigger'   #Optional but need to be explore
#restore_command = 'cp /path_to/archive/%f "%p"' #Optional but need to be explore

H. Start the postgresql server of standby

Troubleshoot and verification:
You can see postgresql log for any issue or to see wether streaming replication is working or not.

Reference:
http://wiki.postgresql.org/wiki/Streaming_Replication

Primary Observations:
1. Even those standby goes down it will pickup a changes from primary at the time of startup.
2. You can not do a any modification on standby.
3. On restart any machine aromatically postgresql services get loaded

Important reference:
    http://wiki.postgresql.org/wiki/Main_Page
    Go to section Database Administration and Maintenance

Postgresql server 9.1 installation from source code

This document will tell you how to install postgresql server 9.1 from source code.

1. Download postgres 9.1

wget http://ftp9.us.postgresql.org/pub/mirrors/postgresql/source/v9.0.1/postgresql-9.0.1.tar.bz2

2. Extract postgres

tar xjvf postgresql-9.0.1.tar.bz2

3. Install following dependent packages which are needed for installation

sudo apt-get install build-essential libreadline-dev zlib1g-dev 

4. To install, go inside of the extracted folder

./configure

make

sudo make install # This will install postgresql in /usr/local/pgsql 

5. Copy server startup script under services

sudo cp contrib/start-scripts/linux /etc/init.d/postgresql

6. Then make the startup script executable.

sudo chmod 775 /etc/init.d/postgresql  

7. add the script to the Server's startup routine (init) with update-rc.d like 

 sudo update-rc.d postgresql defaults 

8. Add A Postgres User (service account) 

Now, we need to add a postgres user. This user runs the postgresql server. Postgres will not run as root. 

sudo adduser postgres --home /usr/local/pgsql 

9. Add Paths to Binaries and Man Pages 

1. sudo nano /etc/profile.d/postgresql.sh 

add lines

PATH=$PATH:/usr/local/pgsql/bin

export PATH

2. sudo nano /etc/profile.d/pgmanual.sh 

add lines

MANPATH=$MANPATH:/usr/local/pgsql/man

export MANPATH

Note : These configs can also be done in /home/<username>/.bashrc

3. Make above file executable 

sudo chmod 775 /etc/profile.d/postgresql.sh

sudo chmod 775 /etc/profile.d/pgmanual.sh 

10. Create the PostgreSQL Database Cluster 

1. Make a directory to contain the databases 

sudo mkdir /usr/local/pgsql/data

sudo chown -R postgres:postgres /usr/local/pgsql/data 

2. Execute the initdb script 

su postgres

initdb -D /usr/local/pgsql/data 

Note: If initdb command not found, use full path i.e '/usr/local/pgsql/bin/initdb'

OR

Load the postgres commands into shell environment using, 

source /etc/profile.d/postgresql.sh 

11. Start and stop serve

sudo /etc/init.d/postgresql start

sudo /etc/init.d/postgresql stop 

Reference: 

For this I have referred a following link. Actually this link shows installation of  postgres 8.3.7 but you can change it appropriately
http://www.xtuple.org/InstallingPostgresFromSource