Last night I presented at Cookpad’s Tokyo Rails event. This installment had the largest turn out I’ve seen so far, with at least forty participants. I think word’s gotten out that Tokyo Rails has the best food of any Ruby group in the city (and great people too)! This time we had some great home cooked Korean food.

My presentation was about converting Rails plugins to gems. Fortuitously, there was an announcement last week that vendor/plugins will be deprecated, and gem plugins would become the standard way of building a Rails plugin. So the timing of my presentation worked out great.

Unfortunately, I don’t think my slides showed up so well on the projector. I’ve attached them below, so if anyone is interested, please have a look at them.

For 請求書.jp, we wanted to provide our users with a way to generate PDF versions of invoices. We used the PDFKit gem (https://github.com/jdpace/PDFKit), which by itself is a thin wrapper around wkhtmltopdf (http://code.google.com/p/wkhtmltopdf/), to easily generate PDFs. With this tool chain it is possible to generate PDFs for any page, using the HTML and CSS you already have in place.

Installation

Installation is quite simple:

1. add PDFKit to your Gemfile, install the wkhtmltopdf-binary gem
2. add the following to your config/application.rb (Rails 3) to hook PDFKit into Rack:
   

   config.middleware.use PDFKit::Middleware
   

3. add an initializer (config/initializers/pdfkit.rb) like this:
   

   PDFKit.configure do |config|
     config.default_options = { page_size: 'A4', print_media_type: true }
   end
   

Basically that’s it, but please see https://github.com/jdpace/PDFKit for more details. Anyway, pretty easy, no?

Getting it to work on Heroku

Well, the tricky part starts when you want to deploy this to Heroku.

First, you’ll need to include a statically linked 64bit Linux version of wkhtmltopdf in your application. You can download it from http://code.google.com/p/wkhtmltopdf/downloads/list (wkhtmltopdf-0.10.0_rc2-static-amd64.tar.bz2 for instance). To use the same binaries in development and production, we removed the wkhtmltopdf-binary gem again, downloaded the binaries for Linux and Mac so all our development environments are covered, and extended config/initializers/pdfkit.rb like this:

PDFKit.configure do |config|
  config.default_options = { page_size: 'A4', print_media_type: true }
  if RUBY_PLATFORM =~ /linux/
    wkhtmltopdf_executable = 'wkhtmltopdf-amd64'
  elsif RUBY_PLATFORM =~ /darwin/
    wkhtmltopdf_executable = 'wkhtmltopdf-osx'
  else
    raise "Unsupported. Must be running linux or intel-based Mac OS."
  end
  config.wkhtmltopdf = Rails.root.join('vendor', 'bin', wkhtmltopdf_executable).to_s
end

After deploying to Heroku, the second issue you are likely to notice is the following: Requesting a PDF page just sits there for a while and then returns an application error. That is because you are most likely using external stylesheets and wkhtmltopdf will request them in order to generate the PDF. But your Dyno is currently busy with generating the PDF, so it can’t respond to the incoming request regarding the stylesheets.

Unfortunately starting a second Dyno or using unicorn on the cedar stack doesn’t really help here, because Heroku’s router doesn’t know which Dynos are busy but rather distributes the workload according to different parameters. So basically doing a request from within your application to your application won’t work reliably. Putting a cache / CDN (like AWS’ cloudfront) in front of your assets mitigates the problem, but users might still see an application error because the cache / CDN needs to request those files from time to time. Serving all your assets altogether from S3 works of course, but makes deployment to Heroku harder.

The solution we employed in the end was to avoid the additional requests entirely and instead embed the printing styles into the HTML itself, using the following code in the layout (HAML):

!!!
%html{:lang => "ja"}
  %head
    %title= 請求書.jp
    - if request_from_pdfkit?
      %style{type: "text/css"}
        = File.read(Rails.root.join("public","stylesheets","print.css"))
    - else
      = javascript_include_tag 'application'
      = stylesheet_link_tag ‘application’

And in the application_helper.rb:

  def request_from_pdfkit?
    # when generating a PDF, PDFKit::Middleware will set this flag
    request.env["Rack-Middleware-PDFKit"] == "true"
  end

The downside of this solution is though that we need to have a static print.css in the public/stylesheets directory and can’t use Rails 3.1’s asset pipeline as before.

The third challenge we had to address was caused by using non-Latin script, Japanese in our case. To have Japanese script in the PDF, you need Japanese fonts installed on the server doing the PDF generation, so it can embed the font. There are no Japanese fonts installed on Heroku though. Fortunately we could come up with a way to install the required font within the Heroku environment. For that to work, we added the font to vendor/fonts in the Rails project and added the following initializer:

if Rails.env.production?
  font_dir = File.join(Dir.home, ".fonts")
  Dir.mkdir(font_dir) unless Dir.exists?(font_dir)

  Dir.glob(Rails.root.join("vendor", "fonts", "*")).each do |font|
    target = File.join(font_dir, File.basename(font))
    File.symlink(font, target) unless File.exists?(target)
  end
end

With this setup we can now reliably generate nice looking PDFs for our users.

請求書.jp allows Japanese freelancers and small businesses to easily create invoices. We’ve built it using Rails 3.0 in conjunction with Coffee Script and Sass, and host the application on Heroku. Although CoffeeScript and Sass have made developing the service easier, getting them setup on Heroku is a bit of a hassle. However, Rails 3.1 introduces the asset pipeline, which not only makes it easier to use CoffeeScript and Sass, but also handles the packaging of these resources into a single file for increased performance.

Although Rails 3.1 has not been officially released yet, it is out of beta and into the fourth release candidate. Given how attractive the asset pipeline was, I decided to give upgrading 請求書.jp a shot.

The biggest challenge was to find information about how the asset pipeline works. I was able to come across a couple articles and a presentation DHH gave, but overall the information was sparse on how it actually worked. The best success I had in understanding how everything worked was to generate a new rails project with 3.1 and then use the scaffold command to generate a simple resource. By studying the generated code, I was able to figure out how to convert our application. The following is a summary of the asset pipeline specific changes I made.

# Gemfile
gem 'rails', '3.1.0.rc4'
gem 'sprockets', '= 2.0.0.beta.10' # rails 3.1.0.rc4 compatible
gem 'sass-rails', "~> 3.1.0.rc"
gem 'coffee-script'
gem 'uglifier'

# app/assets/javascripts/application.js
//= require jquery
//= require jquery_ujs
//= require ../../../vendor/assets/javascripts/externals
//= require_tree .

# vendor/assets/javascripts/externals.js:
//= require ./jshashtable-2.1.js
//= require ./jquery.numberformatter-1.2.2.min.js

# app/assets/javascripts/invoices.coffee
// Invoicing specific CoffeeScript that is dependent on jquery.numberformatter

# app/assets/stylesheets/application.css
/*
 *= require_self
 *= require_tree ./web
 */

# app/assets/stylesheets/_compatibility.css.sass
/* Macros for browser compatibility */

# app/assets/stylesheets/web/*.css.sass
@import compatibility
/* Styling of various elements */

# app/assets/stylesheets/print.css.sass
/* Print specific CSS */

# config/application.rb
config.assets.enabled = true

# config/environments/production.rb
config.assets.compress = true
config.assets.js_compressor = :uglifier

After getting everything migrated to the asset pipeline, the rest of the upgrade involved identifying outstanding issues in Rails 3.1. These issues appeared in places where I was doing something that was a bit unusual, such as accessing a relation in an after_initialize block or storing an unsaved ActiveRecord object to the session. Rather than delving into the internals of Rails, I’ve worked around these issues.

After resolving these issues, we deployed it to Heroku. There we discovered one issue - that the New Relic plugin isn’t compatible with Rails 3.1. Once removing the plugin, we had no other issues.

Upgrading 請求書.jp to Rails 3.1 took me a total of eleven hours. In its current state, Rails 3.1 is usable, but upgrading requires a fair amount of independent research and debugging. Once it is actually released, these hiccups should go away, and upgrading your application should go smoothly.

We have just finished updating one of our bigger projects from Rails 2.3.8 to 3.0.3 and wanted to share our experience.

The update process:

For the update we performed the following tasks:

• basic Rails 3 framework update, including new routes and switch to bundler
• update the gems and plugins, replace some gems / plugins with others or with custom code (see below)
• resolve Rails and shoulda deprecations
• adapt to Rails 3 new escaping behavior
• port all mailers to the new ActionMailer API

For the update process itself, we followed the various advise out there.

Finding replacements for the gems and plugins we used wasn't always straight forward.  The replacements we had to do:

• AppConfig -> RailsConfig
• super exception notifier -> exception_notification and custom code using rescue_from to handle error display
• globalize2 -> globalize3
• subdomain_fu -> custom code; the new router already takes care of some of subdomain_fu's functionality

Some numbers:

To give you an impression of the size of the application, here are the sloccount numbers:

• test: 4047 sloc
• app: 2497 sloc
• vendor: 606 sloc
• config: 332 sloc
• lib: 188 sloc

Change statistics (taken from git, comparing before starting the upgrade with upgrade finished):

• all directories excluding vendor:
  268 of 698 files changed with 2191 insertions, 1886 deletions
• app, config, lib, test only:
  260 of 263 files changes with 1957 insertions, 1801 deletions

As you can see, changes were needed to nearly every file in app, config. The main reason we also had to change so many test cases were the deprecations in shoulda.

In total we spent 51 hours on the upgrade, plus 10 hours debugging a very strange issue.

The basic update and getting our test cases running again took us around 2 days. Another 5 days we then spent mainly on:

• Resolving deprecations and minor issues which were caused by changed behavior in new Rails or gems.
• Updating our own plugins and gems. This was rather challenging because for some parts of Rails the documentation is still lacking or outdated.
• The new escaping behavior wasn't caught by our test cases. So manual testing to find any escaped HTML chunks in was required.
• Locale in URL handling: The new router still isn't flexible enough and we couldn't DRY this up completely. For simpler sites routing-filter works nicely though.

All in all, the update took a bit longer than we expected (which was about a week). We attribute this to the amount of detail that needed to be addressed.

The delayed_job plugin for Rails does a good job for pushing tasks that take some time to process into the background, so that your users (and your Rails processes) can do other things than to wait. It uses daemons to process the backgrounded tasks, so for your system to work correctly it is essential that those daemons are running. Thus you want to make sure that those daemons are getting started when the server boots and are restarted in case they die. Those points are not addressed by delayed_job.

The usual way to get processes started at boot time under Linux is to use an init.d script. But init.d scripts only address the boot process - if the daemon dies, it won't get restarted.

D.J. Bernstein's daemontools make it very simple to create system services which achieve both of the above points: starting your daemons at system boot time and restarting them in case they die. And the best feature: to create a new service, you don't even need to include any of the typical daemon features (such as backgrounding the process) into your program. So while delayed_job uses the daemons library to provide those features, we won't be needing those.

The following steps show how to set up a new service. This assumes that you already installed daemontools (available for Ubuntu/Debian for instance: apt-get install daemontools-run).

1. Create a new directory. This is going to be the service directory.
2. Create a shell script "run" in the service directory which runs your program. This can be as simple as exec /path/to/my/program.
3. Create a symlink from the system service directory (for Ubuntu that'd be /etc/service), pointing to your new service directory.

That's it. To control your service, use the svc tool. See the manpage for more information.

In case of delayed_job, we are using the following run script:

#!/bin/sh
export RAILS_ENV=production
exec 2>/dev/null
exec setuidgid railsuser /srv/railsuser/project/current/script/delayed_job run

The script changes the user to "railsuser" (you don't want to run your delayed job processing under root; change it to match your setup), and then starts the usual delayed_job script, telling it to not put itself into the background.

One specialty to note is the handling of stderr. We redirect it to /dev/null to avoid potential "Broken pipe" exceptions in case something writes to stderr, which isn't available. Redirecting sdterr to stdout did not work.

Now, when updating, you will want to restart the delayed_job service. With capistrano we use the following task definitions:

namespace :delayed_job do
  desc "Start delayed job (if not running)"
  task :start, :roles => :app do
    sudo "svc -u /etc/service/#{application}_#{rails_env}_delayed_job"
  end
  desc "Stop delayed job"
  task :stop, :roles => :app do
    sudo "svc -d /etc/service/#{application}_#{rails_env}_delayed_job"
  end
  desc "Restart delayed job"
  task :restart, :roles => :app do
    sudo "svc -t /etc/service/#{application}_#{rails_env}_delayed_job"
  end
end

after "deploy:start", "delayed_job:start" 
after "deploy:stop", "delayed_job:stop" 
after "deploy:restart", "delayed_job:restart"

This requires that your deployment user will be able to run svc using sudo, so make sure to add this to your sudoers.

Also note that for the service names in the system service directory we use the pattern #{application}_#{rails_env}_delayed_job. Those are links to the service's directory, which are located under /srv/railsuser/project/services for our setup.

With this setup we have a pretty reliable delayed_job, and can use the same framework to run most (if not all) other services we might need with very little effort.

Out of the box, Ruby on Rails uses cookies to store a user's session ID. This is fine for most applications, but doesn't work if your application needs to support browsers that don't support cookies, such as some mobile browsers. Instead of putting the session ID in a cookie, it must be put in the URL. This increases the possibility of session fixation attacks, where one user can take over another's session, however if security isn't paramount, this is an acceptable trade off. See the Ruby on Rails Security Guide for more details on these kinds of attacks.

To enable parameter based sessions in Rails, there are a number of changes you need to make. First, in config/initializers/session_store.rb, change the default file from containing something like

ActionController::Base.session = {
  :key         => '_application_session',
  :secret      => 'secret'
}

to

ActionController::Base.session = {
  :key         => '_application_session',
  :secret      => 'secret',
  :cookie_only => false # allow session to be loaded from params
}

# Overwrite default cookie based store
ActionController::Base.session_store = :active_record_store

Now Rails can read the session ID from the URL parameters, and doesn't store the session in a cookie (the default behaviour). Besides the ActiveRecord session store, other server based stores, such as the memcache store, work as well.

In addition to enabling Rails to read the session ID from the URL parameters, the session ID must be added as a parameter. The most basic way of doing this is to define default_url_options in the ApplicationController:

def default_url_options(options = nil)
  { request.session_options[:key] => request.session_options[:id] }
end

This will ensure that the session ID is always set in the URL.

The session ID can also be added conditionally, by doing something like the following:

def default_url_options(options = nil)
  if cookies_supported?
    super
  else
    { request.session_options[:key] => request.session_options[:id] }
  end
end

If the session ID is not included in the parameters, it will fall back to the cookie.