Ruby 2.1 was released this Christmas, great news everyone! It sports a better GC (RGenGC - gerational GC), hierarchical method caching, some small syntax changes and non-experimental rafinements. All in all one can expect 5% to 15% performance increase which is quite awesome.
As I was reading the comments in the hacker news thread related to the release, one of them caught my eye - we need JIT in the MRI Ruby VM. OK but looks like everyone forgets about Rubinius that sports for some time a LLVM-based JIT, native threads, low-pause generational garbage collector and almost perfect support of C extensions.
Yes we also have JRuby - it might just be the fastest implementation and when coupled with a server like TorqueBox is even faster, the main issue: some C libraries need to be swapped but as one can see later in the article this was just old thinking at play, clearly things are much better now as most gems do support JRuby without any issue.
So we have this awesome middle-ground so to say between Ruby VMs: it supports both C extensions without a problem (yet, empirically at least, some more exotic gems might fail to install) but somehow it gets ignored ?
The plan is simple, take one production Rails 4.0.2 app with all its dependences and convert it to Rubinius, install Puma and do some benchmarking then if all good deploy to staging.
The setup
Rubinius extracted most of the standard library into gems so in order to properly boot any Ruby script that uses them one needs this into the Gemfile:
gem 'racc' gem 'rubysl' gem 'puma'
Notes:
- rubysl - is a rather cryptic name for ruby standard library gem
- racc is a LALR(1) parser generator written by tenderlove - is also a hard requirement for Rubinius or else no booting up Rails.
- puma - well this is the best server choice for Rubinius as it supports native threads
Gems that won't work with Rubinius (will update if I'll find more):
# gem 'oj'
Remember to comment them out or again Rails won't boot up.
For VM install and switching I'm using the good old RVM with the latest version of Rubinius 2.2.3 and Ruby 2.1.0.
On to the benchmarks!
OK for obvious reasons I can't share the source code of the app, at some point I might create a public repository with something meaty for
testing. These benchmarks are for a special case only and for some good fun also so before making any decisions based on them DO TEST first on your own.
The Rails app is actually an API so most of the important parts are disabled (i.e. Streaming DataStreaming Rendering RequestForgeryProtection) also the sprockets railtie.
ApacheBench config
I used the simple ApacheBench, Version 2.3 - clearly not the ideal tool for benchmarking (one should use siege or something similar) but
for a quick glance like this test it fits the job nicely.
The command to start it up:
ab -n400 -c16 -T'application/json' http://localhost:3000/entries
unicorn.rb
# config/unicorn.rb worker_processes Integer(ENV["WEB_CONCURRENCY"] || 3) timeout 15 preload_app true before_fork do |server, worker| # et cetera end
The command to start it up:
unicorn_rails -c config/unicorn.rb -p 3000
Results after some runs:
Concurrency Level: 16
Time taken for tests: 6.769 seconds
Complete requests: 400
Failed requests: 0
Write errors: 0
Total transferred: 3611600 bytes
HTML transferred: 3346800 bytes
Requests per second: 59.09 [#/sec] (mean)
Time per request: 270.766 [ms] (mean)
Time per request: 16.923 [ms] (mean, across all concurrent requests)
Transfer rate: 521.03 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 1.0 0 5
Processing: 49 267 36.8 270 331
Waiting: 44 266 36.8 269 330
Total: 49 267 36.3 270 331
Percentage of the requests served within a certain time (ms)
50% 270
66% 283
75% 288
80% 293
90% 308
95% 315
98% 324
99% 327
100% 331 (longest request)
puma.rb on Rubinius 2.2.3
# config/puma.rb threads 8,32 workers 1 preload_app! on_worker_boot do # et cetera end
The command to start it up:
puma -C config/puma.rb -b tcp://localhost:3000
Results after several runs (so that the JIT can do its magic):
Concurrency Level: 16
Time taken for tests: 9.383 seconds
Complete requests: 400
Failed requests: 0
Write errors: 0
Total transferred: 3590400 bytes
HTML transferred: 3346800 bytes
Requests per second: 42.63 [#/sec] (mean)
Time per request: 375.311 [ms] (mean)
Time per request: 23.457 [ms] (mean, across all concurrent requests)
Transfer rate: 373.69 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 1
Processing: 84 371 105.4 348 731
Waiting: 83 363 104.1 338 728
Total: 84 371 105.3 348 732
Percentage of the requests served within a certain time (ms)
50% 348
66% 390
75% 431
80% 458
90% 526
95% 571
98% 640
99% 683
100% 732 (longest request)
puma.rb on JRuby 1.7.9
# config/puma.rb threads 8,32 preload_app! on_worker_boot do ActiveSupport.on_load(:active_record) do ActiveRecord::Base.establish_connection end end
The command to start it up:
puma -C config/puma.rb -b tcp://localhost:3000
Gems that need replacement:
# gem 'pg' gem 'activerecord-jdbcpostgresql-adapter'
Results after several runs (so that the JIT can do its magic):
Concurrency Level: 16
Time taken for tests: 4.019 seconds
Complete requests: 400
Failed requests: 0
Write errors: 0
Total transferred: 3590400 bytes
HTML transferred: 3346800 bytes
Requests per second: 99.53 [#/sec] (mean)
Time per request: 160.760 [ms] (mean)
Time per request: 10.048 [ms] (mean, across all concurrent requests)
Transfer rate: 872.42 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 1
Processing: 35 158 31.5 157 389
Waiting: 34 151 27.1 149 261
Total: 36 158 31.5 157 389
Percentage of the requests served within a certain time (ms)
50% 157
66% 166
75% 173
80% 177
90% 189
95% 204
98% 232
99% 260
100% 389 (longest request)
Conclusion
The poor Rubinius performance might be related to the racc gem as it might be really slow as detailed in this Github thread
14 req/s vs 60 req/s (I disabled cache and the app produces lots of ActiveRecord objects that's why the numbers are rather low) makes Rubinius, for now, not a good choice for this particular Rails app.
Update
Thanks to headius I've revised the benchmarks:
- apparently my VM was accessing only one core (thus the initial abysmal performance of Rubinius and JRuby) - bumped to four
- updated all benchmarks and also added JRuby 1.7.9
jruby 99.53 ################################# cruby 59.09 ################### rubinius 42.63 ##############
As one can clearly see from the chart above JRuby is the winner by an impressive margin and with only one gem change I think it deserves to be pushed to staging.