Keep your Rails logs free of unwanted noise

I like to use Rollbar 's default plan for small apps or prototypes that are wip. Getting production exceptions in my inbox can be quite useful but most of them are actually 404s - sure there are a multitude of ways for filtering out the noise (Fail2Ban , hand written IPTables rules , etc.)

The main idea here is to filter as much of the Rails's logs noise with simple enough changes — security is a completely different topic (though some of the tools do touch on that realm too).

nginx

First thing I've observed was that my app was getting scanned via the IP and not the domain name — so first the fix comes as a configuration change in nginx to enable responses only via the app's host:

server {
  listen 80;
  server_name {the-ip-address};
  return 404;
}

I'm not really sure whether I should use 404 or 403 — Not Found sounds correct since one should not access the app via its IP but Forbidden is somewhat equally suitable — the resource is here but you cannot access it via the server's IP.

This single configuration change reduced the logging noise by about 200 daily faux-requests — which is quite nice.

Rack::Attack

Next stop: automatic scans for WordPress vulnerabilities — lots and lots of requests for wp-admin and wp-login — just ghastly.

How can we fix this? We get down on the application level i.e. Rack.

The idea is simple: put a Rack middleware in front of the chain that can just drop all those pesky WP scans: Rack::Attack

Installation is simple: add the gem to your Gemfile, create a config file like /config/initialisers/rack_attact.rb and add:

# After 1 blocked requests in 10 minutes, block all requests from that IP for 1 day.
Rack::Attack.blocklist('fail2ban pentesters') do |req|
  # `filter` returns truthy value if request fails, or if it's from a previously banned IP so the request is blocked
  Rack::Attack::Fail2Ban.filter(
    "pentesters-#{req.ip}",
    maxretry: 1,
    findtime: 10.minutes,
    bantime: 1.day
  ) do
    # The count for the IP is incremented if the return value is truthy
    req.path.include?('wp-admin') ||
    req.path.include?('wp-login')
  end
end

Any requests to /{wp-admin | wp-login} will translate into an instant one-day ban for that IP. Clearly you could do this using Fail2Ban but Rack::Attack comes with other niceties too and it ticks the simple checkbox whilst being more portable than Fail2Ban.

The final step is to actually use this middleware by prepending it in config.ru:

require ::File.expand_path('../config/environment',  __FILE__)

use Rack::Attack
run YourApp::Application

Why in config.ru and not in the config files of Rails? Mostly because of semantics — it's not related to the business logic of the app and it makes it explicit that's the first Rack middleware a request going through nginx will hit.

After all of this is set-up — a simple request to '/wp-admin' will result in a 403 with the Forbidden message and a one-day ban.

Conclusion

before: 220 to 240 - 404 exceptions/day
after: 10 or fewer - 404 exceptions/day

and this without completely turning off 404 exceptions.

Tagged under: