Logging and Notifications with Rogger, Graylog2 and Twilio

January 09, 2016

Rogger 0.1.2 was released yes­ter­day, so check it out if you need a way to log your Rails appli­ca­tions / Rake tasks on Gray­log2!

Anyway, this post is a show­case of how we’re using Rogger in pro­duc­tion to log excep­tions raised in Rake tasks and notify us by SMS using Twilio.

We have a few Rake tasks that run daily. Some of them are pretty siz­able, and we want to know if any of them fails and throws an excep­tion. Using Rog­ger’s log_exceptions, we can wrap the task like so:

namespace :important_tasks do
  task :first_task => :environment do
    Rogger.log_exceptions do
      some_function_that_might_fail_and_throw
    end
  end

  task :second_task => :environment do
    Rogger.log_exceptions do
      Rogger.info "Starting task important_tasks:second_task..."
      bad_function_that_might_fail
    end
  end
end

If the task does throw, Rogger will log it in Gray­log2:

log
log

Notice the level. Level 4 cor­re­sponds to the :error level (level 7 for :debug), which is what log_exceptions sends excep­tion mes­sages with.

Gray­log2 Streams

We then create a stream in Gray­log2 with the fol­low­ing rules:

source must match exactly MyImportantApp
level must be smaller than 5

The source name cor­re­sponds to the app_name set­ting as con­fig­ured in rogger.yml, and defaults to <%= Rails.application.class.parent if defined?(Rails) %>.

Twilio Noti­fi­ca­tions

We install the offi­cial Gray­log2 Twilio SMS plugin on all our Gray­log2 nodes. The instruc­tions are in the README but it’s really straight­for­ward - just wget the .jar file into your Gray­log2 plu­g­ins folder and restart the Gray­log2 server process (./graylogctl restart).

Once it’s suc­cess­ful­ly installed on your Gray­log2 nodes, you can add the alert as a call­back, and add your Twilio account details in:

log
log

We can test if the alert is work­ing by using “Send Test Alert”, and even do a full inte­gra­tion test by writ­ing a sure-fail test (some­thing like x = 1/0 will do nicely):

log
log

Con­clu­sion

Now, every­time any of our afore­men­tioned daily Rake tasks throw an excep­tion, we will be noti­fied by SMS - thank­ful­ly we set it up to run daily at a sane time (no, not in the middle of the night).