Quieter Errors with Sidekiq

At my current job, we do a lot of integration with external APIs. In order to keep our UI snappy and our web servers happy, we send a lot of that communication into Sidekiq workers.

That mostly works well, but we do hit temporary problems sometimes (network timeouts, random failures on the other end, etc). Sidekiq really tries hard to let you fix errors in workers, trying 25 times before conceding defeat, so often these temporary failures are best handled by letting Sidekiq do it's thing.

Unfortunately when combined with an error notification service like Rollbar (or Airbrake or Honeybadger) you can end up with some really noisy errors. Below is a small snippet that will only send through the first error to Rollbar.

I'd prefer to be able to make every line of our app handle every possible error that can come up. We're working on that. Until that happens, having a single error per worker means we don't end up prioritising worker errors over web server errors becuase they look 25 times more common!

  # Some Sidekiq middleware to stop us sending errors for every single
  # retry to rollbar
  class SilenceRetryErrorMiddleware
    def call(_worker_class, msg, _queue)
      if msg.key?("retry_count")
        Rollbar.silenced { yield }
      else
        yield
      end
    end
  end

  Sidekiq.configure_server do |config|
    config.server_middleware do |chain|
      chain.add(SilenceRetryErrorMiddleware)
    end
  end