Brad Wilson

Low Tech CI Simulation

Tue, 24 Feb 2015 00:00:00 +0000

Are random CI failures getting you down?

Are you pretty sure it's a timing issue but just can't quite reason it out?

Are you unable to replicate the problem locally on your mac?

Try my new script to slow your machine to a crawl. It'll fix you test problems guaranteed!*

* Not a guarantee

#!/usr/bin/env ruby

# leave one CPU free to run tests:
cpus = `sysctl -n hw.ncpu`.to_i - 1

puts("Forking #{cpus} processes")
(0..cpus).each { Process.spawn("yes > /dev/null") }

Process.wait

Dodgy? Yes.

Handy? Very.

Quieter Errors with Sidekiq

Sat, 14 Feb 2015 00:00:00 +0000

At my current job, we do a lot of integration with external APIs. In order to keep our UI snappy and our web servers happy, we send a lot of that communication into Sidekiq workers.

That mostly works well, but we do hit temporary problems sometimes (network timeouts, random failures on the other end, etc). Sidekiq really tries hard to let you fix errors in workers, trying 25 times before conceding defeat, so often these temporary failures are best handled by letting Sidekiq do it's thing.

Unfortunately when combined with an error notification service like Rollbar (or Airbrake or Honeybadger) you can end up with some really noisy errors. Below is a small snippet that will only send through the first error to Rollbar.

I'd prefer to be able to make every line of our app handle every possible error that can come up. We're working on that. Until that happens, having a single error per worker means we don't end up prioritising worker errors over web server errors becuase they look 25 times more common!

  # Some Sidekiq middleware to stop us sending errors for every single
  # retry to rollbar
  class SilenceRetryErrorMiddleware
    def call(_worker_class, msg, _queue)
      if msg.key?("retry_count")
        Rollbar.silenced { yield }
      else
        yield
      end
    end
  end

  Sidekiq.configure_server do |config|
    config.server_middleware do |chain|
      chain.add(SilenceRetryErrorMiddleware)
    end
  end

Convert Webby to Jekyll

Sun, 08 Feb 2015 00:00:00 +0000

I'm going to blog more. When I think of the countless times other people sharing information helps me every day, I feel bad I don't share more.

One thing that has been acting as a block to more sharing was my old blog software. It was using Webby. Webby is a nice static site generator, but one that hasn't been touched since 2009. Every time I wanted to post, I had to reinstall Ruby 1.8.7, patch together the old gems and hope everything still somehow held together.

That's too high a bar for me - most of the stuff I want to share isn't really enough to warrant that effort. It's not that I want to share total rubbish (I hope!), just that taking 45 minutes before I can even start a post really puts the pressure on.

So I'm finally switching to Github pages and Jekyll. With that combination I can write up some HTML, commit, then push. Hopefully that's convenient enough that I can push up scripts, hacks, random thoughts and ideas that might help others.

A decent way to start seems to be to post the script I used to convert my old posts from Webby to Jekyll. Not fancy, but it'll do the trick.

require "yaml"

root = "[local path to old blog]"
posts = "[local path to new blog]/_posts"

Dir.glob("#{root}**/*.txt").each do |f|
  next if File.basename(f) == "index.txt"

  _junk, header, body = File.read(f).split("---")

  name = File.basename(f)
  date = f.gsub(root, "").gsub("/#{name}", "").gsub("/", "-")
  metadata = YAML.load(header)
  destination = "#{posts}/#{date}-#{name.gsub('.txt', '.html')}"

  File.open(destination, "w") do |out|
    out.puts("---")
    out.puts("layout: post")
    out.puts("title: \"#{metadata['title']}\"")
    out.puts("date: #{metadata['created_at']}")
    out.puts("---")
    out.puts(body.gsub("\r\n", "\n"))
  end

  puts(f)
  puts(destination)
  puts("==============================")
end

MongoDB Is Mental

Sat, 28 Sep 2013 02:02:29 +0000

Pro tip: MongoDB cursors return duplicate records

If you are iterating through a data set, you should switch from:

Object.each { |o| ... }

Object.extras(hint: { _id: 1 }).each { |o| ... }

That example uses Mongoid, but this is a MongoDB problem, not a Mongoid one. It’s likely it affects other MongoDB libraries.

Students get too many awards

Each week, our systems distribute awards to students if they achieve certain goals. We started doing this naively: iterating through each record and calling a method on the object. This has a few benefits:

the code is really easy to follow
for now, it runs quickly enough for our needs
it works

At least, we thought it worked. We’d started to notice more and more reports of people getting two awards where they should only be getting one. The awards were effectively identical, with only slight differences in created_at times to tell them apart.

We ruled out some of the obvious potential causes: our crons were only firing once, the code wasn’t accidentally creating two awards.

The only other possible solution we could see is if our iteration was somehow returning the same object twice. Nah, that can’t be it… that’d be insane… that’d be impossible…

Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth.

Arthur Conan Doyle

The Internet to the rescue!

We stumbled across this post on stack overflow which suggested that was it. By default, mongoDB returns documents more than once if a write operation moves the data.

A few test iterations through “all” records in our db confirmed that iterating without a hint resulted in duplicates. Adding the hint meant we only got each record once. As far as we can tell, this is a MongoDB problem, not a Mongoid problem.

Rant

It’s not impossible that there is somebody out there has a need to return records twice in one iteration. Maybe. Regardless, there’s no way that is most common use case. MongoDB really doesn’t have that pit of success stuff down. This just seems like another time when MongoDB defaults are either wrong, misleading, or useless.

:sigh:

(Thanks to jimoleary for the helpful Stack Overflow comment)

(Originally posted at on the Blake Dev Team Blog)

Dynamic Sites on Nginx With Passenger

Sat, 04 May 2013 04:26:22 +0000

If you are using passenger with nginx, it's possible to set nginx up in a way that allows new apps (or branches of the same app) to be deployed with no change to the server config.

Step 1

Add this config to your nginx.conf. Edit the hostname to match your server, and the root to match your deploy directory

server {
   server_name ~^([-\w]+)\.yourdomain.com;
   root /var/www/$1/current/public;
   passenger_enabled on;
}

Step 2

Deploy your rails app to /var/www/whatever/current. The app will then be available at http://whatever.yourdomain.com.

Similarly, deploying to /var/www/some-internal-app/current means the app will be available at http://some-internal-app.yourdomain.com

Notes

This technique probably isn't ideal for production sites, but for internal and or temporary sites it is quite useful. Passenger takes care of spinning up apps as they are requested, so if one app is getting lots of requests, passenger will naturally spin up more app servers for that one, potentially spinning down unused apps. Providing all apps aren't being constantly accessed, this acts as a kind of resource balancing allowing an underpowered internal server to serve a lot of apps.

Webby on Heroku

Fri, 11 Jun 2010 07:48:47 +0000

The $20 a month more for beer dream

I've been hosting this blog on Slicehost for a while now. They have been a good host, but a full slice is more than I need these days.

The plan

After a bit of thinking, I realised I could pretty easily move this blog across to Heroku. It's just a bunch of static files created by Webby, so I figured a tiny Sinatra app would be just the trick.

The problem

Sinatra is perfect for these kind of tiny jobs, and ended up working without too much effort. I had a bit of a wrestle with Heroku initially. I couldn't seem to get heroku to read my articles. They were stored in a directory called "output".

The solution

After a while, I tried tried renaming the directory to "public". That seemed to do the trick. Heroku mentions that you can read the filesystem but I couldn't find too much mention of exactly what you can read. A directory called "public" seems like a pretty safe bet to be readable forever though. I told Webby to start buiding to the public directory instead. Here's my SiteFile:

SITE.blog_dir = "articles"
SITE.output_dir = "public"

The app

After getting that working, the script worked on Heroku. It's just as simple as I'd hoped it would be.

require "rubygems"
require "sinatra"

DIR = 'public'
set :public, DIR

get '/' do
  read('index.html')
end

get '/articles/*' do
  read('articles', params["splat"], 'index.html')
end

def read(*parts)
  File.read(File.join([ DIR ] + parts))
end

Not much to it - read index.html for the root path, or read the article page for any article. It'd probably be possible to merge these into a single route, but it seems like that would just complicate things unecessarily.

The source

http://github.com/bradx3/blog

Fundation

Mon, 07 Sep 2009 03:40:10 +0000

I've kept my saving in a single, high interest account for a while now. It works particularly well as a contractor in that I can put aside tax and GST and earn interest on that before I have to pay it to the government.

It has been a bit difficult to keep track of how much money was saving, and how much was tax/GST though, so I've written a simple app to keep track.

Fundation lets you set up funds to manage your saving. Each paycheck can be automatically divided up into the different funds in ratios you specify.

Anyway, Fundation is free to use, so if you're having trouble remembering how much money you've got to spend, give it a try here.

If you have any errors in Fundation, or any suggestions for improvements, please use the comments of this post to provide them.

Restart Passenger From Emacs Using Rinari

Thu, 06 Aug 2009 04:27:33 +0000

Being able to completely re-program your editor is fun. I feel like I've got a long way to go before I'm an Emacs Guru (if anybody *ever* becomes a true Emacs Guru), but it's nice being able to easily automate things I do in my editor.

I've been experimenting with using Passenger in development instead of script/server. It's not too bad. It's nice to have a server always ready to go, but it does make dropping down to the debugger difficult.

Anyway, here is a small snippet of emacs lisp to easily restart the Passenger server for the project you are currently working on. It assumes you have Rinari installed.

The function:

(defun restart-passenger ()
  "Restart passenger using the current rinari root" (interactive)
  (shell-command (concat "touch " (rinari-root) "tmp/restart.txt")))

And to bind it to a key

(global-set-key (kbd "C-c ' p") 'restart-passenger)

My Backup Script

Thu, 23 Jul 2009 22:25:51 +0000

This is my simple backup script. It keeps everything I don't already have backed up in version control safe and sound from hardware failure. Not from me though - if I delete a file accidentally, that file is really gone (unless you act fast before the next backup runs!). If I make a bad edit, that file is busted.

But I can live with that.

backup.rb

Backup.rb uses rsync over ssh to backup a number of different server. Rsync is good because it only sends what changed, so we're not transferring gigs every night.

You'll have to make some changes to the SSH_USER, REMOTE_DIR and HOSTS constants. You could also add any file types you want to ignore to the EXCLUDES list.

#!/usr/bin/env ruby

NAME = %x{ hostname }.strip
SSH_USER = "brad"
REMOTE_DIR = "/home/brad/Backups/#{ NAME }"

HOSTS = [ "clamps", "hubert.lucky-dip.net" ]
DIRS = [
        "~/Library", 
        "~/Documents",
        "~/Pictures"
]
EXCLUDES = [
            "*Cache*",
            "*NewsFire*",
            "*Cookies*",
            "*.log",
            "HistoryIndex.ox*",
            "*Virtual Machines*",
            "*.resume",
            "*.ipsw",
            "*.dmg",
            "*.corrupt"
]

HOSTS.each do |host|
  DIRS.each do |dir|
    puts "

Creating a jruby rails project

Thu, 09 Jul 2009 22:08:26 +0000

For some reason, this command:

jruby -S gem install mongrel activerecord-jdbcmysql-adapter rails

didn't install all the required gems I needed.

Manually listing all the basic rails gems needed fixed the problem for me though. Working command:

jruby -S gem install mongrel activerecord-jdbcmysql-adapter rails \
activesupport actionmailer activerecord actionpack activeresource