• Low Tech CI Simulation

    Are random CI failures getting you down?

    Are you pretty sure it's a timing issue but just can't quite reason it out?

    Are you unable to replicate the problem locally on your mac?

    Try my new script to slow your machine to a crawl. It'll fix you test problems guaranteed!*

    * Not a guarantee

    #!/usr/bin/env ruby
    
    # leave one CPU free to run tests:
    cpus = `sysctl -n hw.ncpu`.to_i - 1
    
    puts("Forking #{cpus} processes")
    (0..cpus).each { Process.spawn("yes > /dev/null") }
    
    Process.wait
      

    Dodgy? Yes.

    Handy? Very.

    Comments

  • Quieter Errors with Sidekiq

    At my current job, we do a lot of integration with external APIs. In order to keep our UI snappy and our web servers happy, we send a lot of that communication into Sidekiq workers.

    That mostly works well, but we do hit temporary problems sometimes (network timeouts, random failures on the other end, etc). Sidekiq really tries hard to let you fix errors in workers, trying 25 times before conceding defeat, so often these temporary failures are best handled by letting Sidekiq do it's thing.

    Unfortunately when combined with an error notification service like Rollbar (or Airbrake or Honeybadger) you can end up with some really noisy errors. Below is a small snippet that will only send through the first error to Rollbar.

    I'd prefer to be able to make every line of our app handle every possible error that can come up. We're working on that. Until that happens, having a single error per worker means we don't end up prioritising worker errors over web server errors becuase they look 25 times more common!

      # Some Sidekiq middleware to stop us sending errors for every single
      # retry to rollbar
      class SilenceRetryErrorMiddleware
        def call(_worker_class, msg, _queue)
          if msg.key?("retry_count")
            Rollbar.silenced { yield }
          else
            yield
          end
        end
      end
    
      Sidekiq.configure_server do |config|
        config.server_middleware do |chain|
          chain.add(SilenceRetryErrorMiddleware)
        end
      end
      

    Comments

  • Convert Webby to Jekyll

    I'm going to blog more. When I think of the countless times other people sharing information helps me every day, I feel bad I don't share more.

    One thing that has been acting as a block to more sharing was my old blog software. It was using Webby. Webby is a nice static site generator, but one that hasn't been touched since 2009. Every time I wanted to post, I had to reinstall Ruby 1.8.7, patch together the old gems and hope everything still somehow held together.

    That's too high a bar for me - most of the stuff I want to share isn't really enough to warrant that effort. It's not that I want to share total rubbish (I hope!), just that taking 45 minutes before I can even start a post really puts the pressure on.

    So I'm finally switching to Github pages and Jekyll. With that combination I can write up some HTML, commit, then push. Hopefully that's convenient enough that I can push up scripts, hacks, random thoughts and ideas that might help others.

    A decent way to start seems to be to post the script I used to convert my old posts from Webby to Jekyll. Not fancy, but it'll do the trick.

    require "yaml"
    
    root = "[local path to old blog]"
    posts = "[local path to new blog]/_posts"
    
    Dir.glob("#{root}**/*.txt").each do |f|
      next if File.basename(f) == "index.txt"
    
      _junk, header, body = File.read(f).split("---")
    
      name = File.basename(f)
      date = f.gsub(root, "").gsub("/#{name}", "").gsub("/", "-")
      metadata = YAML.load(header)
      destination = "#{posts}/#{date}-#{name.gsub('.txt', '.html')}"
    
      File.open(destination, "w") do |out|
        out.puts("---")
        out.puts("layout: post")
        out.puts("title: \"#{metadata['title']}\"")
        out.puts("date: #{metadata['created_at']}")
        out.puts("---")
        out.puts(body.gsub("\r\n", "\n"))
      end
    
      puts(f)
      puts(destination)
      puts("==============================")
    end
      

    Comments

  • MongoDB Is Mental

    Pro tip: MongoDB cursors return duplicate records

    If you are iterating through a data set, you should switch from:

    Object.each { |o| ... }
    

    to

    Object.extras(hint: { _id: 1 }).each { |o| ... }
    

    That example uses Mongoid, but this is a MongoDB problem, not a Mongoid one. It’s likely it affects other MongoDB libraries.

    Students get too many awards

    Each week, our systems distribute awards to students if they achieve certain goals. We started doing this naively: iterating through each record and calling a method on the object. This has a few benefits:

    1. the code is really easy to follow
    2. for now, it runs quickly enough for our needs
    3. it works

    At least, we thought it worked. We’d started to notice more and more reports of people getting two awards where they should only be getting one. The awards were effectively identical, with only slight differences in created_at times to tell them apart.

    We ruled out some of the obvious potential causes: our crons were only firing once, the code wasn’t accidentally creating two awards.

    The only other possible solution we could see is if our iteration was somehow returning the same object twice. Nah, that can’t be it… that’d be insane… that’d be impossible…

    Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth.

    Arthur Conan Doyle

    The Internet to the rescue!

    We stumbled across this post on stack overflow which suggested that was it. By default, mongoDB returns documents more than once if a write operation moves the data.

    A few test iterations through “all” records in our db confirmed that iterating without a hint resulted in duplicates. Adding the hint meant we only got each record once. As far as we can tell, this is a MongoDB problem, not a Mongoid problem.

    Rant

    It’s not impossible that there is somebody out there has a need to return records twice in one iteration. Maybe. Regardless, there’s no way that is most common use case. MongoDB really doesn’t have that pit of success stuff down. This just seems like another time when MongoDB defaults are either wrong, misleading, or useless.

    :sigh:

    (Thanks to jimoleary for the helpful Stack Overflow comment)

    (Originally posted at on the Blake Dev Team Blog)

    Comments

  • Dynamic Sites on Nginx With Passenger

    If you are using passenger with nginx, it's possible to set nginx up in a way that allows new apps (or branches of the same app) to be deployed with no change to the server config.

    Step 1

    Add this config to your nginx.conf. Edit the hostname to match your server, and the root to match your deploy directory
    server {
       server_name ~^([-\w]+)\.yourdomain.com;
       root /var/www/$1/current/public;
       passenger_enabled on;
    }

    Step 2

    Deploy your rails app to /var/www/whatever/current. The app will then be available at http://whatever.yourdomain.com.

    Similarly, deploying to /var/www/some-internal-app/current means the app will be available at http://some-internal-app.yourdomain.com

    Notes

    This technique probably isn't ideal for production sites, but for internal and or temporary sites it is quite useful. Passenger takes care of spinning up apps as they are requested, so if one app is getting lots of requests, passenger will naturally spin up more app servers for that one, potentially spinning down unused apps. Providing all apps aren't being constantly accessed, this acts as a kind of resource balancing allowing an underpowered internal server to serve a lot of apps.

    Comments

  • Webby on Heroku

    The $20 a month more for beer dream

    I've been hosting this blog on Slicehost for a while now. They have been a good host, but a full slice is more than I need these days.

    The plan

    After a bit of thinking, I realised I could pretty easily move this blog across to Heroku. It's just a bunch of static files created by Webby, so I figured a tiny Sinatra app would be just the trick.

    The problem

    Sinatra is perfect for these kind of tiny jobs, and ended up working without too much effort. I had a bit of a wrestle with Heroku initially. I couldn't seem to get heroku to read my articles. They were stored in a directory called "output".

    The solution

    After a while, I tried tried renaming the directory to "public". That seemed to do the trick. Heroku mentions that you can read the filesystem but I couldn't find too much mention of exactly what you can read. A directory called "public" seems like a pretty safe bet to be readable forever though. I told Webby to start buiding to the public directory instead. Here's my SiteFile:

    SITE.blog_dir = "articles"
    SITE.output_dir = "public"

    The app

    After getting that working, the script worked on Heroku. It's just as simple as I'd hoped it would be.

    require "rubygems"
    require "sinatra"
    
    DIR = 'public'
    set :public, DIR
    
    get '/' do
      read('index.html')
    end
    
    get '/articles/*' do
      read('articles', params["splat"], 'index.html')
    end
    
    def read(*parts)
      File.read(File.join([ DIR ] + parts))
    end

    Not much to it - read index.html for the root path, or read the article page for any article. It'd probably be possible to merge these into a single route, but it seems like that would just complicate things unecessarily.

    The source

    http://github.com/bradx3/blog

    Comments

  • Fundation

    I've kept my saving in a single, high interest account for a while now. It works particularly well as a contractor in that I can put aside tax and GST and earn interest on that before I have to pay it to the government.

    It has been a bit difficult to keep track of how much money was saving, and how much was tax/GST though, so I've written a simple app to keep track.

    Fundation lets you set up funds to manage your saving. Each paycheck can be automatically divided up into the different funds in ratios you specify.

    Anyway, Fundation is free to use, so if you're having trouble remembering how much money you've got to spend, give it a try here.

    If you have any errors in Fundation, or any suggestions for improvements, please use the comments of this post to provide them.

    Comments

  • Restart Passenger From Emacs Using Rinari

    Being able to completely re-program your editor is fun. I feel like I've got a long way to go before I'm an Emacs Guru (if anybody *ever* becomes a true Emacs Guru), but it's nice being able to easily automate things I do in my editor.

    I've been experimenting with using Passenger in development instead of script/server. It's not too bad. It's nice to have a server always ready to go, but it does make dropping down to the debugger difficult.

    Anyway, here is a small snippet of emacs lisp to easily restart the Passenger server for the project you are currently working on. It assumes you have Rinari installed.

    The function:

    (defun restart-passenger ()
      "Restart passenger using the current rinari root" (interactive)
      (shell-command (concat "touch " (rinari-root) "tmp/restart.txt")))
    

    And to bind it to a key

    (global-set-key (kbd "C-c ' p") 'restart-passenger)
    

    Comments

  • My Backup Script

    This is my simple backup script. It keeps everything I don't already have backed up in version control safe and sound from hardware failure. Not from me though - if I delete a file accidentally, that file is really gone (unless you act fast before the next backup runs!). If I make a bad edit, that file is busted.

    But I can live with that.

    backup.rb

    Backup.rb uses rsync over ssh to backup a number of different server. Rsync is good because it only sends what changed, so we're not transferring gigs every night.

    You'll have to make some changes to the SSH_USER, REMOTE_DIR and HOSTS constants. You could also add any file types you want to ignore to the EXCLUDES list.
    #!/usr/bin/env ruby
    
    NAME = %x{ hostname }.strip
    SSH_USER = "brad"
    REMOTE_DIR = "/home/brad/Backups/#{ NAME }"
    
    HOSTS = [ "clamps", "hubert.lucky-dip.net" ]
    DIRS = [
            "~/Library", 
            "~/Documents",
            "~/Pictures"
    ]
    EXCLUDES = [
                "*Cache*",
                "*NewsFire*",
                "*Cookies*",
                "*.log",
                "HistoryIndex.ox*",
                "*Virtual Machines*",
                "*.resume",
                "*.ipsw",
                "*.dmg",
                "*.corrupt"
    ]
    
    HOSTS.each do |host|
      DIRS.each do |dir|
        puts "
    
            

    Comments

  • Creating a jruby rails project

    For some reason, this command:

    jruby -S gem install mongrel activerecord-jdbcmysql-adapter rails
    
    didn't install all the required gems I needed.

    Manually listing all the basic rails gems needed fixed the problem for me though. Working command:

    jruby -S gem install mongrel activerecord-jdbcmysql-adapter rails \
    activesupport actionmailer activerecord actionpack activeresource
    

    Comments

  • Kill All Rinari Buffers in Emacs

    I've been using Emacs for development for a few months now and am pretty happy with it.

    I use the Rinari mode to provide some help when I'm working on Rails projects. Once I'm finished for the day (or changing to another project, I used to use the kill-some-buffers command and go through them one by one.

    Here's some elisp that I've written and put in my .emacs file that just kills all buffers that link to files that are in the current rinari-root. It's probably not perfect, but it seems to work most of the time.

    ;; some rinari helpers
    (defun kill-buffers-in-subdir (subdir buffer)
      "Kills the given buffer if it is linked to a file in the current rinari project."
      (if (buffer-in-subdir-p subdir buffer)
         (kill-buffer buffer)))
    
    (defun buffer-in-subdir-p (subdir buffer) 
      "Returns true if buffer belongs to the current rinari project"
      (and (buffer-file-name buffer)
           (string-match subdir (buffer-file-name buffer))))
    
    (defun kill-all-rinari-buffers ()
      "Kills all buffers linked to the current rinari project"
      (interactive)
      (let ((path (rinari-root)))
        (if path
    	(dolist (buffer (buffer-list))
    	  (kill-buffers-in-subdir path buffer)))))
    

    Comments

  • Omniweb Using Webkit From Safari 4

    I'm a big fan of the Omniweb web browser. I've had some problems with it being a bit slow in the past, but that seems to have been fixed with the latest Nightly Builds. The latest build has brought in the same Webkit as used by Safari 4, so it's fast. Really fast.

    Add that to all the features I like, and I'm a happy web browser. Things I miss when I'm not using Omniweb.

    • Vertical Tab Bar - I have a wide screen, and a lot of tabs open
    • Workspaces - I can put my house hunting links in one workspace, then keep them handy, but hide them while I work
    • Mac (?) font rendering. Something about Firefox's font just looks wrong
    • Built in ad-blocking. It's no Adblock Plus, but I don't mind seeing ads. It's just the flashing ones I hate
    • Location bar hides, shows when I hit Apple-L, then hides again. I don't know what I'm doing wrong on other browsers, but I can never get this to work reliably.
    • And heaps more...
    It's a good browser. It's good enough I paid for it, but it's free now. Nice one OmniGroup.

    Comments

  • Blog Running on Webby

    The server this blog runs on hosts a few things for - email, a few sites and some git repositories. It's got 512mb or ram, but rubygems and even git can get pretty greedy at times. I finally had enough.

    Typo and mysql were only being used for the blog and were taking up a heap or ram. I looked around and Webby seemed a good fit.

    I've managed to convert the old articles and comments to Webby and Disqus format. There's currently some problems with the comments not displaying, so I'll have to fix that, but I can live with that for now.

    Anyway if anybody notices any problems, please leave a comment and I'll get it fixed up. Unless it's a problem posting comments.

    Update: Comments are display fine now. Wooh.

    Comments

  • Time To Go

    I've just released my first iPhone application.

    Time To Go is designed to be the quickest way to tell when the train you need to catch leaves. I use it just before I leave work to know whether I need to run to the station to catch the next train.

    There'll be a dedicated page up to log any timetable errors soon, but for now please leave any errors or suggestions in the comments for this blog post.

    iTunes Store URL

    Screenshot:

    Comments

  • Badged Dock Icon for IPhone XCode development

    Dr Nic made a good suggestion on Twitter for a way to badge the XCode icon with the one currently in use.

    Here's my attempt: http://gist.github.com/49767

    It relies on ImageMagick being installed (port install ImageMagick) and should be run from the project's directorin in the terminal.

    Comments

  • Element.insert, Prototype and named anchors

    This was a strange one.

    HTML:

    <a name="questions">
    <div class="questions">
    </div>
    

    JS:

    Element.insert("questions", { bottom : "<div>Test</div>" })
    

    With that setup, IE compains and says "Invalid source HTML for this operation". It turns out that prototype was trying to put the div into the a tag. I guess I can see how it happened, but I sure didn't expect it.

    Anyway I removed the a tag completely and everything works fine. Weird.

    Comments

  • Random wallpaper from socwall.com

    I quite like http://socwall.com. They have a rss feed for recent posts, but I wrote a small script to download a random selection instead.

    On OSX I could then set my desktop background to be a random picture from a folder. In my case that folder was /Users/brad/Pictures/Wallpapers/ but that could be easily changed.

    #!/usr/bin/ruby
    
    require 'rubygems'
    require 'hpricot'
    require 'open-uri'
    require 'Pathname'
     
    URL = 'http://socwall.com/browse/index.php?wpSortby=8'
    DEST = "/Users/brad/Pictures/Wallpapers/"
    
    
    def clear_old_images
      Pathname.new(DEST).children.each { |f| f.delete }
    end
    
    def download_images(count)
      count = count.to_i
      
      while count > 0
        doc = Hpricot(open(URL))
        
        doc.search("div.wpThumbnail").each do |thumb|
          next if count == 0
          
          thumbnail = thumb.at("img")['src']
          image = thumbnail.gsub(/\/tb_/, '/')
          image = image.gsub(" ", "%20")
          
          remote_image = open(image, 
                              "User-Agent" => "Ruby/#{RUBY_VERSION}")
          local_image = open("#{ DEST }/#{ count }.png", 'w')
          local_image.write(remote_image.read)
          
          remote_image.close
          local_image.close
          
          count -= 1
        end
      end
    end
    
    count = ARGV[0] || 10
    
    clear_old_images
    download_images(count)
    

    Comments

  • ZSH

    I've been zsh lately. I get the feeling most shells have similar features, but somehow I stumbled across a good config file for this one. It's so good I think it is worth sharing.

    Features that I like in zsh:

    • Standard aliases, reverse search, history work as normal
    • Typos in command names and directories are fixed automatically
    • Auto 'cd' if you just type the directory name.
    • Completion of command options (so the - and -- options!)
    • Copletion of rake task names. This might the same as above, but it's still handy.
    • Completion of remote paths over scp. It's a bit slow for regular use, but can be good if you've forgotten the name of one dir.

    My zshrc is available here on github: http://github.com/bradx3/dotfiles/tree/master/.zshrc

    Comments

  • performSelector in RubyCocoa

    I had a lot of trouble finding documentation for this. If you want to make a call on the main thread in RubyCocoa, the format is something like:

    performSelectorOnMainThread_withObject_waitUntilDone('method_name:', object, true)
    

    So two things that took me a while:

    1. method_name has a trailing colon.
    2. The object the has the method you want to call should be the second argument. I got it in my head it should be the first for some reason.

    Comments

  • Read table row-by-row in ruby

    I needed to export some data from a huge data table. Iterating through MyModel.find took too long and too much ram, so I went straight to the db instead.

    Example code was tough to find, so here's an example:
    sql = "select * from #{ MyModel.table_name }"
    MyModel.connection.execute(sql) do |handle|
      handle.fetch do |row|
        # row is an array of values
        ... export row ...
      end
    end
    

    Comments

  • Named SQL Server Instances with FreeTDS

    A helpful post on accessing named sql server instances using freetds

    I was having a lot of trouble doing that. Turns out the 'instance' key exists. As far as I could find it wasn't in the doc anywhere, so this is just a post in the hope that it'll save somebody else some time in the future.

    From the post:
    [def]
    host = abc
    instance = def
    port = 1433
    client charset = UTF-8
    tds version = 8.0
    

    Comments

  • Using Javascript Code for RJS Instead of IDs

    Some info on how to make RJS output code like:

    new Insertion.Bottom($$('p.welcome b').first(), "Some item"
    

    http://sentia.com.au/2008/03/using-javascript-code-for-rjs.html

    Comments

  • Redish Greenish for ZenTest

    ZenTest is a great tool. I use it and its redgreen plugin a lot to easily keep an eye on my tests.

    I use a green terminal background though, so when my tests pass, rather than a green line, I have to rely on a lack of a red line. This isn't so bad, but I thought it could be improved with a little effort.

    Redish greenish is an extension of redgreen that allows you to specify the colours used for tests passing and/or failing.

    To use it, save this file into your autotest lib directory. (Mine is GEM_PATH/1.8/gems/ZenTest-3.6.1/lib/autotest/). To enable redishgreenish, you'll need to modify your autotest config file. Open "~/.autotest" and change the line:

    require 'autotest/redgreen'
    

    to
    require 'autotest/redishgreenish'
    

    At this point, autotest will behave exactly the same as if redgreen was used. To change the colour of the lines used to signify tests passing and failing, your autotest config file should look like

    PASSED = :blue
    FAILED = :yellow
    require 'autotest/redishgreenish'
    

    Valid values for colours are

    :black, :red, :green, :yellow, :blue, :magenta, :cyan, :white
    

    Comments

  • Haml & Sass Editors 0.5.4

    These editors have been merged into the Aptana project. Please don't post any error reports here.

    Last release didn't go so well. For some reason I figured other people had tried to install Aptana, had it fail and then given up on it like me. Turns out I was alone haha.

    So I spent a bit of time figuring out what changed in RDT and fixed things up. I've tested this with a straight out of the box install of Aptana m8, so hopefully it'll work for others too.

    If anybody does try it, please let me know if it works or not. I'm going to use this blog post as a bit of a test before I post to the Haml mail list. Thanks in advance to anyone who replies!

    (Oh yeah, the update site is still http://haml.lucky-dip.net/).

    Comments

  • Haml & Sass Editor For Eclipse

    These editors have been merged into the Aptana project. Please don't post any error reports here.

    I did some very basic editors for Haml and Sass a while back. At the time I was interested in Haml. It's elegant to write, and the automatically formatted html it outputs just made it a match made in heaven.

    Fast forward a few months and now I'm actually using Haml and Sass. It's as good as I ever hoped it would be. With Haml and a little bit of FormBuilder magic I'm making useful forms in four short lines. Too good.

    Anyway coming with using it is the chance to see some shortcomings in my own editors, so I've spent a bit of time updating them. I've set up a proper Eclipse update site, so you can just add 'http://haml.lucky-dip.net' via the 'Help - Software Updates - Find and Install' menu option. That should help minimize a few errors with the various versions of Eclipse and RDT around. All references to the RadRails plugins have been removed. There was a bit of code that I pulled into my plugin, but with RadRails/Aptana in heavy dev it's easier for me to forget about trying to keep up for a while.

    Also added is code folding. There's a bit of work still to do on this one. You can currently only fold top level elements. This can be a pain when you've got all your code in one file, but if you're using layouts and partial in a sensible way it should hopefully be of some use.

    One thing I found a bit tricky was figuring out which element I was under when I was editing. Some people might like turning on space markers (can you actually do that in Eclipse?), but I figured a character matcher would do the trick. So now as you move around in the editors there'll be a blue box around the element you are currently in. At the moment it just highlights the first letter of the element. I'm going to see if I can get the whole element happening for the next release.

    So I hope it works well for people. Set up an Eclipse update site of 'http://haml.lucky-dip.net' and that should keep you informed as new versions come out.

    Comments

subscribe via RSS