-
Low Tech CI Simulation
Are random CI failures getting you down?
Are you pretty sure it's a timing issue but just can't quite reason it out?
Are you unable to replicate the problem locally on your mac?
Try my new script to slow your machine to a crawl. It'll fix you test problems guaranteed!*
* Not a guarantee
#!/usr/bin/env ruby # leave one CPU free to run tests: cpus = `sysctl -n hw.ncpu`.to_i - 1 puts("Forking #{cpus} processes") (0..cpus).each { Process.spawn("yes > /dev/null") } Process.waitDodgy? Yes.
Handy? Very.
-
Quieter Errors with Sidekiq
At my current job, we do a lot of integration with external APIs. In order to keep our UI snappy and our web servers happy, we send a lot of that communication into Sidekiq workers.
That mostly works well, but we do hit temporary problems sometimes (network timeouts, random failures on the other end, etc). Sidekiq really tries hard to let you fix errors in workers, trying 25 times before conceding defeat, so often these temporary failures are best handled by letting Sidekiq do it's thing.
Unfortunately when combined with an error notification service like Rollbar (or Airbrake or Honeybadger) you can end up with some really noisy errors. Below is a small snippet that will only send through the first error to Rollbar.
I'd prefer to be able to make every line of our app handle every possible error that can come up. We're working on that. Until that happens, having a single error per worker means we don't end up prioritising worker errors over web server errors becuase they look 25 times more common!
# Some Sidekiq middleware to stop us sending errors for every single # retry to rollbar class SilenceRetryErrorMiddleware def call(_worker_class, msg, _queue) if msg.key?("retry_count") Rollbar.silenced { yield } else yield end end end Sidekiq.configure_server do |config| config.server_middleware do |chain| chain.add(SilenceRetryErrorMiddleware) end end -
Convert Webby to Jekyll
I'm going to blog more. When I think of the countless times other people sharing information helps me every day, I feel bad I don't share more.
One thing that has been acting as a block to more sharing was my old blog software. It was using Webby. Webby is a nice static site generator, but one that hasn't been touched since 2009. Every time I wanted to post, I had to reinstall Ruby 1.8.7, patch together the old gems and hope everything still somehow held together.
That's too high a bar for me - most of the stuff I want to share isn't really enough to warrant that effort. It's not that I want to share total rubbish (I hope!), just that taking 45 minutes before I can even start a post really puts the pressure on.
So I'm finally switching to Github pages and Jekyll. With that combination I can write up some HTML, commit, then push. Hopefully that's convenient enough that I can push up scripts, hacks, random thoughts and ideas that might help others.
A decent way to start seems to be to post the script I used to convert my old posts from Webby to Jekyll. Not fancy, but it'll do the trick.
require "yaml" root = "[local path to old blog]" posts = "[local path to new blog]/_posts" Dir.glob("#{root}**/*.txt").each do |f| next if File.basename(f) == "index.txt" _junk, header, body = File.read(f).split("---") name = File.basename(f) date = f.gsub(root, "").gsub("/#{name}", "").gsub("/", "-") metadata = YAML.load(header) destination = "#{posts}/#{date}-#{name.gsub('.txt', '.html')}" File.open(destination, "w") do |out| out.puts("---") out.puts("layout: post") out.puts("title: \"#{metadata['title']}\"") out.puts("date: #{metadata['created_at']}") out.puts("---") out.puts(body.gsub("\r\n", "\n")) end puts(f) puts(destination) puts("==============================") end -
MongoDB Is Mental
Pro tip: MongoDB cursors return duplicate records
If you are iterating through a data set, you should switch from:
Object.each { |o| ... }to
Object.extras(hint: { _id: 1 }).each { |o| ... }That example uses Mongoid, but this is a MongoDB problem, not a Mongoid one. It’s likely it affects other MongoDB libraries.
Students get too many awards
Each week, our systems distribute awards to students if they achieve certain goals. We started doing this naively: iterating through each record and calling a method on the object. This has a few benefits:
- the code is really easy to follow
- for now, it runs quickly enough for our needs
- it works
At least, we thought it worked. We’d started to notice more and more reports of people getting two awards where they should only be getting one. The awards were effectively identical, with only slight differences in created_at times to tell them apart.
We ruled out some of the obvious potential causes: our crons were only firing once, the code wasn’t accidentally creating two awards.
The only other possible solution we could see is if our iteration was somehow returning the same object twice. Nah, that can’t be it… that’d be insane… that’d be impossible…
Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth.
Arthur Conan Doyle
The Internet to the rescue!
We stumbled across this post on stack overflow which suggested that was it. By default, mongoDB returns documents more than once if a write operation moves the data.
A few test iterations through “all” records in our db confirmed that iterating without a hint resulted in duplicates. Adding the hint meant we only got each record once. As far as we can tell, this is a MongoDB problem, not a Mongoid problem.
Rant
It’s not impossible that there is somebody out there has a need to return records twice in one iteration. Maybe. Regardless, there’s no way that is most common use case. MongoDB really doesn’t have that pit of success stuff down. This just seems like another time when MongoDB defaults are either wrong, misleading, or useless.
:sigh:
(Thanks to jimoleary for the helpful Stack Overflow comment)
(Originally posted at on the Blake Dev Team Blog)
-
Dynamic Sites on Nginx With Passenger
If you are using passenger with nginx, it's possible to set nginx up in a way that allows new apps (or branches of the same app) to be deployed with no change to the server config.
Step 1
Add this config to your nginx.conf. Edit the hostname to match your server, and the root to match your deploy directoryserver { server_name ~^([-\w]+)\.yourdomain.com; root /var/www/$1/current/public; passenger_enabled on; }Step 2
Deploy your rails app to /var/www/whatever/current. The app will then be available at http://whatever.yourdomain.com.
Similarly, deploying to /var/www/some-internal-app/current means the app will be available at http://some-internal-app.yourdomain.comNotes
This technique probably isn't ideal for production sites, but for internal and or temporary sites it is quite useful. Passenger takes care of spinning up apps as they are requested, so if one app is getting lots of requests, passenger will naturally spin up more app servers for that one, potentially spinning down unused apps. Providing all apps aren't being constantly accessed, this acts as a kind of resource balancing allowing an underpowered internal server to serve a lot of apps. -
Webby on Heroku
The $20 a month more for beer dream
I've been hosting this blog on Slicehost for a while now. They have been a good host, but a full slice is more than I need these days.
The plan
After a bit of thinking, I realised I could pretty easily move this blog across to Heroku. It's just a bunch of static files created by Webby, so I figured a tiny Sinatra app would be just the trick.
The problem
Sinatra is perfect for these kind of tiny jobs, and ended up working without too much effort. I had a bit of a wrestle with Heroku initially. I couldn't seem to get heroku to read my articles. They were stored in a directory called "output".
The solution
After a while, I tried tried renaming the directory to "public". That seemed to do the trick. Heroku mentions that you can read the filesystem but I couldn't find too much mention of exactly what you can read. A directory called "public" seems like a pretty safe bet to be readable forever though. I told Webby to start buiding to the public directory instead. Here's my SiteFile:
SITE.blog_dir = "articles" SITE.output_dir = "public"
The app
After getting that working, the script worked on Heroku. It's just as simple as I'd hoped it would be.
require "rubygems" require "sinatra" DIR = 'public' set :public, DIR get '/' do read('index.html') end get '/articles/*' do read('articles', params["splat"], 'index.html') end def read(*parts) File.read(File.join([ DIR ] + parts)) endNot much to it - read index.html for the root path, or read the article page for any article. It'd probably be possible to merge these into a single route, but it seems like that would just complicate things unecessarily.
The source
-
Fundation
I've kept my saving in a single, high interest account for a while now. It works particularly well as a contractor in that I can put aside tax and GST and earn interest on that before I have to pay it to the government.
It has been a bit difficult to keep track of how much money was saving, and how much was tax/GST though, so I've written a simple app to keep track.
Fundation lets you set up funds to manage your saving. Each paycheck can be automatically divided up into the different funds in ratios you specify.
Anyway, Fundation is free to use, so if you're having trouble remembering how much money you've got to spend, give it a try here.
If you have any errors in Fundation, or any suggestions for improvements, please use the comments of this post to provide them.
-
Restart Passenger From Emacs Using Rinari
Being able to completely re-program your editor is fun. I feel like I've got a long way to go before I'm an Emacs Guru (if anybody *ever* becomes a true Emacs Guru), but it's nice being able to easily automate things I do in my editor.
I've been experimenting with using Passenger in development instead of script/server. It's not too bad. It's nice to have a server always ready to go, but it does make dropping down to the debugger difficult.
Anyway, here is a small snippet of emacs lisp to easily restart the Passenger server for the project you are currently working on. It assumes you have Rinari installed.
The function:
(defun restart-passenger () "Restart passenger using the current rinari root" (interactive) (shell-command (concat "touch " (rinari-root) "tmp/restart.txt")))
And to bind it to a key
(global-set-key (kbd "C-c ' p") 'restart-passenger)
-
My Backup Script
This is my simple backup script. It keeps everything I don't already have backed up in version control safe and sound from hardware failure. Not from me though - if I delete a file accidentally, that file is really gone (unless you act fast before the next backup runs!). If I make a bad edit, that file is busted.
But I can live with that.backup.rb
Backup.rb uses rsync over ssh to backup a number of different server. Rsync is good because it only sends what changed, so we're not transferring gigs every night.
You'll have to make some changes to the SSH_USER, REMOTE_DIR and HOSTS constants. You could also add any file types you want to ignore to the EXCLUDES list.#!/usr/bin/env ruby NAME = %x{ hostname }.strip SSH_USER = "brad" REMOTE_DIR = "/home/brad/Backups/#{ NAME }" HOSTS = [ "clamps", "hubert.lucky-dip.net" ] DIRS = [ "~/Library", "~/Documents", "~/Pictures" ] EXCLUDES = [ "*Cache*", "*NewsFire*", "*Cookies*", "*.log", "HistoryIndex.ox*", "*Virtual Machines*", "*.resume", "*.ipsw", "*.dmg", "*.corrupt" ] HOSTS.each do |host| DIRS.each do |dir| puts " -
Creating a jruby rails project
For some reason, this command:
jruby -S gem install mongrel activerecord-jdbcmysql-adapter rails
didn't install all the required gems I needed.Manually listing all the basic rails gems needed fixed the problem for me though. Working command:
jruby -S gem install mongrel activerecord-jdbcmysql-adapter rails \ activesupport actionmailer activerecord actionpack activeresource
-
Kill All Rinari Buffers in Emacs
I've been using Emacs for development for a few months now and am pretty happy with it.
I use the Rinari mode to provide some help when I'm working on Rails projects. Once I'm finished for the day (or changing to another project, I used to use the kill-some-buffers command and go through them one by one.
Here's some elisp that I've written and put in my .emacs file that just kills all buffers that link to files that are in the current rinari-root. It's probably not perfect, but it seems to work most of the time.
;; some rinari helpers (defun kill-buffers-in-subdir (subdir buffer) "Kills the given buffer if it is linked to a file in the current rinari project." (if (buffer-in-subdir-p subdir buffer) (kill-buffer buffer))) (defun buffer-in-subdir-p (subdir buffer) "Returns true if buffer belongs to the current rinari project" (and (buffer-file-name buffer) (string-match subdir (buffer-file-name buffer)))) (defun kill-all-rinari-buffers () "Kills all buffers linked to the current rinari project" (interactive) (let ((path (rinari-root))) (if path (dolist (buffer (buffer-list)) (kill-buffers-in-subdir path buffer))))) -
Omniweb Using Webkit From Safari 4
I'm a big fan of the Omniweb web browser. I've had some problems with it being a bit slow in the past, but that seems to have been fixed with the latest Nightly Builds. The latest build has brought in the same Webkit as used by Safari 4, so it's fast. Really fast.
Add that to all the features I like, and I'm a happy web browser. Things I miss when I'm not using Omniweb.
- Vertical Tab Bar - I have a wide screen, and a lot of tabs open
- Workspaces - I can put my house hunting links in one workspace, then keep them handy, but hide them while I work
- Mac (?) font rendering. Something about Firefox's font just looks wrong
- Built in ad-blocking. It's no Adblock Plus, but I don't mind seeing ads. It's just the flashing ones I hate
- Location bar hides, shows when I hit Apple-L, then hides again. I don't know what I'm doing wrong on other browsers, but I can never get this to work reliably.
- And heaps more...
-
Blog Running on Webby
The server this blog runs on hosts a few things for - email, a few sites and some git repositories. It's got 512mb or ram, but rubygems and even git can get pretty greedy at times. I finally had enough.
Typo and mysql were only being used for the blog and were taking up a heap or ram. I looked around and Webby seemed a good fit.
I've managed to convert the old articles and comments to Webby and Disqus format. There's currently some problems with the comments not displaying, so I'll have to fix that, but I can live with that for now.
Anyway if anybody notices any problems, please leave a comment and I'll get it fixed up. Unless it's a problem posting comments.
Update: Comments are display fine now. Wooh.
-
Time To Go
I've just released my first iPhone application.
Time To Go is designed to be the quickest way to tell when the train you need to catch leaves. I use it just before I leave work to know whether I need to run to the station to catch the next train.
There'll be a dedicated page up to log any timetable errors soon, but for now please leave any errors or suggestions in the comments for this blog post.
Screenshot:
-
Badged Dock Icon for IPhone XCode development
Dr Nic made a good suggestion on Twitter for a way to badge the XCode icon with the one currently in use.
Here's my attempt: http://gist.github.com/49767
It relies on ImageMagick being installed (port install ImageMagick) and should be run from the project's directorin in the terminal.
-
Element.insert, Prototype and named anchors
This was a strange one.
HTML:
<a name="questions"> <div class="questions"> </div>
JS:
Element.insert("questions", { bottom : "<div>Test</div>" })With that setup, IE compains and says "Invalid source HTML for this operation". It turns out that prototype was trying to put the div into the a tag. I guess I can see how it happened, but I sure didn't expect it.
Anyway I removed the a tag completely and everything works fine. Weird.
-
Random wallpaper from socwall.com
I quite like http://socwall.com. They have a rss feed for recent posts, but I wrote a small script to download a random selection instead.
On OSX I could then set my desktop background to be a random picture from a folder. In my case that folder was /Users/brad/Pictures/Wallpapers/ but that could be easily changed.
#!/usr/bin/ruby require 'rubygems' require 'hpricot' require 'open-uri' require 'Pathname' URL = 'http://socwall.com/browse/index.php?wpSortby=8' DEST = "/Users/brad/Pictures/Wallpapers/" def clear_old_images Pathname.new(DEST).children.each { |f| f.delete } end def download_images(count) count = count.to_i while count > 0 doc = Hpricot(open(URL)) doc.search("div.wpThumbnail").each do |thumb| next if count == 0 thumbnail = thumb.at("img")['src'] image = thumbnail.gsub(/\/tb_/, '/') image = image.gsub(" ", "%20") remote_image = open(image, "User-Agent" => "Ruby/#{RUBY_VERSION}") local_image = open("#{ DEST }/#{ count }.png", 'w') local_image.write(remote_image.read) remote_image.close local_image.close count -= 1 end end end count = ARGV[0] || 10 clear_old_images download_images(count) -
ZSH
I've been zsh lately. I get the feeling most shells have similar features, but somehow I stumbled across a good config file for this one. It's so good I think it is worth sharing.
Features that I like in zsh:
- Standard aliases, reverse search, history work as normal
- Typos in command names and directories are fixed automatically
- Auto 'cd' if you just type the directory name.
- Completion of command options (so the - and -- options!)
- Copletion of rake task names. This might the same as above, but it's still handy.
- Completion of remote paths over scp. It's a bit slow for regular use, but can be good if you've forgotten the name of one dir.
My zshrc is available here on github: http://github.com/bradx3/dotfiles/tree/master/.zshrc
-
performSelector in RubyCocoa
I had a lot of trouble finding documentation for this. If you want to make a call on the main thread in RubyCocoa, the format is something like:
performSelectorOnMainThread_withObject_waitUntilDone('method_name:', object, true)So two things that took me a while:
- method_name has a trailing colon.
- The object the has the method you want to call should be the second argument. I got it in my head it should be the first for some reason.
-
Read table row-by-row in ruby
I needed to export some data from a huge data table. Iterating through MyModel.find took too long and too much ram, so I went straight to the db instead.
Example code was tough to find, so here's an example:sql = "select * from #{ MyModel.table_name }" MyModel.connection.execute(sql) do |handle| handle.fetch do |row| # row is an array of values ... export row ... end end -
Named SQL Server Instances with FreeTDS
A helpful post on accessing named sql server instances using freetds
I was having a lot of trouble doing that. Turns out the 'instance' key exists. As far as I could find it wasn't in the doc anywhere, so this is just a post in the hope that it'll save somebody else some time in the future.
From the post:[def] host = abc instance = def port = 1433 client charset = UTF-8 tds version = 8.0
-
Using Javascript Code for RJS Instead of IDs
Some info on how to make RJS output code like:
new Insertion.Bottom($$('p.welcome b').first(), "Some item"http://sentia.com.au/2008/03/using-javascript-code-for-rjs.html
-
Redish Greenish for ZenTest
ZenTest is a great tool. I use it and its redgreen plugin a lot to easily keep an eye on my tests.
I use a green terminal background though, so when my tests pass, rather than a green line, I have to rely on a lack of a red line. This isn't so bad, but I thought it could be improved with a little effort.
Redish greenish is an extension of redgreen that allows you to specify the colours used for tests passing and/or failing.
To use it, save this file into your autotest lib directory. (Mine is GEM_PATH/1.8/gems/ZenTest-3.6.1/lib/autotest/). To enable redishgreenish, you'll need to modify your autotest config file. Open "~/.autotest" and change the line:
require 'autotest/redgreen'
to
require 'autotest/redishgreenish'
At this point, autotest will behave exactly the same as if redgreen was used. To change the colour of the lines used to signify tests passing and failing, your autotest config file should look like
PASSED = :blue FAILED = :yellow require 'autotest/redishgreenish'
Valid values for colours are
:black, :red, :green, :yellow, :blue, :magenta, :cyan, :white
-
Haml & Sass Editors 0.5.4
These editors have been merged into the Aptana project. Please don't post any error reports here.
Last release didn't go so well. For some reason I figured other people had tried to install Aptana, had it fail and then given up on it like me. Turns out I was alone haha.
So I spent a bit of time figuring out what changed in RDT and fixed things up. I've tested this with a straight out of the box install of Aptana m8, so hopefully it'll work for others too.
If anybody does try it, please let me know if it works or not. I'm going to use this blog post as a bit of a test before I post to the Haml mail list. Thanks in advance to anyone who replies!
(Oh yeah, the update site is still http://haml.lucky-dip.net/).
-
Haml & Sass Editor For Eclipse
These editors have been merged into the Aptana project. Please don't post any error reports here.
I did some very basic editors for Haml and Sass a while back. At the time I was interested in Haml. It's elegant to write, and the automatically formatted html it outputs just made it a match made in heaven.
Fast forward a few months and now I'm actually using Haml and Sass. It's as good as I ever hoped it would be. With Haml and a little bit of FormBuilder magic I'm making useful forms in four short lines. Too good.
Anyway coming with using it is the chance to see some shortcomings in my own editors, so I've spent a bit of time updating them. I've set up a proper Eclipse update site, so you can just add 'http://haml.lucky-dip.net' via the 'Help - Software Updates - Find and Install' menu option. That should help minimize a few errors with the various versions of Eclipse and RDT around. All references to the RadRails plugins have been removed. There was a bit of code that I pulled into my plugin, but with RadRails/Aptana in heavy dev it's easier for me to forget about trying to keep up for a while.
Also added is code folding. There's a bit of work still to do on this one. You can currently only fold top level elements. This can be a pain when you've got all your code in one file, but if you're using layouts and partial in a sensible way it should hopefully be of some use.
One thing I found a bit tricky was figuring out which element I was under when I was editing. Some people might like turning on space markers (can you actually do that in Eclipse?), but I figured a character matcher would do the trick. So now as you move around in the editors there'll be a blue box around the element you are currently in. At the moment it just highlights the first letter of the element. I'm going to see if I can get the whole element happening for the next release.
So I hope it works well for people. Set up an Eclipse update site of 'http://haml.lucky-dip.net' and that should keep you informed as new versions come out.
subscribe via RSS