Learning about Redis through writing a Twitter clone

Redis is a fast key-value database by Salvatore Sanfilippo with some new ideas. Sometimes it’s called a ‘data-structures database’ because it supports sets and lists. I spent a few hours this morning writing a Twitter clone that uses Redis as its datastore, which is now online at http://retwisrb.danlucraft.com. The code is available on Github.

The lists feature of Redis meant that it was not necessary anywhere in the application to serialize a datastructure into a stored value. This is a tremendous win. I don’t have much experience of writing applications against key-value databases, but I worry that serialization would cause headaches.

I didn’t find a use for the Set structure, simply because I couldn’t see a set of data that I wouldn’t at some point want to page over. The list of a user’s followers was the closest I came. The downside of using a set is that in order to paginate the followers the entire set would have to be sorted first. The downside of using a list is that membership checking (is User A following User B?) requires list traversal by Redis.

There is a SORT command, so sorting sets can be done in the database layer, but I have no desire to sort this data (which could be massive) just to iterate over it in a consistent order. I went with a list for now, but you could use a set and a list to support both types of usage, at the cost of maybe twice the memory usage. I’d like to have an ordered Set structure available, if it could be done in a fast way.

I wonder if a Set and List is all we’ll get, or if Salvatore is planning more structures. Could you implement Trees in any kind of sensible way? Arrays? Would they be useful?

Anyway, I encourage you to try Redis out soon.

New project: Gutkumber - Integration testing for Ruby-GNOME2

Gutkumber is a small set of Ruby-GNOME2 automation helpers. It comes with Cucumber formatters and some steps for writing features. This allows you to test the full application stack, driving the real Gtk GUI with simulated keyboard events, and using assertions that can refer to contents of Gtk widgets.

http://github.com/danlucraft/gutkumber/tree/master

I’ve been using Gutkumber on the Redcar editor project for a while now, and it seems to work well. Here is an example of the kind of features you can write with Gutkumber:

  Scenario: Kill line
    When I type "def foo"
    And I press "Left" then "Left"
    And I press "Ctrl+K"
    ... some Redcar specific assertions

  Scenario: Open a file
    When I press "Ctrl+O"
    And I set the "Open" dialog's filename to "plugins/edit_tab/features/fixtures/file1.rb"
    And I click the button "Open" in the dialog "Open"
    ... some Redcar specific assertions

This is still in early development (it’s probably the most rough and ready software I’ve ever released!). But I’m finding it useful and I thought it would be good to demonstrate that it is possible to thoroughly integration test your Ruby-GNOME2 applications.

I’m writing new steps and helpers as I go along, but they are quite specific to my application, so you will probably find you need to write more. It’d be great if we could build the set of helpers and steps to the point where it contains everything needed to test new Ruby-GNOME2 apps.

Songkick 2.0 in private beta!

You may have noticed that www.songkick.com hasn’t changed a bit in six months. But we’ve not been idle. Friends: email me for an invite code.

Testing Ruby-GNOME2 applications with Cucumber

Testing Ruby-GNOME2 can be a tricky thing to get right. There have been all sorts of annoying issues regarding getting Test::Unit to work, and I know my own efforts with RSpec have not been entirely pain-free.

But getting Cucumber up and running to test Ruby-GNOME2 has been a joy. I think this is because unlike the other two, Cucumber doesn’t use Ruby’s at_exit to determine when to run its tests, giving you more control over the environment.

So here’s how. The problem with testing Ruby-GNOME2 - and indeed any gui - is that they are event based. Once you decide to use Ruby-GNOME2, you hand over control of your application to the gui main-loop. But test frameworks rely on keeping that control so they can dispatch tests.

For instance, you could decide to have a button in your application that reads “Run Tests”. And that button, when pressed, would fire a gui event that sends off to Test::Unit or whatever to say ‘run all tests’, and Test::Unit would march through its test suite, and then the original event would return control to the gui.

But all your tests have been processed within the same gui event handler. So by definition, no other events can have been processed during that time. That means no keypresses or menu clicks or TreeView updates will work in your tests.

What we want is for the gui to process events as needed, but still keep the control with our tests. We do that by replacing the GTK main-loop with a loop that we control.

First you have to write your application to not start the main-loop if features are running, as we will process our events manually. So your code will need to contain a guard like:

  unless $running_features
    Gtk.main
  end

Now in a Test::Unit testcase, you would need to write code like:

  def test_click
    click_button
    process_all_gui_events
    check_dialog_visible
  end

But in Cucumber, we can write something more natural. Our feature will look like this:

Feature: User clicks on something

Scenario: Single click
  When I click on the button
  Then the dialog should be visible

Run this feature using this Gtk Cucumber formatter (requires Cucumber 0.2):

module Cucumber
  module Formatter
    class GtkFormatter < Pretty

      def visit_step(step)
        super
        while Gtk.events_pending?
          Gtk.main_iteration
        end
      end
    end
  end
end

And gui events will automatically be processed between each Cucumber step. Of course this means steps can’t rely upon events being processed within the body of the step, but steps are supposed to be short and atomic anyway, so it should not be a problem.

I have just started using this technique in Redcar, and it has been working very well (example). It’s much more comfortable than the RSpec version that I was using before. I’ll now use RSpec for unit testing in my application, with no gui interaction at all. This seems right because a test that requires the gui like this is an integration test, and Cucumber is our integration test framework!

Sounds of the Server Room

Our CEO’s headphones sounded just like an old 56k modem dialing to the rest of us, so we wondered what he could possibly be listening to…..

(Picture: Gideon Bullock)

features_report

SO your team has diligently been writing Cucumber Features in natural language English, so that non-devs can read and understand them. And you now have a large corpus of readable specs. But guess what, the features are locked in your version control system, and the non-devs either don’t know how to use it or just don’t need to. So this fantastic resource, an always up-to-date executable spec of your website, is actually pretty useless.

So use features_report to unlock those specs and turn them into a lovely PDF. Here’s a sample of the output. Install and use like this:

sudo gem install features_report
features_report features/*.feature

Features:

• Uses Prawn to generate the PDF.
• Has a table of contents of all your features
• Option to add a logo to the front page.
• If you use git, it can add a ‘last changed date’ to the table of contents, along with the name of the committer that last touched it. (Requires the gem ‘mojombo-grit‘ to be installed from Github).

Issues:

• The contents page is at the back. I don’t think Prawn has the capability to go back and add the contents page after the document has been generated, if I’m wrong please enlighten me.
• FIT tables are not handled correctly. The Cucumber project is rewriting the parser at the moment, and it looks like adding this will be lots easier afterwards so we decided to wait.

The source is available at Github and features_report was written by Daniel Lucraft and Niko Felger. Features_report was sponsored by Songkick.com. Thanks to Bryan Helmkamp for some cribbed Cucumber parsing code from features2cards.

RubyConf2008 Videos for your Ipod

Despite the recent love I gave it, Handbrake has let me down. The latest version segfaults on every job I give it. So I’ve had to find a different way to get RubyConf2008 onto my Ipod. I decided to take another look at ffmpeg’s terrifying command line options.

Confreaks have put the RubyConf 2008 videos online here. To convert the larger 960×468 videos to a format playable on your Ipod using ffmpeg:

ffmpeg -i ruby-19-what-to-expect-large.mp4 -cropleft 480  \
-vcodec h264 -b 150k -bufsize 244 -maxrate 768k \
-s 480x368 -bf 0 -level 13 -f mp4 -acodec aac -ar 48000 \
-ab 64 ruby19.mp4 

Confreaks videos have two halves: the slides and the presenter. The cropleft option removes the left hand half of the video, in this case the presenter, leaving just the slides.

If you want to test out your options you can tell ffmpeg to only process a few frames:

-vframes 2000

How to run an IRC channel for your free software project

I recently needed to set up an IRC channel for Redcar (#redcar at freenode.net), and the tutorials I found online were next to useless. I created my channel on freenode.net. Freenode provides free IRC servers for open source projects. Lots of projects I follow (#rubinius, #rspec) are on Freenode, and Freenode has all the services you need to run your channel easily. You don’t have to worry about channel wars or any of that nonsense if you register your channel with them.

Connection

Choose your IRC chat client. I used to use Pidgin, but now I use Irssi. A terminal based client was my choice because it means I can leave it open on my server and connect to it from wherever I am, work or home. That means the client is never closed, so logging can happen continuously and I do not need to disconnect and reconnect periodically. Disconnecting and reconnecting is accomplished using GNU Screen. This is a good tutorial on using Screen and Irssi together.

You do not need to use Irssi, or to have a permanently open connection to run an IRC chat. Having said that, ChanServ has some impenetrable help text about channel successors or somesuch, so may want to test this stuff out.

Register your nickname

Before you can register a channel, you have to have a registered nickname. Send a message to the NickServ bot to register the nickname you are currently using. You can send the message from any channel (though be careful with your typing otherwise you will be broadcasting your password).

/msg nickserv identify mypassword

Create your channel

You create a channel by joining it. So type:

/join #mychannel

Register your channel

Registering a channel secures it from being taken over by unscrupulous types. Freenode runs the ChanServ bot for this purpose. To register your channel send a message to ChanServ

/msg chanserv register #mychannel

You can now use ChanServ to grant yourself admin privileges when you need them, and to make sure that no one else has admin rights. To give yourself admin rights to your channel (called ‘ops’) type:

/msg chanserv op #mychannel

To remove admin rights type this:

/msg chanserv deop #mychannel

According to my sources, admin rights are like sudo, you should grant them to yourself when you need them but otherwise have them turned off.

You can get help about all of ChanServ’s commands:

/msg chanserv help

Maximum Awesome

I just took a look at our website, and I thought I would check off all the aspects of the design that are at all web2.0:

• Stripes: check.
• Reflection: check.
• Beta: check.
• Rounded corners: check.
• Gradients: check.

Gosh. If only there was a new design in the pipeline I could look forward to.

Vala Textmate Bundle

I started working on a Textmate bundle for Vala. It’s almost identical to the C# bundle, as you’d expect. Of course, this is so I can edit my Vala code in Redcar :). Download here.