Friend and colleague, Jake Archibald, has been developing Woosh, which is a JavaScript speed testing framework. Essentially, it's been developed for Glow because we want to make sure that Glow 2 kicks Glow 1's ass (and any else who fancies a piece), but he's open-sourced the work to let everyone benefit from it.
I thought I'd run you through how to set up some basic tests and start benchmarking your own code with Woosh. Bear with me, as it's still quite new to us too.
Setup
Firstly, go and grab the latest copy of Woosh from the Github repo and pop it somewhere to work with it. You're just running scripts, so there's nothing to install or configure. Bear in mind that at the time of writing, Woosh isn't at it's first version yet - so, not that I'm doubting Jake's work, you may find the odd bug and if you do, I'm sure logging it in the issues tracker would be marvellous.
If you're a git user, feel free to include Woosh as a submodule of your own project.
Woosh is primarily designed for comparing libraries, but there's no reason why you can't use it to take a benchmark of your existing scripts and then work up optimised versions to compare. If your code can be unit tested well, it can be speed tested just as easily.
Firstly, you need to let Woosh know about the scripts you want to test. You can just add references to each of your scripts using the Woosh.libs property. For each script to test, though, just make sure they have a unique name so you can reference them later (have a sneaky look in Woosh.js to see which libraries already exist and the formats used - infact, if you're taking your own copy of Woosh, you can just edit your files and add your scripts straight into this file and skip adding them in the test runner page).
Below is how your test runner HTML page should look (also available in the examples directory in the Woosh repo). Notice the reference to your script in the woosh include section, then beneath are links to your individual test files. To make it more manageable, it's probably best to have one test JS for each script you're comparing. Remember that Woosh looks for your scripts relative to where you've got woosh.js.
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<title>My Tests</title>
<!-- include woosh -->
<script src="/where/youve/got/it/lib/woosh/woosh.js" type="text/javascript">
woosh.libs['myTestScript1'] = ['/path/to/scripts/myTestScript.js']
</script>
<!-- Add any CSS you need for the test, but restrict styles to #htmlForTest -->
<!-- Add your tests. The first will be treated as the master -->
<script src="MyTestScipt1-Tests.js" type="text/javascript"></script>
</head>
<body>
<div id="wooshOutput"></div>
<div id="htmlForTest">
<!--
Put elements you want to use in your tests here. The page will be
refreshed for each set of tests, so don't worry about one framework
messing with another.
-->
</div>
</body>
</html>
The final item in the example above would be your intitial test script to be benchmarked (MyTestScript1-Tests.js). This is just a JavaScript file which will call woosh.addTests (as further down).
Now you've got a choice: You can either make minor changes and incrementally watch the improvements, possibly using the save feature, or you can create a copy of your script. I'd recommend the latter, so create a copy of your script, and add a reference to it with Woosh.libs again and also create a file to add the actual tests to for it.
You can add and compare as many scripts as you like, so long as their methods are comparably the same.
Creating tests
Adding tests is easy and in a way, they become an extension of your unit tests, confirming that the return values or behaviours match across the board.
Test files look like this and you can either put all your tests in each file, with a block for each script, or put each block in it's own file. So, below would be your contents of MyTestScript1-Tests.js. You'll need a second for MyTestScript2-Tests.js etc.
woosh.addTests('myTestScript1',
'Test name identifier 1': new woosh.Test(1000, function() {
return myFunc();
}),
'Test name identifier 2': new woosh.Test(1000, function() {
return myOtherFunc();
})
});
Things that matter about these files:
- The name identifier needs to match for each of the tests. Woosh isn't looking for them in order - it's matching on the names to know which should go into each row. i.e. "Test name identifier 1" should be the same in all test files for matching tests for that function.
- The first parameter of addTests should be the name you gave the script in the Woosh.libs command, so Woosh can find your script.
- The first parameter of woosh.Test is the number of times a test is to be run. This should be the same for sets of tests for the same thing. If it's not, Woosh will flag up the test as being unfair.
The value for iteration times is important. It's large because that'll help shake out inaccuracies. Woosh will run the test for the number of times specified, then divide the result by this number to give the average run time for that function. You may find that some browsers don't cope so well with very large numbers of iterations (uh.. IE, we're looking at you) so don't go mad with it and think that running it a million times will help your accuracy. On Glow, we tend to aim to run tests from 100 to 10000 times.
Saving tests
You can save one previous set of tests by clicking the floppy-disk icon. It's just stored in a cookie, and will be over-written if you choose to save another column of tests, but it's useful if you're just doing some small changes and just want to compare before and after.
The hard work
Now, of course, it's down to your hard work. Writing the speed tests is really the easy bit, made ever more so by the simplicity of Woosh. Try your optimisations in the second script and use Woosh to help you benchmark the new script against the old one. All you need to do is load up the test runner page and hit start. The results will pop up as they complete and become colour coded as the results are compared. Keep an eye out for tests that error (they'll go slate grey) or test titles that turn yellow (the titles click to expand further test information). Either of these can indicate that the test isn't fair because the iteration value isn't matching, the return values aren't the same or a method has failed all together. You should aim to have all your tests running without errors or warnings.
Another thing to note is that you'll still need to run all of these tests in all of the browsers you want to optimise for. You'll find massive varience in some cases and it'll be up to you to decide where to keep the speed. Jake's Full Frontal presentation covers some of the things to look out for, so that's definitely worth a look over (most importantly, make sure you're not running developer tools like Firebug when running your tests, since it'll skew your results quite heavily).
Further reading
If you want to have a look at some real tests, Glow 2 has a fair few now for some of the basic modules. They're all up on github, so have a dig around or feel free to clone the repo and run the tests yourself.
The full API has been documented for Woosh, too, although I believe that might be an exclusive as I cannot see reference to it from the github docs at the moment. I recommend taking a look through those to see about running async tests and preparing your code with setup functionality using $preTest, as well as a few other features you might find useful.
On another testing topic, Mat Hampson published an article on A-B testing on the new BBC Web Developer blog.
As with last year, I kept a list of cinema visits for the year.
Films (at the cinema, in seen order):
- The Spirit
- New Shorts: Funny Shit
- Zack and Miri Make a Porno
- Slumdog Millionaire
- Frost/Nixon
- Better Things
- Revolutionary Road
- Hansel and Gretel
- Los Cronocrímenes (Timecrimes)
- The Curious Case of Benjamin Button
- Üç Maymun (Three Monkeys)
- Role Models
- Tokyo Sonata
- Gran Torino
- Surveillance
- Franklyn
- Flame & Citron
- Watchmen
- The International
- Bronson
- In The City of Sylvia
- The Boat That Rocked
- Two Lovers
- Cherry Blossoms
- Martyrs
- Let The Right One In
- Monsters Vs Aliens
- Knowing
- In The Loop
- Star Trek
- The Grocer's Son
- London Sci-Fi Film festival Blink of An Eye Shorts Programme 1
- Kurôn wa kokyô wo mezasu
- Eyeborgs
- The City of Lost Children (with Marc Caro)
- London Sci-Fi Film festival Blink of An Eye Long Shorts/Short Longs
- London Sci-Fi Film festival Blink of An Eye Shorts Programme 2
- Synecdoche, New York
- Coraline
- Observe and Report
- Drag Me To Hell
- Terminator Salvation
- Brüno
- Public Enemies
- Sunshine Cleaning
- Moon
- Inglorious Basterds
- The Hurt Locker
- Jetsam
- District 9
- Away We Go
- Antichrist
- Surrogates
- Up
- Adventureland
- The Imaginarium of Doctor Parnassus
- Fantastic Mr Fox
- 9
- An Education
- The Men Who Stare At Goats
- Cold Souls
- Coco Avant Chanel
- A Serious Man
- Paranormal Activity
- Where The Wild Things Are
- Avatar
- Sherlock Holmes
- Thirst
- Fish Tank
- Unmade Beds
Bonus
Knowing about films can win you prizes! At least, it did for me. The Sonic Tonic had a halloween special podcast, which included snippets from horror flicks in between tracks. Whoever could identify the most won a t-shirt. I got all of them but one. Go me! As promised to the ST guys, here's my best myspace-esque photo of my winnings. Props to Paul Cripps for the tip-off and to my mum for always letting me stay up past my bedtime to watch scary films.
This year, I also recorded all the films I saw via oo5, which is Mike Stenhouse's twitter app for ratings stuff, so all of the films above have a score out of 5.
Best
The start of the year had some really amazing films such as Revolutionary Road, Slumdog Millionaire and Gran Torino. Let The Right One In, which I was hugely anticipating and felt rewarded for doing so, was brilliant, and Away We Go, A Serious Man and happily quite few others kept the year going with some real treats. However, my two favourites this year were sci-fis. The first is an obvious instant classic; Moon. It's 70's-esque styling and Sam Rockwell's staggering performance put it up there with 2001, in my mind.
My other favourite, that I rated fairly averagely initially, is Timecrimes. It's a Spanish language film about time-travel, but it's not your usual showy film about changing the past and getting into a mess by accidentally shooting your aunt or butterfly or something, but a more subtle, accidental story that still has all the clever complexities you'd expect for the genre. I don't want to ruin it, but it's just such a neat film made on a tiny budget. It's shot beautifully and the surprises last right up until the end. I really can't recommend it enough.
Worst
Well, there were quite a few. Some failed because I'd built them up and expected a lot more, such as Hansel and Gretel, Franklyn and Martyrs. Others were just plain stinkers, such as Surrogates, Unmade Beds, Knowing and The Boat That Rocked (the last much more astutely criticised by Patrick over on Talking Animal).
Paranormal Activity took the biscuit for me, though. I'd fallen into the hype trap and seen the trailers and the chatter about it (on twitter, of all places to take seriously) and actually thought that it could be the new Blair Witch. How stupid of me. I'm astounded that anyone thinks that film is good. It's 81 minutes of a hell-of-a-lot of nothing - mostly a couple of pretty terrible actors whimpering and arguing with each other and unconvincingly never leaving their house to, you know, go to school or have jobs, but film every dull moment of their lives up until the final 5 minutes where there's a bit of genuine, well almost, thrill. As with the massively underwhelming Google Wave, it's the hype-machine success of the year that leaves you muttering "...is that it?".
Books
I somewhat failed at reading this year. I'm blaming my commute going from around 40 minutes spent on a tube to get to work to about 10 minutes. It's barely enough time to read your email and scowl at a few commuters, let alone open a book and really get into it. I did read The Outsider, The Road to Serfdom, the new Scott Pilgrim (which I rather nerdily now keep in it's plastic wrapper due to having a limited edition signed inlay plate), Let The Right One In and a couple more K. Dicks. I didn't really think to record what I read, so there is probably others. Must try harder this year.
This year, Glow switched over to using Github as it's code repository when it went open source to make collaboration easier. To be honest, getting used to git and github in general has been somewhat of an uphill battle for the team, but we've really found it's come into it's own recently now that we've adopted using submodules.
Glow 2 now includes Sizzle, Woosh and QUnit all as submodule projects of Glow, and doing so has instantly made Glow feel like a more team-playing product.
The whole many hands make light work approach is really quite good fun. For example, Jake started working up a design for Woosh, his new speed test framework for JS that he's also open-sourced, which was making default QUnit look a bit... uh... under-loved, so I forked QUnit and started to reflect the designs, and over the last couple of weeks worked it up to something that the QUnit team have decided to accept back into the main project. Github made that extremely easy and now Glow, jQuery and the wider JS community benefits from a little bit of time spent making something that was essentially made to please just 3 pairs of eyes.
Picture credits: QUnit Screenshot from John Resig's Flickr Stream, designed by myself.
Last week I was lucky enough to get to attend Full Frontal 2009 in Brighton with a whole bunch of my colleagues to lend a bit of support to friend and fellow Glow developer, Jake, after his previous highly successful first foray into the world of speaking at last year's @media ajax.
The range of topics was excellent. It was really good to see some more people speaking about server-side JavaScript and it seems that Simon Willison's talk and demo of node.js was a particular highlight for a lot of people and certainly seemed inspiring (which is possibly why he's always such a good last speaker).
I'm never terribly good at taking notes, but I didn't think there was an off session of the day. PPK wowed us with the frankly impossible task that is mobile browser testing, Todd Kloots did one of the better and more thorough introductions to ARIA and accessible JavaScript development talks I've seen (and it's a shame he ran out of time), and Robert Nyman did a gallant job of explaining closures and currying (amongst other things, via the inexplicable medium of Ben Affleck) - both of which I think are akin to explaining string-theory to primary-schoolers.
Of course, I'm slightly biased and I really want to tell you everyone else is just being kind and he actually sucked, but even though I'd already seen Jake's "Optimising where it hurts" practice run-throughs and had already heard many of his jokes and examples, I still laughed throughout and thoroughly enjoyed his animated way of explaining the topic (which he does fluently and coherantly) and thought he stole the day.
Generally, I thought the day was a total success. The venue was lovely (although it's a shame it wasn't a slight bit more central, and the rain always sucks) as it was a fully-functional cinema - seats were comfortable and the quality of the screen and sound were excellent. Everything seemed to be organised extremely well and can't really fault it, so big congrats to Remy and his lovely wife, and all the helpers. I really hope they organise one again next year - I'll definitely be there.
Emmy the Great
Not bad. Slightly underwhelmed and the sudden demise of the set was a let-down.
The Decemberists
Something very special. The beautifully orchestrated first set complete play-through of the new album, The Hazards of Love, then a second set of eclectic favourites from the past played by the massively multi-talented group was a sensation.
The Decemberists at The Coronet, 19th November 2009
Older Posts
Newer Posts
