Imagine the following: You’re working on a new feature for your application, dependent on XULRunner 1.9.1. It’s complex, but it ends up being all chrome & JavaScript components. You write tests for it as you’re developing, and you write a specification. Everything’s coming along nicely.

Then you start thinking about moving to XULRunner 1.9.2 beta x. You compile the code, and just to be sure everything’s fine, you run a test you wrote a few weeks ago. The test and the code it is testing haven’t changed at all. The test that passed in 1.9.1.x fails in 1.9.2.x.

Now, what do you do?

This is the situation I faced a few weeks ago. It’s not an idle question. Obviously I want to find out what regressed, but my test case isn’t minimal – it’s approaching maximal. On the one hand, since the bug isn’t in code I’m working on right now (far from it), I could simply ignore it for now and keep working on my feature. On the other hand, my confidence in my code is now badly shaken. Did I misuse an API? Did the API change without my knowledge? Did I make a change in between the 1.9.1 test and the 1.9.2 test? I have to know what broke, so I can work around it for supporting both versions of XULRunner.

(As it turns out, it was a regression in the Mozilla code base itself.)

I started thinking about how to make this a deterministic test: something that I can point to and say “In this set-up, everything works… but in the other set-up, something’s off.” When I think about it though, it’s a bigger problem than just A/B comparisons. My whole approach to developing software is too informal.

More in the extended entry.

Generally speaking, I have code under development for one project that depends on another project. (Verbosio depending on XULRunner, Skyfire depending on Firefox, etc.) When I hit something I didn’t expect and need to explore, though, switching tracks is harder than it might seem. I have to fork internally – the Mozilla code, my code, my tests, fresh builds so I can break stuff… complete copies of the working environment so I can keep “the current work in progress” in one tree, and “reduce the test” or “isolate the bug” in another tree.

In theory, all I need is hg clone. In practice, all that does is copy one repository. We don’t have “subrepos” as the standard part of the Mercurial package yet, and that doesn’t duplicate a build (.mozconfig scripts, compiling, linking, rerunning tests). I think of doing all this manually, and I hesitate, because it’s just going to eat up hours of my time. (Build Mozilla. See how long it takes on Windows.)

I run into this “new goal interrupt” problem quite a lot, enough now that I realize I need to do something about it. I basically need a whole-tree-management tool, something that will let me fork internally. Something that will free me up to continue working on the first problem, while my system prepares a build (or two, for the A/B comparison). Then I can work on the second problem without cross-pollinating either tree. It needs to handle checkouts, clones, fresh builds and running specific tests from a command-line, and A/B comparisons (for failed tests, or for bug fixes). I’m even thinking about symbolic links to remove the risk of accidental discrepancies between copies.

However, I don’t want to fall into that deadly Why are you doing this? trap again. If you know of a tool that does all this on Windows, Linux and Mac (Perl or Python scripts welcome), let me know in the comments. I know I can write a Perl script to do it for me, but I’d rather not reinvent the wheel.

Comments are open.