Category Archives: XUL, XBL, JS and DOM

Making a hash of things, part 2: Multi-part keys

Last year, I wrote about hash tables for JavaScript. This was before I knew about the plans to implement a native hash table for JavaScript called WeakMap in Mozilla Firefox.

WeakMaps are awesome.  They completely obsolete the need for that hashStringKey hack I came up with last year.  When you combine them with JavaScript proxies, you can get something even more awesome called a membrane.  (I didn’t make up the name.)  Through the membrane you can ensure a proxy either returns a primitive value (number, string, boolean, etc.), or another proxy.  In fact, if two underlying “native” objects A and B would refer to each other, through the membrane you can ensure the proxies for them, pA and pB, refer not to A or B, but to each other:  pA.b == pB, and pB.a = pA.

A couple things you can’t do with WeakMap

First, keys cannot be primitive values.  You can’t say:

var map = new WeakMap;
map.set("attributes", []); // throws exception because "attributes" isn't an object.

It just doesn’t work.  I asked Jason Orendorff about it, and the reason has to do with garbage collection.  Simply put, weak maps hold references to their keys loosely:  when no one else knows the key, the JavaScript engine can safely erase the key and the value.  With objects, that’s easy:  they’re unique.  When you copy them, you get a distinct object that isn’t equal to the original.  With primitives like a simple string, that’s not so easy:  you can lose all reference to the original string in memory, but hard-code that string elsewhere.  The weak map would have to remember it.  WeakMap currently deals with the problem by forbidding primitive keys.

Second, it’s one key to one value.  That’s what hash tables are, and what they generally should be.  But there’s really no concept of a two-part key, as there is in public key encryption.  Nor a three-part key, nor an n-part key.  So there’s no way to say that any two keys are related to each other.  Think of a two-dimensional grid: each cell has a row and a column.  The row and the column combine to form a key where you can look up a value for the cell.

My very experimental solutions

For the first problem, I implemented a brute-force primitive-to-singleton-object class, PrimitiveKeySet. It creates an object for every primitive it sees (thankfully, you have to pass it a primitive first), and returns that object. I also implemented a WeakMapWithPrimitives function, which leverages PrimitiveKeySet and wraps around the WeakMap API.  It doesn’t solve the memory leakage problem – nothing can, really – but it at least lets me use primitive keys.  I also tried to be a little smart about it:  when you tell it to delete a primitive key, it really does.

For the second problem, I did a little bootstrapping, using a tree analogy.  I started with a WeakMap (the “root map”).  I used the first part of the key to assign another WeakMap (call this a “branch map”) as a value stored in the root map.  I would use the second part of the key to assign another WeakMap to the branch map.  I repeated this over and over again until I reached the last part of the key and the last WeakMap I needed (the “leaf map”, for lack of a better term). At that point I assigned the user’s value to the last key part, on the leaf map.

I could easily say, then:

var map = new CompositeKeyMap(["row", "column"]);
map.set({
  row: 3,
  column: 4
}, "Row 3, Column 4");

I took it one step further, and added a shortcut, the PartialKey.  With a PartialKey, I wouldn’t have to specify every field, every time:

var map = new CompositeKeyMap(["row", "column", "floor"]);
var floorKey = map.getPartialKey({floor: 6});
floorKey.set({
  row: 3,
  column: 4
}, "Row 3, Column 4");
map.get({row: 3, column: 4, floor: 6}) // returns "Row 3, Column 4"

All of these you can see on my main web site under libraries/CompositeKeyMap, and with a Jasmine test suite.

Should you use these?

If you don’t intend to require Mozilla Firefox 6+, probably not.  This code won’t work without that, and there are no fallbacks.

If you want to use partial keys, I would think the CompositeKeyMap is very useful indeed.  I’d recommend one key you specify be for objects only, at least:  otherwise you might as well just use an ordinary JavaScript object as your map: {}.  Whether it should be the first or the last for memory efficiency, I can’t tell you.

I don’t see much use for WeakMapWithPrimitives, to be honest.  I did that as a proof-of-concept only, a stepping stone to the CompositeKeyMap.

Thanks for reading – and feel free to play with it or review the code as Mozilla Firefox 6 launches next week.  Comments welcome!

Venkman thoughts 2011, part 2

First of all, thanks to everyone who’s responded so far to my statement about Venkman dying yesterday. I’ve had a few thoughts and a few communications since then, and I thought I’d try to answer them here.

Venkman versus Firebug

The most important question I’ve gotten so far is “Well, what does Venkman have that Firebug doesn’t?” As I said in comments, I don’t know, because I have almost never used Firebug. Apparently, several years ago, a few people talked about it (search results courtesy of Google.com). For example:

Since my blog isn’t really a good place to collect this data, I figured I’d start a comparison wiki page where we can collect the features of each. Firebug and Venkman fans, please, help me out with some facts – log in and write them down!

On another note, the question itself bothers me a bit. Eight years ago, you could ask “What does Macintosh do that Windows doesn’t?” We were in a monoculture back then (and still are). You could ask “What does Mozilla do that Internet Explorer doesn’t?” about eight years ago, too. Again, a monoculture existed then.

I agree, Firebug is a very impressive tool, even if I haven’t used it very much. (Something about it’s good, if so many people use it and support it regularly.) Also, remember Firebug itself came years after Venkman… and JavaScript debugging was a monoculture then too. Firebug had a compelling answer then. Venkman, having languished in the shadows for years, doesn’t really have a compelling answer now, but that’s beside the point.

When you have at least two complete, healthy projects using the same interfaces, you’re probably doing something right. The W3C works like this: few specifications reach Proposed Recommendation status without two independent complete implementations. The spec may have bugs, and the implementations certainly will… but it provides a level of confidence that the spec is usable.

Now, someone might write a JS debugger UI independent of both Firebug and Venkman, using jsd2… and that’s great. The question bothers me, and I thought I was answering it above… I can’t put my finger on what it is that bothers me right now, but it’s a gut feeling.

The Venkman community: diehards

The second thing I notice from replies so far is that there are a few enthusiasts still out there. 🙂 It’s nice to know, and it’s appreciated. No one’s committed to working on a rewrite yet (not surprising – it’s a huge task). I certainly haven’t figured out high-level details of a rewrite project yet. My goal yesterday was to start the conversation, but to move on, I need somewhere I and others can at least white-board a bit.

I don’t even have a viable code name for the rewrite yet. (The best I’ve come up with so far is “Spengler”.) I’m open to suggestions – maybe WikiMo, maybe somewhere else.

The only thing I know we need and don’t have right now is a good “dockable XUL panels” implementation. Neil Deakin filed bug 554926 for this. This is not what I would call a “good first bug” by any means, but I suspect a lot of editor applications would love having this. (Komodo Edit, my Verbosio project, BlueGriffon, etc.) I envision using XUL panel elements to replace the floating windows Venkman currently provides. Panels in general could use a lot of help – see XUL:Panel_Improvements for details. I’m sure Neil would welcome patches.

Next steps

I don’t know yet. It’s too soon for me to call anything like a “Town Hall” for Venkman replacement efforts. I’m still trying to identify people willing to actively contribute time and talent. If it were me and Gijs alone, forget about it arriving in the next three years. We need help if it’s going to get done.

R.I.P., Venkman

Almost four years ago, I tried to rally an effort to keep Venkman, ye olde JavaScript debugger, alive. It has been on life support since then. The only people currently working on the code are Gijs Kruitbosch and myself, to my knowledge. For all intents and purposes, I believe the two of us are the current “owners” of Venkman, by default.

Soon, the new “JSD2” JavaScript Debugger interface code will land on mozilla-central. (By “soon”, I mean probably by the end of the year.) When Firebug moves to JSD2, JSD1 (which Venkman relies on) will be deprecated and eventually removed. This would be the final nail in the coffin for Venkman, and it’ll be time to bury it.

Now, I need a Venkman-like tool regardless, and the UI as it was presented to the end-user was fairly well defined. The problems I had were really about how to make improvements on an architecture that’s over ten years old and has been abandonware for years. When
I need something and no one else is building it, I’m likely to build it.
So I’d like to start a new project that looks like Venkman, but works with jsd2 and has a clean, truly XUL-based UI implementation.

The biggest problem we face, by far, is a lack of available developer time. I have a full-time job and I’m about to start college (more on that in a separate blog post). Not to mention a little pet project that I’m obsessed about. To pull this off, we’re going to need some help, particularly from competent JavaScript authors. Previous experience in Mozilla UI hacking not required – I’ll be very happy to teach XUL & XBL to anyone who would offer significant help.

I’m looking for volunteers to help me kick-start a new Mozilla Debugger Developer Community. Who out there is interested?

(P.S. The company I work for is looking to hire developers who are familiar with FF extensions. For anyone who’s not experienced enough, a project like this is a great way to get into the field… )

Paired methods (start*, stop*) are a bad idea

By paired methods, I mean a pattern like this:

manager.startWatching(node);
doSomething();
manager.stopWatching(node);

In pure JS land, this is unnecessary. If your API requires both the start and stop methods execute, around user code, your API is fragile. You might require customers to call both startWatching() and stopWatching(), but you can’t guarantee they’ll do it. It’s far better to have a single watcher method which takes a callback function:

manager.watchObject(node, function() {
doSomething();
});

The watchObject() can guarantee both the start and the stop methods execute, even in the face of an exception from the callback, with a try… finally statement:

watchObject: function(obj, callback) {
innerManager.startWatching(obj);
try {
callback();
}
finally {
innerManager.stopWatching(obj);
}
}

This blog post isn’t a rant against any Mozilla code in use, specifically. This is something I realized when working on my own pet project.

Though to be fair, a lot of XPCOM code requires this pattern – SAX, TransactionManager’s batched transactions… it’s kind of annoying. The only current solution for this would be to introduce a callback interface, with a single method and “function” defined in the interface attributes:

[scriptable, function, uuid(...)]
interface nsICallback : nsISupports {
void callback();
};

Technically, you could reuse nsIRunnable for that, but that violates the intent of the interface. Somehow, I don’t think we’re going to see that pattern introduced into the Mozilla code base anytime soon, even as a new XPIDL argument type. (“lambda”, bsmedberg?)

UPDATE: John J. Barton raised an excellent point in comments: that if the start method and the stop method are from two separate user interface events, such as a start button and a stop button, then it’s appropriate to have two separate methods. I wrote this article in the context of a single event triggering both.

Aggregation by proxy, part 1

(I was tempted to call this Integration by Parts, but I think the other calculus fans reading Planet Mozilla would raise a pitchfork or two.)

In my last blog entry, I talked briefly about an idea I had:

Aggregation by proxy, though, means you could daisy-chain several JS objects in front of the original DOM node, in terms of method and property priorities. If you need to replace a binding, you swap out one object in the daisy-chain. No __defineSetter__ mess, though pulling it off will require a bit of custom JavaScript code. I’d love to see someone else write it up for me – I see it as a trivial extension of the forwarding proxy examples.

Okay, it’s not that trivial. But I do believe it’s doable. If you exclude the DOM node and just daisy-chain a bunch of JavaScript objects, you get something like this. (Note this will only work in recent FF4 beta builds.)

Tom Van Cutsem saw my sample this way in an e-mail he sent back:

If that’s the case, I think your code here actually demonstrates two nice use cases of proxies:

  1. it shows aggregation of object properties.
  2. it shows how you can create a “side-effect-free” proxy for an object (or a graph of objects), which provides the illusion of side-effects (adding/deleting properties) to the proxy client without affecting the wrapped object.

I can imagine proxies that support only 1) or 2) to be useful in their own right. It may even make sense to separate out the functionality into two proxies and then chain them together so that we have:
client -> side-effect-free-proxy -> aggregate-proxy -> target

The version I’m posting here has Tom’s annotations in the source code.

There’s room for improvement. This code will work only with pure JavaScript objects, and can violate rules for setters. I’m thinking this design only takes me part of the way there. I imagine aggregation by proxy as meaning “I can combine implementations of two different IDL interfaces into a single callable object”. That means fun with QueryInterface, for starters. I also need to work on a SingletonHandler (hint: I’ve already prototyped that too), so that document.getElementById(“test”) returns me the same object every time.

I’d love to work on this with fellow Mozilla contributors, to build a complete aggregation by proxy script for the DOM. Anyone interested?

A different way to add bindings to DOM nodes – with proof of concept!

Several years ago, before HTML 5 reached its current adolescence, it was in
diapers. There was a sub-specification called Web Forms 2.0, and it included
a
repetition model for blocks of HTML markup
. The idea has long since
been dropped from HTML 5 (it’s probably in a landfill with someone’s diaper),
but it’s remained an inspiration (or maybe an obsession!) for me ever since.

Making it work is quite challenging. I think a repetition model could be
very useful for any web page editor – or for Verbosio, my experimental XML
editor. There’s a few drawbacks, though:

  • I can’t use XBL to pull it off. XBL wants me to insert content as
    children of the bound node, and I think that could just mess
    things up badly for CSS. Inserting the anonymous content as previous siblings makes more sense, and is cleaner, both for DOM and CSS manipulation.
  • I can’t use XTF for it. Well, I could, but the code gets very
    ugly. Unmaintainably ugly.
  • I really do need to hide these inserted elements from the DOM, and yet
    show them in layout. This is one of the issues that prevents a JS-based XBL
    2.0 implementation, or a JS-based XBL 1.0 implementation.

Given all that, I finally hit on a solution: copy the DOM. Give one
DOM tree to the primary JavaScripts, and another DOM tree to the web browser.
Unfortunately, I couldn’t copy it just once. I had to copy it several times,
maintaining private copies of the full DOM tree, one for each new “layer” of
bindings.

If all this is boring and you want me to show you something, here’s a
screenshot:

Yes, that’s a chrome Mochitest, with a DOM Inspector window. DOM-I (and the
browser window) sees the “anonymous content” DOM, but all the tests I ran
against the “scriptable” DOM.

Let me play with this!

Okay.

hg clone http://hg.mozdev.org/verbosio --rev 06a72a32d6ec unstable
cd unstable
python client.py --target=floor13 --mozilla-rev=--latest-beta checkout
python client.py --target=floor13 build
. run_bindings.sh

Source code for this experiment lives at

http://hg.mozdev.org/verbosio/file/06a72a32d6ec/experimental/floor13
.

How does it work? Can this be used for XBL 2? What about my own
properties?

Extended entry, please. For those who don’t wish to read on, well,
Happy New Year!

Continue reading A different way to add bindings to DOM nodes – with proof of concept!

Implementing a DOM in JavaScript, part two: Two difficult options, with requirements and test specs

A little over a week ago, I had an idea about implementing
a DOM in JavaScript
. Thank goodness I
wasn’t the first
. (Thanks for the links, guys!)

Still, what these libraries provide isn’t enough. It’s “necessary but not
sufficient” for what I really need: using one DOM tree as a public interface to
another, “private” DOM tree. I’m calling this a “DOM Abstraction”, for lack of
a better term. So in classic do-it-yourself fashion, I set out to plan what I
needed to implement.

I just spent several hours (spread over many days) formally defining
requirements and tests. As I was reviewing my article, I asked myself a simple
question: Well, why don’t you just clone a bunch of native DOM nodes and
extend their properties?

More in the extended entry. (Warning: it’s looooooooooooooooooooooong! But I really do need developer’s feedback. I hope I’m not too boring.)

Continue reading Implementing a DOM in JavaScript, part two: Two difficult options, with requirements and test specs

Implementing a DOM in JavaScript?

I know, that sounds crazy, but hear me out…

In recent weeks, I’ve begun to suspect the singularity is near, really, in JavaScript land. After all, we now have just-in-time compilation (in at least two flavors for Mozilla code alone), the Bespin SkyWriter editor project, the new JS Reflect API, Narcissus and Zaphod… it’s a pretty exciting time.

Also at work, I’ve been seriously exposed to jQuery for the very first time. I’d heard of it and similar libraries (Dojo, Yahoo UI widgets come to mind), and what they do is provide abstractions of the underlying DOM for common operations.

Mix that concept (abstracting the DOM) with Narcissus’s concept (a JavaScript engine implemented in JavaScript), and I start to wonder. What if we implemented a complete DOM in JavaScript, on top of the existing DOM – and then used that JS-layer DOM to implement our own features?

A few years ago, when XBL 2 was first announced, someone suggested they could implement the specification in JavaScript. Someone wiser (I think it was bz, but I don’t remember) replied that this implementation really didn’t do it, since the nodes weren’t really hidden. But if we implemented a full DOM layer in JS, and modified that…

For instance, take Node.firstChild:

JSNode.prototype = {
// ...
get firstChild() {
if (this.DOMLayer.hasSpecialFirstChild(this))
return this.DOMLayer.getSpecialFirstChild(this);
var returnNode = this.innerDOM.firstChild;
while (returnNode && this.DOMLayer.isHidden(returnNode)) {
if (this.DOMLayer.hasSpecialNextSibling(returnNode))
returnNode = this.DOMLayer.getSpecialNextSibling(returnNode);
else
returnNode = returnNode.nextSibling;
}
return this.DOMLayer.wrapNode(returnNode);
},
// ...
}

As long as everything was wrapped in a complete and well-tested DOM abstraction layer like this – and only objects from the abstraction layer were returned to the user – you’d have a baseline for creating (or at least emulating) a whole new way of viewing the DOM. XBL 2 could be implemented by manipulating this JS-based DOM.

You could also experiment with other possibilities. I’m running into a problem where I would really like to hide some nodes from the DOM Core interfaces (particularly previousSibling and nextSibling), but expose them through other means. My first thought was to use node filtering to skip past them (you’ll hear more about that in another blog post), but I wonder if I’m just wall-papering over the real problem or actually solving my immediate problem.

I could even use this idea to get past three of my fundamental problems:

  • XTF, while beautiful for my purposes, isn’t well-supported, and XBL 2 is supposed to make it go away. I don’t know when, but I find myself using more and more of XTF’s features – particularly its notifications – in recent months. If XBL 2 really replaces XTF, it’s going to be more and more difficult to update my code. A JS-based DOM means I can just walk away from XTF entirely.
  • Recently, I found myself hacking the core DOM to support observer notifications from Node.cloneNode(), so that I could manipulate the DOM before and after a clone operation. That’s not a very nice thing to do, but I have my reasons. I can modify a JS-based DOM to avoid that step entirely.
  • At some point, I’d like to implement the Entity and EntityReference parts of the DOM Level 1 Core specification. When it comes to editing XML, DTD’s and entities matter (see localization), but they’re a pain to work with.

Of course, there are some logistical problems with building a JS-based DOM. For one thing, there are a lot of DOM specifications to implement or wrap in this way. This abstraction layer would have to pass a pretty large set of tests too.

If you did all that, though, and could run it a couple layers deep (one JS-DOM layer on top of another JS-DOM layer on top of the native DOM layer) accurately… that would be a pretty good starting point for lots of custom DOM manipulations.

A search on Google didn’t turn up anything like what I detail above. Is there anyone crazy enough to write a narcissistic DOM? Given that I might need this anyway at some point in time, should I seriously consider doing this for my own project? Or is this just a really bad idea?

Vendor-provided user agent extensions are gone… now what do we do?

Scenario: A corporate extension modifies the user agent by a general.useragent.extra.foo preference. The sole purpose is for the web page to know whether that extension’s installed or not.

With Firefox 4.0, that’s no longer practical. Someone helpfully removed support for adding extra parameters to the user agent string. So now, the corporate office finds itself in a pickle.

We need some other way of informing a web page that our extension is enabled. (Not every web page, though – a selective list.) So far, we’ve considered a few options:

  • Set the general.useragent.override preference. This is just wrong
  • Adding a new HTTP header
  • Injecting a node into the target web page.
  • Firing a custom DOM event at the target web page.

I’d like your recommendations on the best way for a chrome extension to notify a web page of its existence (or non-existence). If it’s one of the three options above, please say which one. If you have your own ideas, please post them as a reply, with reasoning behind it.

Please, no comments suggesting we should never, ever inform a web page of our extension’s existence. If we develop both the webpage and the extension, knowing the extension’s there can be really important.

DOM Ranges and Document Fragments

I expect this article will get very few replies, indeed. :-p

A few weeks ago, I had this big block of XBL using a DOM range to show a set of nodes, and a DOM document fragment to hold the same set of nodes while the nodes were “hidden” from the user. It was really clunky, though, and a little hard to manage.

To clean it up a bit, I decided to introduce a new module, RangeFragment.jsm. The idea is simple: when you want to hide the set of nodes, you call rf.toFragment(). When you want to show them, you call rf.toRange().

I wrote this for both my <markup:repeat/> and <markup:children/> elements, but based on the hg diff, it doesn’t look like I actually removed or simplified very much. The reorg was only about making it easier to implement and test what I’m working on.

Sadly, when I went to look at the to-do list for Verbosio, this was so small a functionality change that it didn’t rate a spot on the list.

Oh, well. Maybe someone else will find a good use for an object that lets you switch from DOM range to document fragment and back.