A few days ago, I realized, “This Verbosio project is so big, so detailed, that a high-level overview would be really useful.” These high-level ideas haven’t changed in the six or seven years I’ve held them, but there’s been no complete overview of the ideas at once. I decided, therefore, it’s time to take a step up – not back – and try to paint a comprehensive picture of what I’m really trying to do here.

(UPDATE: For those of you who don’t know, Verbosio is an attempt to build a prototype XML editor driven by add-ons – languages and user interfaces coming from extensions to the core architecture.)

I’ve tried before to explain it – and it’s helped in job interviews (more on that later) – but even a five minute summary just doesn’t capture the whole vision I have. Lengthy specification documents and blog posts don’t do it either – ultimately they end up out-of date and forgotten. So I’m going to attempt a new approach. I’m going to create an audio slideshow “fly-by” of Verbosio, explaining the concepts behind the project.

(I credit the JSConf X presentations Brendan Eich and Douglas Crockford gave as the inspiration for this. Slides plus spoken words in an online video is pretty effective.)

On Wednesday night, I wrote out an outline, in preparation for writing a script and slides. When I counted it out, seventy-five different slide items came up (give or take a couple). That’s too many for a single presentation, so as I write the script, I’ll break it up into a few smaller segments. Overall, this could easily run 90 minutes as a whole.

How serious am I about this? I bought a Logitech headset specifically for this audio slideshow, and I’ll be using OpenOffice and some freeware audio editing software to put it all together. That said, for those of you who’ve done audio slideshows before, I’d welcome any advice – including freeware recommendations – on the technical and conversational aspects. I’m dusting off techniques from my Navy Journalism classes – which were way back in 1996 – and my book, in 2001-2002. I could use a refresher course, so please: chime in on the comments!

Storing references to objects

As I’ve said before, I’m building a markup templates system. The idea is to
reduce markup into reusable patterns, and to separate output from the user
interface. The idea is that for every significant node in the output (and thus,
in the user’s markup), there’s a matching user-interface node, and vice
versa.

Keeping track of these node pairings is the challenge. In a simple case, you
have a DOM tree like this:

  <markup:template>
<markup:output>
<xul:groupbox>
<xul:caption/>
<markup:children/>
</xul:groupbox>
</markup:output>
<markup:ui-section>
<xul:groupbox>
<xul:caption>
<xul:textbox markup:xpath="xul:groupbox/xul:caption/@label"
markup:source="@value"
markup:required="@value"
/>
</xul:caption>
<markup:children markup:xpath="xul:groupbox/markup:children"/>
</xul:groupbox>
</markup:ui-section>

The markup in italics indicates one node pairing – between the output’s
caption element (actually, its label attribute) and the user-interface’s
textbox. Another pairing exists between the underlined nodes. The
<markup:output/> and <markup:ui-section/>
elements define yet another pairing.

So, ultimately, I need a way of referencing one node from the other.
(Specifically, I’d needed references to the output nodes from the
user-interface elements.) The most obvious way is to store each desired output
node as a property of the corresponding user-interface node – through the
setUserData() and getUserData() methods DOM 3 Core
provides. However, I scaled the template system upwards to include repeating
fragments (think for (var i = 0; i < x; i++) {...} in XML).
Since users can rearrange those fragments – or even create new ones and remove
existing ones – pretty soon, when sending the updates back to the output
section, it gets… complicated. Now, you have user-interface nodes in a
different order than output nodes, some output nodes without user interface
nodes (because those were removed), and some user interface nodes without
output nodes (because they were inserted).

More details in the extended entry.

Continue reading Making a hash of things →

Has this ever happened to you?

• You have this great idea.
• You understand it deeply – how it should work, how it should be used, etc.
• You can write code to implement it.
• You can write tests for it.
• You can’t quite figure out how to put it in plain English, or in a diagram easy for your audience to understand.

Yeah, this happened to me. I’m trying to emulate XUL’s broadcast/observer model, but in limited parts of a document, and not necessarily in a XUL document. (The audience for this is fellow web and Mozilla developers.)

Basically, I’m adding in the concept of a “carrier” – as in carrier frequency, or relaying tower. (As an analogy, in a XUL document with inline frames and XUL subdocuments, each subdocument represents a carrier – XUL broadcasters in a subdocument can’t touch XUL observers in another subdocument. I’m attempting a similar boundary concept.) Broadcasters and observers live in the scope of a carrier node, but don’t cross the boundaries into other carrier nodes. Also, you can broadcast and observe not only attributes, but JavaScript values.

If you really understand the last two paragraphs completely in one reading, I’d be pretty amazed. So I’m trying to come up with an illustration of what I mean.

Here, the colored backgrounds indicate a carrier that the DOM node belongs to. (No background color means no carrier.) Some nodes within that carrier’s “grouping” have properties they broadcast… and other nodes in that carrier are observing on a particular name (“foo”, “bar” in this example), and picking up the properties. A carrier can observe and broadcast as a member of a parent carrier’s grouping, but not to any of the elements inside its own grouping.

Does this make sense, with the diagram attached? If so, great, and I’d like to know. If not, well, that’s why I’m posting this. I need some help, in making this easier to understand as a concept. That will help greatly both in using and testing the implementation of this model.

You can ask for the code implementing it and testing it, but be warned: it’s pretty hefty and intricate JavaScript. (Plus, I’ve already found a bug, and I’m still trying to figure out where it went wrong.)

(P.S. Yes, there really is a good reason for me to do this. Imagine a XUL grid where each row has a different number of columns. How would you allow the user to set the number of columns for each individual row? How do you make sure your XUL control for row 3 doesn’t affect rows 1, 2, and 4? This is just the most obvious example.)

First off, thanks to everyone who took the time to reply to my original blog post. Several people pointed me to moz4js, and I’m quite pleasantly surprised to find out the Mozilla Thunderbird team uses it quite a bit. I hadn’t heard of it, but when I did through the comments, I found it had the same problem NSPR logging has: single language support.

Then I realized: if we put moz4js in a JavaScript module, then a component can reference that module and expose the same interfaces to XPCOM. Chrome JS and JS components can load the module, and all other languages can use the XPCOM component. Maybe not simple, but certainly workable.

No, I’m not signing up to own the toolkit moz4js bug. I’m not wholly convinced, personally, it’s the right way to do logging. What I do think, though, is that we need a general discussion on logging API’s and consumers.

In the bug, Flock mentioned their own API – which I also didn’t know about. I mentioned Skyfire has an internal logging system. It makes me wonder how many other companies and projects (Songbird, Komodo, Disruptive Innovations, etc.) have implemented their own logging API’s. This is one subject that reasonably affects the whole Mozilla developer community.

If MozPad were still active, this is one subject I’d put right to them and develop a standard logging API for. Since we don’t have MozPad anymore, I would like to put it to the community at large:

1. What do you require logging for?
2. What logging capabilities do you need?
3. What logging infrastructure (API, code) have you already implemented or used?
4. What tests exist and pass for the logging infrastructure you use?
5. How extensively do you use your logging infrastructure?
6. How easy is it to work with your logging infrastructure?
7. If you use someone else’s logging infrastructure, what’s missing?
8. Are you willing to work on getting your logging infrastructure into the Mozilla code base? (Writing code, tests, specifications)
9. Are you willing to work on developing a common Mozilla standard logging API and infrastructure, and getting that into the Mozilla code base? (Writing code, tests, specifications)
10. Are you willing to actively “own” and drive an effort towards adding a Mozilla standard logging API to the Mozilla code base?

Please reply in the comments. The quality and quantity will tell me how seriously I should be taking this.

(P.S. I found a bug in my logging interfaces already. Not the JavaScript code, but the IDL. Bonus points to the first person to spot it besides me – it’s a little undocumented gotcha.)

What we have now

Over the last several years, I’ve learned to appreciate good message logging. Mozilla’s NSPR Logging API is a very good example, and works well, for compiled code.

JavaScript components? They can’t use NSPR. Neither can chrome, PyXPCOM, or any other privileged code. The best alternative I’ve got is a dump() statement. dump() writes to the standard error console, commonly known as “stderr”. In debug builds, though, stderr gets populated with a bunch of other stuff that I really don’t care about at a particular moment. (I’m not entirely clear of blame, either – leftover dump statements from my code pollute that console as much as anything else.)

At Skyfire, I’ve had to implement a couple logging systems as well, and the generated logs have helped us find and isolate bugs relatively quickly. What you log can help you identify bugs in most cases. They’re not a panacea, though: doing it wrong can really cost you.

Over the past few days, I hit a really weird bug in Verbosio’s templates system (which I’ve been quietly working on for about five months now, in a complete rewrite with tests). One test fails at a critical point, but it’s pretty far removed from the starting point. Amid all the other dump() statements and debug output, I can’t quite figure out where the code started going wrong. To pin it down, I think the best approach is to insert some logging.

Oh, and did I mention that Mochitest includes logging for pass, fail, and todo conditions… but conveniently left out an informational-only logging feature? (Having that helps when you’re trying to trace where a log statement came from – and the conditions at the time.)

The five paragraphs above combine to form one idea: Maybe I should implement my own logging service with tests.

Thoughts in the design

• First of all, keep it really simple. Most logging implementations I’ve seen use a module name, a severity level, and a summary string. I figure that plus an optional details string should cover most messages.
• For where those messages go, files are good. Streams are better. With a stream, you can feed it into a file, or read it into dump(), a XUL page’s multiline textbox, or anywhere else.
• Someone reading the logs probably wants a way of separating messages – so if I give them a “multipart” stream, where each part is a separate stream… it’s easy to tell each message apart.
• Make sure the user requesting the logging stream can specify which messages they want to receive – both by module name and severity level, and whether or not they want the details with each message.
• Make sure to cache all the messages, so that a logging stream created later still gets all the messages that came before it.
• Finally, since each message ultimately ends up as a simple string – use a simple string stream interface to construct each message as needed.

Did I overdo the design? Probably. On the other hand, it really maximizes flexibility. I don’t know how much memory multipart input streams consume, though, so I don’t recommend using this in a production environment. If you use this logging service for debugging, make sure you remove all references to it before checking in!

Did it solve my problem?

No.

Oh, the logging itself worked beautifully. But to get some of the data I wanted to log, I had to call other functions. Some of those functions triggered JavaScript garbage collection, which tried to run some cleanup code, which tried to execute JavaScript code I wrote, which resulted in a 100% reproducible crash. I don’t yet know what I’m going to do about that. I’ll probably first try to reduce the testcase and post it.

Also, logging itself will never solve the problem you’ve got. All it will do is provide data, and hints about where you are in the code execution. You still have to tell it what and when to log, and you have to read those logs to find the bug.

The good news is, though, that a logging infrastructure like this only took one afternoon to write. Sometimes, that’s all I need: a quick, simple component to organize the dirty work into one place.

(The code I wrote I am making available under MPL 1.1 / LGPL 3 / GPL 3 tri-license. If you want it under another license, feel free to e-mail me, and I’ll almost certainly say yes. I don’t currently plan on submitting this in a bug for landing in mozilla-central, though.)

It’s the project that Just Won’t Die Already!

A few months ago, I was thinking about Verbosio’s proposed template system. I realized it would be a lot more involved than I was thinking about two years ago. That, combined with Mozilla 1.9.1’s HTML 5 drag and drop support, required I scrap the original wizard design and start something new.

(Plus, it helps to give your users as much control as you can – while still guiding them through the complicated process. I didn’t give the user enough flexibility in the first design.)

It’s not finished yet, but the most difficult parts are done. (Or as we like to say at Skyfire, “feature-complete”.) I’ve had to create a whole new set of XBL bindings and JavaScript modules to support my perception of how some of this stuff should work.

Read on in the extended entry for the bindings and modules.

Continue reading Verbosio progress, 08/06/2009 →

Dear Lazyweb,

I realized that for Verbosio, I have forgotten to explicitly add the MPL/GPL/LGPL tri-license boilerplate to a number of my files. Although the project is open-source and under these licenses, I still see the missing boilerplates as an oversight.

I did a quick Google search for Perl scripts to find these files, and found only this: an Add_License script. It does support MPL, but not the tri-license.

Does anyone out there have a script for identifying these files? I’d prefer to add the boilerplates myself, but I still need to search for where they are not.