Category Archives: Verbosio

Verbosio is dead… but I have a new code name, Aluminium, with the same ambition

Perhaps the fastest evolution in the world today is knowledge, not software.  This has become painfully clear in the last few years.

The way I see it, Mozilla and I are both going through major turns in our respective life cycles:

  1. I turned 40 this past December.  That has all sorts of implications by itself, good and bad, in the current technology sector…
  2. I am roughly 18-24 months away from finishing a Bachelor of Science degree in Computer Science (with no college debt, thank you very much).
  3. Mozilla has ended support for their (admittedly proprietary) add-ons model, XBL is on the way out, and XUL will follow.  This means that the most specialized professional experience I have is not obsolete, but is dying a hard death, faster than I can keep up.  So the bachelor’s degree is coming around at just the right time.
  4. Rust has definitely arrived in the Mozilla toolchain, and Servo is on the way, as a wholly new layout engine.  (By the way, Lin Clark:  thank you so much for your articles on web engine design.  They’re priceless.)
  5. Emscripten is here, and new libraries to bridge between generated WebAssembly code and DOM/JavaScript API’s are under construction.
  6. Mozilla Firefox WebExtensions replaced the old add-ons model, which means a whole new API to learn – and later, when I work in compiled code on my own projects, customize and extend.  I’m really envious:  these are amazing technologies in their own right, but only Rust, WebExtensions and maybe Emscripten are “mature” at this time from this list…
  7. I’m still working on es-membrane (formerly es7-membrane), which I think is pretty interesting in its own right.  It’s maturing nicely as well, despite a lack of my available time.  (I’m still looking for help on that, by the way.)
  8. I’m a little surprised that the D programming language hasn’t received much traction… although until they become part of GCC and LLVM natively, I suppose I should be.  Supposedly, that’s in progress as well.

All of the above means that Verbosio, as a Mozilla Firefox-based XML editor with specific XML languages as add-ons to the editor, is truly and finally dead, and there’s no point trying to believe otherwise.  Similarly, the need for a XUL IDE is dead as well.  (Daniel Glazman and I need to get together to cry over a beer sometime.)

Enter a new code name, “Aluminium”.

I still want to build a stand-alone (but not too complex) web page editor supporting mathematics students at the high school, community college and university levels.  Amaya remains my inspiration.  I want to build a successor to that project, focusing on HTML5, MathML and SVG, with a conscious bias towards assisting students in doing their homework (but not doing the homework for them).

Of course, naming a future web page editor Aluminium, and basing it on arriving Mozilla technologies, leads to all sorts of bad puns:

  • By inheriting the concepts and some of the non-legacy code from Verbosio, I may be able to call Aluminium “powderware”, not “vaporware”.
  • When you mix powdered Aluminium with Rust, and a small spark of inspiration, you get some solid iron (very useful in repairing metal fatigue, I hear) and lots of heat to feed the fire (fox?)…
  • Apologies to all the Mozilla Firefox theme add-ons with a name of “Aluminum” or “Aluminium”.  Obviously, the name isn’t original.
  • I want to use the spelling and pronunciation common in scientific publishing and outside North America.  (This is again because I wish to place an emphasis on mathematics editing.)
  • Aluminium is the name of a metal, like a certain extremely popular web browser.
  • On the other hand, with all the features I want to implement, especially in-page version control for mathematics formulae, I’m not sure I can make it all that lightweight.  (Mozilla is no simple beast to build, either.)

I know, I know, that’s enough jokes for now.  But about the editor project itself, I’m actually quite serious.

Right now, one of the classes I’m taking at California State University, East Bay is titled “Graphical User Interface Programming Using a Rapid Application Development Tool”.  In short, it’s an introduction to building GUI windows (using Qt as a baseline and toolkit) as opposed to a command line application.  This is a course I am extremely lucky, and extremely happy, to get:  it doesn’t appear in the catalogs for any other CSU campus that I could find, much less go to, and the Computer Science department had told me repeatedly it wasn’t supposed to be in the CSUEB Catalog anymore.  All my programming experience and studies to date have either been for command-line applications, inside a web page, or with Mozilla’s platform code.  None of that taught me how to build GUI applications from scratch, or how to embed a web rendering engine like Servo.  That’s going to change…

Later down the line, I’m not planning on taking easy courses either:  I’m looking forward to classes on “Automata and Computation”, “Analysis of Algorithms”, “Numerical Analysis”, “Compiler Design”, and existing mathematics software.  All of these can only be force-multipliers on my computer programming experience going forward.

So yes, the old, non-standardized technologies of the last twenty years are being eliminated in Darwinian fashion… and a whole new generation of standards-based and safe-to-program-in computer languages are arriving.  The way I see it, I’m earning my Bachelor’s of Science degree at exactly the right time.  I hope future employers see it the same way, to work on some truly ground-breaking software.

Thanks for reading!

Alex

Introducing a WebGL-DOM Visualization Tool

Repository: https://bitbucket.org/verbosio/webgl-dom

Home page: https://alexvincent.us/webgl-dom/

tl;dr:  I have a new way of visualizing DOM trees in WebGL, and I’m looking for volunteers to improve the basic tool, especially on the WebGL side with three.js.

I’ve run into some trouble with an experimental Document Object Model.  Specifically, I’m trying to visualize it, but I’m dealing with multiple dimensions:

  • The parent-child node relationships
  • Sibling nodes
  • Attributes of DOM Element nodes
  • Atomic change sets
  • and at least two models of “shadow content”.

About four to six weeks ago, I realized I needed a tool to not only debug the DOM I’m trying to build, but to simulate new ideas that I haven’t yet implemented.  So I started building a WebGL-based visualization tool.

The tool currently has two main tasks:

  1. Transforming a XML document into a specific JSON format
  2. Generating a tree diagram in WebGL from JSON documents in the specified format

Most importantly, hand-editing this JSON should allow me to show ideas that currently cannot be done in the standard DOM.  The “WebGL Inspector sample” link from the home page shows this:  it takes only JSON as input and renders the tree.

It is very primitive right now, a mere starting point.  I’m posting this now, hoping to find a volunteer who’s more familiar with WebGL / three.js than I am to improve on the rendering parts.  The image is static:  there’s no zoom, pan, or rotation support whatsoever.  I really would like some help there.

Also, it doesn’t work in Google Chrome, but that’s because I had to specify type=”application/javascript;version=1.8” to make it work in Mozilla Firefox 39+.  (I like ECMAScript 6th edition and strict mode, thank you very much.  I just wish it worked without versioning.  I understand that’s Coming Soon.)

There is some click support:  clicking on a sphere should give details about the corresponding DOM Node, including the nodeName and nodeType.

If anyone out there likes the 3-D visualization idea and wants to reimplement it in the Firefox Developer Tools, be my guest.  Though the Tilt add-on for Firefox is more practical right now.

Competing with pen & paper

In tonight’s linear algebra class, I made the mistake of leaving my paper notebook home. Ok, I thought, I’ll just use Amaya and see how that goes.

Not so well, it turns out.

Twenty minutes of lecture equals a frantic “where is that thing?”, and nothing learned…

  • The template for a MathML subscript is in a different panel from the template for a MathML summation (“sigma”), and you have to switch between panels constantly.
  • If you want two subscripts (and in linear algebra, two subscripts for an element is common), you get a modal dialog.  (Really? How many subscripts does an element need?)
  • Where’s the special “M” symbol for matrix spaces? (I’d post it, but WordPress eats everything after that U+1D544 character!)  We can get the real number set with ℝ..
  • The UI for Amaya is hard-coded, so I can’t change it at all.
  • Amaya’s copy & paste support is terrible.
  • It takes about 2 seconds to write [Ai]1j with pen & paper. In Amaya that takes somewhere around ten seconds, plus the dialog I mentioned earlier.
  • Oh, and the instructor’s going on, keeping a pace for students using pen & paper… there’s no chance of me keeping up with him.

After twenty minutes of trying to quickly jot down what he’s saying, without comprehension, I end up with some symbolic gobbledygook that’s probably about 80% of a match to what the instructor is actually saying.  But what I was able to write down was complete nonsense.

I ended up switching to scratch paper and pen, where I was not only able to keep up, but ask some insightful questions.

(Incidentally, I glanced at LibreOffice tonight as well.  I saw immediately that I’d have fundamentally the same problems:  unfamiliar UI and lots of context switching.  Too much to really listen to what the instructor’s saying.)

How does a computer compete with pen & paper?

Later tonight, I realized, if it only takes five quick, essentially subconscious penstrokes to draw Ai, and a fair bit of training to teach someone the equivalent keystrokes in an editor… then maybe a keyboard and mouse are the wrong tools to give a student.  Maybe something closer to pen & paper is best for quickly jotting down something, and then translating it to markup later… which sounds like character recognition.

Hmm, isn’t that something digital tablets and styluses are somewhat good at?  Maybe not perfect, but easier for a human to work with than a memorized set of keystrokes.

Now, I am starting to understand why computer manufacturers (and Firefox OS developers) are putting so much effort into supporting touchscreens:  because they’re useful for taking notes, at least.  Once again, I’ve somewhat missed the boat.

How does this impact my editor project?

The good news is this is way too complicated for me to even attempt in my proof-of-concept editor that I’m trying to build.  (The proof of concept is about making each XML language an add-on to the core editor.)

The bad news is if I ever want students to regularly use computers in a mathematics classroom (which is the raison d’être I even started working with computers as a child), I’m going to need to support tablet computers and styluses.  That’s a whole can of worms I’m not even remotely prepared to look at.  This raises the bar extremely high.  I’m writing this blog post mainly for myself as a future reference, but it means I’ve just discovered a Very Hard Problem is really a Much, Much Harder Problem than I ever imagined.

FastEventLog.jsm: A quick and dirty event log viewing tool

About this time last year, I introduced a tree views module to my Verbosio XML editing project, which I’m still building infrastructure for.  One piece of that infrastructure uses the TreeViews module, and adapts it for simple sequences of objects in a common format.  Event logs were the use-case for this.

(This is currently independent of Mozilla Firefox’s own Log.jsm support, which is pretty nice itself!)

Let’s say you have a simple array of simple objects.  They all have a certain minimal set of properties.  If you want to visualize these objects laid out in a table, then each row can represent an object, and each column a property of the object.  XUL trees can build this table-like view, but they need some help.  First, they need a tree view that supports objects of the same basic type (TreeViews.jsm).  Second, they need to be built with the columns you want.

FastEventLog.jsm does this.  There’s two methods:  addPropertyColumn() and appendTree().  The first takes a property name, a label, the column width, and a couple other optional details.  The second method takes a box to hold the XUL tree element, a specialized “id prefix”,  a tree height, and the array of objects you want to show to the user.

If you have two separate arrays with the same structure, you can call appendTree twice. (Think expected results versus actual results.)

When working on my asynchronous transaction manager idea (more on that in a future post), I realized I couldn’t easily visualize what had happened.  The Firefox devtools debugger is awesome, especially with Thunderbird’s Remote Developer Tools Server extension.   But the debugger’s JavaScript object tree showed arrays of objects in an ordinary key: value hierarchy.  The data I wanted was buried very deep, and impractical to really analyze.  So I built FastEventLog.jsm and a helper XUL file to turn the tree branches I needed into XUL trees.

fasteventlog

The above is a screenshot with a bug deliberately introduced into the underlying transactions test to show what FastEventLog and TreeObjectModel (from TreeViews.jsm) can produce.  This little table view made diagnosing bugs in the async transaction manager very easy!

As usual, this FastEventLog.jsm is available under MPL / GPL / LGPL tri-license.

Introducing MicroXUL: A rapid XUL prototyping mini-project

(tl;dr) I’ve started a XUL editing miniproject, and I’m looking for volunteers to help build it out.

screenshot of MicroXUL editor's layout page

Which came first, the chicken or the egg?  It’s an old question:  conceptually, one had to exist to create the other.

My prototype XML editor project, Verbosio, is XUL-based.  XUL is an XML language.  So to edit XUL, I need a XML editor.  To build my XML editor, I need to edit XUL.  So which comes first?

Chicken, meet egg.  I’ve started work on a “MicroXUL” editing tool to help rapidly experiment with XUL user interfaces and XBL bindings.  It’s extremely primitive, but this is a proof-of-concept.

I’ve been using (and continue to use) the excellent Komodo Edit environment to develop pieces of Verbosio, including MicroXUL.  The screenshot above represents one wizard page where the user can type XUL markup into one panel, and see the rendered XUL in another, on-the-fly.  From this one wizard page and Komodo Edit, I hope to build out the rest of MicroXUL (and Verbosio) with a number of useful features:

  • XUL Project Details
  • Basic XUL Layout
  • Widget-specific wizard pages (based on analysis)
  • Images & Styling
  • Event Listener Declarations
  • Label & Access Key Internationalization
  • Keys and Commands
  • JavaScript controller object properties
  • Complete event handlers
  • Packaging of the XUL
  • Testing support
  • Documentation
  • Checklist
  • XUL menus shown through a XUL tree
  • XUL-to-XBL conversion tools
  • Editing existing XUL fragments from Firefox, Thunderbird, etc.
  • Editing MicroXUL itself
  • Drag & Drop of known XUL markup elements into the editing space

This project uses XULRunner (27+ required, currently in Aurora), Python 2.7 for building & launching, Jasmine 1.3 for automated testing, and CodeMirror 3.19 for syntax highlighting.

I started this because I realized I needed new XUL controls and XBL bindings to support new capabilities, such as a XUL user interface for the new Log.jsm architecture in Firefox 27.  I couldn’t use Verbosio yet, nor its templates system, since those haven’t been built out yet.  But I had just enough pieces from previous work, including my virtual:// protocol handler concepts from seven years ago, to put together a quick & dirty XUL editing interface.

Checkout and build instructions:

  1. hg clone http://hg.code.sf.net/p/verbosio/microxul/code microxul
  2. cd microxul
  3. ./project.py –update-sdk=aurora run
  4. (Mac users for the moment) ./project.py –browser (path/to/Firefox/27/Contents/MacOS/firefox)

Obviously, there’s a lot of work to be done on MicroXUL before it’s usable.  But the project is open under MPL / LGPL / GPL.  If you have a pain point with authoring XUL, this is a good playground to try fixing those pain points!  Your help and experimentation are both welcome.

Project page: https://sourceforge.net/p/verbosio/microxul .  If you need help and the SourceForge project pages don’t provide what you need, please leave a comment here and I’ll try to fix it.

Verbosio progress, October 8, 2013: Milestone 2 for template editing complete

Since my last progress update, I’ve slowly moved forward.  Some new goodies:

codemirror+positioning

I’m using CodeMirror (current version is 3.18) to act as my source code editor.  The scripted DOM is still there, this time showing attributes as children before a XUL tree separator.  On the right side, we start with relevant SAX events to capture details of the DOM I’m building.  Then there are logging events for the “Node Positioning Service”, which tries to align source code to DOM nodes.  The lower right box captures change events from CodeMirror.

Over the last couple weeks, I’ve compared what I’m working on with my project’s immediate goal of building a XML templates editor, and realized that I was starting to get distracted.  So I filed a few tracking tickets, checked my roadmap, and discovered I’d essentially finished the second milestone: the DOM implementation in JavaScript.

The next major phase will be about handling XML entities (&button.label; for example).  This will require a first introduction of a “shadow DOM”, where some DOM nodes will hide others from direct access.  (Indirectly these other nodes will still be accessible.)  This also means I’m starting to catch up with progress I made five years ago… that’s what happens when you do a rewrite.

I’m still looking for help!  In particular, I would like JavaScript developers to help out with:

  • Code reviews (Python code reviews also welcome)
  • CodeMirror’s API’s
  • Mozilla’s transaction managers
  • Encoding support (UTF-16, Unicode, etc., since I’m definitely “doing it wrong”)

By the way, I enjoyed the Mozilla Summit, and thank you for letting me present Verbosio at the Innovation Fair in Santa Clara.

Verbosio: It’s time to ask for help.

Yesterday afternoon, I spotted a sentence in a Planet Mozilla Projects page which shocked me to no end.  Benoit Jacob was advocating the end of MathML in Mozilla code.  The thread has attracted a lot of responses, and the tone largely opposed to his proposal.

Personally, I’m opposed to it as well.  MathML is one of those frontiers which has immense unexplored potential.  Can you imagine writing e-mails to instructors with inline mathematics formulae, or including equations in an instant messaging chat?  I can and have imagined exactly that for as long as I can remember.  I tried once upon a time to bring MathML into Mozilla Composer with my Abacus project, but determined it was too hard and too hacky to be a true solution.

This is precisely why I’ve been working on my prototype XML editor, Verbosio.  It’s supposed to be a complete rewrite of how we create and edit web pages.  The idea is that a language like MathML is simply a Mozilla add-on to the editor.  Unfortunately I’ve been buried with both full-time work and college to make any real progress on my Verbosio project on my own.

I’ve said for years that I didn’t want to attract a larger audience on an unproven principle.  Maybe that’s the wrong decision in this open-source Mozilla community.  While I still believe in the idea, I’ve become my own bottleneck.  It’s far past time for me to swallow my pride and admit that.

What I need to continue development is some help – and I don’t care how junior that help is, as long as they’re capable of writing JavaScript and willing to learn.  Two to five people who can work with me by e-mail and are patient can achieve far, far more than I can on my own.  I can train other engineers in this technology.  I can teach and explain what I’m trying to do and why at a deep level.

We’ve seen major improvements to browsers over the last five years:  HTML 5 form controls, audio and video, faster JavaScript performance, etc.  All of these areas are attractive.  Editing web pages?  Not so much – except to me.  The ability to write efficiently is still as important as the ability to read efficiently.

So, if you’re a budding JavaScript developer who wants to get into something experimental with someone who won’t quit on the idea, please leave a comment.  I should’ve asked you years ago.

Preserving source formatting: I need samples of real-world markup

Imagine the following:

  1. You have a XML or HTML document from a version control system. (SVN, Hg, git, etc.)
  2. You have an editor which parses that into a DOM document.
  3. You edit the document using a GUI, not looking at the source code – just adding in paragraphs, new widgets, math formulae, vector graphics, etc. with a visual editor of some kind.
  4. You finish your changes and save the document.
  5. You get ready to commit your altered document back to the source code repository, looking over the change set, and wonder what your editor did to your document:  it only looks vaguely like what you started with.

This is a hard problem, I think, and for the most part it comes down to handling white space inside a XML tag of some kind.  For instance, attributes may appear all on the same line as the element they belong to, or on new lines by themselves, with different indentations.

On the one hand, a visual editor like this isn’t required to preserve source code formatting.  As long as two different formats of the same XML document’s white space parse to equal DOM nodes, the editor normally does not care.  On the other hand, version control systems don’t often deal with DOM nodes – only with the source markup that an editor generates.

In trying to build a better XML editor (Verbosio) which supports editing in visual modes, I feel I have to respect both visual editing and existing source formatting.  Now, there may be a solution out there – a set of algorithms and a specification – but I have certainly not heard of it.  On the other hand, many source-controlled versions of XML documents have certain patterns enforced from coding patterns their authors require.  I am much less concerned with formatting a document the way I think it should be than with honoring existing patterns of source formatting.

Given a few samples of real-world markup, I can make some educated guesses – write a few JavaScript objects into a model which tries to respect the original source.  What I’m asking is for diverse samples of human-generated and human-edited markup – not computer-generated 1MB+ size files, but something more reasonable for people to work with.  It’s especially significant if these samples live in (or are derived from) version control repositories.

So, would you please post relevant links as comments to this blog entry?  Either an existing specification for this sort of problem, or diverse XML samples where people have their hands on the code regularly.

Verbosio progress, June 2, 2012

My prototype XML editor reboot continues!  In putting together my templates editing subproject, I’ve finished building out the basic infrastructure for a partial Document Object Model implementation in JavaScript.  I have a roadmap, and a decent development environment… life is good.  If you’re going to check out the code, you might want to wait until Tuesday… as that’s when Firefox 13 and built-in JavaScript Map() objects become available to the general public.

I’m not going dark.

I wish I’d had a chance to attend the recent Fluent 2012 conference in San Francisco.  I just found out about it too late.  I’m rather curious to see where some of my infrastructure work could be improved with ECMAScript Harmony, or whether I’ve had an idea that Harmony might find useful.  There’s a lot going on in JavaScript these days, and it’s exciting.

DOMRelations: A new design for the Document Object Model’s Node objects

For those of you who don’t care how the DOM is implemented, only that it generally works: this is definitely a “tl;dr” blog post.

For those who do care: I’ve been thinking long and hard about how XBL’s shadow content, and how undo and redo operations, work in Gecko.  Ultimately, I find the current implementation just doesn’t support what I need for XML templates.  So I am working on a very different model, one where I base Node objects on “DOM relations” objects.  These latter objects control the relationships between nodes, instead of the nodes themselves doing it.  They also make undo/redo and shadow content much easier to maintain in my opinion (at the cost of additional memory, which in JavaScript land I don’t have to worry about… too much).

I’m not suggesting Gecko DOM implement this (though that would be very nice of them).  I’m going to prototype it for my own needs, and see how well it works.  I hope this explanation is in clear enough language for the average developer to understand.