Category Archives: Technobabble

Help wanted with HTML user interface for es7-membrane

I’ve continued to work on es7-membrane in my spare time, to the point where I released version 0.8 (“first beta”) without announcing it on my blog… oops.  (I also forgot to tag the 0.8.1 release on GitHub.)  For those who don’t know what it is about, just read the first few paragraphs of the 0.7 release announcement.

I also have a low-traffic Google Groups mailing list for release announcements and general support.

I’m looking for unpaid-intern-level help.  Not in the membrane implementation itself, but in crafting the es7-membrane distortions user interface.  (“Distortions” is a relatively new term in the Membranes lexicon:  it means altering proxies so that they don’t exactly match the original value, such as a whitelist for hiding properties.)   The distortions user interface is a subproject for configuring a Membrane instance, and for persisting that configuration for future edits… all with a static GitHub website.

This means JavaScript, HTML, CSS, SVG, modern Web API’s (FileReader, Blob, CSS grids, etc.), build configuration, continuous integration, and more JavaScript (Jasmine, CodeMirror).  It means in particular almost no HTTP server code (so no Python, PHP, etc.)

It does not mean including a library like jQuery.  Call me biased if you want, but I think libraries like jQuery or YUI are unnecessary with recent advances in web browser technologies, and even less so in the future with Web Components evolving.  These libraries were written for older Web API’s, and have to support thousands of websites… I don’t mind reinventing the wheel a little bit, as long as it’s tightly written code.

I’m looking for help because while I could do all of this on my own, I have time constraints in the form of a full-time job and university classes.  On the other hand, I am an experienced Mozilla developer and JavaScript expert, so I can definitely mentor people… and this is cutting-edge JavaScript we’re dealing with here.  Already, I have two interested customers for this open-source project (besides myself, of course), and one fellow student who took over a small widget (a “multistate” HTML button).

What I’m looking for are people who don’t have a lot of experience, but do have the time, an open mind and the willingness to do some of the grunt work in exchange for mentorship and letters of recommendation and/or equivalent written credit good for a résumé.  I just recently added a few “good-first-bug” labels to the 0.9 milestone list of tickets.

If this fits your bill, please reach out through the Google Groups link above, or through my GitHub user page… and thank you.

Validating directory inputs?

A quick thought here.  I spent several hours today trying to figure out why a simple Firefox toolkit application wouldn’t work.  (I don’t know what to call “-app application.ini” applications anymore, as “XULRunner” has definitely fallen from favor…)  It took me far too long to realize that the “default” subdirectory should’ve been named “defaults” – something that I already know about these apps, but I only build them from scratch every two years or so…

Catching this sort of rookie mistake is, fundamentally, an argument validation exercise:  the main difference is instead of the argument being an object of some kind, it’s a directory on the filesystem.  If Mozilla has a module or component for validating a directory’s structure in general, I haven’t heard of it…

Which is the point of my post here.  I’m wondering what general-purpose libraries exist for validating a directory tree’s structure and contents at a basic level.  Somebody out there must have run into this problem before and created libraries for this.  I’d love to see libraries written in C++, D, Python, NodeJS and/or privileged JavaScript.  Please reply to my post if you can point me to them.  (For once, a quick search on the world’s most popular search engine fails me…)  Bonus points for libraries that allow passing in callbacks for file-specific validation. (“Is there a syntactically correct .ini file at (root)/application.ini?”)

A practical whitelist in JavaScript: es7-membrane, version 0.7

Several months ago, I announced es7-membrane, a new project for letting JavaScript developers control, through JS proxies, what their customers see of their own libraries.  A proxy, as you may recall, lets its creator define rules for looking up properties, defining properties, calling methods, etc., often with a real object underneath which the proxy’s handler internally refers to.

First off, a little bit of basics: an object is a collection of properties (some of which are functions, and officially named “methods”). This may sound obvious, but it’s important: you refer to other values by the object and a property name.

Proxies allow you to rewrite the rules for referring to other values by that tuple of the (containing) object and a property name. For instance, you can hide “private” members behind a proxy.

(Stack Overflow)

A membrane presents a one-to-one relationship between each proxy and a corresponding real object.  If you’re given a proxy to a document, and not the document itself, you can get other proxies, and those proxies can offer other properties that let you get back to the proxy of the document… but you can’t break out of any of the proxies to the underlying set of objects (or “object graph”).

When I made the announcement back in August of this new project, the membrane I presented was quite useless, implementing only a mirroring capability.  Not anymore.  There’s a few features that the latest version currently supports:

  1. Membrane owners can replace any proxy the membrane generates with a custom proxy, using the .modifyRules API
    • .createChainHandler(…) creates a new ProxyHandler derived from a handler used for an object graph the membrane already knows about.
    • .replaceProxy(oldProxy, newHandler) takes the new handler and returns a new proxy for the original value, provided the new proxy handler was created via .createChainHandler().
  2. Membrane owners can require a proxy to store new properties locally on the proxy, instead of propagating them through to the underlying object.  (.storeUnknownAsLocal(…) )
  3. Membrane owners can require a proxy to delete properties locally, instead of on the underlying object.  (.requireLocalDelete(…) )
  4. Membrane owners can hide existing properties of an object from the proxy’s users, as if those properties do not exist.  (.filterOwnKeys(…) )
  5. Membrane owners can be notified when a new proxy is about to go to the customer, and set up new rules for that proxy before the customer ever sees it.  (ObjectGraphHandler.prototype.addProxyListener(callback), ..removeProxyListener(callback) )
  6. Membrane owners can define as many object graphs as they want.  Traditionally, a membrane in JavaScript has supported only two object graphs.  (“wet”/”dry”, or “protected”/”public”, or “internal”/”external”… you get the idea)  But there’s no technical reason for that to be the upper limit.  The initial design of es7-membrane allowed the owner to define an object graph by name with a string.
    • Having more than one object graph could have a few applications:  different privileges for different users or customers, for example.
    • The es7-membrane test code uses “dry”, “wet” and “damp” object graphs.
    • For lack of a better name, es7-membrane calls these “multisided” membranes.  I have a document explaining how this works at a very temporary location.
  7. A new feature of ECMAScript 6, however, is symbols – which can be used as valid keys to an object, but are not strings.  Per a Github issue by Dr. Mark Miller of Google (who has been advising me on applications every now and then – thanks, Mark), now membrane owners can define object graphs by a “private” JavaScript symbol they create, as well.

Features two through five above combine to make whitelisting of properties in JavaScript very doable:  properties you don’t want exposed you filter out, and customers receiving proxies configured for local properties won’t propagate their changes to the your objects.  The listeners also mean you can apply those filters and local property rules before the customer ever sees any newly created proxies.  So properties you want private and/or protected really are private and/or protected from the malicious end-user.

Version 0.7 added the symbol support today, along with a probable memory leak clean-up and a little more protection for revoked object graphs.

The production files are available in the dist subdirectory of the es7-membrane project’s Github’ repository, or directly usable as a npm module.

I’m still looking for help in a few areas:

  • Building a static interactive demo site on Github
  • Code and documentation reviews
  • More testing
    • additional tests in Jasmine
    • converting tests to test-262 format (someone suggested the standard tests for ECMAScript might use some integration tests, such as a membrane implementation)
    • if there are weaknesses, where a customer who has only “dry” proxies and good old ECMAScript 6+ (no membrane access, no other proxies) can break out to reach “wet” proxies… no one’s seriously explored that yet.
  • Using the membrane module to protect itself from evildoers (dogfooding bonus)
  • Implementing other use cases for a multisided membrane
  • And just in general, another volunteer developer or two would be extremely welcome!

Introducing a WebGL-DOM Visualization Tool

Repository: https://bitbucket.org/verbosio/webgl-dom

Home page: https://alexvincent.us/webgl-dom/

tl;dr:  I have a new way of visualizing DOM trees in WebGL, and I’m looking for volunteers to improve the basic tool, especially on the WebGL side with three.js.

I’ve run into some trouble with an experimental Document Object Model.  Specifically, I’m trying to visualize it, but I’m dealing with multiple dimensions:

  • The parent-child node relationships
  • Sibling nodes
  • Attributes of DOM Element nodes
  • Atomic change sets
  • and at least two models of “shadow content”.

About four to six weeks ago, I realized I needed a tool to not only debug the DOM I’m trying to build, but to simulate new ideas that I haven’t yet implemented.  So I started building a WebGL-based visualization tool.

The tool currently has two main tasks:

  1. Transforming a XML document into a specific JSON format
  2. Generating a tree diagram in WebGL from JSON documents in the specified format

Most importantly, hand-editing this JSON should allow me to show ideas that currently cannot be done in the standard DOM.  The “WebGL Inspector sample” link from the home page shows this:  it takes only JSON as input and renders the tree.

It is very primitive right now, a mere starting point.  I’m posting this now, hoping to find a volunteer who’s more familiar with WebGL / three.js than I am to improve on the rendering parts.  The image is static:  there’s no zoom, pan, or rotation support whatsoever.  I really would like some help there.

Also, it doesn’t work in Google Chrome, but that’s because I had to specify type=”application/javascript;version=1.8” to make it work in Mozilla Firefox 39+.  (I like ECMAScript 6th edition and strict mode, thank you very much.  I just wish it worked without versioning.  I understand that’s Coming Soon.)

There is some click support:  clicking on a sphere should give details about the corresponding DOM Node, including the nodeName and nodeType.

If anyone out there likes the 3-D visualization idea and wants to reimplement it in the Firefox Developer Tools, be my guest.  Though the Tilt add-on for Firefox is more practical right now.

Compacting XUL interfaces?

For my day job, I work at a startup, basically as an expert in Mozilla technologies.  I love what I do, too.  But whenever I do user-interface work, I frequently run into a simple problem:  screen real estate.

Case in point, my latest work with XML entities and entity references on the Verbosio XML editor project.  (I can’t really talk about details of my work-related code here, but I can talk about the pet project.)  The demo panel for this, which doubles as a laboratory for my experiments, has four major sections.  The upper left corner holds a CodeMirror instance for one DTD document.  The upper right corner holds another CodeMirror instance, for a second DTD.  The middle horizontal section holds a third CodeMirror instance, for the XML document that might load the DTD documents.  Each of these has some ordinary XUL buttons on the right edge and a menulist, to help me control the individual sections.  In the bottom third, I have a XUL deck which can toggle between an iframe and a XUL tree showing the document object model of the XML I’ve parsed.  To the right of this XUL tree, I plan on adding a bit more, for showing the entities defined in the doctype or the entity references on a node.

EntityParsePreview

All of this UI lives inside a tabbox.  As you can see, it’s already a little crowded, though fortunately, it’s almost complete in terms of features I need.

I can’t do a fair comparison against the Mozilla Firefox user-interface; the main windows don’t have textboxes for source code or trees for DOM views. So their controls’ relative sizes don’t come close to mine:  they’re much flatter.

The built-in developer tools, though, do have an elegance to them, and are a fair comparison.  The right side panel, showing variables and events, can collapse away (and the animation’s pretty nice, too).  The left side panel has a listbox (I think) of scripts to choose from, and when you select one (either in call stack or in sources), the equivalent source code appears in the center.  Plus, they have some really tiny icon buttons in the UI, much smaller than the standard XUL buttons I use.  The devtools UI gives you basically what you need and otherwise tries to get out of your way.

Dear lazyweb of Mozilla UI specialists:  Can you point me to a developer.mozilla.org document with some guidelines for efficiently using the screen real estate?  My user-interface works, but I need some tips & tricks for making the most of it.  It could be simple stuff like shrinking buttons, or it could be creating new XBL bindings to masterfully present common ideas together.  I’m not willing to go to HTML5’s Canvas for this.  But my experience has largely been in components and JavaScript, not in crafting UI’s…

Or maybe it’s time for me to re-read Jenifer Tidwell’s excellent book from 2006, “Designing Interfaces”.   (I have the first edition.)

Verbosio progress, October 8, 2013: Milestone 2 for template editing complete

Since my last progress update, I’ve slowly moved forward.  Some new goodies:

codemirror+positioning

I’m using CodeMirror (current version is 3.18) to act as my source code editor.  The scripted DOM is still there, this time showing attributes as children before a XUL tree separator.  On the right side, we start with relevant SAX events to capture details of the DOM I’m building.  Then there are logging events for the “Node Positioning Service”, which tries to align source code to DOM nodes.  The lower right box captures change events from CodeMirror.

Over the last couple weeks, I’ve compared what I’m working on with my project’s immediate goal of building a XML templates editor, and realized that I was starting to get distracted.  So I filed a few tracking tickets, checked my roadmap, and discovered I’d essentially finished the second milestone: the DOM implementation in JavaScript.

The next major phase will be about handling XML entities (&button.label; for example).  This will require a first introduction of a “shadow DOM”, where some DOM nodes will hide others from direct access.  (Indirectly these other nodes will still be accessible.)  This also means I’m starting to catch up with progress I made five years ago… that’s what happens when you do a rewrite.

I’m still looking for help!  In particular, I would like JavaScript developers to help out with:

  • Code reviews (Python code reviews also welcome)
  • CodeMirror’s API’s
  • Mozilla’s transaction managers
  • Encoding support (UTF-16, Unicode, etc., since I’m definitely “doing it wrong”)

By the way, I enjoyed the Mozilla Summit, and thank you for letting me present Verbosio at the Innovation Fair in Santa Clara.

Preserving source formatting: I need samples of real-world markup

Imagine the following:

  1. You have a XML or HTML document from a version control system. (SVN, Hg, git, etc.)
  2. You have an editor which parses that into a DOM document.
  3. You edit the document using a GUI, not looking at the source code – just adding in paragraphs, new widgets, math formulae, vector graphics, etc. with a visual editor of some kind.
  4. You finish your changes and save the document.
  5. You get ready to commit your altered document back to the source code repository, looking over the change set, and wonder what your editor did to your document:  it only looks vaguely like what you started with.

This is a hard problem, I think, and for the most part it comes down to handling white space inside a XML tag of some kind.  For instance, attributes may appear all on the same line as the element they belong to, or on new lines by themselves, with different indentations.

On the one hand, a visual editor like this isn’t required to preserve source code formatting.  As long as two different formats of the same XML document’s white space parse to equal DOM nodes, the editor normally does not care.  On the other hand, version control systems don’t often deal with DOM nodes – only with the source markup that an editor generates.

In trying to build a better XML editor (Verbosio) which supports editing in visual modes, I feel I have to respect both visual editing and existing source formatting.  Now, there may be a solution out there – a set of algorithms and a specification – but I have certainly not heard of it.  On the other hand, many source-controlled versions of XML documents have certain patterns enforced from coding patterns their authors require.  I am much less concerned with formatting a document the way I think it should be than with honoring existing patterns of source formatting.

Given a few samples of real-world markup, I can make some educated guesses – write a few JavaScript objects into a model which tries to respect the original source.  What I’m asking is for diverse samples of human-generated and human-edited markup – not computer-generated 1MB+ size files, but something more reasonable for people to work with.  It’s especially significant if these samples live in (or are derived from) version control repositories.

So, would you please post relevant links as comments to this blog entry?  Either an existing specification for this sort of problem, or diverse XML samples where people have their hands on the code regularly.

XPCOM Stream Guide: Help Wanted

I’ve just started putting together a guide on XPCOM streams over at MDN.  We already have wonderful guides on strings, hashtables, and arrays.  Streams are also really important, but there’s not much documentation on them.  So I figured, why not start some?

I’m going to be working with streams a bit over the next few weeks as I convert from SAXXMLReader.parseFromString() to SAXXMLReader.parseAsync().  Apparently, the parseFromString() method is not very nice.  So I’m dusting off some old tools and ideas, and I’ll be writing down what I run into that I think others can use.

The XPCOM Stream Guide is right now little more than an outline.  My own time is very limited; if anyone else wants to fill in some of the blanks, I’d really appreciate it!

 

Where do we debate computer science?

Both the web technologies space and myself are undergoing a fair bit of change right now.  I’m in college learning C++’s fundamentals which I missed, and picking up on concepts in computer science I never really worried about before (order N^2, order n log n, etc.).  People who are familiar with my blogging are also aware I like to understand best practices that are language agnostic (i.e. “each function should do only one core thing, and leave the rest to other functions”).  I can say I’ve learned a lot of what’s from talented Mozilla contributors, but not necessarily a lot of why’s (at a level higher than specific languages).

On the web technologies side, new capabilities are arriving in Mozilla code very rapidly (WebIDL, “DOM Bindings”, the new C++11 standard, Opus audio, direct proxies and hashtables in JavaScript) – so fast that I definitely haven’t been able to keep up.  There’s some very interesting stuff going on under the hood, and I’m beginning to think I won’t recognize a lot of that code in a couple years.

So I’m writing to ask the Web and the programming community at large (yes, bigger than just Mozilla) where we talk amongst each other about computer science as a practice – not about specific languages, but about why we do what we do.  Keep in mind I’m not referring to one-way conversations, where someone writes a blog entry (CodingHorror) or a Wikipedia article or a book (thanks, Code Simplicity, seriously), but more of a freeform discussion among professionals.  Sort of like an IRC channel (with politeness), or a newsgroup, or even a  meal at the local bar.

I’m experienced enough as a developer now to mentor others, but I recognize that I am not yet the Wise Bearded One.  (Nor will I be in December, thank you very much.)  I’d really love to find a place to hang out with other professionals in a free-form exchange of ideas.  Somewhere to step into the abstract without getting too technical.

Ideas welcome!