A bit of background: our research group at Brown has been doing a fair bit of work on the semantics of web programming, and at last year's WebApps workshop in Boston, I presented the results of our effort in formally modeling the dispatch mechanism of DOM 3 Events (https://www.usenix.org/conference/webapps12/modeling-and-reasoning-about-dom-events).  After a conversation off-list, I was suggested to forward to this list a description of the main stumbling blocks we found.  Ultimately, there were two issues, one straightforward and the other more systemic.  (Note: These comments are based on the 04 Sep 2012 draft of the spec.) 

First, in the definition of "candidate event listeners",
This list is captured or frozen before event listeners on the target object are dispatched, and released or un-frozen after this set of of candidate event handlers have been dispatched (allowing these event listeners to add or remove additional listeners on other objects in an event's propagation chain, but not affect the order of listeners that will be invoked on the target object).
(Typo, "of of".)  This list isn't quite "frozen"; this wording could be read to say that if event listeners on a current target call addEventListener or removeEventListener on the current node, it will cause some kind of error.  Perhaps a better wording is, "This list is a temporary copy of the list of event listeners installed on the current target node; any changes to the current target (via add/removeEventListener) during dispatch will affect the target but not this copied list: this list is effectively immutable once dispatch reaches a given current target node."

Second, one piece of terminology worries me considerably, which is the apparently interchangeable nature of "event listener" and "event handler", even though they have semantically different behavior.  E.g, D3E, Sec 3.1:

Exceptions thrown inside event listeners must not stop the propagation of the event or affect the propagation path. The exception itself must not propogate outside the scope of the event handler.
and later,

Next, the implementation must determine the current target's candidate event listeners. This must be the list of all event listeners that have been registered on the current target in their order of registration. [HTML5] defines the ordering of listeners registered through event handler attributes. Once determined, the candidate event listeners must not be changed; adding or removing listeners does not affect the current target's candidate event listeners.

Finally, the implementation must process all candidate event handlers in order and trigger each handler if all the following conditions are met:

Why the change in terms?  Then, HTML5 strongly distinguishes handlers from listeners (sec handlers wind up registering the same listener, and the relative ordering of handlers and listeners are carefully defined.  And that listener can never be removed; it can merely be made impotent if the event handler content attribute is set to null.

Another discrepancy: section 3.2,
Authoring Note: Many implementations additionally interpret an event listener's return value, such as the value false, to mean that the default action of cancelable events will be cancelled (though window.onerror handlers are cancelled by returning true).
But the definition of EventListener (section 4.4) says that listeners have no return value.  I suspect you meant event handlers, in both places in section 3.2.

The discrepancies that we found between browsers frequently had to do with the mistaken treatment of adding and removing handlers, as opposed to listeners.  As part of our research, our repository (https://github.com/brownplt/DOM-Semantics) contains a test-generation tool.  Attached is one of those generated tests, which produces different outputs in IE9, Fx17, Chrome 24, and our model -- at least one browser is wrong, and possibly our model, too.  And it hinges on setting event handler attributes, not just event listeners.

It seems to me that you can simplify the D3E spec by eliminating the term "event handler" altogether, in favor of "event listener".  You can still make the point that languages such as HTML5 may define other ways of implicitly defining listeners.  E.g. (this isn't spec-ready text, but you hopefully catch my meaning), "HTML5 defines event handlers via content attributes and IDL attributes.  When either of these forms of attribute are first set to a non-null value, they install a single event listener, whose behavior is the event handler processing algorithm (HTML5, sec  When these attributes are modified or set to a null value, that event listener is not removed.  In fact, this listener can never be removed by mechanisms in HTML, and, since it is anonymous, cannot be removed via removeEventListener.  The rules for ordering event listeners therefore imply the proper ordering behavior of HTML5 event handlers with D3E event listeners."

Eliminating the term "event handler" also lets you get rid of the note in 4.3, regarding host languages allowing you to "register event listeners by the use of attributes"...

Hope this helps,
Ben Lerner