Re: Your proposal

Yaron--

  Thanks for time you put into writing this response--I'm more familiar (due to reasons of political correctness) with the browser other than IE, so it's good to hear about all of this technology in IE and other Microsoft products.

  I've found that the XML Jedi mind trick works pretty well, second only to the Java mind trick--"You will write all applications in Java", to which the reply is "Yes, we must use cutting edge Web technologies like Java".  However, your mileage may vary in Redmond ;-)

  My response to all of this is <monotone>"Yes, I will use new HTTP methods"</monotone>.

--Eric

----- Original Message ----- 
  From: Yaron Goland 
  To: 'Eric Sedlar' 
  Cc: w3c-dist-auth@w3.org ; 'yaron@goland.org' 
  Sent: Thursday, February 24, 2000 12:13 AM
  Subject: RE: Your proposal


  (Note: I found this in my drafts directory, figured I would send it out)
   
  See below 
  [WARNING: O.k. I admit it, Eric's letter sent me down memory lane. So this e-mail is seriously long. However if you are interested in some of the history behind how XML ended up the way it is, how XML ended up in WebDAV and some of the work I and others at MS did to make WebDAV actually happen, read on!]
    -----Original Message-----
    From: Eric Sedlar [mailto:esedlar@us.oracle.com]
    Sent: Mon, January 03, 2000 4:33 PM
    To: Yaron Goland
    Cc: w3c-dist-auth@w3.org
    Subject: Re: Your proposal


    Now we get into our areas of disagreement. 
    * I mentioned HTTP as having lousy performance WITHIN A SINGLE PROCESS.  I guarantee you that a function call is much cheaper than formatting the request into a stream, writing into a loopback socket, reading the result back, parsing it,  and then the reverse.  Another example is SQL access to properties of a resource.  If I want to query the resource data efficiently, I need to be able to treat properties of the resource as "virtual data", not make a series of function calls on each item in the query set.  If I want to write a servlet that takes an HTML form input and changes WebDAV properties, I definitely want a functional API (in Java).  Everyone knows that SMB sucks and is incredibly chatty.  As far as HTTP performance, it mostly depends on the amount of functionality you can pack into one request.  If I have to issue a GETLOCKED name request followed by a bunch of other requests to get all of the state of a particular resource, HTTP will suck.   

    My point was that the WebDAV model is actually bigger than HTTP--other data access systems (like SQL and XML technologies) should also be able to manipulate the data easily.   

    <Yaron>

    I certainly wouldn't want someone accessing a DB on their local machine to have to run their requests through DAV. What a waste, a full trip through the protocol stack just for a local request? Of course they should be able to work through SQL and other mechanisms. In fact, in fulfilling my role as the lead author of the WebDAV specification and the chief WebDAV evangelist at Microsoft I was worked hand-in-hand with the ODBC/OLE DB/ADO teams at Microsoft to ensure that the model we were putting together would be easily accessible through their APIs. That is why, for example, OLD DB 2.5 has a native WebDAV provider. This is what was used to implement the WebDAV support in Web Folders. I also worked hand-in-hand with the SMB folks at Microsoft to make sure that the WebDAV model was such that we could run WebDAV directly over the file system and at the same time access it through the Win32 file APIs and SMB without any conflicts. That is why Win 2000 is able to ship a native WebDAV implementation that works directly off the file system allowing for simultaneous access by FTP/SMB/WebDAV/Win 32. At the same time I also coordinated with the Exchange group at Microsoft to ensure that the WebDAV model was consistent with POP/IMAP/MAPI. That is why Exchange 2000 (the upcoming release of Exchange) has a full WebDAV implementation that enables you to access your e-mail box simultaneously through POP/IMAP/MAPI/WebDAV. I also coordinated with our Active Directory team to ensure that the WebDAV model was consistent with LDAP/ADSI. I also coordinated with the SQL team to ensure that WebDAV could be easily layered on top of SQL so that one could do direct SQL queries as well as normal work. There are a number of other groups that I also coordinated with but unfortunately I can't release any data about that. But look forward to some interesting uses of WebDAV coming down the line. My favorite current example is pointing Office 2000 against Exchange 2000. Office sees Exchange as just another file store while Exchange sees Office as just another WebDAV client. The result is that you can save your files in your e-mail directories thus having your files automatically replicated and backed up for you. Very cool use of DAV. 

    I think we will see even cooler uses of DAV when DASL starts to get deployed. With DAV/DASL you can control/search your E-mail store, file store, directory or database with the same protocol using the same data model. I have been working for over 3 years to bring this vision to fruition. So obviously I am deeply interested in the issues of enabling access to stores that support WebDAV through many APIs and protocols..  

    That having been said I disagree with your statement that having to make multiple requests to describe a resource will destroy performance. A properly implemented HTTP server will support pipelining which, as repeated experiments have demonstrated, has equivalent performance to having one big message but with much cleaner protocol semantics. So long as the connection is kept open (default in HTTP/1.1) and the process is not spun down after each request there is no difference between processing one huge message and processing a string of messages that are pipelined in. In fact, in the WebDAV Book of Why I discuss in detail how depth came about and why we used depth instead of just pipelining. It is an interesting story, you will see that while performance was involved, it had nothing to do with the cost of parsing messages. You might want to read that section entitled "<"This is another fine protocol you've gotten me into!">" available in http://lists.w3.org/Archives/Public/w3c-dist-auth/1998OctDec/0303.html. Keep in mind, btw, that the server does need to be a bit smart. For example, it should wait a little bit before processing a message in order to see if the next message can be bundled into a single API call to the underlying system. There are lots of tricks but these tricks are all well known. So it isn't a big deal.

    </Yaron>

    * Another reason for making all information about a resource available as properties is using XSL to format an HTML page giving access to all of this data.  There is a lot of support for not requiring a lot of programming (ala JSPs or ASPs) to format data within a page.   

    <Yaron> 

    No one I'm aware of is arguing against making the information available through XML. The argument is simply this: Do we get the information through a PROPFIND or do we get it through multiple requests, each of which are free to return XML. My response is that we must do the later because of the need to support content negotiation. In order to keep the WebDAV property model as simple as possible it does not support any form of content negotiation. This means that I can't say "I want the name property in French" or "I want the picture you have stored in this property returned as  JPEG." The only way to get this sort of negotiation is through individual method requests. If we only provide these properties through PROPFIND then we dumb DAV down to the level of a static database table and prevent people from developing more intelligent systems. The only real counter argument I see against using multiple methods is the possible performance ramifications but as I argued in the previous paragraph the introduction of pipelining into HTTP/1.1 makes this largely a non-issue. 

    </Yaron> 

    * I don't understand why you think it will be easier to handle protocol extensions on the client side by using additional HTTP methods rather than resources.  XML applications are used to ignoring elements that they don't care about--extensibility is one of the key wins of XML vs. traditional OO languages.  It allows DOM objects to be passed around multiple areas of the system better.  For example, let's say that I have some client library which implements the client side of the HTTP protocol, and allows plugins to that client to access the XML DOM for each resource.  I can install a new client plugin to handle new properties that appear in that DOM, but I won't be able to extend the HTTP protocol.  HTTP protocol implementation is typically at a lower level of the client or server than properties.   

    <Yaron> 

    The main reason  that I believe that adding new methods is easy is that I have implemented a number of XML/DAV systems and it has repeatedly proven simpler to support new methods than new elements. This is mostly because the HTTP model is flat so that I can access things like the method name or a head directly where as with XML I have to navigate through a tree. This is the same logic that leads people to use attributes in XML, which as I argue in the WebDAV book of why is a really stupid thing to do. But I digress. 

    As for ignoring elements, it is funny to hear someone say that XML is used to ignoring elements. I first became involved with XML before it was called XML. I had helped write a proposal called Web Collections that would later be put together with a bunch of other proposals in order to become what we now know as XML. In fact, my boss at the time was Thomas Reardon who was MS's representative to the W3C and the leading proponent both at MS and the W3C for XML. Now a days only the old timers remember the work he did in making XML real but in my estimation without him XML never would have taken off. As such I have been involved in helping with the XML design since day one. One of the problems we ran into with XML when it was first introduced was the question of handling unrecognized elements. At the time the SGML heads had total control over XML much to the anger and dismay of those of us born after the turn of the last century. At the time the SGML heads were arguing hard and fast against have a definition of well formidness. What they wanted to do was to require a DTD be available for all XML documents. These were the same geniuses who argued against the introduction of namespaces. In fact it was their sheer cussedness that prevented us from being able to just name an XML element as a URL. They refused to allow in the forward slash and other characters that would have been necessary to allow URLs in the name of elements. That is where the namespace hack came from. Anyway, I'm digressing. So the SGML heads were arguing that you always had to have a DTD so therefore, naturally, if you had an "unrecognized" element it was an error and you should fail the entire XML document! Those of us from the post-Diluvium age were aghast. Unfortunately Tim Berners-Lee (TBL) was convinced that we had to keep support of the SGML community. I remember getting lectured by Andrew Layman (who hated what the SGML guys were doing as well) about the "millions" of SGML documents out there and the need to work with them. Of course now nobody knows or cares about those ***BLEEP*** SGML documents, but a lot of the really awful crap you see in XML is the result of the deals that TBL made with the SGML guys to keep their support. TBL touches on this topic a little bit in his book. Anyway, the SGML guys were finally argued into accepting the idea of "well formidness" and thus we were freed of needing a DTD. However this left open the question of what to do when you hit an element you didn't recognize. Oh man were there flame fests on that topic. People in the protocol business tend to think of XML as just a tree so for us it is natural to just prune the XML tree at an unrecognized node. But the SGML guys don't see a tree when they see XML, they see a markup language. So, for example, imagine I have the following entry in my XML document "<p>He was a <bold>grand</bold> man!</p>. Now let us imagine that your XML parser supports the <p> tag but doesn't support <bold>. If you just pruned the tree you would end up with "<p>He was a man!</p>". That is clearly not what you want. The result was just a real mess. Some people demanded that you always prune. Some demanded that you only removed the tags but not the text. Some argued that you should have an attribute you would put on each element that would specify what you should do. There never was much of a resolution to the issue. Eventually I just got really pissed off and added what is now the first paragraph of section 14 of RFC 2518 to WebDAV declaring that we would prune any sub-tree whose root we didn't recognize. Man, did we ever catch hell for that! The XML guys told us that we wouldn't be compatible with their XML. That we were creating my own world that wouldn't work with the rest of the XML universe. Oh man, what a bloody mess. So to hear you say that the WebDAV ignore rule (as we called it) is a strength of XML is pretty damn ironic. 

    As for HTTP generally appearing at a lower layer and not being available from the DOM, you may want to check out http://msdn.microsoft.com/xml/reference/scriptref/XMLHttpRequest_object.asp and http://msdn.microsoft.com/xml/reference/cvbref/IXMLHttpRequest_interface.asp. The links describe the XMLHTTP interface available through IE 5.0's DOM. The first link describes the interface for script programmers (JScript/ECMAScript, VBscript, Python, Perl, etc.) and the second for C/VB programmers. This interface allows you to send and receive generic HTTP messages directly through IE 5.0's DOM. What is really cool about it is that while it can handle arbitrary HTTP messages it is smart so that if you want to send or if you receive a message with an XML body it will automatically detect that and automatically parse out, lode up and pass back a DOM pointer to the contents. So actually HTTP access is as free and ready as XML access, at least in IE 5.0. BTW, massive credit goes to Alex Hopmann who nearly single handedly convinced the IE guys to ship this feature. Hats off to Alex! 

    </Yaron> 

    HTTP request/response parsers are not generic technology like XML parsers are   

    <Yaron> 

    I'm not sure what leads you to this conclusion. For example, as the author of the HTTP over UDP spec I found that HTTP request/response parsers were incredibly generic. This is what made HTTP over UDP so compelling. We could take existing parsers, do a little tweaking and bam, it just worked. In fact I would argue that there is no better proof of the generic nature of HTTP parsers then the existence of WebDAV. We didn't have to do any major work to support WebDAV in our HTTP request/response parsers. HTTP already understood the concept of methods and headers so we were able to use the existing parsers to implement WebDAV. In fact the biggest problem we ran into is that some folks had implemented checks on what methods you could send as an error detection procedure! Sigh... In WinInet we had the problem that the devs, in order to optimize performance, had built a method mapping table that turned the method names into tokens and then moved the tokens around. Unfortunately the tokens were fixed, which meant that WinInet (our HTTP client stack) couldn't handle new methods. But the problem was easily fixed so modern versions of WinInet can handle arbitrary methods. 

    </Yaron> 

    * It sounds like you are arguing against the general trend of making all of your data available via XML, which is counter to the general strategies most technology companies are following now, including Microsoft and Oracle.  With your proposal, you are going to render all of the burgeoning XML technology base useless, which is why the WebDAV people built around XML in the first place.   

    <Yaron> 

    I was the one who introduced XML into WebDAV. Bonus points to anyone who remembers the original meeting when I proposed Web Collections (a precursor to XML) to WebDAV. This was the first time WebDAV was introduced to what would become XML.  

    As for Microsoft's XML strategy, that is a whole other story. Because my boss, Thomas Reardon, lead the XML effort at MS for a long time I was actually part of creating our XML strategy. When Adam Bosworth took over XML I ended up having weekly meetings with him just so that we could synch up on Microsoft XML corporate strategy. I also worked with both Andrew Layman and Jean Paoli on XML issues. I still remember the tongue lashing I gave Andrew over fully qualified end tags.  It wasn't Andrew's fault, he hated it too, but he was the messenger and I was really angry. Basically the SGML heads forced XML to adopt fully qualified end tags. In the original proposal XML markup looked like <tag>stuff</>. But the SGML heads argued that this didn't make XML readable so we should make the already mind blowingly bloated XML format even more bloated by adding fully qualified end tags, i.e. <tag>stuff</tag>. Yet another fight that they won. 

    As such I can safely assure you that I would not do anything to "render all of the burgeoning XML technology base useless". Having helped birth XML I would certainly be reticent about killing it. After all, infanticide is a messy business. I will admit, however and Jim Whitehead will back me up, that there were plenty of times when I would have have liked to have given XML a severe spanking followed by a good amount of time in the corner. 

    I used to joke about the "XML Jedi Mind Trick." You just wave your hand in front of someone and say "You will use XML". Their eyes loose focus and they repeat in a monotone voice "I will use XML." This helped a lot in early WebDAV adoption. If I couldn't convince someone to adopt WebDAV because of its own merits I would just pull the XML Jedi Mind Trick and they would adopt it because it used XML. After all, if it uses XML it must be cool!  There is apparently something indescribably sexy about angle brackets.  

    </Yaron> 

    Is anyone thinking about a thin-client WebDAV implementation (HTML only) running in a browser?  I should be able to do stuff like changing RSRs, manipulate directories, etc. with a web app.   

    <Yaron> 

    Thinking? We have already written it! Using XMLHTTP Exchange has written up an entire E-Mail client for Exchange 2000 that speaks WebDAV to their server. The whole thing is written 100% in nothing but script!!!! What is really amazing is that it uses DHTML so the UI looks almost 100% like Outlook, even down to the multiple Outlook views and the Outlook bar. It is seriously cool. I have also written, on my own, a generic WebDAV client library in JScript. Unfortunately I never finished it. Too many other things to do.  

    </Yaron> 

    --Eric   

            Yaron 

      

    Yaron Goland wrote: 

      I wouldn't accept a blanket statement that HTTP has lousy performance. I have seen it repeatedly beat the pants off all competing systems. For example, we did a speed comparison of a super optimized SMB implementation and DAV in W2K. DAV won hands down. The reason is that SMB, like most protocols of its ilk, is unbelievably chatty. They are basically RPCs rather than protocols. DAV, on the other hand, is extremely optimized. So even though the SMB implementation could process some ridiculous number of messages per second DAV still had better performance because it sent a hell of a lot less messages.As for the use of properties, I assert that there is no difference between executing a GETLOCKEDNAME method and getting the lockedname property in terms of how you write your back end. However, the first is a hell of a lot easier to deal with in terms of specifying and extending the protocol than the second. So the issue isn't one of back end implementation, it is one of front end convenience.In other words, just say NO to live properties.            Yaron 
        -----Original Message----- 
        From: Eric Sedlar [mailto:esedlar@us.oracle.com] 
        Sent: Monday, January 03, 2000 1:09 PM 
        To: Yaron Goland 
        Cc: w3c-dist-auth@w3.org 
        Subject: Re: Your proposal 
         
        One problem with your qualms about properties is that we are trying to map WebDAV data to object representation systems that do not have functional semantics, like XML.  We should define an interface that doesn't rely on the distinction between functional interfaces and properties for maximum implementability on various servers.  (This distinction is something may programmers have trouble with--does everyone always bother to create accessor methods for everything?  ...)  The benefit of using live properties as a representation is that object properties are more "portable" to the other types of systems that may want to access the same data, presumably through another means than the HTTP protocol (which isn't particularly efficient).  (Which brings me to another unrelated issue--should there be a functional interface to WebDAV methods for programs living in the same server as the data repository, given the performance costs of HTTP within a single process--more on that later). Yes you need a set of clear rules for how live properties are used, and unless their use is rigorously controlled, you will have compatability problems of the type you cite, but this is a problem with any loosely written standard.I think of properties in the JavaBeans sense--in an OO language binding, they would actually be functional interfaces to set and retrieve them, but could be overridden to provide customized behaviour.  Any JavaBeans user has no idea whether or not this piece of data is live or not, and this model works well. --Eric 
          ----- Original Message -----
          From: Yaron Goland
          To: 'Eric Sedlar'
          Sent: Monday, January 03, 2000 11:18 AM
          Subject: Your proposal
           Eric, I read your analysis of Geoff's proposal and was really impressed by your deep grasp of both HTTP and WebDAV. 
          I have a series of issues with your counter-proposal but I'm going to hold off on commenting until we can build up more of a common base for conversation. Please see my post on the mailing list in regards to this. 

          I did, however, want to point out a general design issue regarding your proposal that isn't directly related to locks. In your proposal you suggest using properties to provide various bits of protocol information, such as which names are currently locked. I would caution against using properties in this way, see http://lists.w3.org/Archives/Public/w3c-dist-auth/1998OctDec/0302.html for more details. For a history of how we ended up in this mess in the first place see http://lists.w3.org/Archives/Public/w3c-dist-auth/1998OctDec/0074.html and http://lists.w3.org/Archives/Public/w3c-dist-auth/1998OctDec/0303.html. BTW, all of these posts are collected in the WebDAV Book of Why available at http://lists.w3.org/Archives/Public/w3c-dist-auth/1999JanMar/0129.html. 

                          Thanks, 
                                          Yaron