Forms & Data Entry & Validation

Hi,

I've been doing some research on Forms implementation and have been
carefully reviewing the XForms emails and specifications.

I do not see a way in MosquitoX to do actual page layout with XForms as the
entry engine.  Due to the way the examples render in the browser I also do
not see a way to copy an existing working form and modify it due to the code
translation not preserving source.

There has also been much discussing about Validation.  In a standard
database application the only really correct way to do data validation is on
a field-by-field basis with procedural entry.  Existing <FORM> markup in
HTML is nothing more than a one form-submit batch.  While XForms transcends
this by enabling field by field entry with dynamic display of the results it
does not appear to enable complex validation of the data in any particular
field.  In many cases, the validation would need to rely on a huge database
for any particular field and an example would be that of Zip Codes in the
US.  One certainly would not want to implement that within an XML format,
but would want to make a call against an existing database and one would
want to be able to populate corresponding fields with data such as automatic
fill in of the City and State fields based on the return from the Zip Code
field just as many contact managers do quite effectively.

One of the biggest issues involved with any types of forms on the web is the
STATE OF THE USER within an application/form.  HTML forms are essentially
stateless and what is needed is a persistent state to make a browser act
like a terminal with a dedicated connection.

Microsoft has played with its own proprietary way to do this with their
Remote Scripting "XML Download Behavior" but this is not acceptable as a
universal solution as it requires IE proprietary code, but it is interesting
to look at what they have done as their general approach is valid.

The biggest thing that I can see as missing within XForms is abstract
identification of each particular field with a name that is distinct from
either the NAME or VALUE pair and which provide coordinates such that the
field can be determined for later manipulation based on its abstract
identification within an array.

One of the most useful and interesting approaches to this has been done by
one of the Pick Database developers, www.pixieware.com,  which has
established a protocol in which servers can control and rewrite the supplied
template web page with delimited string commands and in which INPUT FIELDS
have ids of "fx1, fx2, fx3" etc. and in which other active objects use
prefixes of "gx8" for a grid and then ids of gx8x1, gx8x2 for rows within
the grid and input fields within rows have ids like "gx8x2x4" which allows
precise positioning and identification of any one particular piece of data
from any given field.

At the heart of how to collect data is WHAT DATA you are transferring and to
where.  The LEAST amount of data is ALWAYS BEST.  The absolute standard for
any type of data involving fields is to use a string with a single unique
delimiting character.  XML is problematic when it tries to wrap every piece
of data with extraneous and grossly verbose and redundant definitions which
can be NORMALIZED and eliminated from the transfer enabling fast and
efficient parsing - up to 10 times faster than XML encoded data fields.

The pixieware approach calls its method the FBP protocol and features fully
bi-directional such as server calling browser functions as well as
vice-versa.  Microsoft's SOAP is uni-directional with only support for
browser calling server functions.  Moreover the FBP method is identical on
both IE and Netscape and is JavaScript based and most importantly it routes
the data to and from the browser in essentially a dynamic variable format.

On the topic of CSS positioning, HTML version 4 and above support absolute
x-y positioning of labels (<DIV..>s).  IE5+ also supports direct positioning
of input fields, while Netscape 6 and IE4 require them to be parked inside a
positioned <DIV..>.

A very clever solution that pixieware utilized for translation of existing
terminal screen designs and layouts to web forms is via replacing @(x,Y)
with a Web equivalent, and they created a SUBROUTINE WEBDU for labels and a
SUBROUTINE WEBIU for input text fields, text area boxes, buttons, and
checkboxes.  They used a browser pop-up window workspace of 720 x 500 pixels
which enables multiplying existing x values by 9 and existing y values by 22
which nicely translates character-based coordinates to Web forms.

An example of this is as follows:

Pick/Basic code from WEBDU
Translate positioning from character coordinates:
x = (xraw * 9);'px'
y = (yraw * 22);'px'

In the web application this becomes:
call WEBDU (4,2,0,0, "Supplier Code","","",SC)
Then to define an active GUI input field:
call WEBIS(22,2,0,0,"","fx0", "",SC)

The pixieware FBF is available freely as an open source model and working
example web applications and more details are available at
http://203.109.148.138:7301/pixieinfo/rsdemointro.htm

Some of their approaches are valid to consider for XForms.

Best regards,

Jim R. Benfer Jr.
www.maxframe.com


----- Original Message -----
From: joern turner (by way of "Masayasu Ishikawa" <mimasa@w3.org>)
<joern@webman.de>
To: <www-forms@w3.org>
Sent: Tuesday, August 21, 2001 1:47 AM
Subject: Re: Form function and presentation - Schema usage


> Hello Josef,
>
> excuse my late reply (but my collegues and me are actually in the
> deadline for delivering a basic, working 'XFormish' implementation, so
> that's why) and thanx for your detailed reply but it seems to me that
> maybe i somewhat haven't made clear our/my intensions:
>
> let me try to clarify...
>
> Josef Dietl wrote:
>
>  > Hello Joern,
>  >
>  >
>  >>the former WD still defined a 'Simple Syntax' for defining
>  >>XForms-models,which was removed in the last WD. For my
>  >>personal opinion
>  >>this was an unlucky move...
>  >>Let me explain:
>  >>- it may be a personal issue but the removal left a gap which makes it
>  >>harder to imagine how the pieces (model,instance,ui) fit together. i
>  >>finally returned to a kind of 'simple syntax' for internal processing.
>  >>
>
>  >
>  > The current perceived gap is IMO mostly an editorial issue (nothing
>  > against our editor here :-): We have removed the simple syntax when we
>  > knew that we couldn't sustain it, but we hadn't had the replacement
yet.
>  > A joint task force was founded between the Schema Working Group and the
>  > XForms Working Group, and that task force has delivered its results
>  > recently. We are currently preparing a new draft that among others
fills
>  > the gap. Usual disclaimers apply, from what I can see holdig your
breath
>  > is in order.
>
>
>
> i think i understand the status of a Working Draft but as i've committed
> to the basic ideas of XForms and try to deliver a xformsish ;) thing as
> soon as possible,  we had to 'guess' something in the meantime in order
> to meet this goal (and adapt later). we would be happy to integrate a
> well-defined Schema syntax as soon as it's fixed.
>
>
> Our struggles/experiences with down-to-the-ground problems of the
> implementation brought up some differences/extensions to the current WD
> and i've tried to transfer these back...
>
>
>  > True, Schema is said to be complicated. But looking closely we found
>  > that
>  >  - given our requirements, the thinking process is the difficult part,
>  > the actual syntax is "just syntax". Maybe we could have simpliefied the
>  > syntax, but we could never have simplified the thinking.
>  >  - we have to support Schemas anyway. Simply because the use case that
>  > someone has a Schema for his or her data already is so overwhelmingly
>  > significant.
>
>
> Agreed, the thinking is the hard part - but after thinking when it comes
> to implementation efficiency and real-world constraints (limitations in
> clients/protocol/robustness of tools) are the other. sure, it's only
> syntax but it even brings a reasonable powerful machine into sweat if
> you implement all this server-side. so why hang the apple so
> un-necessarily high?
>
> if you're really reject the idea of an 'abbreviated syntax' you
> should better skip the following, but i think there are arguments ;)
> that go much beyond.
> ---
> everything in the following is induced through some kind of real-world
> constraint that exists TODAY, so some issues will solve through newly
> delivered technology in the future while other issues will remain. some
> can be overcome with common solutions while others may not. to list a
> few when you want to deliver for today:
>
> - support for CSS is in most cases either incomplete and unreliable.
> this makes the layout part in XForms hard.
>
> - only really advanced clients can deliver the 'XForms Full' conformance
> level in networked applications - with DOM-Events, the
> 'immediate'-functions and such. this implies use of RPC of some kind and
> some really serious processing...
>
> - ... or the use in an stand-alone client. but which client could not
> network today, if it's so powerful, that it can deliver full XForms
> processing?
>
> - ...otherwise the client must understand XML. unluckily it's likely
> that for some time (maybe one maybe two years, doesn't matter) from now
> it will be senseless to send XML directly to your client, cause they
> will not or only partially understand it.
>
> - so it maybe not completely wrong to assume a 'devided' architecture to
> become common: some processing and advanced functions server-side and
> styling, interactivity, simple validation client-side for example.
>
> - a lot of clients rely on http for simple to medium, even complex
> networked applications today and is IMHO one of the most interesting
> environment for XForms-processing, cause http often is used not only for
> browsers but for small devices too. but unfortunately this leads to all
> this well-known (and already mentioned) problems which i don't want to
> repeat here.
>
>
> the 'abbreviated' syntax lowers the memory consumption of our processor
> by degrees through the simple fact that we use three times less tags for
> expressing the same semantics; the memory consumption slowly reaches
> acceptable sizes: one testcase contains one example for most of the
> XForms UI elements and is about 500 lines of XML. these would have been
> easily about at least the double in full Schema syntax.
>
> - this memory consumption has naturally some unwanted effects on
> processing-time, searches and evaluations on the tree.
>
> - fully! Schema-aware, robust and open technology will be some time
> away, as is common support for CSS, XPath, XML, XSL on the client
> (sorry, if i repeat myself)
>
>
>  >
>  > Essentially, there are two dominant use cases, probably with not much
in
>  > between:
>  >  - the really-simple form, say, a search form. Doesn't need input
>  > validation to begin with
>
>  >  - the user interface to an XML-based application. In all likelihood,
>  > already has a Schema: the Schema attached to the application in the
>  > background.
>  >
>  > You will find the details of the solution in the next draft, roughly
>  > what we've done is a tiered approach:
>  >  - if you don't know about validation, don't worry - you don't have to
>  > provide it.
>  >  - if you do, you may live with just Schema Datatypes (part 2).
>  >  - if you have a Schema, use it.
>  >  - if you have someone who knows how to handle Schemas, work with him:
>  > the interface between the UI and the Schema is as lean as can be.
>
>
> i'm really looking forward to that!
>
>
>  >
>  > Again, it's not the solution that is difficult but the problem that is
>  > hard, and by using XPath we have at least the people who know about
>  > XPointer and the people who know about XSL-T on our side. We also leave
>  > the door open for future developments in the area of XPath together
with
>  > XML Query, which is a extremely exciting prospect to me. Ideally, one
>  > day in the future, your form fields could adress directly into the
>  > database...
>  >
>  > XPath is currently the only know-how bottleneck.
>  >
>  > With all due respect, but tasked with writing something for XML I take
>  > XML for granted, and tons of Web designers have managed to master CSS
so
>  > far. You can live well in an XForms context without knowing anything
>  > about Schemas, it just shouldn't surprise you that you can't access
>  > Schema's features in that case.
>  >
>
>
> i don't really intended to make this an 'all-or-nothing'-case. for one,
> i widely agree that new technologies offer new possibilities but for the
> price of learning. we are in the lucky situation to have all these
> xml,xsl,Schema,xpath, java expert ready, so that's not the problem to
> us. but even then the solution should scale to meet a wide range of needs.
>
> second, but i believe that a spec also gains through
> 'self-containedness' - maybe one reason why html was adapted so fast.
>
>
>  >>- a lot of the power of XForms comes from its abstractions:
>  >>e.g. the UI
>  >>abstracts a concrete representation in a UI-toolkit. Why not let the
>  >>model abstract the concrete usage of a schema (with little s)
>  >>mechanism?
>  >>This would nicely solve the issue with different clients: like the
>  >>processor maps the UI to the clients-format it could map a XForms
>  >>model-syntax to a version compatible with the client (may it
>  >>be Schema,
>  >>Relax,T-Rex, DTD, instance-attributes,..., according to needs).
>  >>
>  >
>  > Interesting approach. But as far as I can tell, it is difficult to
>  > realize and will not provide an XForms author with the necessary
>  > reliability. For example, if (s)he writes a form and assumes that the
>  > model takes care of checking that a date-of-birth is actually a date,
>  > then a translation to DTDs will just not cut it. Faced with that
>  > prospect, the designer would have to write the code for the date
>  > validation again anyway, so under these circumstances the model doesn't
>  > add value.
>
>   >
>   > If we provide the feature, we want people to be able to depend on it.
>   >
>
> absolutely.
>
> on the contrary a transformation from an internal representation to some
> kind of client-schema allows some advanced uses:
>
> consider following scenario:
> a client connects to a server and requests an XForms session. through
> user-agent recognition the server may fetch a XForm and the appropriate
> transformation for the client. for the server a transformation to Schema
> will be used to validate the incoming-data. for the client another
> transformation will map all validations that the client supports into an
> appropriate schema syntax and transfer it to it. this allows to reduce
> the contacts during the form-filling session and allows to delegate at
> least some immediate validations to the client.
>
> to cut it short it would allow to plugin a schema mechanism into the
> processor may it be Schema (very likely in the future), some lightweigth
> mechanism to support PDAs,DTDs when your application just requests that.
> just imagine to map even to database-models directly. this way a
> processor could also export different model-representations for
> data-consumers.
>
>
> this scenario has surely some more interesting aspects which
>
> allow a flexible adaptation to the clients capabilities and would go
> beyond all-or-nothing.
>
>
>
>  >
>  >>- we've been building some test-forms with repeats and
>  >>switches and even
>  >>in our simplified syntax, the appropriate model-markup becomes really
>  >>complicated (we're using just one element where Schema syntax needs at
>  >>least 3).
>  >>
>  >
>  > As I said earlier: the tricky thing is not the markup, the tricky thing
>  > is the thinking behind. Repeat and switch are extremely powerful
>  > constructs, and exploiting that power takes some thought. A switch in
>  > the UI for example is pretty much equivalent to a choice in the Schema,
>  > and repeat corresponds to maxOccurs>1. So, fundamentally, it's easy.
But
>  > you can nest and recurse these constructs - that is where the power
>  > comes from, but it's also where the brainware is challenged.
>  >
>
>
> the power of XForms is why we committed us to it and i can only agree
> that it really challenged us but we managed to walk through all this
> recursions and get it running but not without the price of 'breaking
> some rules' of the spec.
>
> with all respect for the WD and their authors, i can only sum up our
> experience: it's not only the thinking it's also the thinking of how to
> implement it!
>
>
> Joern
>
>
>  >
>  > Josef
>  >
>  >
>  >
>
>
>
>
>
>