Re: N3.js 0.2.0: now with browser support

Ruben, thank you very much for the extensive and insightful response.

On 04.12.2013, at 12:37, Ruben Verborgh <ruben.verborgh@ugent.be> wrote:

> Hi Michael,
> 
> Thanks and interesting questions!
> 
>> I am right now using N3.js through rdfstore-js actively. We are building a library aiming to make visualizations of graphs easy (like everybody tries), but also adaptive to the user and device and should also present not yet seen structured graphs in a more or less human readable way. (I am personally annoyed by all the graph representations of graph based information. This because I think graphs representations are not optimized for the human but for …?)
> 
> That sounds cool. Would you have a demo somewhere?

We will soon (in less than two weeks) put a first demo online, until then I like to clean-up and prepare the code a bit.

The main goal/vision in some phrases (work in progress and input appreciated)

# One Page
Visualize a graph with the amount what can fit roughly on one page or screen. Everything you input in the visualization process needs to be visualized. Basically the decision what is needed by the users needs to be made upfront by another component.

# Separation between content structure and visualization
We have a Matcher which looks for known structures (basically extract known trees out of possible cyclic graphs). These structures then are linked to one or multiple visualizations (dependent on device, user, language) which the visualizer with the current context marries and then renders. The first basic heuristic how to decide the order of matching is that bigger matched structures most probably have a nicer visualization knowledge. Meaning that the match which consumes the most triples gets precedence.

# Extensibility
All the knowledge about the structures and also how to visualize these structures can build each on top of another and can be extended in an easy way.


> Also, I like the word "adaptive" in there and I wonder whether it could be a nice application of streaming.

In the libraries internal it would not make sense to use streaming in any way, as the libraries goal is to take a input graph in visualize it as good as possible with the provided "visualization knowledge". 

> We could imagine the simple case of the visualization updating as a large Turtle file comes in,
> but also updating iteratively as more and more data is explored.

But yes there is nothing what hinders you to take a stream and call our library over and over, every time you get some new triples.

> For instance, if a user activates a node, data is pulled from the Web through dereferencing.
> 
> N3.js supports a streaming interface, I'm not aware on how rdfstore-js implements this.
> Could you maybe describe what functionality of rdfstore-js you are using?

Mostly we use the rdfstore-js right know to use the SPARQL queries to access the graph. Like this we have the full possibility to access structures in the graph. This might be in a near future not anymore necessary as the amount of different kinds of access to the graph might flatten out based on the use cases for this kind of visualization framework.

> While I'm not aiming at the moment to cover all rdfstore-js functionality,
> maybe the N3.js library could receive additions that make it useful for scenarios like yours.

As stated above, probably it does already. What possibilities to access a parsed graph are there right now? What for sure is a needed feature for our visualization framework are more formats. 

> 
>> Anyway, to the point: As this library also is build in JavaScript I would like to profit from your learning curve in the way of what tools you did use for your development. I don't speak about editors or such (even if might be interesting)
> 
> Well actually, I'd like to talk a little about editors too.
> Specifically, the fact that I use a simple text editor as opposed to a full-fledged IDE for JavaScript development.
> This fact alone heavily influences any JavaScript development I do.
> In particular, this means that my code is designed to be developed without an interactive debugger.
> Small tests are more important than any debugger, because they focus on the result, not the details. 

Here we are walking on the same turf ...

> 
> The reasons for not using an IDE are beautifully summarized in “IDEs Are a Language Smell” [1]:
>    “I'm not opposed to the concept of an IDE, in fact I wouldn't go near a Java codebase without one. But the fact that I need one to auto-generate boilerplate and navigate hundreds if not thousands of little files certainly is a language smell.”
> 
>> but much more about stuff like the test environment, how to make a library useful in Node and the browser (which build scripts), what is necessary to look at to make it browserify compatible, in a nutshell I would like to hear from a "senior" how to head start the development of a JavaScript library.
> 
> Testing is the most crucial aspect when developing a library.
> When programming in dynamically typed languages, you give up compiler checks (that only help a little),
> so tests are the only way to verify that what you've written works.
> 
> But it even starts before that: I use JSHint to ensure that all code (main and test) conforms to my own rules.
> I'm pretty strict about that, even enforcing stylistic things such as proper indentation [2].
> I have a pre-commit hook that a) runs JSHint and b) executes unit tests. I cannot commit if either fails.
> 
> Testing-wise, I have the same attitude as with backups: a file that is not backed up is lost.
> Hence, a feature that is not tested is lost. That said, always remember Dijkstra when you test:
> “Program testing can be used to show the presence of bugs, but never to show their absence.”
> I create tests that verify desired functionality, emphasizing the highlighting of likely bugs.
> 
> However, the most important test for the parser is the Turtle spec test suite.
> Before I implemented the spec tests, I thought I had a decent library,
> but then discovered that I was failing more than half of the spec.
> While passing the spec gave me this confidence, a few bugs have still been found.
> Have a look at the tests to see how they systematically go through the functionalities.
> Furthermore, you should make it easy to write tests. Write test helpers that keep the code clear [3].
> 
> As for browser-compatibility, I didn't really care at first, as my goal was to have a Node library.
> However, I always kept in mind not to make it too difficult, so I didn't rely on Node-specific things where possible.
> Actually, it turned out the library was in itself compatible with browserify, if not for a recent breaking change [4].
> As browserify relies on the abstract syntax tree, it couldn't extract `require`s anymore. A trick fixed this [5].
> However, following the backup/testing observation above, I couldn't say it worked with browsers if I didn't test it.
> This is where Testling comes in [6]. Basically, it browserifies your tests and executes them remotely on different browsers.
> It was then that I noticed that my testing framework wasn't browser-compatible, so I switched from Vows.js to Mocha [7].
> Honestly, I don't see any reason why you would use Vows.js anymore; it's more code and it's incompatible.
> Then I had to make minor browser-specific adjustments, and now I have an automated indicator whether it works or not [8].
> Honestly, never rely on opening a browser manually for browser tests. You won't do it, and things will break.
> In addition to browserify, which generates "heavy code", I also have a custom build script that excludes Node-specific things.
> In my library, this means Streams. I purposely build Streams as a layer around my code [9] for performance (!) and compatibility.
> 
> Then, some main things to know for a good head start.
> Have a vision of where you want to go. Have an architecture, a master plan where things fit in.
> 

These are all really help full tips, I definitely have too look at jshint, as also have now already made the easy decision to use Mocha. Testling looks also promising.

Another additional question, what kind of Design Pattern to use for libraries today? I did see that Module patterns are kind of common. But I am overwhelmed regarding the different styles and possibilities. Also I like to start out with a code documentation right from the beginning. And am back and forth with usedocjs.org and simpler stuff like https://github.com/jashkenas/docco ?

> And the most crucial thing: discipline.
> Never be tempted to write bad code. Think maintainability, not short-term wins.
> Everybody can make complex things. Few can make complex things simple.
> I care a lot about aesthetics in my code, because beautiful code can be managed better [10].
> 
> Cheers,
> 
> Ruben
> 
> [1] http://www.recursivity.com/blog/2012/10/28/ides-are-a-language-smell/
> [2] https://github.com/RubenVerborgh/N3.js/blob/master/.jshintrc
> [3] https://github.com/RubenVerborgh/N3.js/blob/master/test/N3Lexer-test.js
> [4] https://github.com/RubenVerborgh/N3.js/commit/cb5ad3c328b7646d18f1f93a9a23829b9d6ab9c5
> [5] https://github.com/RubenVerborgh/N3.js/blob/master/N3.js
> [6] http://ci.testling.com/
> [7] http://vowsjs.org/ and http://visionmedia.github.io/mocha/,
>     as seen in https://github.com/RubenVerborgh/N3.js/commit/bc876b22fd2b8a864dc3135457b164def86ce679
> [8] https://github.com/RubenVerborgh/N3.js#license-status-and-contributions
> [9] https://github.com/RubenVerborgh/N3.js/blob/master/lib/N3StreamParser.js
> [10] http://ruben.verborgh.org/blog/2013/02/21/programming-is-an-art/

All best,
Michael

Received on Wednesday, 4 December 2013 14:05:37 UTC