Re: stitching together APIs

I agree with David's general point, though I'm wary of issues creeping up from different OS impacting the evaluation of the API itself. Perhaps using something like CloudFoundry or even a VM with a pretty vanilla stack would serve the same purpose and level the playing field some. Also, are we assuming the API is running on the HPC system or as a hosted service?

The auth, data, jobs, and metadata services seem to be a good starting place. We might also want some information services such as system discovery and monitoring. Given that this is meant to drive web apps…and hopefully future ones, perhaps supporting event and pub/sub services would also be helpful. Lastly, is the api in charge of monitoring itself or are we assuming that's a production detail the centers would implement themselves? One of the things we've done with AGAVE is provide both real time and historical uptime reports for our users. This service is deployed outside the api, and lets us know the ongoing stability of our services and the systems and services we depend on. We find that it also helps build trust with our users. I'm not sure that this service is really in the scope of the API, but it's one of those things that, until we had it, we always missed, but never knew it. What are other people's thoughts on this?

https://foundation.iplantcollaborative.org/monitor/history

--
Rion
@deardooley

On Oct 2, 2012, at 1:24 PM, David Skinner <deskinner@lbl.gov> wrote:

> 
> 
> On Tue, Oct 2, 2012 at 11:11 AM, Annette Greiner <amgreiner@lbl.gov> wrote:
> Hi folks,
> To frame the discussion for the October 11 conference call, I've started thinking about how to go about putting together a first draft of a standard API. It seems to me that it would be logical to simply blend the two APIs we currently have, NEWT and the iPlant API (Agave). There's a lot they have in common, though of course they have different terms for things. I would suggest we choose our terms based on three principles:
> coherence: terms in the API should have grammatical commonality with other terms of similar function in the API
> clarity: terms should be unambiguous
> memorability: terms should be easy to associate mentally with their meaning in the API
> cross-center generalizability: terms should make sense in the context of any HPC center
> 
> 
> Good points. One step toward the last one is to make a fake HPC center stubbed out in the software itself. This serves two purposes. 1) you get to try the software or develop on your laptop without touching the guts of your HPC center. 2) It provides a common meeting ground for all of us as a plain vanilla idealization of an HPC center. To be a little more specific I am suggesting that auth, data, and job functions should have stub implementations that operate locally and while ineffectual they should be processed in a way that mimics a real HPC center. 
> 
> auth: just use an install-time configured password with a test user
> data: just move local files on disk 
> jobs: just run the command (fork/exec). 
> KVP store: use a couch or mongo local instance. 
> 
> Once we have that stub implementation down and packaged people can download and try the API without herculean efforts. 
> 
> We'll also need to discuss the scope of the standard API. How much should it cover? Clearly, centers should be free to do their own implementations; we are just defining a set of REST calls that can be re-used across implementations. But what functions should be left out of the standard? I'm thinking here of functions that are not specific to HPC. One example is the iPlant PostIt, which generates disposable URLs. I think that's a great service to offer people, but I would suggest we leave it out of a standard for HPC, since it isn't a function that arises from the HPC context. The iPlant Apps and Profile features strike me similarly. NEWT has a liststore feature that could also be seen as a non-HPC aspect of that API.
> 
> 
> The guiding model for NEWT thus far has been to stick to the core things you see in HPC center documentation. How do I log in, how do I move files, how do I run things. We don't need to be rigid about that but having a guiding principle with a decent level of simplicity seems prudent. 
>  
> We've also advocated an exception mechanism whereby you can step outside the API and do whatever you like. That provides some demarcation as to where the API stops and where custom machinery begins. 
> 
> -David
> 
> What do other people think? How should we define what is in/out of the spec?
> -Annette
> --
> Annette Greiner
> Outreach, Software, and Programming Group
> NERSC, LBNL
> amgreiner@lbl.gov
> 510-495-2935
> 
> 
> 
> 
> 
>