Re: VCF and RDF, at Clinical Pharmacogenomics TF, Wed Apr 3rd

Thanks Michael

this is helpful, we are thinking about various ways of selecting what gets mapped … 

While it is blindingly obvious that to each person looking at a VCF file "the vast majority are going to turn out to be uninteresting" the difficulty I am having is whether we (as essentially a middleware provider) can make the call as to which are the boring bits, or will it vary wildly from person to person.

I mapped the whole of my initial sample files but then threw away the results.
Motives:
a) get a feel for the size of the problem
b) check that there wasn't some input half way through that I hadn't coded for
c) make sure that I had some understanding of all the data

Jeremy J Carroll
Principal Architect
Syapse, Inc.



On Apr 3, 2013, at 1:20 PM, Michael Miller <Michael.Miller@systemsbiology.org> wrote:

> hi jeremy,
>  
> sorry i missed your talk this morning, it was early on the west coast.
>  
> 'I am assuming that the data lines can largely be addressed by reading the INFO, FORMAT and FILTER definitions at the top of the file, and handling appropriately'
>  
> yes, certainly for the standard annotations but providers aren't bound by what's in the spec for the INFO and the FILTERing can vary, so between two different providers, the info tags provided could be entirely different, which is allowed by the VCF 4.1 spec.  some of those may be equivalent between the two providers so might be mapped to the same RDF, others not and the mapping would be specific to one or the other provider.  i just find that it keeps my code saner by having a subclass per provider.  the base class still takes care of the majority of the parsing and RDF production but the subclasses do provider specific tasks.
>  
> i'm curious why you are interested in mapping all the variants in the VCF file, the vast majority are going to turn out to be uninteresting.  of course the trick is how does one determine this.  we are primarily first filter to protein coding variants (yes ENCODE has 'shown' that 90+% of the genome is coding in some sense but...) which cuts the variants down to a couple hundred thousand from seven and a half million.  one can even then decide to stick with the variants associated with the top hits from whatever uni- or multi-variant analysis is preformed.  those variants can then form a basis to go back to the filtered variants to look for similarity.
>  
> cheers,
> michael
>  
> Michael Miller
> Software Engineer
> Institute for Systems Biology
>  
>  
> From: Jeremy J Carroll [mailto:jjc@syapse.com] 
> Sent: Tuesday, April 02, 2013 2:22 PM
> To: Michael Miller
> Cc: Kingsley Idehen; HCLS hcls; Chris Mungall
> Subject: Re: VCF and RDF, at Clinical Pharmacogenomics TF, Wed Apr 3rd
>  
>  
> Hi Michael
>  
> I am curious … about "use a base
> class to handle most of the parsing then have a derived class per provider
> and try to drive everything through a configuration file"
>  
> I have not yet seen enough variety of VCF files, but so far I am assuming that the data lines can largely be addressed by reading the INFO, FORMAT and FILTER definitions at the top of the file, and handling appropriately.
> There still seems a little way to go before that is fully automatable, because of overly smart definitions such as:
>  
> ##INFO=<ID=SAO,Number=1,Type=Integer,Description="Variant Allele Origin: 0 - unspecified, 1 - Germline, 2 - Somatic, 3 - Both">
>  
> ##INFO=<ID=EA_AC,Number=.,Type=String,Description="European American Allele Count in the order of AltAlleles,RefAllele. For INDELs, A1, A2, or An refers to the N-th alternate allele while R refers to the reference allele.">
>  
> ##INFO=<ID=MAF,Number=.,Type=String,Description="Minor Allele Frequency in percent in the order of EA,AA,All">
> ##INFO=<ID=GTS,Number=.,Type=String,Description="Observed Genotypes. For INDELs, A1, A2, or An refers to the N-th alternate allele while R refers to the reference allele.">
> ##INFO=<ID=EA_GTC,Number=.,Type=String,Description="European American Genotype Counts in the order of listed GTS">
>  
> which introduce non-standard parsing rules. 
>  
> I am hoping that there are relatively few of them, and one can do a reasonable job of recognizing key phases: like the enum in the first seems something which could be automatically detected, and the  AltAlleles,RefAllele idiom seems
>  
> To me it seems that the intent is that the header declarations gives you enough to understand the content, and I imagine that there is in practice sufficiently little variation for that to be automatable ...
>  
>  
> Jeremy J Carroll
> Principal Architect
> Syapse, Inc.
>  
>  
>  
> On Apr 1, 2013, at 4:24 PM, Michael Miller <Michael.Miller@systemsbiology.org> wrote:
> 
> 
> hi jeremy,
> 
> unfortunately, it isn't just the metadata that can be different between
> different producers.
> 
> for a project i'm working on, similar in scope to 1000 genomes, variants
> at the same location for different subjects are collapsed into one VCF
> record and there is a list of possible alleles, one per collapsed variant.
> i'm not working directly with the VCF but there is also an annotation
> field to tell whether the variant is in a coding region, an exon or an
> intron, or outside a coding region, whether the change makes a difference
> in the amino acid, plus that all depends on which isoforms are possible.
> even tho it is reasonably documented, there still are gotcha's.
> 
> the one thing i have found tho, is that one can base one's mapping on the
> source of the document.  two vcf's from the same provider will likely be
> parsable by the same code.  so i usually do what most would do, use a base
> class to handle most of the parsing then have a derived class per provider
> and try to drive everything through a configuration file.
> 
> cheers,
> michael
> 
> Michael Miller
> Software Engineer
> Institute for Systems Biology
> 
> 
> -----Original Message-----
> From: Jeremy J Carroll [mailto:jjc@syapse.com]
> Sent: Monday, April 01, 2013 2:02 PM
> To: Chris Mungall
> Cc: Kingsley Idehen; HCLS hcls
> Subject: Re: VCF and RDF, at Clinical Pharmacogenomics TF, Wed Apr 3rd
> 
> 
> This looks really helpful ..
> 
> I suspect that the key questions remain about how to effectively use the
> data
> 
> FALDO seems at first glance to be fairly simplistic compared with the
> model
> 
> implicit in VCF.
> Maybe FALDO is actually more useful, concentrating on the important
> stuff
> 
> ...
> 
> thanks
> 
> 
> Jeremy J Carroll
> Principal Architect
> Syapse, Inc.
> 
> 
> 
> On Apr 1, 2013, at 12:40 PM, Chris Mungall <cjmungall@lbl.gov> wrote:
> 
> 
> 
> Apologies if this has been covered already I haven't been following
> the
> 
> whole discussion.
> 
> 
> Genome variant data is just a subset of genome data. My understanding
> is
> 
> that the semweb BioHackathon group looked at a variety of different
> kinds
> 
> of genomic data and came up with FALDO[1]. This model looks pretty good
> to me, and importantly there is a converter from GFF3[2,3]. Of all the
> commonly used genome feature formats out there, GFF3 is by far the best
> at
> 
> encouraging provision of relevant metadata using standard
> ontologies/terminologies.
> 
> 
> VCF is convertible to GVF[4,5] which is a subset of GFF3 with
> additional
> 
> recommended metadata. It's supported by Ensembl, gbGap and others, and
> the 1000genomes data is available in GVF[6].
> 
> 
> As GFF3 is convertible to RDF/OWL that uses FALDO and SO, it follows
> that
> 
> GVF is too (though the converter may need tweaking to take advantage of
> the additional GVF metadata).
> 
> 
> I just wanted to make sure you were aware of all this previous work
> before reinventing anything.
> 
> 
> [1] https://github.com/JervenBolleman/FALDO
> [2] http://www.sequenceontology.org/gff3.shtml
> [3] https://code.google.com/p/gff3-to-owl/
> [4] http://www.ncbi.nlm.nih.gov/pubmed/20796305 - A standard variation
> file format for human genome sequences - Reese at al
> 
> [5] http://www.sequenceontology.org/resources/gvf.html
> [6] ftp://ftp.ensembl.org/pub/current_variation/gvf/homo_sapiens/
> 
> On Apr 1, 2013, at 10:59 AM, Jeremy J Carroll wrote:
> 
> 
> Hi Kingsley,
> 
> I wasn't going to but since you ask:
> 
> http://www.slideshare.net/JeremyJCarroll/vcf-and-rdf
> 
> or
> 
> http://lists.w3.org/Archives/Public/www-archive/2013Apr/att-
> 0002/W3C-JJC-LifeSci.pdf
> 
> 
> 
> Jeremy J Carroll
> Principal Architect
> Syapse, Inc.
> 
> 
> 
> On Apr 1, 2013, at 10:13 AM, Kingsley Idehen
> <kidehen@openlinksw.com> wrote:
> 
> 
> 
> On 4/1/13 1:05 PM, Jeremy J Carroll wrote:
> 
> Hi
> 
> I am hoping to present the work I am currently doing on VCF and RDF
> at
> 
> the Clinical Pharamcogenomics TF telecom on Wednesday.
> 
> 
> My presentation should cover:
> 
> - business background, Syapse Discovery
> - some background on VCF as a knowledge representation format
> - and some initial results on mapping 1000 genomes into RDF
> 
> I will circulate slides shortly
> 
> 
> Jeremy J Carroll
> Principal Architect
> Syapse, Inc.
> 
> 
> 
> 
> 
> 
> 
> Hopefully you'll publish to Slideshare?
> 
> --
> 
> Regards,
> 
> Kingsley Idehen
> Founder & CEO
> OpenLink Software
> Company Web: http://www.openlinksw.com
> Personal Weblog: http://www.openlinksw.com/blog/~kidehen
> Twitter/Identi.ca handle: @kidehen
> Google+ Profile:
> https://plus.google.com/112399767740508618350/about
> 
> LinkedIn Profile: http://www.linkedin.com/in/kidehen
>