VarSome API Python client rewrite #7

ckopanos · 2018-01-12T09:39:40Z

Includes new vcf annotator object
Several code fixes
Unit tests
updated documentation

…pper classes for json to python objects to improve access of annotation results

…r forgotten static method

…y are installed under the venv bin directory so that their always within the current PATH

…name to varsome-api-client

vtsiolkas · 2018-01-12T09:52:47Z

README.md


-Without an API key you will not be able to perform batch requests as well.
+The script should complete without errors and display aprox 6,700 lines of data from dann, dbnsfp, ensemble_transcripts, gerp, gnomad_exomes, gnomad_exomes_coverage, icgc_somatic, ncbi_clinvar2, pub_med_articles, refseq_transcripts, sanger_cosmic_public, uniprot_variants, wustl_civic etc.


It's ensembl_transcripts

vtsiolkas · 2018-01-12T10:44:49Z

scripts/varsome_api_run.py

+
+
+def annotate_variant(argv):
+    parser = argparse.ArgumentParser(description='Sample Variant API calls')


description='Sample Varsome API calls'

vtsiolkas · 2018-01-12T12:06:49Z

varsome_api/models/elements/ncbi.py

+    rsid = fields.ListField(items_types=(int,), help_text="RS ID")
+
+
+class ClinVarDisease(models.Base):


ClinVar and ClinVarDisease are not needed anymore, we only have them for old annotations for the portal, right?

Yes correct.

vtsiolkas · 2018-01-12T12:07:52Z

varsome_api/models/variant.py

+    sanger_cosmic = fields.ListField(required=False, items_types=(Cosmic,), help_text="Sanger Cosmic")
+    sanger_cosmic_public = fields.ListField(required=False, items_types=(CosmicPublic,), help_text="Cosmic")
+    sanger_cosmic_licensed = fields.ListField(required=False, items_types=(CosmicLicensed,), help_text="Cosmic")
+    ncbi_clinvar = fields.ListField(required=False, items_types=(ClinVar,), help_text="ClinVar")


ncbi_clinvar is not in the API anymore

Well the schema still returns them, and the available get parameters too. I will remove from the json models. Maybe we should also patch the api and remove the parameters, but we should keep the serialzers for the portal..

Ok done, removed clinvar (1). I will prepare a patch for the api as well

Made some changes to the text of the rEADME file

chapplec · 2018-01-12T10:01:03Z

README.md

+
+    varsome_api_annotate_vcf.py -g hg19 -k api_key -i input.vcf -o annotated_vcf.vcf -p add-all-data=1
+
+Notice however that not all available annotations will be present in the annotated_vcf file. Only a subset


This is confusing. Why won't all annotations be present? Which ones will be there and which ones will be missing? How does the script "decide"? Is it random? Is it a specific set of annotations, the same each time?

chapplec · 2018-01-12T10:02:24Z

README.md

        # proceed with your code flow e.g.
        print(e) # 404 (invalid reference genome)

-### Example Usage
+To view available request parameters (used in the params method parameter) refer to an example at [api.varsome.com](https://api.varsome.com)


If "params" is code, it should be params (`params`). I can't tell here if the method os parameter or params.

chapplec · 2018-01-12T10:04:09Z

README.md

+    api = VarSomeAPIClient(api_key)
+    # fetch information about a variant into a dictionary
+    result = api.lookup('chr7-140453136-A-T', params={'add-source-databases': 'gnomad-exomes,refseq-transcripts'}, ref_genome='hg19')
+    annotated_variant = AnnotatedVariant(**result)


The ** here make al subsequent lines bold. It looks like the code is not actually in a code block, so it is parsed as markdown.

chapplec · 2018-01-12T10:05:33Z

scripts/varsome_api_run.py

@@ -0,0 +1,112 @@
+#!/usr/bin/env python


Shouldn't this (and all other shebang lines) be #!/usr/bin/env python3? env python will likely still resolve to python2 on many systems.

Well if you are on a system where python2 is still the default. which is none except Macs, and dont create a python3 virtual env then yes it will fail.
So leave it on pyton3. The setup.py script has a requirement for python3 but I guess its ok to change that to python3

chapplec · 2018-01-12T10:31:29Z

README.md


-for a list of available options
+Please visit the [api documentation](https://api.varsome.com) to find out how to use the api and
+what values does the api provide as a response to lookup requests


Change

what values does the api provide as a response to lookup requests

to

what values the api provides as a response to lookup requests

added syntax highlighting to readme examples

RikMaxSpeed · 2018-01-12T15:43:27Z

README.md

+This client is still in beta but it is a good for playing around with the API.
+
+### Python versions
+Python version 3 is supported.


Requires at least Python 3.5, you can download the latest version from www.python.org

ie: Python 3.4 doesn't work.

RikMaxSpeed · 2018-01-12T15:44:32Z

README.md

-within your code, or do
+There are several ways to create a virtual environment, but you can refer to [pip installation](https://pip.pypa.io/en/stable/installing/) and
+[virtualenv installation](https://virtualenv.pypa.io/en/stable/installation/) to first install these 2 tools if you don't
+have them already installed via a package manager (Linux) or HomeBrew (MacOS), etc.


Remember to use "sudo -H" when installing on Mac.

RikMaxSpeed · 2018-01-12T15:48:33Z

README.md

+    varsome_api_run.py -g hg19 -k api_key -i variants.txt -o annotations.txt -p add-all-data=1
+
+The command above will read variants from `variants.txt` and dump the annotations to `annotations.txt`. 
+If you don't use the `-k` parameter, the script will do as many requests as there are variants in `variants.txt`,


For any substantial number of variants you will need to register for an API key. You can try the software without the -k apiKey parameter but you will quickly bump into safeguard limits...

or something like that.

RikMaxSpeed · 2018-01-12T15:49:03Z

README.md

+    varsome_api_annotate_vcf.py -g hg19 -k api_key -i input.vcf -o annotated_vcf.vcf -p add-all-data=1
+
+Notice, however, that not all available annotations will be present in the `annotated_vcf.vcf` file. Only a subset
+of the returned annotations will be available when running this script. See the "Using the client in your code" 


What's missing? Why?

I have added only a subset of annotations in the output.vcf the possible number of annotations are too many to add them all and header info in the vcf file for all of them it might take several more days with tests and changes etc.. Besides you dont know what a user might need from all of these annotations, this is why the user is referred to the section using the client,where he is instructed how to override the VCFAnnotator class with the annotations he wants.
After all the intended use is that the end user will get the python class objects to develop an app of his own not use the run.py and annotate_vcf scripts. Even if he wants to do that he can copy the code and use his overriden vcf annotator class

RikMaxSpeed · 2018-01-12T15:50:33Z

README.md

+
+You will also not be able to perform batch requests without an API key.
+
+To obtain an API key please [contact us](mailto:support@saphetor.com)


Move this to the top of this section.

RikMaxSpeed · 2018-01-12T15:51:45Z

README.md

+
+### How to get an API key
+
+You are generally not required to have an API key to use the API but, without one, the number of requests 


Change this to: "You can use the API without an API key, but performance and the number of queries will be limited in order to safeguard our platform's reliability."

RikMaxSpeed · 2018-01-12T15:52:46Z

setup.py

        'Operating System :: OS Independent',
        'License :: OSI Approved :: Apache License',
-        'Programming Language :: Python :: 2.7',
        'Programming Language :: Python :: 3',
        'Programming Language :: Python :: 3.2',
        'Programming Language :: Python :: 3.3',
        'Programming Language :: Python :: 3.4',


You probably need to remove up to 3.4 included.

RikMaxSpeed · 2018-01-12T15:57:06Z

scripts/varsome_api_annotate_vcf.py

@@ -0,0 +1,41 @@
+#!/usr/bin/env python


Many (all?) of these files are missing (c) 2018 Saphetor at the top.
We need to decide which license we are using for these.
And then include the appropriate header in all files.
Preferably something that means that:
1- We own the original code.
2- If somebody modifies the code they still have to include the copyright message.
3- Would be nice if they send bug-fixes back to us.
4- They can sell the code to 3rd party uses, but they must credit Saphetor.

Not sure if we need all this info in all files. The repository contains a license file which can cover all the code

e.g. https://github.com/elastic/elasticsearch-py/blob/master/LICENSE

…nings in requests as possible

ckopanos and others added 15 commits November 29, 2017 19:17

API-106 #comment Initial implementation of a vcf annotator class. Wra…

5f45117

…pper classes for json to python objects to improve access of annotation results

API-106 Changed file permissions + tweaked the README file.

da01bec

Refactoring Work in progress. closes #3 closes #4

f944cec

Refactoring Work in progress. closes #3 closes #4

075aab6

Modified run.py to accept input and output files. Fix in client.py fo…

d83f6a6

…r forgotten static method

Transformed setup.py so that scripts former run.py and annotate_vcf.p…

7f7ae69

…y are installed under the venv bin directory so that their always within the current PATH

Transformed setup.py so that scripts former run.py and annotate_vcf.p…

eaf1e36

…y are installed under the venv bin directory so that their always within the current PATH

Minor fixes in command line tools

13e4b08

Updated documentation

4a4f4f7

Updated documentation

6cfb6d8

Added unit tests

9bb0c88

Updated readme.md

b956aa2

API-106 #comment finalized client. Writter unittests, updated doc, re…

b5fb6ef

…name to varsome-api-client

Updated readme

2bd13f3

Updated readme

4751bc4

ckopanos assigned vtsiolkas Jan 12, 2018

ckopanos requested review from vtsiolkas, RikMaxSpeed and chapplec January 12, 2018 09:39

vtsiolkas reviewed Jan 12, 2018

View reviewed changes

ckopanos added 2 commits January 12, 2018 11:54

Fixed typo in readme

e68df2f

Updated setup.py

d1a35d7

vtsiolkas reviewed Jan 12, 2018

View reviewed changes

ckopanos added 2 commits January 12, 2018 14:15

changed description in varsome_api_run

fc2c091

Removed clivar v1 from the client

793634e

vtsiolkas approved these changes Jan 12, 2018

View reviewed changes

chapplec and others added 2 commits January 12, 2018 13:42

Made some changes to the text of the rEADME file

7272f10

Merge pull request #9 from saphetor/minor-changes-to-readme

8e00d86

Made some changes to the text of the rEADME file

chapplec approved these changes Jan 12, 2018

View reviewed changes

chapplec and others added 4 commits January 12, 2018 15:40

added syntax highlighting to readme examples

18649af

changed shebang line to point to python3

f8c99ce

updated python version requirement

f7fe98a

Merge pull request #10 from saphetor/minor-changes-to-readme

fc0f676

added syntax highlighting to readme examples

RikMaxSpeed requested changes Jan 12, 2018

View reviewed changes

ckopanos and others added 8 commits January 12, 2018 18:39

Changed clien to work with python v 3.3 and up

210761c

Changed clien to work with python v 3.3 and up

a7b3820

Fixes for python 3.3, 3.4 compatibility. Supress as many annoying war…

2caef25

…nings in requests as possible

Changed default logging handler to StreamHandler

c9f06e5

Changes in readme

650e4cf

Changes in readme

79e23b7

Update LICENSE

9474093

Added license header in files

46a010c

RikMaxSpeed changed the title ~~Rewritted python client~~ VarSome API Python client rewrite Jan 15, 2018

Fixed copyright message

fee15da

RikMaxSpeed approved these changes Jan 15, 2018

View reviewed changes

RikMaxSpeed merged commit ff57060 into master Jan 15, 2018

ckopanos deleted the vcf-annotator branch January 17, 2018 11:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VarSome API Python client rewrite #7

VarSome API Python client rewrite #7

ckopanos commented Jan 12, 2018

vtsiolkas Jan 12, 2018

ckopanos Jan 12, 2018

vtsiolkas Jan 12, 2018

ckopanos Jan 12, 2018

vtsiolkas Jan 12, 2018

ckopanos Jan 12, 2018

vtsiolkas Jan 12, 2018

ckopanos Jan 12, 2018

ckopanos Jan 12, 2018

chapplec Jan 12, 2018

chapplec Jan 12, 2018

chapplec Jan 12, 2018

chapplec Jan 12, 2018

ckopanos Jan 12, 2018

chapplec Jan 12, 2018

RikMaxSpeed Jan 12, 2018

RikMaxSpeed Jan 12, 2018

RikMaxSpeed Jan 12, 2018

RikMaxSpeed Jan 12, 2018

RikMaxSpeed Jan 12, 2018

ckopanos Jan 12, 2018

RikMaxSpeed Jan 12, 2018

RikMaxSpeed Jan 12, 2018

RikMaxSpeed Jan 12, 2018

RikMaxSpeed Jan 12, 2018

ckopanos Jan 14, 2018

ckopanos Jan 14, 2018


		Without an API key you will not be able to perform batch requests as well.
		The script should complete without errors and display aprox 6,700 lines of data from dann, dbnsfp, ensemble_transcripts, gerp, gnomad_exomes, gnomad_exomes_coverage, icgc_somatic, ncbi_clinvar2, pub_med_articles, refseq_transcripts, sanger_cosmic_public, uniprot_variants, wustl_civic etc.



		def annotate_variant(argv):
		parser = argparse.ArgumentParser(description='Sample Variant API calls')

		rsid = fields.ListField(items_types=(int,), help_text="RS ID")


		class ClinVarDisease(models.Base):


		varsome_api_annotate_vcf.py -g hg19 -k api_key -i input.vcf -o annotated_vcf.vcf -p add-all-data=1

		Notice however that not all available annotations will be present in the annotated_vcf file. Only a subset


		You will also not be able to perform batch requests without an API key.

		To obtain an API key please [contact us](mailto:support@saphetor.com)


		### How to get an API key

		You are generally not required to have an API key to use the API but, without one, the number of requests

VarSome API Python client rewrite #7

VarSome API Python client rewrite #7

Conversation

ckopanos commented Jan 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment