Genome Collector

Genome Collector (full documentation here) is a Python library to download and manage reference genome data for specific TaxIDs, in particular nucleotide and protein sequences (in fasta/genbank/gff formats), and alignment databases (BLAST, Bowtie1/2).

The data is downloaded automatically on a need-to basis, making it very easy for Python projects to use and re-use reference genomes of E. coli, S. cerevisiae, and so on, without the worry of manually downloading from NCBI.

Examples

Let's get Biopython records of all protein sequences in E. coli:

from genome_collector import GenomeCollection
collection = GenomeCollection()
records = collection.get_taxid_biopython_records(511145, "protein_fasta")

And that's it! If the protein data wasn't already on your machine, Genome Collector downloaded it from NCBI, and stored it in your "collection" for the next time you need it.

Now let's get a path to a local BLAST database for S. cerevisiae:

from genome_collector import GenomeCollection
collection = GenomeCollection()
db_path = collection.get_taxid_blastdb_path(taxid=559292, db_type='nucl')

If there was no S. cerevisiae database on your machine, Genome Collector downloaded the genome data and built the database. It is now in your collection, and the returned path db_path is pointing to it. So you can use it to run a BLAST process:

import subprocess
process = subprocess.run([
    'blastn', '-db', db_path, '-query', 'queries.fa', '-out', 'results.txt'
])

Usage tips

Changing the data storage directory

You can decide where a collection's local files will be stored with the data_dir parameter of GenomeCollection. Note that the default value for data_dir is highly recommended as it always points to the same local user data folder. As a consequence, all librairies and applications using the default will be able to pick genomes from the same folder. The path of this default collection.data_dir is platform-specific:

• ~/.local/share/genome_collector on Linux
• ~/Library/Application Support/genome_collector on MacOS
• C:\Documents and Settings\<User>\Application Data\Local Settings\EGF\genome_collector on Windows

You can set the local default path globally at the beginning of your Python script with:

from genome_collector import GenomeCollection
GenomeCollection.default_dir = '/my/new/dir'

Finally, you can set a default path as an environment variable (so it will be shared by different Python processes):

env GENOME_COLLECTOR_DATA_DIR = /my/other/path

TaxIDs with more than one genome assembly

Even when TaxIDs correspond to a well-identified strain or species, they may have more than one genome assembly available on NCBI. For instance, the Lambda Phage with taxID 10710 has 2 assemblies available.

In this case, most autodownload functions will fail, and you should dowload explicitly the genome assembly informationat the beginning of your script, by providing the an assembly ID in download_taxid_genome_infos_from_ncbi:

collection.download_taxid_genome_infos_from_ncbi(taxid, assembly_id="456094")
# Or to download whichever assembly comes first in the NCBI list:
collection.download_taxid_genome_infos_from_ncbi(taxid, assembly_id="#1")

Preventing auto-download

When using Genome Collector in a particular project, for instance a web app, you may want to pre-download only a few genomes, and prevent users from using other genomes. This can be done by setting a collection's autodownload attribute to False. To globally prevent Genome Collector from downloadind data files, set this attribute at class level:

GenomeCollection.autodownload = False

Command line interface

The very basic command-line interface enables to use Genome Collector to pre-download genomes and pre-build BLAST databases on a machine. This can be particularly useful in Dockerfiles to set up docker containers.

python -m genome_collector genome 511145
python -m genome_collector blast_db 511145 nucl

By default these genomes will be downloaded to the platform-specific local data folder. This can be changed by adding a data_dir at the end:

python -m genome_collector genome 511145 /path/to/some/dir/

Similar projects

ncbi-genome-download and Genomepy are previous projects with a similar goal. In comparison, Genome Collector is more opinionated, it uses TaxID first, and is meant for use in Python scripts/Jupyter notebooks (while the other tools are primarily for command-line usage), with practical features like Biopython records loading and "on-demand" genome downloading and database building.

Installation

You can install genome_collector through PIP

sudo pip install genome_collector

Alternatively, you can unzip the sources in a folder and type

sudo python setup.py install

For the BLAST-related features to work, you must have the NCBI BLAST software installed. For instance on Ubuntu install with:

sudo apt-get install ncbi-blast+

License = MIT

genome_collector is an open-source software originally written at the Edinburgh Genome Foundry by Zulko and released on Github under the MIT licence (copyright Edinburgh Genome Foundry).

Everyone is welcome to contribute !

More biology software

genome_collector is part of the EGF Codons synthetic biology software suite for DNA design, manufacturing and validation.