Hostile accurately removes host sequences from short and long read (meta)genomes, consuming single-read or paired fastq[.gz] input. Batteries are included – a human reference genome is downloaded when run for the first time. Hostile is precise by default, removing an order of magnitude fewer microbial reads than existing approaches while removing >99.5% of real human reads from 1000 Genomes Project samples. For the best possible retention of microbial reads, use an existing index masked against bacterial and/or viral genomes, or make your own using the built-in masking utility. Read headers can be replaced with integers (using --rename) for privacy and smaller FASTQs. Heavy lifting is done with fast existing tools (Minimap2/Bowtie2 and Samtools). Bowtie2 is the default aligner for short (paired) reads while Minimap2 is default aligner for long reads. In benchmarks, bacterial Illumina reads were decontaminated at 32Mbp/s (210k reads/sec) and bacterial ONT reads at 22Mbp/s, using 8 alignment threads. By default, Hostile requires 4GB of RAM for decontaminating short reads and 13GB for long reads (Minimap2). Further information and benchmarks can be found in the paper and blog post. Please open an issue to report problems or otherwise reach out for help and advice.
The default index human-t2t-hla comprises T2T-CHM13v2.0 and IPD-IMGT/HLA v3.51, and is downloaded automatically when running Hostile unless another index is specified. Slightly higher microbial sequence retention is may be possible using masked indexes, listed below. The index human-t2t-hla-argos985 is masked against 985 reference grade bacterial genomes including common human pathogens, while human-t2t-hla.argos-bacteria-985_rs-viral-202401_ml-phage-202401 is further masked comoprehensively against all known virus and phage genomes. The latter should be used when retention of viral sequences is a priority. To use a standard index, simply pass its name as the value of the --index argument which takes care of downloading and caching the relevant index. Automatic download can be disabled using the --offline flag, and --index can accept a path to a custom reference genome or Bowtie2 index. Object storage is provided by the ModMedMicro research unit at the University of Oxford.
| Name | Composition | Date | Masked positions |
|---|---|---|---|
human-t2t-hla (default) |
T2T-CHM13v2.0 + IPD-IMGT/HLA v3.51 | 2023-07 | 0 (0%) |
human-t2t-hla-argos985 |
human-t2t-hla masked with 150mers for 985 FDA-ARGOS bacterial genomes |
2023-07 | 317,973 (0.010%) |
human-t2t-hla.rs-viral-202401_ml-phage-202401 |
human-t2t-hla masked with 150mers for 18,719 RefSeq viral and 26,928 Millard Lab phage genomes |
2024-01 | 1,172,993 (0.037%) |
human-t2t-hla.argos-bacteria-985_rs-viral-202401_ml-phage-202401 |
human-t2t-hla masked with 150mers for 985 FDA-ARGOS bacterial, 18,719 RefSeq viral, and 26,928 Millard Lab phage genomes |
2024-01 | 1,473,260 (0.046%) |
human-t2t-hla-argos985-mycob140 |
human-t2t-hla masked with 150mers for 985 FDA-ARGOS bacterial & 140 mycobacterial genomes |
2023-07 | 319,752 (0.010%) |
mouse-mm39 |
GRCm39 (GCF_000001635.27) |
2024-11-29 | 0 (0%) |
Performance of human-t2t-hla and human-t2t-hla-argos985-mycob140 was evaluated in the paper
Installation with conda/mamba or Docker is recommended due to non-Python dependencies (Bowtie2, Minimap2, Samtools and Bedtools). Hostile is tested with Ubuntu Linux 22.04, MacOS 12, and under WSL for Windows.
Conda/mamba
conda create -y -n hostile -c conda-forge -c bioconda hostile
conda activate hostileDocker
git clone https://github.com/bede/hostile.git
cd hostile
docker build . --platform linux/amd64A Biocontainer image is also available, but beware that this often lags behind the latest released version
Hostile automatically downloads and caches the default index human-t2t-hla when run for the first time, meaning that there is no need to download an index in advance. Neverthless:
-
To download and cache the default index (
human-t2t-hla), runhostile fetch -
To list available indexes, run
hostile fetch --list -
To download and cache another standard index, run e.g.
hostile fetch --name human-t2t-hla-argos985. This will download and cache both short read (Bowtie2) and long read (Minimap2) indexes, unless restricted to one or the other using e.g.--aligner minimap2. -
To use a custom genome (made with
hostile maskor otherwise), runhostile cleanwith--index path/to/genome.fa(for minimap2) or--index path/to/bowtie2-index-name(for Bowtie2). Note that Minimap2 mode accepts a path to a genome in fasta format, whereas Bowtie2 mode accepts a path to a precomputed index, minus the.x.bt2suffix. A Bowtie2 index can be built for use with Hostile using e.g.bowtie2-build genome.fa index-name. -
To change where indexes are stored, set the environment variable
HOSTILE_CACHE_DIRto a directory of your choice. Runhostile fetch --listto verify.
$ hostile clean -h
usage: hostile clean [-h] --fastq1 FASTQ1 [--fastq2 FASTQ2] [--aligner {bowtie2,minimap2,auto}] [--index INDEX] [--invert] [--rename] [--reorder] [--out-dir OUT_DIR]
[--threads THREADS] [--aligner-args ALIGNER_ARGS] [--force] [--offline] [--debug]
Remove reads aligning to an index from fastq[.gz] input files.
options:
-h, --help show this help message and exit
--fastq1 FASTQ1 path to forward fastq[.gz] file
--fastq2 FASTQ2 optional path to reverse fastq[.gz] file (short reads only)
(default: None)
--aligner {bowtie2,minimap2,auto}
alignment algorithm. Defaults to minimap2 (long read) given fastq1 only or bowtie2 (short read)
given fastq1 and fastq2. Override with bowtie2 if cleaning single/unpaired short reads
(default: auto)
--index INDEX name of standard index or path to custom genome (Minimap2) or Bowtie2 index
(default: human-t2t-hla)
--invert keep only reads aligning to the target genome (and their mates if applicable)
(default: False)
--rename replace read names with incrementing integers
(default: False)
--reorder ensure deterministic output order
(default: False)
--out-dir OUT_DIR path to output directory
(default: ./)
--threads THREADS number of alignment threads. A sensible default is chosen automatically
(default: 5)
--aligner-args ALIGNER_ARGS
additional arguments for alignment
(default: )
--force overwrite existing output files
(default: False)
--offline disable automatic index download
(default: False)
--debug show debug messages
(default: False)Short paired reads, default index
$ hostile clean --fastq1 human_1_1.fastq.gz --fastq2 human_1_2.fastq.gz
INFO: Hostile version 1.0.0. Mode: paired short read (Bowtie2)
INFO: Found cached standard index human-t2t-hla
INFO: Cleaning…
INFO: Cleaning complete
[
{
"version": "1.0.0",
"aligner": "bowtie2",
"index": "human-t2t-hla",
"options": [],
"fastq1_in_name": "human_1_1.fastq.gz",
"fastq1_in_path": "/Users/bede/human_1_1.fastq.gz",
"fastq1_out_name": "human_1_1.clean_1.fastq.gz",
"fastq1_out_path": "/Users/bede/human_1_1.clean_1.fastq.gz",
"reads_in": 2,
"reads_out": 0,
"reads_removed": 2,
"reads_removed_proportion": 1.0,
"fastq2_in_name": "human_1_2.fastq.gz",
"fastq2_in_path": "/Users/bede/human_1_2.fastq.gz",
"fastq2_out_name": "human_1_2.clean_2.fastq.gz",
"fastq2_out_path": "/Users/bede/human_1_2.clean_2.fastq.gz"
}
]Short paired reads, masked index, save log
$ hostile clean --fastq1 human_1_1.fastq.gz --fastq2 human_1_2.fastq.gz --index human-t2t-hla-argos985 > log.json
INFO: Hostile version 1.0.0. Mode: paired short read (Bowtie2)
INFO: Found cached standard index human-t2t-hla
INFO: Cleaning…
INFO: Cleaning completeShort single/unpaired reads, save log
By default, single/unpaired fastqs are assumed to be long reads. Override this by specifying --aligner bowtie2 when decontaminating unpaired short reads.
$ hostile clean --aligner bowtie2 --fastq1 tests/data/human_1_1.fastq.gz > log.json
INFO: Hostile version 1.0.0. Mode: short read (Bowtie2)
INFO: Found cached standard index human-t2t-hla
INFO: Cleaning…
INFO: Cleaning completeLong reads
$ hostile clean --fastq1 tests/data/tuberculosis_1_1.fastq.gz
INFO: Hostile version 1.0.0. Mode: long read (Minimap2)
INFO: Found cached standard index human-t2t-hla
INFO: Cleaning…
INFO: Cleaning complete
[
{
"version": "1.0.0",
"aligner": "minimap2",
"index": "human-t2t-hla",
"options": [],
"fastq1_in_name": "tuberculosis_1_1.fastq.gz",
"fastq1_in_path": "/Users/bede/Research/git/hostile/tests/data/tuberculosis_1_1.fastq.gz",
"fastq1_out_name": "tuberculosis_1_1.clean.fastq.gz",
"fastq1_out_path": "/Users/bede/Research/git/hostile/tuberculosis_1_1.clean.fastq.gz",
"reads_in": 1,
"reads_out": 1,
"reads_removed": 0,
"reads_removed_proportion": 0.0
}
]
from pathlib import Path
from hostile.lib import clean_fastqs, clean_paired_fastqs
# Long reads, defaults
clean_fastqs(
fastqs=[Path("reads.fastq.gz")],
)
# Paired short reads, various options, capture log
log = clean_paired_fastqs(
fastqs=[(Path("reads_1.fastq.gz"), Path("reads_2.fastq.gz"))],
index="human-t2t-hla-argos985",
out_dir=Path("decontaminated-reads"),
rename=True,
force=True,
threads=4
)
print(log)The mask subcommand makes it easy to create custom-masked indexes in order to achieve maximum retention of specific target organisms:
hostile mask human.fasta lots-of-bacterial-genomes.fasta --threads 8You may wish to use one of the existing reference genomes as a starting point. Masking uses Minimap2 to align 150mers of the supplied target genomes with the reference genome, and bedtools to mask all aligned regions with N. Both a masked genome (for Minimap2) and a masked Bowtie2 index is created.
- Hostile prioritises retaining microbial sequences above discarding host sequences. If you strive to remove every last human sequence, other approaches may serve you better (blog post).
- Performance is not always improved by using all available CPU cores. A sensible default is therefore chosen automatically at runtime based on the number of available CPU cores.
- Minimap2 has an overhead of 30-90s for human genome indexing prior to starting decontamination, which frustrate users processing large numbers of small samples. Support for automatic Minimap2 index caching is planned for a future release.
Please cite Hostile if you find it useful.
Bede Constantinides, Martin Hunt, Derrick W Crook, Hostile: accurate decontamination of microbial host sequences, Bioinformatics, 2023; btad728, https://doi.org/10.1093/bioinformatics/btad728
@article{10.1093/bioinformatics/btad728,
author = {Constantinides, Bede and Hunt, Martin and Crook, Derrick W},
title = {Hostile: accurate decontamination of microbial host sequences},
journal = {Bioinformatics},
volume = {39},
number = {12},
pages = {btad728},
year = {2023},
month = {12},
issn = {1367-4811},
doi = {10.1093/bioinformatics/btad728},
url = {https://doi.org/10.1093/bioinformatics/btad728},
eprint = {https://academic.oup.com/bioinformatics/article-pdf/39/12/btad728/54850422/btad728.pdf},
}git clone https://github.com/bede/hostile.git
cd hostile
conda env create -y -f environment.yml
conda activate hostile
pip install --editable '.[dev]'
pytest
pre-commit install