anvi-dereplicate-genomes

Authors
Requires
Can use
Provides
Usage

Input Options
Output Format
Required Parameters and Example Runs
Other parameters

Command line interface

Command line usage
Parameters and their descriptions

Additional Resources

Identify redundant (highly similar) genomes.

🔙 To the main page of anvi’o programs and artifacts.

Authors

Evan Kiefl

Mahmoud Yousef

Requires

external-genomes fasta

Can use

internal-genomes genome-similarity

Provides

fasta

Usage

This program uses the user’s similarity metric of choice to identify genomes that are highly similar to each other, and groups them together into redundant clusters. The program finds representative sequences for each cluster and outputs them into fasta files.

Input Options

You have two options for the input to this program:

the results of anvi-compute-genome-similarity (a genome-similarity directory). If you used fastANI or pyANI when you ran anvi-compute-genome-similarity, provide this using the parameter --ani-dir; if you used sourmash, use the parameter --mash-dir.
an internal-genomes, external-genomes or a series of fasta files (each of which represents a genome), in which case anvi’o will run anvi-compute-genome-similarity for you. When providing these inputs, you can also provide any of the parameters that anvi-compute-genome-similarity can take, including the --program you want to use (out of PyANI, fastANI, sourmash) and their parameters. Details about all of this can be found in the help menu for anvi-compute-genome-similarity.

Output Format

By default, the output of this program is a directory containing two descriptive text files (the cluster report and fasta report) and a subdirectory called GENOMES:

-The cluster report describes is a tab-delimited text file where each row describes a cluster. This file contains four columns: the cluster name, the number of genomes in the cluster, the representative genome of the cluster, and a list of the genomes that are in the cluster. Here is an example describing 11 genomes in three clusters:

cluster	size	representative	genomes
cluster_000001	1	G11_IGD_MAG_00001	G11_IGD_MAG_00001
cluster_000002	8	G11_IGD_MAG_00012	G08_IGD_MAG_00008,G33_IGD_MAG_00011,G01_IGD_MAG_00013,G06_IGD_MAG_00023,G03_IGD_MAG_00021,G05_IGD_MAG_00014,G11_IGD_MAG_00012,G10_IGD_MAG_00010
cluster_000003	2	G03_IGD_MAG_00011	G11_IGD_MAG_00013,G03_IGD_MAG_00011

-The subdirectory GENOMES contains fasta files describing the representative genome from each cluster. For example, if your original set of genomes had two identical genomes, this program would cluster them together, and the GENOMES folder would only include one of their sequences.

-The fasta report describes the fasta files contained in the subdirectory GENOMES. By default, this describes the representative sequence of each of the final clusters. It tells you the genome name, its source, its cluster (and the representative sequence of that cluster), and the path to its fasta file in GENOMES. So, for the example above, the fasta report would look like this:

name	source	cluster	cluster_rep	path
G11_IGD_MAG_00001	fasta	cluster_000001	G11_IGD_MAG_00001	GENOMES/G11_IGD_MAG_00001.fa
G11_IGD_MAG_00012	fasta	cluster_000002	G11_IGD_MAG_00012	GENOMES/G11_IGD_MAG_00012.fa
G03_IGD_MAG_00011	fasta	cluster_000003	G03_IGD_MAG_00011	GENOMES/G03_IGD_MAG_00011.fa

You can also choose to report all genome fasta files (including redundant genomes) (with --report-all) or report no fasta files (with --skip-fasta-report). This would change the fasta files included in GENOMES and the genomes mentioned in the fasta report. The cluster report would be identical.

Required Parameters and Example Runs

You are required to set the threshold for two genomes to be considered redundant and put in the same cluster.

For example, if you had the results from an anvi-compute-genome-similarity run where you had used pyANI and wanted the threshold to be 90 percent, you would run:

anvi-dereplicate-genomes --ani-dir genome-similarity \ -o path/to/output \ --program pyANI \ --similiarity-threshold 0.90

If instead you hadn’t yet run anvi-compute-genome-similarity and instead wanted to cluster the genomes in your external-genomes file with similarity 85 percent or more (no fasta files necessary) using sourmash, you could run:

anvi-dereplicate-genomes -e external-genomes \ --skip-fasta-report \ --program sourmash \ -o path/to/output \ --similiarity-threshold 0.85

Other parameters

You can change how anvi’o picks the representative sequence from each cluster with the parameter --representative-method. For this you have three options:

Qscore: picks the genome with highest completion and lowest redundancy
length: picks the longest genome in the cluster
centrality (default): picks the genome with highest average similiarty to every other genome in the cluster

You can also choose to skip checking genome hashes (which will warn you if you have identical sequences in separate genomes with different names), provide a log path for debug messages or use multithreading (relevant only if not providing --ani-dir or --mash-dir).

Edit this file to update this information.

Command line interface

The following is the list of parameters and their descriptions in the latest version of anvi-dereplicate-genomes.

The code that generates this web page ensures that this information is the latest available in 9-dev (eunice). Which means, in some cases the information in this section (generated automatically) might be more comprehensive or up-to-date compared to the descriptions above (written by humans).

Command line usage

anvi-dereplicate-genomes [-h] [-i FILE_PATH] [-e FILE_PATH] [-f FASTA_TEXT_FILE] [--ani-dir PATH] [--mash-dir PATH] -o DIR_PATH [--skip-fasta-report] [--report-all] [--program {pyANI,fastANI,sourmash}] [--fastani-kmer-size FASTANI_KMER_SIZE] [--fragment-length FRAGMENT_LENGTH] [--min-fraction MIN_FRACTION] [--method {ANIm,ANIb,ANIblastall,TETRA}] [--min-alignment-fraction NUM] [--significant-alignment-length INT] [--use-full-percent-identity] [--min-full-percent-identity FULL_PERCENT_IDENTITY] [--kmer-size INT] [--scale INT] --similarity-threshold SIMILARITY_THRESHOLD [--cluster-method {simple_greedy}] [--representative-method {Qscore,length,centrality}] [-T NUM_THREADS] [--just-do-it] [--skip-checking-genome-hashes] [--log-file FILE_PATH]

Parameters and their descriptions

INPUT OPTIONS Tell anvi'o what you want.

-i, --internal-genomes FILE_PATH: A five-column TAB-delimited flat text file. The header line must contain these columns: 'name', 'bin_id', 'collection_id', 'profile_db_path', 'contigs_db_path'. Each line should list a single entry, where 'name' can be any name to describe the anvi'o bin identified as 'bin_id' that is stored in a collection.
-e, --external-genomes FILE_PATH: A two-column TAB-delimited flat text file that lists anvi'o contigs databases. The first item in the header line should read 'name', and the second should read 'contigs_db_path'. Each line in the file should describe a single entry, where the first column is the name of the genome (or MAG), and the second column is the anvi'o contigs database generated for this genome.
-f, --fasta-text-file FASTA_TEXT_FILE: A two-column TAB-delimited file that lists multiple FASTA files to import for analysis. If using for `anvi-dereplicate-genomes` or `anvi-compute-distance`, each FASTA is assumed to be a genome. The first item in the header line should read 'name', and the second item should read 'path'. Each line in the field should describe a single entry, where the first column is the name of the FASTA file or corresponding sequence, and the second column is the path to the FASTA file itself.

IMPORT RESULTS Alternatively, if you have previous ANI or mash similarity computations on your genomes, you can import the result directory here to use. Please note that file names must remain unchanged for anvi'o to find them

--ani-dir PATH: You can import the directory created by `anvi-compute-genome-similarity` if `--program` parameter was set to `fastANI` or `pyANI` and use it for dereplication
--mash-dir PATH: You can import the directory created by `anvi-compute-genome-similarity` if `--program` parameter was set to `sourmash` and use it for dereplication

OUTPUT OPTIONS Tell anvi'o where to store your results.

-o, --output-dir DIR_PATH required: Directory path for output files
--skip-fasta-report: By default, if any sequence source is provided, FASTA files of non-redundant genomes are reported. With this flag, no FASTA files are reported.
--report-all: By default, only FASTA files of non-redundant genomes are reported, i.e. single representatives from each cluster. With this flag, all genome FASTAS will be reported.

Program Tell anvi'o which similarity program to run.

--program PROGRAM: Tell anvi'o which program to run to process genome similarity. For ANI, you can either use pyANI or fastANI. If accuracy is paramount (for example, distinguishing things less than 1 percent different), or for dealing with genomes < 80 percent similar, pyANI is what we recommend. However, fastANI is much faster. If you for some reason want to use mash similarity, you can use sourmash, but its really not intended for genome comparisons. Choices: pyANI fastANI sourmash

fastANI Settings Tell anvi'o to tell fastANI what settings to set. Only if `--program` is set to `fastANI`

--fastani-kmer-size FASTANI_KMER_SIZE: Choose a kmer. The default is 16. Default: 16
--fragment-length FRAGMENT_LENGTH: Choose a fragment length. The default is 3000. Default: 3000
--min-fraction MIN_FRACTION: Minimum fraction of alignment to be shared between genome pairs to calculate ANI. If reference and query genome size differ, smaller one among the two is considered. The default is 0.25. Default: 0.25

pyANI Settings Tell anvi'o to tell pyANI what method you wish to use and what settings to set. Only if `--program` is set to `pyANI`

--method METHOD: Method for pyANI. The default is ANIb. You must have the necessary binary in path for whichever method you choose. According to the pyANI help for v0.2.7 at https://github.com/widdowquinn/pyani, the method 'ANIm' uses MUMmer (NUCmer) to align the input sequences. 'ANIb' uses BLASTN+ to align 1020nt fragments of the input sequences. 'ANIblastall': uses the legacy BLASTN to align 1020nt fragments Finally, 'TETRA': calculates tetranucleotide frequencies of each input sequence Choices: ANIm ANIb ANIblastall TETRA Default: ANIb
--min-alignment-fraction NUM: In some cases you may get high raw ANI estimates (percent identity scores) between two genomes that have little to do with each other simply because only a small fraction of their content may be aligned. This filter will set all ANI scores between two genomes to 0 if the alignment fraction is less than you deem trustable. When you set a value, anvi'o will go through the ANI results, and set percent identity scores between two genomes to 0 if the alignment fraction *between either of them* is less than the parameter described here. The default is 0.25. Default: 0.25
--significant-alignment-length INT: So --min-alignment-fraction discards any hit that is coming from alignments that represent shorter fractions of genomes, but what if you still don't want to miss an alignment that is longer than an X number of nucleotides regardless of what fraction of the genome it represents? Well, this parameter is to recover things that may be lost due to --min-alignment-fraction parameter. Let's say, if you set --min-alignment-fraction to '0.05', and this parameter to '5000', anvi'o will keep hits from alignments that are longer than 5000 nts, EVEN IF THEY REPRESENT less than 5 percent of a given genome pair. Basically if --min-alignment-fraction is your shield to protect yourself from incoming garbage, --significant-alignment-length is your chopstick to pick out those that may be interesting, and you are a true warrior here.
--use-full-percent-identity: Usually, percent identity is calculated only over aligned regions, and this is what is used as a distance metric by default. But with this flag, you can instead use the *full* percent identity as the distance metric. It is the same as percent identity, except that regions that did not align are included in the calculation. This means *full* percent identity will always be less than or equal to percent identity. How is it calculated? Well if P is the percentage identity calculated in aligned regions, L is the length of the genome, and A is the fraction of the genome that aligned to a compared genome, the full percent identity is P * (A/L). In other words, it is the percent identity multiplied by the alignment coverage. For example, if the alignment is a whopping 97 percent identity but only 8 percent of the genome aligned, the *full* percent identity is 0.970 * 0.080 = 0.078, which is just 7.8 percent.
--min-full-percent-identity FULL_PERCENT_IDENTITY: In some cases you may get high raw ANI estimates (percent identity scores) between two genomes that have little to do with each other simply because only a small fraction of their content may be aligned. This can be partly alleviated by considering the *full* percent identity, which includes in its calculation regions that did not align. For example, if the alignment is a whopping 97 percent identity but only 8 percent of the genome aligned, the *full* percent identity is 0.970 * 0.080 = 0.078 OR 7.8 percent. *full* percent identity is always included in the report, but you can also use it as a filter for other metrics, such as percent identity. This filter will set all ANI measures between two genomes to 0 if the *full* percent identity is less than you deem trustable. When you set a value, anvi'o will go through the ANI results, and set all ANI measures between two genomes to 0 if the *full* percent identity *between either of them* is less than the parameter described here. The default is 20. Default: 20.0

sourmash settings Tell anvi'o to run sourmash with specific settings. Only if `--program` is set to `sourmash`

--kmer-size INT: Set the k-mer size for mash similarity checks. The default is 13. Default: 13
--scale INT: Set the compression ratio for fasta signature file computations. The default is 1000. Smaller ratios decrease sensitivity, while larger ratios will lead to large fasta signatures. Default: 1000

Dereplication Parameters Some parameters to guide your dereplication

--similarity-threshold SIMILARITY_THRESHOLD required: If two genomes have a similarity greater than or equal to this threshold, they will belong to the same cluster. Since measures of 'similarity' depend strongly on what method is used for calculation, and since the threshold at which two genomes should be considered 'similar enough' to be considered redundant will depend on the application, anvi'o refuses to provide a default parameter. If you're using pyANI, maybe 0.90 is what you're after. If you're using sourmash, maybe 0.25 is what you're after. Or maybe not? Anvi'o is feeling nervous about this decision.
--cluster-method CLUSTER_METHOD: Currently, genomes are clustered based on a simple greedy algorithm. Let's say your similarity threshold is 0.90. If genome A is 0.95 similar to B, and B is 0.95 similar to C, and C is 0.95 similar to D, then {A,B,C,D} will form a cluster. This is *even though* D may share a similarity to A of merely 0.80, which is below similarity threshold. You want better alternatives? Contact the developers with your ideas. Choices: simple_greedy Default: simple_greedy
--representative-method REPRESENTATIVE_METHOD: After genomes are grouped into redundancy clusters, you can define how anvi'o picks the representative genome from the cluster. 'Qscore' computes the genome with the highest completion and lowest redundancy as the representative. 'length' returns the longest genome. 'centrality' returns the genome with the highest average similarity to everything in the cluster, i.e. the most central. The default is centrality Choices: Qscore length centrality Default: centrality

OTHER IMPORTANT STUFF Yes. You're almost done.

-T, --num-threads NUM_THREADS: Maximum number of threads to use for multithreading whenever possible. Very conservatively, the default is 1, unless the environmental variable `ANVIO_THREADS` sets another positive integer. If you use this parameter, it will override `ANVIO_THREADS`. It is a good idea to not exceed the number of CPUs / cores on your system. Plus, please be careful with this option if you are running your commands on a SGE -- if you are clusterizing your runs, and asking for multiple threads to use, you may deplete your resources very fast. Default: 1
--just-do-it: Don't bother me with questions or warnings, just do it.
--skip-checking-genome-hashes: Use this flag if you would like anvi'o to skip checking genome hashes. This is only relevant if you may have genomes in your internal or external genomes files that have identical sequences with different names AND if you are OK with it. You may be OK with it, for instance, if you are using `anvi-dereplicate-genomes` program to dereplicate genomes desribed in multiple collections in an anvi'o profile database that may be describing the same genome multiple times (see https://github.com/merenlab/anvio/issues/1397 for a case).
--log-file FILE_PATH: File path to store debug/output messages.

Additional Resources

Are you aware of resources that may help users better understand the utility of this program? Please feel free to edit this file on GitHub. If you are not sure how to do that, find the __resources__ tag in this file to see an example.