anvi-compute-genome-similarity

Export sequences from sequence sources and compute a similarity metric (e.g. ANI). If a Pan Database is given anvi'o will write computed output to misc data tables of Pan Database.

🔙 To the main page of anvi’o programs and artifacts.

Authors

Evan Kiefl
WebEmailTwitterGithub

Requires

external-genomes

Can use

internal-genomes pan-db

Provides

genome-similarity

Usage

This program uses the user’s similarity metric of choice to calculate the similarity between the input genomes.

The currently available programs for calculating similarity metrics include, chosen can be chosen with --program:

  • PyANI) to calculate the average nucleotide identity (ANI) (i.e. what portion of orthologous gene pairs align)
  • fastANI also to calcualte the ANI but at a faster speed (at the drawback of a slight reduction in accuracy)
  • sourmash to calculate the mash distance between genomes. Though we provide this option, we don’t recommend using sourmash for genome comparisons–it excels at other tasks–yet it remains as a legacy option.

Input/Output

The expected input is any combination of external-genomes, internal-genomes, and text files that contains paths to fasta files that describe each of your genomes. This is a tab-delimited file with two columns (name and path to the fasta files, each of which is assumed to be a single genome).

The program outputs a directory with genome-similarity data. The specific contents will depend on how similarity scores are computed (specified with --program), but generally contains tab-separated files of similarity scores between genomes and related metrics.

You also have the option to provide a pan-db, in which case the output data will additionally be stored in the database as misc-data-layers and misc-data-layer-orders data. This was done in the pangenomic tutorial.

Here is an example run with pyANI from an external-genomes without any parameter changes:

anvi-compute-genome-similarity -e external-genomes \ -o path/for/genome-similarity \ --program pyANI

Genome similarity metrics: parameters

Parameters have been divided up based on which --program you use.

pyANI

You have the option to change any of the follow parameters:

  • The method used for alignment. The options are:
    • ANIb (default): uses BLASTN+ to align 1020 nt fragments of the inputs
    • ANIm: uses MUMmer to align
    • ANIblastall: Uses legacy BLASTN to align 1020 nt fragments
    • TETRA: Alignment free. This calculates similarity scores by comparing tetranucleotide frequencies for each input

  • The minimum alignment fraction (all percent identity scores lower than this will be set to 0). The default is 0.

  • If you want to keep alignments that are long, despite them not passing the minimum alignment fraction filter, you can supply a --significant-alignment-length to override --min-alignment-fraction.

  • Similarly, you can discard all results less than some full percent identity (percent identity of aligned segments * aligned fraction).

fastANI

You can change any of the following fastANI parameters:

  • The kmer size. The default is 16.

  • The fragment length. The default is 30.

  • The minimum number of fragments for a result to count. The default is 50.

sourmash

You have the option to change the kmer-size. This value should depend on the relationship between your samples. The default is 31 (as recommended by sourmash for genus-level distances, but we found that 13 most closely parallels the results from an ANI alignment.

You can also set the compression ratio for your fasta files. Decreasing this from the default (1000) will decrease sensitivity.

Other Parameters

Once calculated, the similarity matrix is used to create dendrograms via hierarchical clustering, which are stored in the output directory (and in the pan-db, if provided). You can choose to change the distance metric or linkage algorithm used for this clustering.

If you’re getting a lot of debug/output messages, you can turn them off with --just-do-it or helpfully store them into a file with --log-file.

Edit this file to update this information.

Command line interface

The following is the list of parameters and their descriptions in the latest version of anvi-compute-genome-similarity.

The code that generates this web page ensures that this information is the latest available in 9-dev (eunice). Which means, in some cases the information in this section (generated automatically) might be more comprehensive or up-to-date compared to the descriptions above (written by humans).

Command line usage

anvi-compute-genome-similarity [-h] [-i FILE_PATH] [-e FILE_PATH] [-f FASTA_TEXT_FILE] -o DIR_PATH [-p PAN_DB] [--program {pyANI,fastANI,sourmash}] [--fastani-kmer-size FASTANI_KMER_SIZE] [--fragment-length FRAGMENT_LENGTH] [--min-num-fragments MIN_NUM_FRAGMENTS] [--method {ANIm,ANIb,ANIblastall,TETRA}] [--min-alignment-fraction NUM] [--significant-alignment-length INT] [--min-full-percent-identity FULL_PERCENT_IDENTITY] [--kmer-size INT] [--scale INT] [--distance DISTANCE_METRIC] [--linkage LINKAGE_METHOD] [-T NUM_THREADS] [--just-do-it] [--skip-checking-genome-hashes] [--log-file FILE_PATH]

Parameters and their descriptions

INPUT OPTIONS Tell anvi'o what you want.
-i, --internal-genomes FILE_PATH
A five-column TAB-delimited flat text file. The header line must contain these columns: 'name', 'bin_id', 'collection_id', 'profile_db_path', 'contigs_db_path'. Each line should list a single entry, where 'name' can be any name to describe the anvi'o bin identified as 'bin_id' that is stored in a collection.
-e, --external-genomes FILE_PATH
A two-column TAB-delimited flat text file that lists anvi'o contigs databases. The first item in the header line should read 'name', and the second should read 'contigs_db_path'. Each line in the file should describe a single entry, where the first column is the name of the genome (or MAG), and the second column is the anvi'o contigs database generated for this genome.
-f, --fasta-text-file FASTA_TEXT_FILE
A two-column TAB-delimited file that lists multiple FASTA files to import for analysis. If using for `anvi-dereplicate-genomes` or `anvi-compute-distance`, each FASTA is assumed to be a genome. The first item in the header line should read 'name', and the second item should read 'path'. Each line in the field should describe a single entry, where the first column is the name of the FASTA file or corresponding sequence, and the second column is the path to the FASTA file itself.
OUTPUT OPTIONS Tell anvi'o where to store your results.
-o, --output-dir DIR_PATH required
Directory path for output files
-p, --pan-db PAN_DB
This is totally optional, but very useful when applicable. If you are running this for genomes for which you already have an anvi'o pangeome, then you can show where the pan database is and anvi'o would automatically add the results into the misc data tables of your pangenome. Those data can then be shown as heatmaps on the pan interactive interface through the 'layers' tab.
Program Tell anvi'o which similarity program to run.
--program PROGRAM
Tell anvi'o which program to run to process genome similarity. For ANI, you should either use pyANI or fastANI. If accuracy is paramount (for example, distinguishing things less than 1 percent different), or for dealing with genomes < 80 percent similar, pyANI is what we recommend. However, fastANI is much faster. If you for some reason want to use mash similarity, you can use sourmash, but its really not intended for genome comparisons. If you don't choose anything here, anvi'o will reluctantly set the program to pyANI, but you really should be the one who is on top of these things. Choices: pyANI fastANI sourmash Default: pyANI
fastANI Settings Tell anvi'o to tell fastANI what settings to set. Only if `--program` is set to `fastANI`
--fastani-kmer-size FASTANI_KMER_SIZE
Choose a kmer. The default is 16. Default: 16
--fragment-length FRAGMENT_LENGTH
Choose a fragment length. The default is 3000. Default: 3000
--min-num-fragments MIN_NUM_FRAGMENTS
Choose the minimum number of fragment lengths that can be trusted. The default is 50. Default: 50
pyANI Settings Tell anvi'o to tell pyANI what method you wish to use and what settings to set. Only if `--program` is set to `pyANI`
--method METHOD
Method for pyANI. The default is ANIb. You must have the necessary binary in path for whichever method you choose. According to the pyANI help for v0.2.7 at https://github.com/widdowquinn/pyani, the method 'ANIm' uses MUMmer (NUCmer) to align the input sequences. 'ANIb' uses BLASTN+ to align 1020nt fragments of the input sequences. 'ANIblastall': uses the legacy BLASTN to align 1020nt fragments Finally, 'TETRA': calculates tetranucleotide frequencies of each input sequence Choices: ANIm ANIb ANIblastall TETRA Default: ANIb
--min-alignment-fraction NUM
In some cases you may get high raw ANI estimates (percent identity scores) between two genomes that have little to do with each other simply because only a small fraction of their content may be aligned. This filter will set all ANI scores between two genomes to 0 if the alignment fraction is less than you deem trustable. When you set a value, anvi'o will go through the ANI results, and set percent identity scores between two genomes to 0 if the alignment fraction *between either of them* is less than the parameter described here. The default is 0. Default: 0.0
--significant-alignment-length INT
So --min-alignment-fraction discards any hit that is coming from alignments that represent shorter fractions of genomes, but what if you still don't want to miss an alignment that is longer than an X number of nucleotides regardless of what fraction of the genome it represents? Well, this parameter is to recover things that may be lost due to --min-alignment-fraction parameter. Let's say, if you set --min-alignment-fraction to '0.05', and this parameter to '5000', anvi'o will keep hits from alignments that are longer than 5000 nts, EVEN IF THEY REPRESENT less than 5 percent of a given genome pair. Basically if --min-alignment-fraction is your shield to protect yourself from incoming garbage, --significant-alignment-length is your chopstick to pick out those that may be interesting, and you are a true warrior here.
--min-full-percent-identity FULL_PERCENT_IDENTITY
In some cases you may get high raw ANI estimates (percent identity scores) between two genomes that have little to do with each other simply because only a small fraction of their content may be aligned. This can be partly alleviated by considering the *full* percent identity, which includes in its calculation regions that did not align. For example, if the alignment is a whopping 97 percent identity but only 8 percent of the genome aligned, the *full* percent identity is 0.970 * 0.080 = 0.078 OR 7.8 percent. *full* percent identity is always included in the report, but you can also use it as a filter for other metrics, such as percent identity. This filter will set all ANI measures between two genomes to 0 if the *full* percent identity is less than you deem trustable. When you set a value, anvi'o will go through the ANI results, and set all ANI measures between two genomes to 0 if the *full* percent identity *between either of them* is less than the parameter described here. The default is 0. Default: 0.0
Sourmash Settings Tell anvi'o to tell sourmash what settings to set. Only if `--program` is set to `sourmash`
--kmer-size INT
Set the k-mer size for mash similarity checks. We found 13 in almost all cases correlates best with alignment-based ANI.
--scale INT
Set the compression ratio for fasta signature file computations. The default is 1000. Smaller ratios decrease sensitivity, while larger ratios will lead to large fasta signatures. Default: 1000
HIERARCHICAL CLUSTERING anvi-compute-genome-similarity outputs similarity matrix files, which can be clustered into nice looking dendrograms to display the relationships between genomes nicely (in the anvi'o interface and elsewhere). Here you can set the distance metric and the linkage algorithm for that.
--distance DISTANCE_METRIC
The distance metric for the hierarchical clustering. The default is "euclidean". Default: euclidean
--linkage LINKAGE_METHOD
The linkage method for the hierarchical clustering. The default is "ward". Default: ward
OTHER IMPORTANT STUFF Yes. You're almost done.
-T, --num-threads NUM_THREADS
Maximum number of threads to use for multithreading whenever possible. Very conservatively, the default is 1, unless the environmental variable `ANVIO_THREADS` sets another positive integer. If you use this parameter, it will override `ANVIO_THREADS`. It is a good idea to not exceed the number of CPUs / cores on your system. Plus, please be careful with this option if you are running your commands on a SGE -- if you are clusterizing your runs, and asking for multiple threads to use, you may deplete your resources very fast. Default: 1
--just-do-it
Don't bother me with questions or warnings, just do it.
--skip-checking-genome-hashes
Use this flag if you would like anvi'o to skip checking genome hashes. This is only relevant if you may have genomes in your internal or external genomes files that have identical sequences with different names AND if you are OK with it. You may be OK with it, for instance, if you are using `anvi-dereplicate-genomes` program to dereplicate genomes desribed in multiple collections in an anvi'o profile database that may be describing the same genome multiple times (see https://github.com/merenlab/anvio/issues/1397 for a case).
--log-file FILE_PATH
File path to store debug/output messages.

Additional Resources

Are you aware of resources that may help users better understand the utility of this program? Please feel free to edit this file on GitHub. If you are not sure how to do that, find the __resources__ tag in this file to see an example.

Copyleft, The anvi'o Project