anvi-gen-variability-profile

Generate a table that comprehensively summarizes the variability of nucleotide, codon, or amino acid positions. We call these single nucleotide variants (SNVs), single codon variants (SCVs), and single amino acid variants (SAAVs), respectively.

🔙 To the main page of anvi’o programs and artifacts.

Authors

Requires

contigs-db profile-db

Can use

structure-db bin variability-profile splits-txt genes-of-interest-txt

Provides

variability-profile-txt

Usage

This program takes the variability data stored within a profile-db and compiles it from across samples into a single matrix that comprehensively describes your SNVs, SCVs or SAAVs (a variability-profile-txt).

This program is described in detail in this blog post, which also covers the biological motivation, output column definitions, and example use cases.

A note on default filtering of variability signal during profiling

Before this program even runs, anvi’o applies a dynamic, coverage-dependent filter during the profiling step (i.e., when you run anvi-profile on your BAM files) to reduce the reporting of variation due to sequencing errors and other factors that are difficult to distinguish frmo noise. Specifically, the base frequencies observed at a position in the read recruitment data are only stored in the profile-db if departure_from_reference value for that position, which quantifies the fraction of reads that differ from the reference nucleotide at a given position, meets or exceeds a coverage-dependent minimum threshold defined by:

\[y = \left(\frac{1}{b}\right)^{x^{\frac{1}{b}} - m} + c\]

where $x$ is the coverage depth and the model parameters are $b = 2$, $m = 1.45$, $c = 0.05$. This gives a threshold of approximately 0.17 at 20X coverage, 0.07 at 50X, and asymptotically approaches 0.05 at very high coverage. Positions that do not meet this threshold are silently excluded from the database and will never appear in the output of this program.

The user can bypass this filter entirely and store all observed variation regardless of frequency by including --report-variability-full when running anvi-profile. But the use of this parameter will dramatically increase database size, and should only be used with extreme care and attention (this is a very politically correct way to say “please do not use this parameter unless you are certain that this is what you need”).

Basic usage

Here is a basic run with no bells or whistles:

anvi-gen-variability-profile -p profile-db \ -c contigs-db \ -C DEFAULT \ -b EVERYTHING

Note that this program requires you to specify a subset of the data to focus on, so to work with everything in the databases, first run anvi-script-add-default-collection and use the resulting collection and bin as shown above.

You can also specify an output file path, which is useful when running multiple times with different --engine settings:

anvi-gen-variability-profile -p profile-db \ -c contigs-db \ -C DEFAULT \ -b EVERYTHING \ --output-file /path/to/your/variability.txt

Choosing what to analyze

Focusing on a subset of the input

Instead of a collection and bin, there are three alternatives:

  1. Provide gene caller IDs directly or as a file:

    anvi-gen-variability-profile -p profile-db \ -c contigs-db \ --gene-caller-ids 1,2,3

    anvi-gen-variability-profile -p profile-db \ -c contigs-db \ --genes-of-interest genes-of-interest-txt

  2. Provide a splits-txt to focus on a specific set of splits:

    anvi-gen-variability-profile -p profile-db \ -c contigs-db \ --splits-of-interest splits-txt

  3. Provide any collection and bin:

    anvi-gen-variability-profile -p profile-db \ -c contigs-db \ -C collection \ -b bin

Restricting to specific samples

You can limit the analysis to a subset of samples by providing a file with one sample name per line:

anvi-gen-variability-profile -p profile-db \ -c contigs-db \ -C collection \ -b bin \ --samples-of-interest my_samples.txt

where my_samples.txt looks like:

DAY_17A DAY_18A DAY_22A …

Excluding intergenic positions

For nucleotide-level analyses, you can restrict output to positions that fall within gene calls:

anvi-gen-variability-profile -p profile-db \ -c contigs-db \ -C collection \ -b bin \ --exclude-intergenic

Choosing the variability type (engine)

Which type of variants you analyze depends on the --engine parameter:

Engine Variant type Requires
NT (default) Single nucleotide variants (SNVs) standard profiling
CDN Single codon variants (SCVs) --profile-SCVs at profiling time
AA Single amino acid variants (SAAVs) --profile-SCVs at profiling time

To analyze SAAVs:

anvi-gen-variability-profile -p profile-db \ -c contigs-db \ -C collection \ -b bin \ --engine AA

To analyze SCVs:

anvi-gen-variability-profile -p profile-db \ -c contigs-db \ -C collection \ -b bin \ --engine CDN

Adding structural annotations

You can add structural annotations by providing a structure-db:

anvi-gen-variability-profile -p profile-db \ -c contigs-db \ -C DEFAULT \ -b EVERYTHING \ -s structure-db

When a structure-db is provided, you can also limit your analysis to only genes that have structures in the database:

anvi-gen-variability-profile -p profile-db \ -c contigs-db \ -s structure-db \ --only-if-structure

Filtering the output

By departure from reference or consensus

departure_from_reference is the fraction of reads at a position that differ from the reference nucleotide. departure_from_consensus is similar but measured against the most frequent allele in that sample. You can set minimum and maximum bounds on either:

anvi-gen-variability-profile -p profile-db \ -c contigs-db \ -C collection \ -b bin \ --min-departure-from-reference 0.05 \ --max-departure-from-reference 0.90

By minimum occurrence across samples

To keep only positions that are variable in at least N samples (useful for reducing stochastic noise):

anvi-gen-variability-profile -p profile-db \ -c contigs-db \ -C collection \ -b bin \ --min-occurrence 3

By coverage across all samples

To remove any position that has insufficient coverage in even one sample (requires --quince-mode to have coverage data for all samples at all positions):

anvi-gen-variability-profile -p profile-db \ -c contigs-db \ -C collection \ -b bin \ --quince-mode \ --min-coverage-in-each-sample 10

By number of positions per split

To randomly subsample variable positions and keep at most N per split:

anvi-gen-variability-profile -p profile-db \ -c contigs-db \ -C collection \ -b bin \ --num-positions-from-each-split 100

Special reporting modes

–quince-mode

By default, if a position is variable in only some samples, the other samples will have no entry for that position in the output. With --quince-mode, the program goes back to the raw data and fills in allele frequencies for every sample at every reported position, even those where no variation was detected. This is essential for statistical approaches that require a complete matrix.

anvi-gen-variability-profile -p profile-db \ -c contigs-db \ -C collection \ -b bin \ --quince-mode

Note that --quince-mode substantially increases runtime and output file size.

–kiefl-mode

When using --engine AA or --engine CDN, the default behavior reports only positions that had detectable variation during profiling. With --kiefl-mode, all positions in the analyzed genes are reported, with invariant positions given a reference allele frequency of 1. This is useful for analyses that need a complete picture of every codon or amino acid position, not just the variable ones. Incompatible with --quince-mode.

anvi-gen-variability-profile -p profile-db \ -c contigs-db \ -C collection \ -b bin \ --engine CDN \ --kiefl-mode

This flag was added in this pull request where you can read about the tests performed to validate its behavior.

Additional output columns

Contig and split names

anvi-gen-variability-profile -p profile-db \ -c contigs-db \ -C collection \ -b bin \ --include-contig-names \ --include-split-names

Contig names are excluded by default since they can nearly double file size.

Gene-level coverage statistics

anvi-gen-variability-profile -p profile-db \ -c contigs-db \ -C collection \ -b bin \ --compute-gene-coverage-stats

This appends per-gene coverage statistics to each row. It is computationally expensive and off by default.

Per-site pN/pS values (CDN engine only)

anvi-gen-variability-profile -p profile-db \ -c contigs-db \ -C collection \ -b bin \ --engine CDN \ --include-site-pnps

This adds 12 columns of per-site synonymous and nonsynonymous substitution information, computed relative to the reference, the consensus, and the most common consensus across samples.

Additional data from the database

anvi-gen-variability-profile -p profile-db \ -c contigs-db \ -C collection \ -b bin \ --engine AA \ --include-additional-data

This appends any data stored in the amino_acid_additional_data table as extra columns. Currently only supported for the AA engine.

Edit this file to update this information.

Command line interface

The following is the list of parameters and their descriptions in the latest version of anvi-gen-variability-profile.

The code that generates this web page ensures that this information is the latest available in 9-dev (eunice). Which means, in some cases the information in this section (generated automatically) might be more comprehensive or up-to-date compared to the descriptions above (written by humans).


Command line usage

anvi-gen-variability-profile [-h] -p PROFILE_DB -c CONTIGS_DB [-s STRUCTURE_DB] [-C COLLECTION_NAME] [-b BIN_NAME] [--splits-of-interest FILE] [--genes-of-interest FILE] [--gene-caller-ids GENE_CALLER_IDS] [--only-if-structure] [--samples-of-interest FILE] [--engine ENGINE] [--num-positions-from-each-split INT] [-r FLOAT] [-z FLOAT] [-j FLOAT] [-a FLOAT] [-x NUM_SAMPLES] [--min-coverage-in-each-sample INT] [--quince-mode] [--kiefl-mode] [--exclude-intergenic] [-o VARIABILITY_PROFILE] [--include-contig-names] [--include-split-names] [--include-additional-data] [--include-site-pnps] [--compute-gene-coverage-stats]

Parameters and their descriptions

DATABASES Declaring relevant anvi'o databases. First things first. Some are mandatory, some are optional.
-p, --profile-db PROFILE_DB required
Anvi'o profile database
-c, --contigs-db CONTIGS_DB required
Anvi'o contigs database generated by 'anvi-gen-contigs-database'
-s, --structure-db STRUCTURE_DB
Anvi'o structure database.
FOCUS :: BIN You need to pick someting to focus. You can ask anvi'o to work with a bin in a collection.
-C, --collection-name COLLECTION_NAME
Collection name.
-b, --bin-id BIN_NAME
Bin name you are interested in.
FOCUS :: SPLIT NAMES Alternatively you can declare split names to focus.
--splits-of-interest FILE
A file with split names. There should be only one column in the file, and each line should correspond to a unique split name.
FOCUS :: GENE CALLER IDs Alternatively you can declare gene caller IDs to focus.
--genes-of-interest FILE
A file with anvi'o gene caller IDs. There should be only one column in the file, and each line should correspond to a unique gene caller id (without a column header).
--gene-caller-ids GENE_CALLER_IDS
Gene caller ids to focus on. Multiple of them can be declared separated by a delimiter (the default is a comma, and if there is no option to change the delimiter, you must use a comma).In anvi-gen-variability-profile, if you declare nothing you will get all genes matching your other filtering criteria. In other programs, you may get everything, nothing, or an error. It really depends on the situation. Fortunately, mistakes are cheap, so it's worth a try.
--only-if-structure
If provided, your genes of interest will be further subset to only include genes with structures in your structure database, and therefore must be supplied in conjunction with a structure database, i.e. `-s <your_structure_database>`. If you did not specify genes of interest, ALL genes will be subset to those that have structures.
SAMPLES You can ask anvi'o to focus only on a subset of samples.
--samples-of-interest FILE
A file with samples names. There should be only one column in the file, and each line should correspond to a unique sample name (without a column header).
ENGINE Set your engine. This is important as it will define the output profile you will get from this program. The engine can focus on nucleotides (NT), codons (CDN), or an amino acids (AA).
--engine ENGINE
Variability engine. The default is 'NT'. Default: NT
FILTERS Parameters that will help you to do a very precise analysis. If you declare nothing from this bunch, you will get "everything" to play with, which is not necessarily a good thing...
--num-positions-from-each-split INT
Each split may have one or more variable positions. By default, anvi'o will report every SNV position found in a given split. This parameter will help you to define a cutoff for the maximum number of SNVs to be reported from a split (if the number of SNVs is more than the number you declare using this parameter, the positions will be randomly subsampled). Default: 0
-r, --min-departure-from-reference FLOAT
Takes a value between 0 and 1, where 1 is maximum divergence from the reference. Default is 0.000000. The reference here observation that corresponds to a given position in the mapped context. Default: 0
-z, --max-departure-from-reference FLOAT
Similar to '--min-departure-from-reference', but defines an upper limit for divergence. The default is 1.000000. Default: 1
-j, --min-departure-from-consensus FLOAT
Takes a value between 0 and 1, where 1 is maximum divergence from the consensus for a given position. The default is 0.000000. The consensus is the most frequent observation at a given position. Default: 0
-a, --max-departure-from-consensus FLOAT
Similar to '--min-departure-from-consensus', but defines an upper limit for divergence. The default is 1.000000. Default: 1
-x, --min-occurrence NUM_SAMPLES
Minimum number of samples a nucleotide position should be reported as variable. Default is 1. If you set it to 2, for instance, each eligible variable position will be expected to appear in at least two samples, which will reduce the impact of stochastic, or unintelligible variable positions. Default: 1
--min-coverage-in-each-sample INT
Minimum coverage of a given variable nucleotide position in all samples. If a nucleotide position is covered less than this value even in one sample, it will be removed from the analysis. Default is 0. Default: 0
--quince-mode
The default behavior is to report allele frequencies only at positions where variation was reported during profiling (which by default uses some heuristics to minimize the impact of error-driven variation). So, if there are 10 samples, and a given position has been reported as a variable site during profiling in only one of those samples, there will be no information will be stored in the database for the remaining 9. When this flag is used, we go back to each sample, and report allele frequencies for each sample at this position, even if they do not vary. It will take considerably longer to report when this flag is on, and the use of it will increase the file size dramatically, however it is inevitable for some statistical approaches and visualizations.
--kiefl-mode
The default behavior is to report codon/amino-acid frequencies only at positions where variation was reported during profiling (which by default uses some heuristics to minimize the impact of error-driven variation). When this flag is used, all positions are reported, regardless of whether they contained variation in any sample. The reference codon for all such entries is given a codon frequency of 1. All other entries (aka those with legitimate variation to be reported) remain unchanged. This flag can only be used with `--engine AA` or `--engine CDN` and is incompatible wth --quince-mode.
--exclude-intergenic
Use this flag to exclude SNVs that occur in intergenic regions (i.e., nucleotide positions that do not fall within any gene call, whether coding or non-coding). This flag can only be used with `--engine NT`.
OUTPUT Output file and style
-o, --output-file VARIABILITY_PROFILE
File path to store results. Default: variability.txt
--include-contig-names
Use this flag if you would like contig names for each variable position to be included in the output file as a column. By default, we do not include contig names since they can practically double the output file size without any actual benefit in most cases.
--include-split-names
Use this flag if you would like split names for each variable position to be included in the output file as a column.
--include-additional-data
Use this flag if you would like to append data stored in the `amino_acid_additional_data` table as additional columns to your output. NOTE: This is not yet implemented for the `nucleotide_additional_data` table.
--include-site-pnps
Use this flag if you want per-site pN and pS added as additional columns. Synonymity will be calculated with respect to the reference, to the consenus, and to the most common consensus seen at that site across samples (popular consensus). The number of synonymous and nonsynonymous sites will also be stored for each case. This makes a total of 12 added columns. This flag will be ignored if --engine is not CDN.
--compute-gene-coverage-stats
If provided, gene coverage statistics will be appended for each entry in variability report. This is very useful information, but will not be included by default because it is an expensive operation, and may take some additional time.

Additional Resources

Are you aware of resources that may help users better understand the utility of this program? Please feel free to edit this file on GitHub. If you are not sure how to do that, find the __resources__ tag in this file to see an example.