# nf-core/metatdenovo: Output

## Introduction

This document describes the output produced by the pipeline.

The directories listed below will be created in the results directory after the pipeline has finished.
All paths are relative to the top-level results directory.

## Pipeline overview

The pipeline is built using [Nextflow](https://www.nextflow.io/) and the results are organized as follow:

- [Module output](#module-output)
  - [Preprocessing](#preprocessing)
    - [FastQC](#fastqc) - Read quality control
    - [Trim galore!](#trim-galore) - Primer trimming
    - [MultiQC](#multiqc) - Aggregate report describing results
    - [BBduk](#bbduk) - Filter out sequences from samples that matches sequences in a user-provided fasta file (optional)
    - [BBnorm](#bbnorm) - Normalize the reads in the samples to use less resources for assembly (optional)
  - [Assembly step](#assembly-step) - Generate contigs with an assembler program
    - [Megahit](#megahit) - Output from Megahit assembly (default)
    - [Spades](#spades) - Output from Spades assembly (optional)
  - [ORF Caller step](#orf-caller-step) - Identify protein-coding genes (ORFs) with an ORF caller
    - [Prodigal](#prodigal) - Output from Prodigal (default)
    - [Prokka](#prokka) - Output from Prokka (optional)
    - [TransDecoder](#transdecoder) - Output from transdecoder (optional)
  - [Quantification](#quantification) - Quantify expression of features
    - [BBMap](#bbmap) - Align reads to contigs
    - [FeatureCounts](#featurecounts) - Quantify features
  - [Functional and taxonomic annotation](#functional-and-taxonomic-annotation) - Predict the function and the taxonomy of ORFs
    - [EggNOG](#eggnog) - Output from EggNOG-mapper (default; optional)
    - [KOfamSCAN](#kofamscan) - Output KOfamSCAN (optional)
    - [EUKulele](#eukulele) - Output from EUKulele taxonomy annotation (default; optional)
    - [Diamond taxonomy](#diamond-taxonomy) - Output from the Diamond-based taxonomy processing (optional)
    - [Hmmsearch](#hmmsearch) - Output from HMMER run with user-supplied HMM profiles (optional)
- [Custom metatdenovo output](#metatdenovo-output)
  - [Summary tables folder](#summary-tables) - Tab separated tables ready for further analysis in tools like R and Python
  - [Pipeline information](#pipeline-information) - Report metrics generated during the workflow execution

## Module output

### Preprocessing

#### FastQC

[FastQC](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/) gives general quality metrics about your sequenced reads. It provides information about the quality score distribution across your reads, per base sequence content (%A/T/G/C), adapter contamination and overrepresented sequences. For further reading and documentation see the [FastQC help pages](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/).

<details markdown="1">
<summary>Output files</summary>

- `trimgalore/fastqc/`
  - `*_fastqc.html`: FastQC report containing quality metrics for your untrimmed raw fastq files.
  - `*_fastqc.zip`: Zipped FastQC data.
- `summary_tables/`

</details>

#### Trim galore!

[Trimgalore](https://github.com/FelixKrueger/TrimGalore) is a fastq preprocessor for read/adapter trimming and quality control. It is used in this pipeline for trimming adapter sequences and discard low-quality reads. Its output is in the results folder and part of the MultiQC report.

<details markdown="1">
<summary>Output files</summary>

- `trimgalore/`: directory containing log files with retained reads, trimming percentage, etc. for each sample.
  - `*trimming_report.txt`: report of read numbers that pass trimgalore.
- `summary_tables/`

</details>

#### MultiQC

[MultiQC](http://multiqc.info) is a visualization tool that generates a single HTML report summarising all samples in your project. Most of the pipeline QC results are visualised in the report and further statistics are available in the report data directory.

Results generated by MultiQC collate pipeline QC from supported tools e.g. FastQC. The pipeline has special steps which also allow the software versions to be reported in the MultiQC output for future traceability. For more information about how to use MultiQC reports, see <http://multiqc.info>.

<details markdown="1">
<summary>Output files</summary>

- `multiqc/`
  - `multiqc_report.html`: a standalone HTML file that can be viewed in your web browser.
  - `multiqc_data/`: directory containing parsed statistics from the different tools used in the pipeline.
  - `multiqc_plots/`: directory containing static images from the report in various formats.
- `summary_tables/`

</details>

:::note
The FastQC plots displayed in the MultiQC report shows _untrimmed_ reads. They may contain adapter sequence and potentially regions with low quality.
:::

#### BBduk

[BBduk](https://jgi.doe.gov/data-and-tools/software-tools/bbtools/bb-tools-user-guide/bbnorm-guide/) is a filtering tool that removes specific sequences from the samples using a reference fasta file.
BBduk is built-in tool from BBmap.

<details markdown="1">
<summary>Output files</summary>

- `bbmap/`
  - `bbduk/*.fastq.gz`: fastq files after removal of sequences specified with `--sequence_filter`. Only saved if you set `--save_bbduk_fastq`.
  - `bbduk/*.bbduk.log`: text files with the results from BBduk analysis. The number of filtered reads can be found here.
- `summary_tables/`

</details>

#### BBnorm

[BBnorm](https://jgi.doe.gov/data-and-tools/software-tools/bbtools/bb-tools-user-guide/bbduk-guide/) is a tool from BBmap that allows to reduce the coverage of highly abundant sequence kmers and remove sequences representing kmers that are below a threshold.
It can be useful if the data set is too large to assemble but also potentially improve an assembly.
N.B. the digital normalization is done only for the assembly and the non-normalized sequences will be used for quantification.
BBnorm is a BBmap tool.

<details markdown="1">
<summary>Output files</summary>

- `bbmap/bbnorm/`
  - `all_samples.bbnorm.log`: Log file for the run.
  - `all_samples.fastq.gz`: Sequences kept after normalization. Only saved if you set `--save_bbnorm_fastq`.
- `summary_tables/`

</details>

### Assembly step

#### Megahit

[Megahit](https://github.com/voutcn/megahit) is used to assemble the cleaned and trimmed FastQ reads into contigs.

<details markdown="1">
<summary>Output files</summary>

- `megahit/megahit_out/`
  - `*.log`: log file of Megahit run.
  - `megahit_assembly.contigs.fa.gz`: contigs created by Megahit.

</details>

#### Spades

[Spades](https://github.com/ablab/spades) is another option to assemble reads into contigs.

<details markdown="1">
<summary>Output files</summary>

- `spades/`
  - `spades.assembly.gfa.gz`: gfa file output from Spades.
  - `spades.spades.log`: log file output from Spades.
  - `spades.transcripts.fa.gz`: contigs created by Spades.
  - `spades.yaml`: configuration file used by Spades.

</details>

### ORF caller step

#### Prodigal

As default, [Prodigal](https://github.com/hyattpd/Prodigal) is used to identify ORFs in the assembly.

<details markdown="1">
<summary>Output files</summary>

- `prodigal/`
  - `<assembly_name>.prodigal_all.txt.gz`: ORF summary
  - `<assembly_name>.prodigal.fna.gz`: ORFs in nucleotide format fasta file
  - `<assembly_name>.prodigal.faa.gz`: ORFs in amino acids format fasta file
  - `<assembly_name>.prodigal.gff.gz`: ORFs in genome feature file format
  - `<assembly_name>.prodigal_format.gff.gz`: ORFs in alternative genome feature file format

</details>

#### Prokka

As one alternative, you can use [Prokka](https://github.com/tseemann/prokka) to identify ORFs in the assembly.
In addition to calling ORFs (done with Prodigal) Prokka will filter ORFs to only retain quality ORFs and will functionally annotate the ORFs.
NB: Prodigal or Prokka are recomended for prokaryotic samples

<details markdown="1">
<summary>Output files</summary>

- `prokka/`
  - `*.ffn.gz`: ORFs in nucleotide format fasta file
  - `*.faa.gz`: ORFs in amino acids format fasta file
  - `*.gff.gz`: all features in genome feature file format
- `summary_tables/`
  - `<assembly_name>.<orfcaller_name>.prokka-annotations.tsv.gz`: functional annotation of ORFs parsed from the `.gff.gz` file

</details>

#### TransDecoder

Another alternative is [TransDecoder](https://github.com/sghignone/TransDecoder) to find ORFs in the assembly.
N.B. TransDecoder is recommended for eukaryotic samples

<details markdown="1">
<summary>Output files</summary>

- `transdecoder/`
  - `*.bed.gz`: ORFs in bed format
  - `*.cds.gz`: ORFs in nucleotide format fasta file
  - `*.pep.gz`: ORFs in amino acid format fasta file
  - `*.gff3.gz`: ORFs in genome feature format

</details>

### Quantification

#### BBMap

Align reads to contigs with [BBMap](https://sourceforge.net/projects/bbmap/)

<details markdown="1">
<summary>Output files</summary>

- `bbmap/bbmap/`
  - `*.bam`: alignments in bam format if enabled with `--save_bam`.
  - `*.bbmap.log`: log files

</details>

#### FeatureCounts

Quantification of CDS features with `featureCounts` from the [subread](https://subread.sourceforge.net) package

<details markdown="1">
<summary>Output files</summary>

- `featurecounts/`
  - `*.featureCounts.tsv`: tab separated file with count data
  - `*.featureCounts.tsv.summar`: summary statistics
- `summary_tables/`
  - `<assembly_name>.<orfcaller_name>.counts.tsv.gz`: reformatted count data

</details>

### Functional and taxonomical annotation

#### EggNOG

[EggNOG-mapper](https://github.com/eggnogdb/eggnog-mapper) will perform an analysis to assign functions to the ORFs.

<details markdown="1">
<summary>Output files</summary>

- `eggnog/`
  - `<assembly_name>.<orfcaller_name>.emapper.annotations.gz`: a file with the results from the annotation phase, see the [EggNOG-mapper documentation](https://github.com/eggnogdb/eggnog-mapper/wiki/).
  - `<assembly_name>.<orfcaller_name>.emapper.hits.gz`: a file with the results from the search phase, from HMMER, Diamond or MMseqs2.
  - `<assembly_name>.<orfcaller_name>.emapper.seed_orthologs.gz`: a file with the results from parsing the hits. Each row links a query with a seed ortholog. This file has the same format independently of which searcher was used, except that it can be in short format (4 fields), or full.
  - `<assembly_name>.<orfcaller_name>.emapper.hits.gz`: the `*.emapper.annotations.gz` a little bit tidied up.
- `summary_tables/`
  - `<assembly_name>.<orfcaller_name>.emapper.tsv.gz`: reformatted eggnog-mapper output

</details>

#### KOfamScan

[KOfamScan](https://github.com/takaram/kofam_scan) will perform an analysis to assign KEGG orthologs to ORFs.

<details markdown="1">
<summary>Output files</summary>

- `kofamscan/`
  - `<assembly_name>.<orfcaller_name>.kofamscan_output.tsv.gz`: kofamscan output.
- `summary_tables/`
  - `<assembly_name>.<orfcaller_name>.kofamscan.tsv.gz`: reformatted kofamscan output
  - `<assembly_name>.<orfcaller_name>.kofamscan-uniq.tsv.gz`: reformatted kofamscan output subset to the best hit for each ORF

</details>

#### EUKulele

[EUKulele](https://github.com/AlexanderLabWHOI/EUKulele) will perform an analysis to assign taxonomy to the ORFs.
A number of databases are supported: MMETSP, PhyloDB and GTDB.
GTDB currently only works as a user provided database, i.e. data must be downloaded before running nf-core/metatdenovo.

<details markdown="1">
<summary>Output files</summary>

- `eukulele/<assembly_name>.<orfcaller_name>_<database>`
  - `<assembly_name>.<orfcaller_name>/mets_full/diamond/proteins.diamond.out.gz`: Diamond output
  - `<assembly_name>.<orfcaller_name>/taxonomy_counts/*.csv.gz`: counts for different ranks, see [EUKulele documentation](https://eukulele.readthedocs.io)
  - `<assembly_name>.<orfcaller_name>/taxonomy_estimation/proteins-estimated-taxonomy.out.gz`: EUKulele taxonomy assignment
- `summary_tables/`
  - `<assembly_name>.<orfcaller_name>.<database>.eukulele.tsv.gz`: reformatted EUKulele output

</details>

#### Diamond taxonomy

[Diamond](https://github.com/bbuchfink/diamond) is a fast protein sequence aligner that can also assign taxonomy based on a "Last Common Ancestor" (LCA)
algorithm.
At the time of writing, users of the pipeline need to craft their own databases, see the usage documentation.

<details markdown="1">
<summary>Output files</summary>

- `diamond_taxonomy/`
  - `<assembly_name>.<orfcaller_name>.<database>.tsv.gz`: Output directly from the Diamond aligner
  - `<assembly_name>.<orfcaller_name>.<database>.lineage.tsv.gz`: Output after `taxonkit lineage` added the full taxonomic lineage to the above
- `summary_tables/`
  - `<assembly_name>.<orfcaller_name>.<database>.diamond.taxonomy.tsv.gz`: Cleaned up output, including addition of a header row and, if a list of ranks was given, the taxonomy parsed into individual taxa
  - `<assembly_name>.<orfcaller_name>.<database>.taxonomy-taxdump.tsv.gz`: Only if `parse_with_taxdump` was set in the input file; like the above, but with the taxonomy parsed using the taxonomy dump files

</details>

#### Hmmsearch

You can run [hmmsearch](https://www.ebi.ac.uk/Tools/hmmer/search/hmmsearch) on ORFs using a set of HMM profiles provided to the pipeline (see the `--hmmdir`, `--hmmpatern` and `--hmmfiles` parameters).

<details markdown="1">
<summary>Output files</summary>

- `hmmer/`
  - `*.tbl.gz`: results from individual HMMER runs in tabular format.
  - `hits/*.faa.gz`: Sequences of the best ranked hits to the different HMMER profiles.
- `summary_tables/`
  - `<assembly_name>.<orfcaller_name>.hmmrank.tsv.gz`: ranked hmmsearch hits

</details>

After the search, hits for each ORF and HMM will be summarised and ranked based on scores for the hits (see output in [summary tables](#summary-tables)).

## Metatdenovo output

### Summary tables

Consistently named and formated output tables in tsv format ready for further analysis.
Filenames start with assembly program and ORF caller, to allow reruns of the pipeline with different parameter settings without overwriting output files.

<details markdown="1">
<summary>Output files</summary>

- `summary_tables/`
  - `<assembly_name>.<orfcaller_name>.overall_stats.tsv.gz`: overall statistics from the pipeline, e.g. number of reads, number of called ORFs, number of reads mapping back to contigs/ORFs etc.
  - `<assembly_name>.<orfcaller_name>.counts.tsv.gz`: read counts per ORF and sample.
  - `<assembly_name>.<orfcaller_name>.emapper.tsv.gz`: reformatted output from EggNOG-mapper.
  - `<assembly_name>.<orfcaller_name>.kofamscan.tsv.gz`: reformatted output from Kofamscan.
  - `<assembly_name>.<orfcaller_name>.kofamscan-uniq.tsv.gz`: reformatted output from Kofamscan with a _single_ row per ORF in contrast to the above.
  - `<assembly_name>.<orfcaller_name>.{db}_eukulele.tsv.gz`: taxonomic annotation per ORF for specific database.
  - `<assembly_name>.<orfcaller_name>.prokka-annotations.tsv.gz`: reformatted annotation output from Prokka.
  - `<assembly_name>.<orfcaller_name>.<database>.diamond.taxonomy.tsv.gz`: diamond taxonomy parsed into individual taxa
  - `<assembly_name>.<orfcaller_name>.<database>.taxonomy-taxdump.tsv.gz`: diamond taxonomy parse with taxdump data. Only if `parse_with_taxdump` was set in the input file
  - `<assembly_name>.<orfcaller_name>.hmmrank.tsv.gz`: ranked summary table from HMMER results.

</details>

### Pipeline information

<details markdown="1">
<summary>Output files</summary>

- `pipeline_info/`
  - Reports generated by Nextflow: `execution_report.html`, `execution_timeline.html`, `execution_trace.txt` and `pipeline_dag.dot`/`pipeline_dag.svg`.
  - Reports generated by the pipeline: `pipeline_report.html`, `pipeline_report.txt` and `software_versions.yml`. The `pipeline_report*` files will only be present if the `--email` / `--email_on_fail` parameter's are used when running the pipeline.
  - Reformatted samplesheet files used as input to the pipeline: `samplesheet.valid.csv`.
  - Parameters used by the pipeline run: `params.json`.

</details>
