Skip to content

Overview

It doesn't matter what data type you started with. Once your data is prepared, you receive three zipped files, which we sometimes call the three Rs:

  • Raw data
  • Report
  • Results
<ProjectNumber>_<RunDate>_<Gene>_[RawData/Results/Reports].zip

Sequencer
    │
    ▼
RawData.zip
    │
    ├───────────────────┐
    ▼                   ▼
Reports.zip         Results.zip
    │                   │
    ▼                   ▼
QC logs,            Count tables,
parameters,         trees, FASTA,
statistics          Krona charts

As mentioned on the Data Prep Short-Reads (PE) page, our processing pipeline is modular rather than fixed, with modules and parameters assembled to fit the data at hand. One consequence shows up here directly: your output can look somewhat different from another project's, or even from an earlier run of your own project, since not every module runs every time and file sets can vary accordingly. If you're having trouble understanding your specific output, let us know. We're happy to walk through it with you.

Quick Start: What Do I Actually Need?

The Results package alone can contain dozens of files, but most people only ever touch a handful of them.

I want to... Use...
Analyse in R (e.g. phyloseq) The annotated count table (*_Count_Sintax.txt, see note below) and the map file
Do phylogenetic-aware analyses (e.g. UniFrac) Also add the tree file (*.tre)
Export or BLAST specific sequences The FASTA file (*.fa.gz)
Get a first impression of diversity The Krona pie charts (*.html)
Troubleshoot or document your run The report files (y_help/*.report)

Files for Data Import

Together with the map (meta) file, three files are recommended for importing your data into phyloseq. Only the annotated count/OTU table and the map file are strictly required; the tree and sequence files are optional. We usually include the tree file, since some analysis methods (e.g. UniFrac) require it. If you have many (z)OTUs and long amplicons, the sequence file can get large and slow down analysis.

Count table    : e_OTU/*_*OTU*_Count_*.txt
Tree file      : e_OTU/*_*OTU*.tre
FASTA sequences: e_OTU/*_*OTU*.fa*

We usually provide a working map file template (*_MapFileTemplate.txt). It might be neither complete nor entirely correct for your purposes, so adjust it as needed, but please don't change the first three columns or the last column. Everything in between is yours to edit freely. One piece of advice: use short column names without whitespace. "tissue" is better than "sample_tissue", and both are better than "sample tissue".

Unless you have a specific reason to prefer OTUs, for example matching an existing dataset that was already OTU-clustered, we generally recommend starting with the zOTU table for new analyses.

A Note on Filenames

Sintax in *_Count_Sintax.txt is a placeholder for whichever classifier and reference were actually used, and is often replaced with the reference name instead, for example *_Count_Silva138.txt or *_Count_UNITE10.txt. If you don't see a file literally named Sintax, look for _Count_<reference name>.txt instead, it's the same file.

The rest of this page explains where these files come from and what everything else in your results package is for.

Raw Data

This is a copy of your sequencing data. Submit it to a read archive and keep a copy for yourself (see the Data Download page for how). For Illumina or AVITI data, you should have two FASTQ files (R1 and R2); for PacBio, one CCS BAM file. That's the common case, but there are special exceptions where what you get looks different:

  • Sometimes data is not demultiplexed at all, or is pooled, for example samples from different loci (16S and ITS) sharing the same barcode, which can't be separated by demultiplexing alone.
  • Sometimes the sequencing facility applies its own demultiplexing strategy, for example orienting reads without correspondingly updating the strand information in the read headers, which leaves the files internally inconsistent if taken at face value.

In any of these cases, we provide data in whatever form is actually usable for archive submission, which may not match the simple two-file or one-file pattern above. If what you received doesn't look like what's described here, ask rather than assume something went wrong.

Original filenames from the sequencer can be long, redundant, and not very meaningful, so we usually simplify them to include only the relevant information, typically the sample ID. For example:

MSQ184049_Sample_RH01_S1_L001_R1_001.fastq.gz → RH01_R1.fq.gz

Secure Storage

Raw data is important and must be kept safe. You, and only you, are responsible for your data, so we leave it up to you to decide what "safe" means in your specific case, though we're happy to advise. A copy should also be deposited at the European Nucleotide Archive (ENA) or NCBI's Sequence Read Archive (SRA) as early as possible.

Reports

We divide data preparation into several steps, and a report file is generated for each one. These files matter: they list the program used, including its version, the parameters applied, any warnings or errors, and relevant statistics. We rely on them ourselves to evaluate the data processing, and you'll need them for documentation purposes. Please don't alter the report files.

Have a closer look at the files:

# All steps together
ll y_help/[ABCDEFG]*.report
# Or a specific step, e.g. B
less y_help/B_*.report

There is also a detailed read (loss) report. The text file is semicolon-delimited and can be imported straight into a spreadsheet editor. It documents data loss at each processing step, for each sample:

Raw → Clean → Merged → Primer → Filter

cat y_help/*ReadStats.report
Pos Header Meaning
$1 Sample Sample ID
$2 Raw Raw reads
$3 Clean PhiX and low-complexity reads removed
$4 Merged R1 and R2 merged (paired-end data only)
$5 Primer Primer site removed
$6 Filter Q-filter, size, and GC filter
$7 MeanLength Mean amplicon length

Data Loss

Overall loss should be below 15%. Some samples, especially negative controls or samples with low read counts to begin with, can have much higher losses, which can also pull up the mean loss rate across your project. See the Data Prep Short-Reads (PE) page for how to think about data loss more generally: the goal is understanding why loss happened at each step, not minimising the number itself.

Pooled samples need particular care. If different amplicons sharing the same index were pooled together on the run, the raw read counts for that index cover all of them combined, not just one sample. You need to add together the counts for every sample sharing an index before those loss percentages mean anything, otherwise a sample can look like it lost far more data than it actually did.

Results

During data processing, we store various results you can use for analysis or to better understand the data and how it was processed. Each clustering approach produces a corresponding set of output files (e.g. count tables for both UPARSE and UNOISE), plus a set of supporting files to help you interpret and evaluate the results. Grouped by purpose rather than listed flat, the files under e_OTU/<ProjectNumber>_<RunDate>_<Gene> break down like this:

Primary analysis
  ..._ZOTU_Count_Sintax.txt       -> count table with annotation (Sintax may be replaced by the reference name, e.g. Silva138)
  ..._ZOTU_Count.txt              -> count table (no annotation)
  ..._ZOTU.fa.gz                  -> cluster sequences (FASTA)

Trees
  ..._ZOTU_CLU.tre                -> cluster-based tree (see report E3)
  ..._ZOTU_MSA.tre                -> alignment-based tree (see report E3)

Supporting files
  ..._ZOTU_Count.summary          -> count table summary
  ..._ZOTU.tax                    -> SINTAX reports (unfiltered)
  ..._ZOTU.mx.gz                  -> cluster assignment reports

Diagnostics
  ..._ZOTU_chimera.txt            -> additional chimera evaluation (see report F)
  ..._ZOTU_Count.map.gz           -> read mapping reports
  ..._ZOTU_CrossTalk.txt.gz       -> cross-talk-corrected counts (see cross-talk HTML reports)

Count Tables (With and Without Annotation)

For each clustering method, there's a set of corresponding files, including the (annotated) count tables:

OTU      : the classic 97% identity (3% radius) clustering method (UPARSE)
ZOTU     : the newer zero-radius (zOTU) or amplicon sequence variant (ASV) method (UNOISE3)
ZOTU_cXX : the zOTU method with additional clustering

See the Overview Workflows page for the conceptual difference between OTU and zOTU clustering, and the Quick Start section above for which one to start with. On our data specifically, the number of OTUs typically comes out close to the number of zOTUs additionally clustered at 98% or 97% identity, and the number of zOTUs clustered at 97% is usually slightly lower still, likely a result of UNOISE's built-in error correction step.

Uncross

UNCROSS detects and filters cross-talk (sample misassignment) in an OTU table, and it's worth knowing about even though many sequencing facilities never mention it. In a typical run, roughly 2% of reads end up assigned to the wrong sample ID. When some samples carry large numbers of reads for a given OTU, those reads can "bleed" into other samples that may not actually contain that OTU at all, producing spurious counts that should really be zero and inflating estimates of richness and alpha/beta diversity. This correction is still experimental, but it can meaningfully change what your count table implies about which samples actually contain which taxa.

Krona Pie Charts

Krona charts are meant for quick exploration, not formal analysis. They give a first impression of the diversity in your data as interactive Krona pie charts in HTML format:

ls -lh y_help/*_Krona_Counts.html                            # per sample
ls -lh y_help/*_OTU_Krona_TaxPrevalence_TotalAbundance.html  # all samples together
ls -lh y_help/*_ZOTU_Krona_TaxPrevalence_TotalAbundance.html # all samples together
  • Taxonomic prevalence: how often a particular taxon occurs (frequency).
  • Total abundance: the sum of all counts for a given taxon.

Taxonomic Assignments

Please don't blindly trust taxonomic assignments. They are, at best, informed suggestions, and error rate increases with taxonomic depth (see the caution on this in Data Prep Short-Reads (PE), Step F). The assignments added to the (z)OTUs in the count tables are already filtered, but still carry uncertainty. The confidence threshold used is in report file F and should be somewhere between 70% and 90%. The unfiltered assignments are also available as a text file (*.tax), useful for examining confidence values at different taxonomic levels.

grep "Tax Filter" y_help/F_*_TAX.report
head e_OTU/*_ZOTU.tax

It's worth knowing which reference database was used, also documented in report file F. Reference diversity, accuracy, and sequence coverage all shape the result, and a larger reference isn't automatically a better one. Its species composition should cover your expected diversity and include appropriate outgroups. References with incorrect, missing, or vague labels (e.g. "environmental sample") aren't much use, and reference sequences ideally should cover the full amplicon, not just part of it.

grep "Application" -A 2 y_help/F_*_TAX.report

Count Table Summary

Summary reports exist for the (z)OTU tables, giving statistics such as number of reads (amplicons), number of samples, number of (z)OTUs, and a detailed count breakdown.

cat results/*_Count.summary

Tree Files

For each clustering method, there are two kinds of tree files: an alignment-based tree (MSA) and a cluster-based tree (CLU). Both can help you understand and refine OTU clustering, and spot outliers.

ls -1 e_OTU/*.tre
# *_CLU.tre
# *_MSA.tre

Both trees are approximate, and are meant for ecological distance calculations (e.g. UniFrac) and exploratory visualisation rather than as a rigorous phylogenetic inference. For most analyses of that kind, their exact accuracy doesn't matter much, but don't treat them as publication-quality phylogenies.