转录组数据分析—fastp v0.23.1

用于执行FASTQ数据 的质量控制,支持单端和双端短读数据,作为一体化的 FASTQ预处理器,fastp提供了 诸如质量分析,接头修整,读 取过滤和基本校正等功能。 在处理完所有读取后,这些值 将合并,并且报告者将生成 HTML和JSON格式的报告 。它可以仅仅扫描FASTQ文件一次,就完成比FASTQC+cutadapt+Trimmomatic这三个软件加起来的功能还多很多的功能,而且速度上比仅仅使用Trimmomatic一个软件还要快3倍左右,因为它使用C++开发,处处使用了高效算法,而且完美支持多线程!


Fig.1
Fig.2

安装

使用conda安装conda install -c bioconda fastp=0.23.2

#看一下是否成功,顺便看一下help文档
fastp
fastp: an ultra-fast all-in-one FASTQ preprocessor
version 0.23.2
usage: fastp [options] ...
options:
  -i, --in1                            read1 input file name (string [=])
  -o, --out1                           read1 output file name (string [=])
  -I, --in2                            read2 input file name (string [=])
  -O, --out2                           read2 output file name (string [=])
      --unpaired1                      for PE input, if read1 passed QC but read2 not, it will be written to unpaired1\. Default is to discard it. (string [=])
      --unpaired2                      for PE input, if read2 passed QC but read1 not, it will be written to unpaired2\. If --unpaired2 is same as --unpaired1 (default mode), both unpaired reads will be written to this same file. (string [=])
      --overlapped_out                 for each read pair, output the overlapped region if it has no any mismatched base. (string [=])
      --failed_out                     specify the file to store reads that cannot pass the filters. (string [=])
  -m, --merge                          for paired-end input, merge each pair of reads into a single read if they are overlapped. The merged reads will be written to the file given by --merged_out, the unmerged reads will be written to the files specified by --out1 and --out2\. The merging mode is disabled by default.
      --merged_out                     in the merging mode, specify the file name to store merged output, or specify --stdout to stream the merged output (string [=])
      --include_unmerged               in the merging mode, write the unmerged or unpaired reads to the file specified by --merge. Disabled by default.
  -6, --phred64                        indicate the input is using phred64 scoring (it'll be converted to phred33, so the output will still be phred33)
  -z, --compression                    compression level for gzip output (1 ~ 9). 1 is fastest, 9 is smallest, default is 4\. (int [=4])
      --stdin                          input from STDIN. If the STDIN is interleaved paired-end FASTQ, please also add --interleaved_in.
      --stdout                         stream passing-filters reads to STDOUT. This option will result in interleaved FASTQ output for paired-end output. Disabled by default.
      --interleaved_in                 indicate that  is an interleaved FASTQ which contains both read1 and read2\. Disabled by default.
      --reads_to_process               specify how many reads/pairs to be processed. Default 0 means process all reads. (int [=0])
      --dont_overwrite                 don't overwrite existing files. Overwritting is allowed by default.
      --fix_mgi_id                     the MGI FASTQ ID format is not compatible with many BAM operation tools, enable this option to fix it.
  -V, --verbose                        output verbose log information (i.e. when every 1M reads are processed).
  -A, --disable_adapter_trimming       adapter trimming is enabled by default. If this option is specified, adapter trimming is disabled
  -a, --adapter_sequence               the adapter for read1\. For SE data, if not specified, the adapter will be auto-detected. For PE data, this is used if R1/R2 are found not overlapped. (string [=auto])
      --adapter_sequence_r2            the adapter for read2 (PE data only). This is used if R1/R2 are found not overlapped. If not specified, it will be the same as  (string [=auto])
      --adapter_fasta                  specify a FASTA file to trim both read1 and read2 (if PE) by all the sequences in this FASTA file (string [=])
      --detect_adapter_for_pe          by default, the auto-detection for adapter is for SE data input only, turn on this option to enable it for PE data.
  -f, --trim_front1                    trimming how many bases in front for read1, default is 0 (int [=0])
  -t, --trim_tail1                     trimming how many bases in tail for read1, default is 0 (int [=0])
  -b, --max_len1                       if read1 is longer than max_len1, then trim read1 at its tail to make it as long as max_len1\. Default 0 means no limitation (int [=0])
  -F, --trim_front2                    trimming how many bases in front for read2\. If it's not specified, it will follow read1's settings (int [=0])
  -T, --trim_tail2                     trimming how many bases in tail for read2\. If it's not specified, it will follow read1's settings (int [=0])
  -B, --max_len2                       if read2 is longer than max_len2, then trim read2 at its tail to make it as long as max_len2\. Default 0 means no limitation. If it's not specified, it will follow read1's settings (int [=0])
  -D, --dedup                          enable deduplication to drop the duplicated reads/pairs
      --dup_calc_accuracy              accuracy level to calculate duplication (1~6), higher level uses more memory (1G, 2G, 4G, 8G, 16G, 24G). Default 1 for no-dedup mode, and 3 for dedup mode. (int [=0])
      --dont_eval_duplication          don't evaluate duplication rate to save time and use less memory.
  -g, --trim_poly_g                    force polyG tail trimming, by default trimming is automatically enabled for Illumina NextSeq/NovaSeq data
      --poly_g_min_len                 the minimum length to detect polyG in the read tail. 10 by default. (int [=10])
  -G, --disable_trim_poly_g            disable polyG tail trimming, by default trimming is automatically enabled for Illumina NextSeq/NovaSeq data
  -x, --trim_poly_x                    enable polyX trimming in 3' ends.
      --poly_x_min_len                 the minimum length to detect polyX in the read tail. 10 by default. (int [=10])
  -5, --cut_front                      move a sliding window from front (5') to tail, drop the bases in the window if its mean quality < threshold, stop otherwise.
  -3, --cut_tail                       move a sliding window from tail (3') to front, drop the bases in the window if its mean quality < threshold, stop otherwise.
  -r, --cut_right                      move a sliding window from front to tail, if meet one window with mean quality < threshold, drop the bases in the window and the right part, and then stop.
  -W, --cut_window_size                the window size option shared by cut_front, cut_tail or cut_sliding. Range: 1~1000, default: 4 (int [=4])
  -M, --cut_mean_quality               the mean quality requirement option shared by cut_front, cut_tail or cut_sliding. Range: 1~36 default: 20 (Q20) (int [=20])
      --cut_front_window_size          the window size option of cut_front, default to cut_window_size if not specified (int [=4])
      --cut_front_mean_quality         the mean quality requirement option for cut_front, default to cut_mean_quality if not specified (int [=20])
      --cut_tail_window_size           the window size option of cut_tail, default to cut_window_size if not specified (int [=4])
      --cut_tail_mean_quality          the mean quality requirement option for cut_tail, default to cut_mean_quality if not specified (int [=20])
      --cut_right_window_size          the window size option of cut_right, default to cut_window_size if not specified (int [=4])
      --cut_right_mean_quality         the mean quality requirement option for cut_right, default to cut_mean_quality if not specified (int [=20])
  -Q, --disable_quality_filtering      quality filtering is enabled by default. If this option is specified, quality filtering is disabled
  -q, --qualified_quality_phred        the quality value that a base is qualified. Default 15 means phred quality >=Q15 is qualified. (int [=15])
  -u, --unqualified_percent_limit      how many percents of bases are allowed to be unqualified (0~100). Default 40 means 40% (int [=40])
  -n, --n_base_limit                   if one read's number of N base is >n_base_limit, then this read/pair is discarded. Default is 5 (int [=5])
  -e, --average_qual                   if one read's average quality score =1000), a sequential number prefix will be added to output name ( 0001.out.fq, 0002.out.fq...), disabled by default (long [=0])
  -d, --split_prefix_digits            the digits for the sequential number padding (1~10), default is 4, so the filename will be padded as 0001.xxx, 0 to disable padding (int [=4])
      --cut_by_quality5                DEPRECATED, use --cut_front instead.
      --cut_by_quality3                DEPRECATED, use --cut_tail instead.
      --cut_by_quality_aggressive      DEPRECATED, use --cut_right instead.
      --discard_unmerged               DEPRECATED, no effect now, see the introduction for merging.
  -?, --help                           print this message

功能

  • 对数据自动进行全方位质控,生成人性化的报告。

  • 过滤功能(低质量,太短,太多N……)。

  • 对每一个序列的头部或尾部,计算滑动窗内的质量均值,并将均值较低的子序列进行切除(类似Trimmomatic的做法,但是快非常多)。

  • 全局剪裁 (在头/尾部,不影响去重),对于Illumina下机数据往往最后一到两个cycle需要这样处理。

  • 去除接头污染。厉害的是,你不用输入接头序列,因为算法会自动识别接头序列并进行剪裁。

  • 对于双端测序(PE)的数据,软件会自动查找每一对read的重叠区域,并对该重叠区域中不匹配的碱基对进行校正。

  • 去除尾部的polyG。对于Illumina NextSeq/NovaSeq的测序数据,因为是两色法发光,polyG是常有的事,所以该特性对该两类测序平台默认打开。

  • fastp支持对PE数据的每一对read进行分析,查找它们的overlap区间,然后对于overlap区间中不一致的碱基,如果发现其中一个质量非常高,而另一个非常低,则可以将非常低质量的碱基改为相应的非常高质量值的碱基值,该校正功能默认没有开启使用-c参数可以启用,对于一些对噪声容忍度低的应用,比如液体活检,建议开启。

  • 可以对带分子标签(UMI)的数据进行预处理,不管UMI在插入片段还是在index上,都可以轻松处理。

  • 可以将输出进行分拆,而且支持两种模式,分别是指定分拆的个数,或者分拆后每个文件的行数。

使用

fastp使用非常简单,如果不清楚具体参数该如何设置,直接定义好输入及输出数据就可以快速实现。

##单端数据
fastp -i R1.fastq.gz -o R1_clean.fastq.gz
##双端数据
fastp -i R1.fastq.gz -o R1_clean.fastq.gz -I R2.fastq.gz -O R2_clean.fastq.gz

本次测试使用之前下载好的原始数据(GSE176393),我的代码

#批量操作
cat SRR_Acc_List.txt | while read line;
do
fastp -i 0-rawdata/$line\_1.fastq.gz -o 1-fastp/$line\_clean_1.fastq.gz -I 0-rawdata/$line\_2.fastq.gz -O 1-fastp/$line\_clean_2.fastq.gz -j 1-fastp/$line.json -h 1-fastp/$line.html -z 9 -f 0 -t 0 -T 0 -F 0 -l 75 -p --thread=10
done
##这里我添加了-p参数,查看过度表达序列的情况,也可不加,开启会增加运行时间

输出文件如下,两个版本的报告和输出结果文件。

Fig.3

报告详解

报告全部包括这些内容,点击每一个条目可以收缩或展开


Fig.4

Summary

Summary中包含了本次分析的情况概要。

Fig.5
  • General

版本号、序列循环数、质控之前的平均长度、质控之后的平均长度、插入片段的峰值

  • Before filtering

数据质控之前的(反应测序质量):总的reads长度、总碱基长度、Q20合格率、Q30合格率、GC含量

  • After filtering

质控之后的:内容同上

  • Filtering result

reads的通过率、低质量的reads、含太多N值的reads

Adapters

这里列出了read1和read2从1到几十位的adapters的发生次数,以及其他未列出的接头数

Fig.6

Insert size estimation

配对末端重叠分析,不同长度的Insert在reads中占的比例,相当于是DNA被打断后的长度分布。当插入片段大小<30或> 270,或包含太多错误,则不能被读取,比如我这里就有33.870706%的不可读reads)

Fig.7

Before filtering

质控之前的数据质量、碱基含量以及kmer分析等,可直接在网页上用鼠标拖动放大缩小以及查看具体数据细节,或进行图片保存等操作

reads质量

在不同位置上的碱基质量分布,一般来讲质量应 >30 且波动较小为不错的数据


Fig.8

碱基质量

read各个位置上碱基比例分布,这个是为了分析碱基的分离程度。何为碱基分离?已知AT配对,CG配对,假如测序过程是比较随机的话(随机意味着好),那么在每个位置上A和T比例应该差不多,C和G的比例也应该差不多,如上图所示,两者之间即使有偏差也不应该太大,最好平均在1%以内,如果过高,除非有合理的原因,比如某些特定的捕获测序所致,否则都需要注意是不是测序过程有什么偏差。

Fig.9

KMER计数

fastp对5个碱基长度的所有组合的出现次数进行了统计,然后把它放在了一张表格中,表格的每一个元素为深背景白字,背景越深,则表示重复次数越多。这样,一眼望去,就可以发现有哪些异常的信息。鼠标可停留在某一具体组合上看出现次数和平均占比。


Fig.10

过表达序列

提供了这些overrepresented sequence的序列个数和占比,还提供了他们在测序cycles中的分布情况,这有利于分析各种问题

Fig.11

参考资料:

生信学习笔记:fastp质控处理生成的report结果解读

fastp: 一款超快速全功能的FASTQ文件自动化质控+过滤+校正+预处理软件

你可能感兴趣的:(转录组数据分析—fastp v0.23.1)