高通量数据库的数据，下载方法有三种：
- 常规下载(wget,迅雷、IDM)
- aspera
- SRA Toolkit 的prefetch
wget下载数据速度很慢，比较容易断。
wget -c 下载地址
保持断点下载
ENA数据下载方法汇总 https://ena-docs.readthedocs.io/en/latest/retrieval/file-download.html#using-wget
下载数据库
优先选择 快速下载fq格式的数据，EBI数据库下载。
从EBI数据库直接获取到aspera的下载代码，复制到本地服务器，可以直接运行。
cd ~/wes_cancer/project/1.raw_fq
ascp -QT -l 300m -P33001 -i $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh [email protected]:vol1/fastq/SRR318/008/SRR3182418/SRR3182418_2.fastq.gz .
ascp -QT -l 300m -P33001 -i $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh [email protected]:vol1/fastq/SRR318/003/SRR3182423/SRR3182423_1.fastq.gz .
注意：上述是apera的下载格式，最后面的.代表保存的路径是本目录，一定不能省略，不然会报错。
网络有时候会提示SSH或者UDP错误，可能是服务器的33001端口没有开，打开这个端口即可。
上面这种下载时候，搜索框需要输入SRA号，最近有一个项目的SRA号达到了1785个显然不能收到一个个搜索地址。
新的工具ffq可以用来批量获取下载地址。
ffq github
直接pip安装即可 pip install ffq
ffq 参数说明
-o 输出文件名称
-t 指定输入的编号类型，默认是：SRR,可用选项：SRR, ERR, DRR, SRP, ERP, DRP, GSE, DOI
ffq输出的结果格式是json
#获取一个SRR的下载地址
ffq SRR12455819
输出信息如下：直接包括文章的标题摘要，数据类型，数据下载地址和md5
{
"SRR12455819": {
"accession": "SRR12455819",
"experiment": {
"accession": "SRX8950230",
"title": "Illumina HiSeq 4000 sequencing; GH0202",
"platform": "ILLUMINA",
"instrument": "Illumina HiSeq 4000"
},
"study": {
"accession": "SRP275570",
"title": "Genebank resequencing of tetraploid cottons",
"abstract": "Modern cultivated tetraploid cottons contain two species, Gossypium barbadense and Gossypium hirsutum. Among them, G. hirsutum is the significant species cultivated worldwide and contributes more than 90% natural fiber for the industry. To completely reveal the genetic diversity and population divergence within cultivated tetraploid cotton, we resequenced more than 1,700 accessions, which mostly included the South-China landraces, the elite introgression lines, and the obsoleted historic varieties. After integrating them with the major public resequencing data (PRJNA257154, PRJNA336461, PRJNA375965, PRJNA399050, and PRJNA414461), we obtained a whole Genebank variation map contained 3,248 tetraploid cottons (included 2,922 G. hirsutum). This variation map covered more than 1/3 of current G. hirsutum Genebank in China (~9,000 accessions), which could represent the genetic diversity of G. hirsutum."
},
"sample": {
"accession": "SRS7128615",
"title": "Gossypium hirsutum",
"organism": "Gossypium hirsutum",
"attributes": {
"isolate": "not applicable",
"cultivar": "GH0202",
"ecotype": "not applicable",
"age": "not applicable",
"dev_stage": "not applicable",
"geo_loc_name": "not applicable",
"tissue": "Leaf",
"BioSampleModel": "Plant",
"ENA-SPOT-COUNT": "231824444",
"ENA-BASE-COUNT": "34773666600",
"ENA-FIRST-PUBLIC": "2020-12-09",
"ENA-LAST-UPDATE": "2020-12-09"
}
},
"title": "Illumina HiSeq 4000 sequencing; GH0202",
"files": [
{
"url": "ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR124/019/SRR12455819/SRR12455819.fastq.gz",
"md5": "056333e1c9fba8b9ff930496073cbb95",
"size": "7552277314"
}
]
}
}
获取的地址ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR124/019/SRR12455819/SRR12455819.fastq.gz和上面使用aspera下载的地址[email protected]:vol1/fastq/SRR124/019/SRR12455819/SRR12455819.fastq.gz只有前面的ftp://ftp修改为era-fasp@fasp即可。
批量获取下载地址输出到1个json文件
ffq -o SRA_Acc.json SRR12455818 SRR12455819 SRR12455820 SRR12455821 SRR12455822 SRR12455823 SRR12455824
所有的结果输出到SRA_Acc.json文件
批量获取下载地址，并自动生成ascp的下载命令行
#输出文件在NCBI.json
ffq -o NCBI.json SRR12455818 SRR12455819 SRR12455820 SRR12455821 SRR12455822 SRR12455823 SRR12455824
json2tab.py脚本内容如下：
此脚本目前适配的是SRR和ERP的输出，其他的输出如果不一致，需要自己手动修改。
#!/usr/bin/python3
##用法：python3 json2tab.py NCBI.json
#输入文件是ffq输出的json文件
#输出是"projectID","sampleID","fq_url","size","md5","organism"
import sys
import pandas as pd
import json
#jsonfile = "NCBI.json"
jsonfile = sys.argv[1]
#filename = jsonfile.split('.')[0]
#解析输入的json文件
with open(jsonfile,"r") as load_f:
load_dict = json.load(load_f)
#输出文件头部信息
#print("projectID","sampleID","fq1_url","fq2_url","size_1","size_1","md5_1","md5_2")
for i in load_dict.keys(): #i是项目的编号，当有多个NCBI的号的时候
idlist=[]
for j in load_dict[i].keys():
idlist.append(j) #把每一个的keys输出到idlist,如果keys里有files选项，则直接输出，如果有runs，需要多解析一层
if "files" in idlist:
data_ID=load_dict[i] #此时有files直接解析即可
for list_url in data_ID['files']:
print(load_dict[i]['accession'],data_ID['accession'],list_url['url'],list_url['size'],list_url['md5'],data_ID['sample']['organism'],sep='\t')
elif "runs" in idlist: #此时是需要多解析一层
for id in load_dict[i]['runs'].keys(): #id是每个项目里，对应的文件的号
data_ID=load_dict[i]['runs'][id]
for list_url in data_ID['files']:
print(load_dict[i]['accession'],data_ID['accession'],list_url['url'],list_url['size'],list_url['md5'],data_ID['sample']['organism'],sep='\t')
else: #此时可能是其他情况，自行解决
print("不符合已知的字段，需要自行解析如下内容：")
print(load_dict[i])
脚本输出的格式如下：
SRR12455818 SRR12455818 ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR124/018/SRR12455818/SRR12455818.fastq.gz 5425719230 67a1ba5cafd70f6137916589bf9bb437 Gossypium hirsutum
SRR12455819 SRR12455819 ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR124/019/SRR12455819/SRR12455819.fastq.gz 7552277314 056333e1c9fba8b9ff930496073cbb95 Gossypium hirsutum
直接生成转换后的下载地址
python3 json2tab.py NCBI.json >info.list
cat info.list|awk '{print $3}'|sed 's/ftp:\/\/ftp.sra.ebi.ac.uk/ascp -QT -l 30m -P33001 -i \$HOME\/.aspera\/connect\/etc\/asperaweb_id_dsa.openssh era-fasp\@fasp.sra.ebi.ac.uk:/g;s/$/ ./g' >download.sh
nohup bash download.sh & #直接即可下载
download.sh的内容如下：
ascp -QT -l 30m -P33001 -i $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh [email protected]:/vol1/fastq/SRR124/018/SRR12455818/SRR12455818.fastq.gz .
ascp -QT -l 30m -P33001 -i $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh [email protected]:/vol1/fastq/SRR124/019/SRR12455819/SRR12455819.fastq.gz .
md5检测下载的数据是否完整
awk '{print $3,$5}' info.list|rev|cut -d "/" -f1|rev|awk '{print $2,$1}' >md5.txt
md5sum -c md5.txt
如果全部ok,就说明数据没问题。
下面内容可忽略
批量获取下载地址分别输出到json文件
ffq -o srr_split --split SRR12455818 SRR12455819 SRR12455820 SRR12455821 SRR12455822 SRR12455823 SRR12455824
所有的结果输出到srr_split的目录里，分别是SRR12455818.json …… SRR12455824.json
不推荐 下载原始的SRA格式，NCBI数据库
在NCBI数据库会获取到Accession List里面是SRR的列表。
使用prefetch下载
##单行手动下载
prefetch SRR3182423
##批量自动化下载
cat SRR_Acc_List.txt | while read id
do
prefetch ${id} -O ./
done
数据格式
SRA数据格式，是NCBI数据库的格式，我们下载之后，需要自己手动转换成fq格式。
SRA的数据是每一个SRR数据，是一个文件夹。