NCBI NR文件下载和批量md5检验

【1】 Aspera下载NCBI数据

单个文件下载命令：

ascp -i "C:\Users\Administrator\AppData\Local\Programs\Aspera\Aspera Connect\etc\asperaweb_id_dsa.openssh" -k 1 -T -l 200m [email protected]:/sra/sra-instant/reads/ByRun/sra/SRR/SRR121/SRR1218151/SRR1218151.sra E:\aspera

多个文件下载命令：

ascp -QT -k1 -l 200m -i "C:\Users\Administrator\AppData\Local\Programs\Aspera\Aspera Connect\etc\asperaweb_id_dsa.putty" --mode=recv --host=ftp-trace.ncbi.nlm.nih.gov --user=anonftp --file-list=E:\aspera\filelist.txt H:\NCBI

• 下载文件列表存放在文件E:\aspera\filelist.txt
• windows CMD路径中如果有空格Aspera Connect，全路径用双引号包起来

硬件所限，只能下载处理好的BLAST db文件nr.00.tar.gz~nr.113.tar.gz，共有文件114个

无标题.png

【2】批量校验

100多个大文件，每个都有独立的md5校验码，手动校验太费劲。也没找到其他批量校验工具，就自己写了个python脚本（新手上路，代码不是很python）。代码如下：

#!/usr/bin/env python           
#coding : utf-8 

import hashlib
import linecache
#import os #本来想通过OS调用外部md5程序，路径搞不定，放弃了
import subprocess #subprocess调用测试成功，下面注释掉的第二部分

count = 0
wrong_c = 0
fail_list = [] 
for i in range(0,113):
    m = hashlib.md5()
    if i<10:
        filename1 = "I:\\NCBI\\nr.0"+str(i)+".tar.gz"
        filename2 = "I:\\NCBI\\nr.0"+str(i)+".tar.gz.md5"
    else:
        filename1 = "I:\\NCBI\\nr."+str(i)+".tar.gz"
        filename2 = "I:\\NCBI\\nr."+str(i)+".tar.gz.md5"
    with open(filename1,'rb') as f:
        for line in f:
            m.update(line)
    real_md5 = m.hexdigest() #print(m.hexdigest()) #47a6b079cc33a4f312786b46e61e0305
    print(filename1 + " is verifying.......")
    print(real_md5)
    """
    md5_cmd = "D:\\md5\\md5.exe -n" + filename1
    rs = os.popen(md5_cmd).read()
    os_md5 = rs.strip().lower()
    print(os_md5)
    """
    '''
    obj = subprocess.Popen(['D:\\md5\\md5.exe','-n',filename1], shell = True, stdin=subprocess.PIPE, stdout=subprocess.PIPE ,stderr=subprocess.PIPE)
    raw_md5 = obj.stdout.read()
    str_md5 = raw_md5.strip().decode("utf-8").lower()
    print(str_md5)
    '''
    the_line = linecache.getline(filename2, 1)
    ori_md5 = the_line.split(" ")[0].strip()
    print(ori_md5)
    if real_md5 == ori_md5:
        count += 1
        print("The file nr."+str(i)+".tar.gz is OK!")
    else:
        wrong_c += 1
        fail_list.append("nr."+str(i)+".tar.gz")
        print("The file nr."+str(i)+".tar.gz has been damaged!")
    sum_files = count + wrong_c
    print("Validation of %d files in total, %d passed, %d failed!" %         (sum_files, count, wrong_c))
if not fail_list:
    print("Go on analyzing with all the files!")
else:
    print("Failed files include:")
    print(",".join(str(i) for i in fail_list))