KeepLearningBigData

基因数据处理100之bwamem算法处理100万条paired-reads数据GRCH38chr1L3556522N1000000L100paired12

运行记录：

hadoop@Master:~/xubo/ref/GRCH38L1Index/pe$ bwa mem ../GRCH38chr1L3556522.fasta GRCH38chr1L3556522N1000000L100paired1.fastq GRCH38chr1L3556522N1000000L100paired2.fastq >GRCH38chr1L3556522N1000000L100paired12.sam
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[M::process] read 100000 sequences (10000000 bp)...
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44602, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (498.79, 49.84)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 38.011 CPU sec, 37.988 real sec
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44446, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 498, 532)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (331, 666)
[M::mem_pestat] mean and std.dev: (498.57, 49.70)
[M::mem_pestat] low and high boundaries for proper pairs: (264, 733)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 38.287 CPU sec, 38.187 real sec
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44553, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (499.09, 49.89)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 37.447 CPU sec, 37.335 real sec
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44490, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (498.78, 49.86)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 37.639 CPU sec, 37.529 real sec
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44537, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (499.18, 49.73)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 37.181 CPU sec, 37.063 real sec
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44493, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (466, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (332, 667)
[M::mem_pestat] mean and std.dev: (499.19, 49.67)
[M::mem_pestat] low and high boundaries for proper pairs: (265, 734)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 37.731 CPU sec, 37.645 real sec
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44395, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (499.19, 49.91)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 38.868 CPU sec, 38.783 real sec
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44428, 1, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (498.75, 49.92)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 37.643 CPU sec, 39.199 real sec
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44455, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (498.90, 49.75)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 38.059 CPU sec, 38.147 real sec
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44497, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (498.75, 49.95)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 38.032 CPU sec, 38.342 real sec
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44452, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (466, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (332, 667)
[M::mem_pestat] mean and std.dev: (499.25, 49.76)
[M::mem_pestat] low and high boundaries for proper pairs: (265, 734)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 38.383 CPU sec, 41.257 real sec
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44624, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (498.90, 49.83)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 37.706 CPU sec, 37.771 real sec
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44548, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 498, 532)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (331, 666)
[M::mem_pestat] mean and std.dev: (498.50, 49.73)
[M::mem_pestat] low and high boundaries for proper pairs: (264, 733)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 38.009 CPU sec, 38.306 real sec
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44369, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (466, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (332, 667)
[M::mem_pestat] mean and std.dev: (499.29, 49.65)
[M::mem_pestat] low and high boundaries for proper pairs: (265, 734)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 38.027 CPU sec, 38.429 real sec
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44525, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 532)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (331, 666)
[M::mem_pestat] mean and std.dev: (498.71, 49.67)
[M::mem_pestat] low and high boundaries for proper pairs: (264, 733)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 37.977 CPU sec, 39.461 real sec
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44586, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 532)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (331, 666)
[M::mem_pestat] mean and std.dev: (498.80, 49.59)
[M::mem_pestat] low and high boundaries for proper pairs: (264, 733)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 38.294 CPU sec, 40.004 real sec
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44519, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (464, 498, 532)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (328, 668)
[M::mem_pestat] mean and std.dev: (498.21, 50.11)
[M::mem_pestat] low and high boundaries for proper pairs: (260, 736)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 37.618 CPU sec, 43.286 real sec
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44469, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (498.90, 49.89)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 38.862 CPU sec, 39.607 real sec
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44503, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (466, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (332, 667)
[M::mem_pestat] mean and std.dev: (498.94, 49.63)
[M::mem_pestat] low and high boundaries for proper pairs: (265, 734)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 38.381 CPU sec, 41.691 real sec
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44510, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 532)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (331, 666)
[M::mem_pestat] mean and std.dev: (498.66, 49.88)
[M::mem_pestat] low and high boundaries for proper pairs: (264, 733)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 37.654 CPU sec, 41.754 real sec
[main] Version: 0.7.13-r1126
[main] CMD: bwa mem ../GRCH38chr1L3556522.fasta GRCH38chr1L3556522N1000000L100paired1.fastq GRCH38chr1L3556522N1000000L100paired2.fastq
[main] Real time: 790.344 sec; CPU: 760.405 sec

flagstat：

hadoop@Master:~/xubo/ref/GRCH38L1Index/pe$ samtools flagstat GRCH38chr1L3556522N1000000L100paired12.sam 
2000000 + 0 in total (QC-passed reads + QC-failed reads)
0 + 0 secondary
0 + 0 supplementary
0 + 0 duplicates
2000000 + 0 mapped (100.00% : N/A)
2000000 + 0 paired in sequencing
1000000 + 0 read1
1000000 + 0 read2
1999960 + 0 properly paired (100.00% : N/A)
2000000 + 0 with itself and mate mapped
0 + 0 singletons (0.00% : N/A)
0 + 0 with mate mapped to a different chr
0 + 0 with mate mapped to a different chr (mapQ>=5)

参考

【1】https://github.com/xubo245/AdamLearning
【2】https://github.com/bigdatagenomics/adam/ 
【3】https://github.com/xubo245/SparkLearning
【4】http://spark.apache.org
【5】http://stackoverflow.com/questions/28166667/how-to-pass-d-parameter-or-environment-variable-to-spark-job  
【6】http://stackoverflow.com/questions/28840438/how-to-override-sparks-log4j-properties-per-driver

研究成果：

【1】 [BIBM] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Chao Wang, and Xuehai Zhou, "Distributed Gene Clinical Decision Support System Based on Cloud Computing", in IEEE International Conference on Bioinformatics and Biomedicine. (BIBM 2017, CCF B)
【2】 [IEEE CLOUD] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Xuehai Zhou. Efficient Distributed Smith-Waterman Algorithm Based on Apache Spark (CLOUD 2017, CCF-C).
【3】 [CCGrid] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Jinhong Zhou, Xuehai Zhou. DSA: Scalable Distributed Sequence Alignment System Using SIMD Instructions. (CCGrid 2017, CCF-C).
【4】more: https://github.com/xubo245/Publications

Help

If you have any questions or suggestions, please write it in the issue of this project or send an e-mail to me: [email protected]
Wechat: xu601450868
QQ: 601450868

R语言零基础基因/数据差异分析（一） Frms R语言零基础基因/数据差异分析 r语言数据分析数据可视化
文章目录介绍环境搭建软件下载结果展示基因数据下载流程基因数据处理利用GEO分析绘制拟火山图注意，本系列有连贯性，每一步都很详细，每一步都很重要，请耐心读完！！介绍本系列文主要依据真实论文制图流程，详细说明制图过程，其中包括：1.基因数据下载2.制图所需数据格式3.火山图制作流程4.聚类热图制作流程环境搭建软件下载移步至此学习结果展示基因数据处理注意删除末行注释基因数据下载流程以GSE137578基
PyVCF 变异基因数据处理 loong_XL 生信生信
PyVCF是一个用于处理VCF(VariantCallFormat)文件的python库。它提供了许多功能来读取，过滤和修改VCF文件中的变异PyVCF是一个用于读取和写入VCF格式文件的Python库。主要功能包括:1、读取VCF文件：PyVCF提供了一个vcf.Reader()函数，可以用来打开并读取VCF文件。读取后的文件可以进行遍历，每个元素是一个vcf.model._Record的实例2
云上弹性高性能计算，支持生命科学产业高速发展、降本增效阿里云开发者人工智能运维数据可视化安全 BI 数据处理调度云计算数据中心容器阿里云开发者
简介：生命科学的研究早已离不开高性能计算（HighPerformanceComputing，HPC）的辅助。从计算机辅助药物设计、疫苗研发，到基因数据处理与分析，再到提供精准医疗服务于肿瘤治疗、产前筛查等医疗技术，高性能计算HPC在生命科学研究中扮演着十分重要的角色。而云，又能如何助力产业发展，帮助产业降本增效呢？随着云计算技术服务及实践的日趋成熟，越来越多的行业通过上云实现了整个产业的转型升级，
云上弹性高性能计算，支持生命科学产业高速发展、降本增效程序员
简介：生命科学的研究早已离不开高性能计算（HighPerformanceComputing，HPC）的辅助。从计算机辅助药物设计、疫苗研发，到基因数据处理与分析，再到提供精准医疗服务于肿瘤治疗、产前筛查等医疗技术，高性能计算HPC在生命科学研究中扮演着十分重要的角色。而云，又能如何助力产业发展，帮助产业降本增效呢？image.png随着云计算技术服务及实践的日趋成熟，越来越多的行业通过上云实现了整
云上弹性高性能计算，支持生命科学产业高速发展、降本增效人工智能运维数据可视化安全bi
简介：生命科学的研究早已离不开高性能计算（HighPerformanceComputing，HPC）的辅助。从计算机辅助药物设计、疫苗研发，到基因数据处理与分析，再到提供精准医疗服务于肿瘤治疗、产前筛查等医疗技术，高性能计算HPC在生命科学研究中扮演着十分重要的角色。而云，又能如何助力产业发展，帮助产业降本增效呢？随着云计算技术服务及实践的日趋成熟，越来越多的行业通过上云实现了整个产业的转型升级，
相似度融合网络：用于聚合不同的基因数据类型：Similarity network fusion for aggregating data types on a genomic scale Ice-iron 论文笔记机器学习神经网络深度学习
论文标题：Similaritynetworkfusionforaggregatingdatatypesonagenomicscale.论文下载地址论文总结论文以计算机视觉多视图方式为启发，设计了一种图融合网络用于解决基因数据不能综合处理的困难。现有的基因数据非常丰富，有各种类型的基因数据可以利用。但现有的基因数据处理方式大多数是只利用一种基因数据，例如只使用DNA或者是只使用mRNA，不能综合所有
基因数据处理80之disease的DataProcessing KeepLearningBigData 基因数据处理
1.代码：/***@authorxubo*morecode:https://github.com/xubo245/SparkLearning*moreblog:http://blog.csdn.net/xubo245*/packageorg.gcdss.cli.diseaseimportjava.text.SimpleDateFormatimportjava.util.Dateimportorg.
基因数据处理85之adam-0.18.2无法读取0.14.0使用adamSave存储的数据 KeepLearningBigData 基因数据处理
1.介绍：cs-bwamem依赖的是adam-0.14.0里面的adamSave存储之后无法用adam-0.18.2的结果读取：2.adam-0.18.2adamsave可以用loadParquetAlignments读取。但无法用loadParquetAlignments读取adam-0.14.23.代码：packageorg.gcdss.cli.testimportjava.nio.file.
基因数据处理73之从HDFS读取fasta文件存为Adam的parquet文件 KeepLearningBigData 基因数据处理
1.GRCH38chr14：hadoop@Master:~/xubo/project/load$./load.shstart:1SLF4J:Failedtoloadclass"org.slf4j.impl.StaticLoggerBinder".SLF4J:Defaultingtono-operation(NOP)loggerimplementationSLF4J:Seehttp://www.sl
基因数据处理94之使用kmer分析SRR003161数据的kmer分布 KeepLearningBigData 基因数据处理
1.分两组(1)kmer长度为：5to21(2)kmer长度为：5to55by102.代码：packageorg.gcdss.cliimportjava.text.SimpleDateFormatimportjava.util._importorg.apache.spark._importorg.bdgenomics.adam.projections.{AlignmentRecordField,P
基因数据处理57之BWA-MEM运行single-end(1千万条100bp的reads) KeepLearningBigData 基因数据处理
```hadoop@Master:~/cloud/adam/xubo/data/GRCH38Sub/cs-bwamem$bwamemGRCH38BWAindex/GRCH38chr1L3556522.fastag38L100c10000000Nhs20.fq>g38L100c10000000Nhs20.bwamem.sam[M::bwa_idx_load_from_disk]read0ALTcon
基因数据处理75之从HDFS读取vcf文件存为Adam的parquet文件（成功） KeepLearningBigData 基因数据处理
1.参考：packageorg.bdgenomics.adam.cliclassFlattenSuiteextendsADAMFunSuite{valloader=Thread.currentThread().getContextClassLoadervalinputPath=loader.getResource("small.vcf").getPathvaloutputFile=File.cre
基因数据处理74之从HDFS读取vcf文件存为Adam的parquet文件（有问题） KeepLearningBigData 基因数据处理 spark 基因数据处理 adam
1.small.vcf:没记录2.读取：5loadtime:3287ms{"variant":{"variantErrorProbability":139,"contig":{"contigName":"1","contigLength":null,"contigMD5":null,"referenceURL":null,"assembly":null,"species":null,"refere
基因数据处理2之ftp数据快速查找 KeepLearningBigData 基因数据处理
基因数据处理2之ftp数据快速查找linux下可以用grep:curl-s"ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/current.tree"|grepNA12878运行结果：hadoop@Mcnode1:~/cloud/adam/xubo/testAdam34/TestBaiBas$curl-s"ftp://ftp.1000genomes.ebi.ac.
基因数据处理83之移动GRCH38Index到每个节点 KeepLearningBigData 基因数据处理
1.从cloud/adam移出到xubo/ref:hadoop@Master:~/cloud/adam/xubo/data/test20160310$mkdir-p~/xubo/ref/GRCH38Index/hadoop@Master:~/cloud/adam/xubo/data/test20160310$mvGCA_000001405.15_GRCh38/*~/xubo/ref/GRCH38I
基因数据处理54之bwa-mem运行paird-end（1千万条100bp的reads） KeepLearningBigData 基因数据处理 Spark问题
指令：```hadoop@Master:~/cloud/adam/xubo/data/GRCH38Sub/cs-bwamem$bwamemGRCH38BWAindex/GRCH38chr1L3556522.fastag38L100c10000000Nhs20Paired1.fqg38L100c10000000Nhs20Paired2.fq>g38L100c10000000Nhs20Paired12
基因数据处理1之mapping_to_cram KeepLearningBigData 基因数据处理
基因数据处理1之mapping_to_cram参考资料：AWorkedExampleObtainsomepublicdataWewillusethefirst100,000read-pairsfromayeastdataset.curlftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR507/SRR507778/SRR507778_1.fastq.gz|gzip-d|he
基因数据处理72之GATK安装成功 KeepLearningBigData 基因数据处理
1.下载：gitclonehttps://github.com/broadgsa/gatk-protected.git2.安装：gitcheckout3.5mvncleanpackage-DskipTests3.安装成功：[INFO]ReactorSummary:[INFO][INFO]GATKRoot..........................................SUCCES
基因数据处理77之从vcf文件中提取某条染色体的数据 KeepLearningBigData 基因数据处理
1.代码：/***@authorxubo*/packageorg.gcdss.cli.vcfimportorg.apache.spark.{SparkConf,SparkContext}/***Createdbyxuboon2016/5/23.*/objectextractGRCH38chr20vcf{defmain(args:Array[String]){valconf=newSparkConf
基因数据处理34之使用samtools和bcftools进行变异分析 KeepLearningBigData 基因数据处理
1.指令：（1）samtoolsmpileup-vfHomo_sapiens_assembly19chr20.fastaNA12878_snp_A2G_chr20_225058.sorted.bam>NA12878_snp_A2G_chr20_225058.variants或者：samtoolsmpileup-vfHomo_sapiens_assembly19chr20.fastaNA12878_
基因数据处理116之重新运行SparkBWA Yarn集群模式 KeepLearningBigData 基因数据处理
更多代码请见：https://github.com/xubo245基因数据处理系列1.解释很久没运行SparkBWA了，系统文件有点多，重新运行。2.代码：endhadoop@Master:~/disk2/xubo/project/alignment/sparkBWA$vig38L100c100000Nhs20Paired12SparkBWAYarnPartition0.shecho"start"
基因数据处理117之重新多次运行SparkBWA Yarn集群 KeepLearningBigData 基因数据处理
更多代码请见：https://github.com/xubo245基因数据处理系列1.解释重新运行，跟换了文件地址后来终端，需要运行其他的2.代码：hadoop@Master:~/disk2/xubo/project/alignment/sparkBWA$visparkBWA.shforjin10000100000100000010000000doforiin50100doecho$iecho$j
基因数据处理112之运行gcdss的avocado编译识别报错getRecordGroupSample空指针异常解决办法 KeepLearningBigData 基因数据处理
前面一片已经讲了遇到的问题，之前也遇到过：基因数据处理31之avocado运行avocado-cli中的avocado问题3-变异识别找不到RecordGroupSample（null）http://blog.csdn.net/xubo245/article/details/51525241解决办法：在读入的sam/bam/adam数据进行判定，如果RecordGroup等数据为空，则加上默认值在
基因数据处理110之批量测试SparkBWA和CS-bwamem算法 KeepLearningBigData 基因数据处理
1.总的脚本hadoop@Master:~/xubo/project/alignment$catrunSparkBWAAndCsbwamem.shecho"startsparkBWA"./sparkBWA/sparkBWA.shecho"startCsbwamemalgin"./cs-bwamem/csbwamemAlign.shecho"startCsbwamemmerge"./cs-bwame
基因数据处理106之bwa-mem运行paird-end（1千万条100bp的reads g38L100c10000000Nhs20Paired12） KeepLearningBigData 基因数据处理
脚本：hadoop@Master:~/xubo/project/alignment/sparkBWA$catg38L100c10000000Nhs20Paired12Bwamem.shecho"start"startTime4=`date+"%s.%N"`time4=`date+"%Y%m%d%H%M%S"`#spark-submit--classorg.apache.spark.examples
基因数据处理113之对avocado识别的SparkBWA变异数据进行疾病分析_ KeepLearningBigData 基因数据处理
（一）问题问题1：avocado数据读取：avocado存到磁盘是：RDD[Genotype]valprocessedGenotypes:RDD[Genotype]=postProcessVariants(calledVariants,stats).flatMap(variantContext=>variantContext.genotypes)//savevariantstooutputfile
基因数据处理109之SparkBWA运行成功的部分log KeepLearningBigData 基因数据处理
数据来自：http://219.219.220.149:18080/history/application_1466866953605_0001/executors/http://219.219.220.248:8042/node/containerlogs/container_1466866953605_0001_01_000003/hadoop/stderr/?start=0SLF4J:Cla
Spark问题14之Spark stage retry问题 KeepLearningBigData Spark问题
更多代码请见：https://github.com/xubo245基因数据处理系列之SparkBWA1.解释1.1简述当partitions超过节点数量的时候Lostexecutor的问题，已经提交到SparkBWA中，https://github.com/citiususc/SparkBWA/issues/35另外发现，tmp里面有临时文件没有删除，而且stageretry未解决2.记录完整报错
基因数据处理102之SparkBWA本地运行100万条paired-reads实例 KeepLearningBigData 基因数据处理
脚本：spark-submit--classSparkBWA\--masterlocal\--archivesbwa.zip\SparkBWA.jar\-algorithmmem-readspaired\-index/home/hadoop/xubo/ref/GRCH38L1Index/GRCH38chr1L3556522.fasta\-partitions3\/xubo/alignment/sp
基因数据处理101之SparkBWA本地运行配置和实例 KeepLearningBigData 基因数据处理
1.修改Makefile.common：将LIBBWA_LIBS=-lrt改为LIBBWA_LIBS=-lrt-lz不然会报错误【5】2.make之后修改java.library.path步骤：vi/etc/profile加入exportLD_LIBRARY_PATH=/home/hadoop/xubo/tools/SparkBWA/build:$LD_LIBRARY_PATH使生效：source
PHP，安卓，UI，java，linux视频教程合集 cocos2d-x小菜 java UI linux PHP android
╔-----------------------------------╗┆
zookeeper admin 笔记 braveCS zookeeper
Required Software 1) JDK>=1.6 2)推荐使用ensemble的ZooKeeper(至少3台)，并run on separate machines 3)在Yahoo!，zk配置在特定的RHEL boxes里，2个cpu，2G内存，80G硬盘数据和日志目录 1)数据目录里的文件是zk节点的持久化备份，包括快照和事务日
Spring配置多个连接池 easterfly spring
项目中需要同时连接多个数据库的时候，如何才能在需要用到哪个数据库就连接哪个数据库呢？ Spring中有关于dataSource的配置： <bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" &nb
Mysql 171815164 mysql
例如，你想myuser使用mypassword从任何主机连接到mysql服务器的话。 GRANT ALL PRIVILEGES ON *.* TO 'myuser'@'%'IDENTIFIED BY 'mypassword' WI TH GRANT OPTION; 如果你想允许用户myuser从ip为192.168.1.6的主机连接到mysql服务器，并使用mypassword作
CommonDAO（公共/基础DAO） g21121 DAO
好久没有更新博客了，最近一段时间工作比较忙，所以请见谅，无论你是爱看呢还是爱看呢还是爱看呢，总之或许对你有些帮助。 DAO(Data Access Object)是一个数据访问（顾名思义就是与数据库打交道）接口，DAO一般在业
直言有讳永夜-极光感悟随笔
1.转载地址:http://blog.csdn.net/jasonblog/article/details/10813313 精华: “直言有讳”是阿里巴巴提倡的一种观念，而我在此之前并没有很深刻的认识。为什么呢？就好比是读书时候做阅读理解，我喜欢我自己的解读，并不喜欢老师给的意思。在这里也是。我自己坚持的原则是互相尊重，我觉得阿里巴巴很多价值观其实是基本的做人
安装CentOS 7 和Win 7后，Win7 引导丢失随便小屋 centos
一般安装双系统的顺序是先装Win7，然后在安装CentOS，这样CentOS可以引导WIN 7启动。但安装CentOS7后，却找不到Win7 的引导，稍微修改一点东西即可。一、首先具有root 的权限。即进入Terminal后输入命令su，然后输入密码即可二、利用vim编辑器打开/boot/grub2/grub.cfg文件进行修改 v
Oracle备份与恢复案例 aijuans oracle
Oracle备份与恢复案例一. 理解什么是数据库恢复当我们使用一个数据库时，总希望数据库的内容是可靠的、正确的，但由于计算机系统的故障（硬件故障、软件故障、网络故障、进程故障和系统故障）影响数据库系统的操作，影响数据库中数据的正确性，甚至破坏数据库，使数据库中全部或部分数据丢失。因此当发生上述故障后，希望能重构这个完整的数据库，该处理称为数据库恢复。恢复过程大致可以分为复原(Restore)与
JavaEE开源快速开发平台G4Studio v5.0发布無為子
我非常高兴地宣布,今天我们最新的JavaEE开源快速开发平台G4Studio_V5.0版本已经正式发布。访问G4Studio网站 http://www.g4it.org 2013-04-06 发布G4Studio_V5.0版本功能新增 (1). 新增了调用Oracle存储过程返回游标，并将游标映射为Java List集合对象的标
Oracle显示根据高考分数模拟录取百合不是茶 PL/SQL编程 oracle例子模拟高考录取学习交流
题目要求: 1,创建student表和result表 2,pl/sql对学生的成绩数据进行处理 3,处理的逻辑是根据每门专业课的最低分线和总分的最低分数线自动的将录取和落选 1,创建student表,和result表学生信息表; create table student( student_id number primary key,--学生id
优秀的领导与差劲的领导 bijian1013 领导管理团队
责任优秀的领导：优秀的领导总是对他所负责的项目担负起责任。如果项目不幸失败了，那么他知道该受责备的人是他自己，并且敢于承认错误。差劲的领导：差劲的领导觉得这不是他的问题，因此他会想方设法证明是他的团队不行，或是将责任归咎于团队中他不喜欢的那几个成员身上。努力工作优秀的领导：团队领导应该是团队成员的榜样。至少，他应该与团队中的其他成员一样努力工作。这仅仅因为他
js函数在浏览器下的兼容 Bill_chen jquery 浏览器 IE DWR ext
做前端开发的工程师，少不了要用FF进行测试，纯js函数在不同浏览器下，名称也可能不同。对于IE6和FF，取得下一结点的函数就不尽相同： IE6：node.nextSibling,对于FF是不能识别的； FF：node.nextElementSibling,对于IE是不能识别的；兼容解决方式：var Div = node.nextSibl
【JVM四】老年代垃圾回收：吞吐量垃圾收集器(Throughput GC) bit1129 垃圾回收
吞吐量与用户线程暂停时间衡量垃圾回收算法优劣的指标有两个：吞吐量越高，则算法越好暂停时间越短，则算法越好首先说明吞吐量和暂停时间的含义。垃圾回收时，JVM会启动几个特定的GC线程来完成垃圾回收的任务，这些GC线程与应用的用户线程产生竞争关系，共同竞争处理器资源以及CPU的执行时间。GC线程不会对用户带来的任何价值，因此，好的GC应该占
J2EE监听器和过滤器基础白糖_ J2EE
Servlet程序由Servlet，Filter和Listener组成，其中监听器用来监听Servlet容器上下文。监听器通常分三类：基于Servlet上下文的ServletContex监听，基于会话的HttpSession监听和基于请求的ServletRequest监听。 ServletContex监听器 ServletContex又叫application
博弈AngularJS讲义(16) - 提供者 boyitech js AngularJS api Angular Provider
Angular框架提供了强大的依赖注入机制，这一切都是有注入器(injector)完成. 注入器会自动实例化服务组件和符合Angular API规则的特殊对象，例如控制器，指令，过滤器动画等。那注入器怎么知道如何去创建这些特殊的对象呢？ Angular提供了5种方式让注入器创建对象，其中最基础的方式就是提供者(provider), 其余四种方式(Value, Fac
java-写一函数f(a,b)，它带有两个字符串参数并返回一串字符，该字符串只包含在两个串中都有的并按照在a中的顺序。 bylijinnan java
public class CommonSubSequence { /** * 题目：写一函数f(a,b)，它带有两个字符串参数并返回一串字符，该字符串只包含在两个串中都有的并按照在a中的顺序。 * 写一个版本算法复杂度O(N^2)和一个O(N) 。 * * O(N^2)：对于a中的每个字符，遍历b中的每个字符，如果相同，则拷贝到新字符串中。 * O(
sqlserver 2000 无法验证产品密钥 Chen.H sql windows SQL Server Microsoft
在 Service Pack 4 (SP 4), 是运行 Microsoft Windows Server 2003、 Microsoft Windows Storage Server 2003 或 Microsoft Windows 2000 服务器上您尝试安装 Microsoft SQL Server 2000 通过卷许可协议 (VLA) 媒体。这样做, 收到以下错误信息CD KEY的 SQ
[新概念武器]气象战争 comsci
气象战争的发动者必须是拥有发射深空航天器能力的国家或者组织.... 原因如下: 地球上的气候变化和大气层中的云层涡旋场有密切的关系,而维持一个在大气层某个层次
oracle 中 rollup、cube、grouping 使用详解 daizj oracle grouping rollup cube
oracle 中 rollup、cube、grouping 使用详解 -- 使用oracle 样例表演示转自namesliu -- 使用oracle 的样列库，演示 rollup, cube, grouping 的用法与使用场景 --- ROLLUP ，为了理解分组的成员数量，我增加了分组的计数 COUNT(SAL)
技术资料汇总分享 Dead_knight 技术资料汇总分享
本人汇总的技术资料，分享出来，希望对大家有用。 http://pan.baidu.com/s/1jGr56uE 资料主要包含： Workflow->工作流相关理论、框架(OSWorkflow、JBPM、Activiti、fireflow...) Security->java安全相关资料(SSL、SSO、SpringSecurity、Shiro、JAAS...) Ser
初一下学期难记忆单词背诵第一课 dcj3sjt126com english word
could 能够 minute 分钟 Tuesday 星期二 February 二月 eighteenth 第十八 listen 听 careful 小心的，仔细的 short 短的 heavy 重的 empty 空的 certainly 当然 carry 携带；搬运 tape 磁带 basket 蓝子 bottle 瓶 juice 汁，果汁 head 头；头部
截取视图的图片, 然后分享出去 dcj3sjt126com OS Objective-C
OS 7 has a new method that allows you to draw a view hierarchy into the current graphics context. This can be used to get an UIImage very fast. I implemented a category method on UIView to get the vi
MySql重置密码 fanxiaolong MySql重置密码
方法一: 在my.ini的[mysqld]字段加入： skip-grant-tables 重启mysql服务，这时的mysql不需要密码即可登录数据库然后进入mysql mysql>use mysql; mysql>更新 user set password=password('新密码') WHERE User='root'; mysq
Ehcache（03）——Ehcache中储存缓存的方式 234390216 ehcache MemoryStore DiskStore 存储驱除策略
Ehcache中储存缓存的方式目录 1 堆内存（MemoryStore） 1.1 指定可用内存 1.2 驱除策略 1.3 元素过期 2 &nbs
spring mvc中的@propertysource jackyrong spring mvc
在spring mvc中，在配置文件中的东西，可以在java代码中通过注解进行读取了： @PropertySource 在spring 3.1中开始引入比如有配置文件 config.properties mongodb.url=1.2.3.4 mongodb.db=hello 则代码中 @PropertySource(&
重学单例模式 lanqiu17 单例 Singleton 模式
最近在重新学习设计模式，感觉对模式理解更加深刻。觉得有必要记下来。第一个学的就是单例模式，单例模式估计是最好理解的模式了。它的作用就是防止外部创建实例，保证只有一个实例。单例模式的常用实现方式有两种，就人们熟知的饱汉式与饥汉式，具体就不多说了。这里说下其他的实现方式静态内部类方式: package test.pattern.singleton.statics; publ
.NET开源核心运行时，且行且珍惜 netcome java .net 开源
背景 2014年11月12日，ASP.NET之父、微软云计算与企业级产品工程部执行副总裁Scott Guthrie，在Connect全球开发者在线会议上宣布，微软将开源全部.NET核心运行时，并将.NET 扩展为可在 Linux 和 Mac OS 平台上运行。.NET核心运行时将基于MIT开源许可协议发布，其中将包括执行.NET代码所需的一切项目——CLR、JIT编译器、垃圾收集器（GC）和核心
使用oscahe缓存技术减少与数据库的频繁交互 Everyday都不同 Web 高并发 oscahe缓存
此前一直不知道缓存的具体实现，只知道是把数据存储在内存中，以便下次直接从内存中读取。对于缓存的使用也没有概念，觉得缓存技术是一个比较”神秘陌生“的领域。但最近要用到缓存技术，发现还是很有必要一探究竟的。缓存技术使用背景：一般来说，对于web项目，如果我们要什么数据直接jdbc查库好了，但是在遇到高并发的情形下，不可能每一次都是去查数据库，因为这样在高并发的情形下显得不太合理——
Spring+Mybatis 手动控制事务 toknowme mybatis
@Override public boolean testDelete(String jobCode) throws Exception { boolean flag = false; &nbs
菜鸟级的android程序员面试时候需要掌握的知识点 xp9802 android
熟悉Android开发架构和API调用掌握APP适应不同型号手机屏幕开发技巧熟悉Android下的数据存储熟练Android Debug Bridge Tool 熟练Eclipse/ADT及相关工具熟悉Android框架原理及Activity生命周期熟练进行Android UI布局熟练使用SQLite数据库；熟悉Android下网络通信机制，S

基因数据处理100之bwamem算法处理100万条paired-reads数据GRCH38chr1L3556522N1000000L100paired12

参考

研究成果：

Help

你可能感兴趣的:(基因数据处理)