KeepLearningBigData
KeepLearningBigData

基因数据处理102之SparkBWA本地运行100万条paired-reads实例

脚本：

spark-submit --class SparkBWA \
--master local \
--archives bwa.zip \
SparkBWA.jar \
-algorithm mem -reads paired \
-index /home/hadoop/xubo/ref/GRCH38L1Index/GRCH38chr1L3556522.fasta \
-partitions 3 \
/xubo/alignment/sparkBWA/GRCH38chr1L3556522N1000000L100paired1.fastq /xubo/alignment/sparkBWA/GRCH38chr1L3556522N1000000L100paired2.fastq \
/xubo/alignment/output/sparkBWA/GRCH38chr1L3556522N1000000L100paired12

运行记录：

hadoop@Master:~/xubo/tools/SparkBWA/build$ ./pairedGRCH38L1N1000000Local.sh             
[Stage 3:>                                                          (0 + 1) / 3][Java_BwaJni_bwa_1jni] Arg 0 'bwa'
[Java_BwaJni_bwa_1jni] Algorithm found 1 'mem'
[Java_BwaJni_bwa_1jni] Arg 1 'mem'
[Java_BwaJni_bwa_1jni] Filename parameter -f found 2 '-f'
[Java_BwaJni_bwa_1jni] Arg 2 '-f'
[Java_BwaJni_bwa_1jni] Filename found 3 '/home/hadoop/cloud/workspace/tmpSparkBWA_GRCH38chr1L3556522N1000000L100paired1.fastq-3-NoSort-local-1466761536651-0.sam'
[Java_BwaJni_bwa_1jni] Arg 3 '/home/hadoop/cloud/workspace/tmpSparkBWA_GRCH38chr1L3556522N1000000L100paired1.fastq-3-NoSort-local-1466761536651-0.sam'
[Java_BwaJni_bwa_1jni] Arg 4 '/home/hadoop/xubo/ref/GRCH38L1Index/GRCH38chr1L3556522.fasta'
[Java_BwaJni_bwa_1jni] Arg 5 '/home/hadoop/cloud/workspace/tmplocal-1466761536651-RDD0_1'
[Java_BwaJni_bwa_1jni] Arg 6 '/home/hadoop/cloud/workspace/tmplocal-1466761536651-RDD0_2'
[Java_BwaJni_bwa_1jni] option[0]: bwa.
[Java_BwaJni_bwa_1jni] option[1]: mem.
[Java_BwaJni_bwa_1jni] option[2]: /home/hadoop/xubo/ref/GRCH38L1Index/GRCH38chr1L3556522.fasta.
[Java_BwaJni_bwa_1jni] option[3]: /home/hadoop/cloud/workspace/tmplocal-1466761536651-RDD0_1.
[Java_BwaJni_bwa_1jni] option[4]: /home/hadoop/cloud/workspace/tmplocal-1466761536651-RDD0_2.
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[M::process] read 100000 sequences (10000000 bp)...
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44550, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (499.07, 49.90)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 40.231 CPU sec, 40.252 real sec
[M::process] read 100000 sequences (10000000 bp)...
[Stage 3:>                                                          (0 + 1) / 3][M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44518, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (498.90, 49.73)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 40.499 CPU sec, 40.241 real sec
[M::process] read 100000 sequences (10000000 bp)...
[Stage 3:>                                                          (0 + 1) / 3][M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44507, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 532)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (331, 666)
[M::mem_pestat] mean and std.dev: (498.72, 49.73)
[M::mem_pestat] low and high boundaries for proper pairs: (264, 733)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 40.312 CPU sec, 40.061 real sec
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44480, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (499.20, 49.64)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 40.590 CPU sec, 40.329 real sec
[M::process] read 100000 sequences (10000000 bp)...
[Stage 3:>                                                          (0 + 1) / 3][M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44530, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (466, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (332, 667)
[M::mem_pestat] mean and std.dev: (499.18, 49.79)
[M::mem_pestat] low and high boundaries for proper pairs: (265, 734)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 40.252 CPU sec, 39.980 real sec
[M::process] read 66664 sequences (6666400 bp)...
[Stage 3:>                                                          (0 + 1) / 3][M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44367, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (499.10, 49.91)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 40.689 CPU sec, 40.409 real sec
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 29620, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 498, 532)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (331, 666)
[M::mem_pestat] mean and std.dev: (498.56, 49.76)
[M::mem_pestat] low and high boundaries for proper pairs: (264, 733)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 66664 reads in 27.274 CPU sec, 27.130 real sec
[main] Version: 0.7.12-r1044
[main] CMD: bwa mem /home/hadoop/xubo/ref/GRCH38L1Index/GRCH38chr1L3556522.fasta /home/hadoop/cloud/workspace/tmplocal-1466761536651-RDD0_1 /home/hadoop/cloud/workspace/tmplocal-1466761536651-RDD0_2
[main] Real time: 282.681 sec; CPU: 346.015 sec
[Java_BwaJni_bwa_1jni] Return code from BWA 0.
[Stage 3:===================>                                       (1 + 1) / 3][Java_BwaJni_bwa_1jni] Arg 0 'bwa'
[Java_BwaJni_bwa_1jni] Algorithm found 1 'mem'
[Java_BwaJni_bwa_1jni] Arg 1 'mem'
[Java_BwaJni_bwa_1jni] Filename parameter -f found 2 '-f'
[Java_BwaJni_bwa_1jni] Arg 2 '-f'
[Java_BwaJni_bwa_1jni] Filename found 3 '/home/hadoop/cloud/workspace/tmpSparkBWA_GRCH38chr1L3556522N1000000L100paired1.fastq-3-NoSort-local-1466761536651-1.sam'
[Java_BwaJni_bwa_1jni] Arg 3 '/home/hadoop/cloud/workspace/tmpSparkBWA_GRCH38chr1L3556522N1000000L100paired1.fastq-3-NoSort-local-1466761536651-1.sam'
[Java_BwaJni_bwa_1jni] Arg 4 '/home/hadoop/xubo/ref/GRCH38L1Index/GRCH38chr1L3556522.fasta'
[Java_BwaJni_bwa_1jni] Arg 5 '/home/hadoop/cloud/workspace/tmplocal-1466761536651-RDD1_1'
[Java_BwaJni_bwa_1jni] Arg 6 '/home/hadoop/cloud/workspace/tmplocal-1466761536651-RDD1_2'
[Java_BwaJni_bwa_1jni] option[0]: bwa.
[Java_BwaJni_bwa_1jni] option[1]: mem.
[Java_BwaJni_bwa_1jni] option[2]: /home/hadoop/xubo/ref/GRCH38L1Index/GRCH38chr1L3556522.fasta.
[Java_BwaJni_bwa_1jni] option[3]: /home/hadoop/cloud/workspace/tmplocal-1466761536651-RDD1_1.
[Java_BwaJni_bwa_1jni] option[4]: /home/hadoop/cloud/workspace/tmplocal-1466761536651-RDD1_2.
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[M::process] read 100000 sequences (10000000 bp)...
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44527, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 498, 532)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (331, 666)
[M::mem_pestat] mean and std.dev: (498.57, 49.60)
[M::mem_pestat] low and high boundaries for proper pairs: (264, 733)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 37.971 CPU sec, 37.765 real sec
[M::process] read 100000 sequences (10000000 bp)...
[Stage 3:===================>                                       (1 + 1) / 3][M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44541, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (498.65, 49.71)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 38.467 CPU sec, 38.186 real sec
[M::process] read 100000 sequences (10000000 bp)...
[Stage 3:===================>                                       (1 + 1) / 3][M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44503, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (498.76, 49.80)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 39.133 CPU sec, 38.846 real sec
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44619, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (498.90, 49.93)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 37.928 CPU sec, 37.648 real sec
[M::process] read 100000 sequences (10000000 bp)...
[Stage 3:===================>                                       (1 + 1) / 3][M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44326, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (498.85, 49.90)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 38.490 CPU sec, 38.212 real sec
[M::process] read 66668 sequences (6666800 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44544, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 532)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (331, 666)
[M::mem_pestat] mean and std.dev: (498.79, 49.59)
[M::mem_pestat] low and high boundaries for proper pairs: (264, 733)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[Stage 3:===================>                                       (1 + 1) / 3][M::mem_process_seqs] Processed 100000 reads in 38.020 CPU sec, 37.772 real sec
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 29608, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (498.91, 50.00)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 66668 reads in 25.369 CPU sec, 25.211 real sec
[main] Version: 0.7.12-r1044
[main] CMD: bwa mem /home/hadoop/xubo/ref/GRCH38L1Index/GRCH38chr1L3556522.fasta /home/hadoop/cloud/workspace/tmplocal-1466761536651-RDD1_1 /home/hadoop/cloud/workspace/tmplocal-1466761536651-RDD1_2
[main] Real time: 263.547 sec; CPU: 608.818 sec
[Java_BwaJni_bwa_1jni] Return code from BWA 0.
[Stage 3:=======================================>                   (2 + 1) / 3][Java_BwaJni_bwa_1jni] Arg 0 'bwa'
[Java_BwaJni_bwa_1jni] Algorithm found 1 'mem'
[Java_BwaJni_bwa_1jni] Arg 1 'mem'
[Java_BwaJni_bwa_1jni] Filename parameter -f found 2 '-f'
[Java_BwaJni_bwa_1jni] Arg 2 '-f'
[Java_BwaJni_bwa_1jni] Filename found 3 '/home/hadoop/cloud/workspace/tmpSparkBWA_GRCH38chr1L3556522N1000000L100paired1.fastq-3-NoSort-local-1466761536651-2.sam'
[Java_BwaJni_bwa_1jni] Arg 3 '/home/hadoop/cloud/workspace/tmpSparkBWA_GRCH38chr1L3556522N1000000L100paired1.fastq-3-NoSort-local-1466761536651-2.sam'
[Java_BwaJni_bwa_1jni] Arg 4 '/home/hadoop/xubo/ref/GRCH38L1Index/GRCH38chr1L3556522.fasta'
[Java_BwaJni_bwa_1jni] Arg 5 '/home/hadoop/cloud/workspace/tmplocal-1466761536651-RDD2_1'
[Java_BwaJni_bwa_1jni] Arg 6 '/home/hadoop/cloud/workspace/tmplocal-1466761536651-RDD2_2'
[Java_BwaJni_bwa_1jni] option[0]: bwa.
[Java_BwaJni_bwa_1jni] option[1]: mem.
[Java_BwaJni_bwa_1jni] option[2]: /home/hadoop/xubo/ref/GRCH38L1Index/GRCH38chr1L3556522.fasta.
[Java_BwaJni_bwa_1jni] option[3]: /home/hadoop/cloud/workspace/tmplocal-1466761536651-RDD2_1.
[Java_BwaJni_bwa_1jni] option[4]: /home/hadoop/cloud/workspace/tmplocal-1466761536651-RDD2_2.
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[M::process] read 100000 sequences (10000000 bp)...
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44486, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (498.84, 49.88)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 38.847 CPU sec, 38.695 real sec
[M::process] read 100000 sequences (10000000 bp)...
[Stage 3:=======================================>                   (2 + 1) / 3][M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44536, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (499.12, 49.82)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 39.578 CPU sec, 39.324 real sec
[M::process] read 100000 sequences (10000000 bp)...
[Stage 3:=======================================>                   (2 + 1) / 3][M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44545, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (498.84, 50.12)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 38.928 CPU sec, 38.646 real sec
[M::process] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44524, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (466, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (332, 667)
[M::mem_pestat] mean and std.dev: (499.18, 49.89)
[M::mem_pestat] low and high boundaries for proper pairs: (265, 734)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 39.112 CPU sec, 38.839 real sec
[M::process] read 100000 sequences (10000000 bp)...
[Stage 3:=======================================>                   (2 + 1) / 3][M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44500, 1, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (466, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (332, 667)
[M::mem_pestat] mean and std.dev: (499.06, 49.72)
[M::mem_pestat] low and high boundaries for proper pairs: (265, 734)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 39.338 CPU sec, 39.066 real sec
[M::process] read 66666 sequences (6666600 bp)...
[Stage 3:=======================================>                   (2 + 1) / 3][M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44478, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (464, 498, 532)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (328, 668)
[M::mem_pestat] mean and std.dev: (498.47, 49.71)
[M::mem_pestat] low and high boundaries for proper pairs: (260, 736)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 100000 reads in 39.372 CPU sec, 39.132 real sec
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 29691, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 498, 532)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (331, 666)
[M::mem_pestat] mean and std.dev: (498.29, 49.72)
[M::mem_pestat] low and high boundaries for proper pairs: (264, 733)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 66666 reads in 26.443 CPU sec, 26.296 real sec
[main] Version: 0.7.12-r1044
[main] CMD: bwa mem /home/hadoop/xubo/ref/GRCH38L1Index/GRCH38chr1L3556522.fasta /home/hadoop/cloud/workspace/tmplocal-1466761536651-RDD2_1 /home/hadoop/cloud/workspace/tmplocal-1466761536651-RDD2_2
[main] Real time: 269.126 sec; CPU: 879.037 sec
[Java_BwaJni_bwa_1jni] Return code from BWA 0.
hadoop@Master:~/xubo/tools/SparkBWA/build$

参考

【1】https://github.com/xubo245/AdamLearning
【2】https://github.com/bigdatagenomics/adam/ 
【3】https://github.com/xubo245/SparkLearning
【4】http://spark.apache.org
【5】http://stackoverflow.com/questions/28166667/how-to-pass-d-parameter-or-environment-variable-to-spark-job  
【6】http://stackoverflow.com/questions/28840438/how-to-override-sparks-log4j-properties-per-driver

研究成果：

【1】 [BIBM] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Chao Wang, and Xuehai Zhou, "Distributed Gene Clinical Decision Support System Based on Cloud Computing", in IEEE International Conference on Bioinformatics and Biomedicine. (BIBM 2017, CCF B)
【2】 [IEEE CLOUD] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Xuehai Zhou. Efficient Distributed Smith-Waterman Algorithm Based on Apache Spark (CLOUD 2017, CCF-C).
【3】 [CCGrid] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Jinhong Zhou, Xuehai Zhou. DSA: Scalable Distributed Sequence Alignment System Using SIMD Instructions. (CCGrid 2017, CCF-C).
【4】more: https://github.com/xubo245/Publications

Help

If you have any questions or suggestions, please write it in the issue of this project or send an e-mail to me: [email protected]
Wechat: xu601450868
QQ: 601450868

你可能感兴趣的:(基因数据处理)

  • R语言零基础基因/数据差异分析（一） Frms R语言零基础基因/数据差异分析r语言数据分析数据可视化
    文章目录介绍环境搭建软件下载结果展示基因数据下载流程基因数据处理利用GEO分析绘制拟火山图注意，本系列有连贯性，每一步都很详细，每一步都很重要，请耐心读完！！介绍本系列文主要依据真实论文制图流程，详细说明制图过程，其中包括：1.基因数据下载2.制图所需数据格式3.火山图制作流程4.聚类热图制作流程环境搭建软件下载移步至此学习结果展示基因数据处理注意删除末行注释基因数据下载流程以GSE137578基
  • PyVCF 变异基因数据处理 loong_XL 生信生信
    PyVCF是一个用于处理VCF(VariantCallFormat)文件的python库。它提供了许多功能来读取，过滤和修改VCF文件中的变异PyVCF是一个用于读取和写入VCF格式文件的Python库。主要功能包括:1、读取VCF文件：PyVCF提供了一个vcf.Reader()函数，可以用来打开并读取VCF文件。读取后的文件可以进行遍历，每个元素是一个vcf.model._Record的实例2
  • 云上弹性高性能计算，支持生命科学产业高速发展、降本增效 阿里云开发者 人工智能运维数据可视化安全BI数据处理调度云计算数据中心容器阿里云开发者
    简介：生命科学的研究早已离不开高性能计算（HighPerformanceComputing，HPC）的辅助。从计算机辅助药物设计、疫苗研发，到基因数据处理与分析，再到提供精准医疗服务于肿瘤治疗、产前筛查等医疗技术，高性能计算HPC在生命科学研究中扮演着十分重要的角色。而云，又能如何助力产业发展，帮助产业降本增效呢？随着云计算技术服务及实践的日趋成熟，越来越多的行业通过上云实现了整个产业的转型升级，
  • 云上弹性高性能计算，支持生命科学产业高速发展、降本增效 程序员
    简介：生命科学的研究早已离不开高性能计算（HighPerformanceComputing，HPC）的辅助。从计算机辅助药物设计、疫苗研发，到基因数据处理与分析，再到提供精准医疗服务于肿瘤治疗、产前筛查等医疗技术，高性能计算HPC在生命科学研究中扮演着十分重要的角色。而云，又能如何助力产业发展，帮助产业降本增效呢？image.png随着云计算技术服务及实践的日趋成熟，越来越多的行业通过上云实现了整
  • 云上弹性高性能计算，支持生命科学产业高速发展、降本增效 人工智能运维数据可视化安全bi
    简介：生命科学的研究早已离不开高性能计算（HighPerformanceComputing，HPC）的辅助。从计算机辅助药物设计、疫苗研发，到基因数据处理与分析，再到提供精准医疗服务于肿瘤治疗、产前筛查等医疗技术，高性能计算HPC在生命科学研究中扮演着十分重要的角色。而云，又能如何助力产业发展，帮助产业降本增效呢？随着云计算技术服务及实践的日趋成熟，越来越多的行业通过上云实现了整个产业的转型升级，
  • 相似度融合网络：用于聚合不同的基因数据类型：Similarity network fusion for aggregating data types on a genomic scale Ice-iron 论文笔记机器学习神经网络深度学习
    论文标题：Similaritynetworkfusionforaggregatingdatatypesonagenomicscale.论文下载地址论文总结论文以计算机视觉多视图方式为启发，设计了一种图融合网络用于解决基因数据不能综合处理的困难。现有的基因数据非常丰富，有各种类型的基因数据可以利用。但现有的基因数据处理方式大多数是只利用一种基因数据，例如只使用DNA或者是只使用mRNA，不能综合所有
  • 基因数据处理80之disease的DataProcessing KeepLearningBigData 基因数据处理
    1.代码：/***@authorxubo*morecode:https://github.com/xubo245/SparkLearning*moreblog:http://blog.csdn.net/xubo245*/packageorg.gcdss.cli.diseaseimportjava.text.SimpleDateFormatimportjava.util.Dateimportorg.
  • 基因数据处理85之adam-0.18.2无法读取0.14.0使用adamSave存储的数据 KeepLearningBigData 基因数据处理
    1.介绍：cs-bwamem依赖的是adam-0.14.0里面的adamSave存储之后无法用adam-0.18.2的结果读取：2.adam-0.18.2adamsave可以用loadParquetAlignments读取。但无法用loadParquetAlignments读取adam-0.14.23.代码：packageorg.gcdss.cli.testimportjava.nio.file.
  • 基因数据处理73之从HDFS读取fasta文件存为Adam的parquet文件 KeepLearningBigData 基因数据处理
    1.GRCH38chr14：hadoop@Master:~/xubo/project/load$./load.shstart:1SLF4J:Failedtoloadclass"org.slf4j.impl.StaticLoggerBinder".SLF4J:Defaultingtono-operation(NOP)loggerimplementationSLF4J:Seehttp://www.sl
  • 基因数据处理94之使用kmer分析SRR003161数据的kmer分布 KeepLearningBigData 基因数据处理
    1.分两组(1)kmer长度为：5to21(2)kmer长度为：5to55by102.代码：packageorg.gcdss.cliimportjava.text.SimpleDateFormatimportjava.util._importorg.apache.spark._importorg.bdgenomics.adam.projections.{AlignmentRecordField,P
  • 基因数据处理57之BWA-MEM运行single-end(1千万条100bp的reads) KeepLearningBigData 基因数据处理
    ```hadoop@Master:~/cloud/adam/xubo/data/GRCH38Sub/cs-bwamem$bwamemGRCH38BWAindex/GRCH38chr1L3556522.fastag38L100c10000000Nhs20.fq>g38L100c10000000Nhs20.bwamem.sam[M::bwa_idx_load_from_disk]read0ALTcon
  • 基因数据处理75之从HDFS读取vcf文件存为Adam的parquet文件（成功） KeepLearningBigData 基因数据处理
    1.参考：packageorg.bdgenomics.adam.cliclassFlattenSuiteextendsADAMFunSuite{valloader=Thread.currentThread().getContextClassLoadervalinputPath=loader.getResource("small.vcf").getPathvaloutputFile=File.cre
  • 基因数据处理74之从HDFS读取vcf文件存为Adam的parquet文件（有问题） KeepLearningBigData 基因数据处理spark基因数据处理adam
    1.small.vcf:没记录2.读取：5loadtime:3287ms{"variant":{"variantErrorProbability":139,"contig":{"contigName":"1","contigLength":null,"contigMD5":null,"referenceURL":null,"assembly":null,"species":null,"refere
  • 基因数据处理2之ftp数据快速查找 KeepLearningBigData 基因数据处理
    基因数据处理2之ftp数据快速查找linux下可以用grep:curl-s"ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/current.tree"|grepNA12878运行结果：hadoop@Mcnode1:~/cloud/adam/xubo/testAdam34/TestBaiBas$curl-s"ftp://ftp.1000genomes.ebi.ac.
  • 基因数据处理83之移动GRCH38Index到每个节点 KeepLearningBigData 基因数据处理
    1.从cloud/adam移出到xubo/ref:hadoop@Master:~/cloud/adam/xubo/data/test20160310$mkdir-p~/xubo/ref/GRCH38Index/hadoop@Master:~/cloud/adam/xubo/data/test20160310$mvGCA_000001405.15_GRCh38/*~/xubo/ref/GRCH38I
  • 基因数据处理54之bwa-mem运行paird-end（1千万条100bp的reads） KeepLearningBigData 基因数据处理Spark问题
    指令：```hadoop@Master:~/cloud/adam/xubo/data/GRCH38Sub/cs-bwamem$bwamemGRCH38BWAindex/GRCH38chr1L3556522.fastag38L100c10000000Nhs20Paired1.fqg38L100c10000000Nhs20Paired2.fq>g38L100c10000000Nhs20Paired12
  • 基因数据处理1之mapping_to_cram KeepLearningBigData 基因数据处理
    基因数据处理1之mapping_to_cram参考资料：AWorkedExampleObtainsomepublicdataWewillusethefirst100,000read-pairsfromayeastdataset.curlftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR507/SRR507778/SRR507778_1.fastq.gz|gzip-d|he
  • 基因数据处理72之GATK安装成功 KeepLearningBigData 基因数据处理
    1.下载：gitclonehttps://github.com/broadgsa/gatk-protected.git2.安装：gitcheckout3.5mvncleanpackage-DskipTests3.安装成功：[INFO]ReactorSummary:[INFO][INFO]GATKRoot..........................................SUCCES
  • 基因数据处理77之从vcf文件中提取某条染色体的数据 KeepLearningBigData 基因数据处理
    1.代码：/***@authorxubo*/packageorg.gcdss.cli.vcfimportorg.apache.spark.{SparkConf,SparkContext}/***Createdbyxuboon2016/5/23.*/objectextractGRCH38chr20vcf{defmain(args:Array[String]){valconf=newSparkConf
  • 基因数据处理34之使用samtools和bcftools进行变异分析 KeepLearningBigData 基因数据处理
    1.指令：（1）samtoolsmpileup-vfHomo_sapiens_assembly19chr20.fastaNA12878_snp_A2G_chr20_225058.sorted.bam>NA12878_snp_A2G_chr20_225058.variants或者：samtoolsmpileup-vfHomo_sapiens_assembly19chr20.fastaNA12878_
  • 基因数据处理116之重新运行SparkBWA Yarn集群模式 KeepLearningBigData 基因数据处理
    更多代码请见：https://github.com/xubo245基因数据处理系列1.解释很久没运行SparkBWA了，系统文件有点多，重新运行。2.代码：endhadoop@Master:~/disk2/xubo/project/alignment/sparkBWA$vig38L100c100000Nhs20Paired12SparkBWAYarnPartition0.shecho"start"
  • 基因数据处理117之重新多次运行SparkBWA Yarn集群 KeepLearningBigData 基因数据处理
    更多代码请见：https://github.com/xubo245基因数据处理系列1.解释重新运行，跟换了文件地址后来终端，需要运行其他的2.代码：hadoop@Master:~/disk2/xubo/project/alignment/sparkBWA$visparkBWA.shforjin10000100000100000010000000doforiin50100doecho$iecho$j
  • 基因数据处理112之运行gcdss的avocado编译识别报错getRecordGroupSample空指针异常解决办法 KeepLearningBigData 基因数据处理
    前面一片已经讲了遇到的问题，之前也遇到过：基因数据处理31之avocado运行avocado-cli中的avocado问题3-变异识别找不到RecordGroupSample（null）http://blog.csdn.net/xubo245/article/details/51525241解决办法：在读入的sam/bam/adam数据进行判定，如果RecordGroup等数据为空，则加上默认值在
  • 基因数据处理110之批量测试SparkBWA和CS-bwamem算法 KeepLearningBigData 基因数据处理
    1.总的脚本hadoop@Master:~/xubo/project/alignment$catrunSparkBWAAndCsbwamem.shecho"startsparkBWA"./sparkBWA/sparkBWA.shecho"startCsbwamemalgin"./cs-bwamem/csbwamemAlign.shecho"startCsbwamemmerge"./cs-bwame
  • 基因数据处理106之bwa-mem运行paird-end（1千万条100bp的reads g38L100c10000000Nhs20Paired12） KeepLearningBigData 基因数据处理
    脚本：hadoop@Master:~/xubo/project/alignment/sparkBWA$catg38L100c10000000Nhs20Paired12Bwamem.shecho"start"startTime4=`date+"%s.%N"`time4=`date+"%Y%m%d%H%M%S"`#spark-submit--classorg.apache.spark.examples
  • 基因数据处理113之对avocado识别的SparkBWA变异数据进行疾病分析_ KeepLearningBigData 基因数据处理
    （一）问题问题1：avocado数据读取：avocado存到磁盘是：RDD[Genotype]valprocessedGenotypes:RDD[Genotype]=postProcessVariants(calledVariants,stats).flatMap(variantContext=>variantContext.genotypes)//savevariantstooutputfile
  • 基因数据处理109之SparkBWA运行成功的部分log KeepLearningBigData 基因数据处理
    数据来自：http://219.219.220.149:18080/history/application_1466866953605_0001/executors/http://219.219.220.248:8042/node/containerlogs/container_1466866953605_0001_01_000003/hadoop/stderr/?start=0SLF4J:Cla
  • Spark问题14之Spark stage retry问题 KeepLearningBigData Spark问题
    更多代码请见：https://github.com/xubo245基因数据处理系列之SparkBWA1.解释1.1简述当partitions超过节点数量的时候Lostexecutor的问题，已经提交到SparkBWA中，https://github.com/citiususc/SparkBWA/issues/35另外发现，tmp里面有临时文件没有删除，而且stageretry未解决2.记录完整报错
  • 基因数据处理102之SparkBWA本地运行100万条paired-reads实例 KeepLearningBigData 基因数据处理
    脚本：spark-submit--classSparkBWA\--masterlocal\--archivesbwa.zip\SparkBWA.jar\-algorithmmem-readspaired\-index/home/hadoop/xubo/ref/GRCH38L1Index/GRCH38chr1L3556522.fasta\-partitions3\/xubo/alignment/sp
  • 基因数据处理101之SparkBWA本地运行配置和实例 KeepLearningBigData 基因数据处理
    1.修改Makefile.common：将LIBBWA_LIBS=-lrt改为LIBBWA_LIBS=-lrt-lz不然会报错误【5】2.make之后修改java.library.path步骤：vi/etc/profile加入exportLD_LIBRARY_PATH=/home/hadoop/xubo/tools/SparkBWA/build:$LD_LIBRARY_PATH使生效：source
  • LeetCode[位运算] - #137 Single Number II Cwind javaAlgorithmLeetCode题解位运算
    原题链接：#137 Single Number II  要求： 给定一个整型数组，其中除了一个元素之外，每个元素都出现三次。找出这个元素 注意：算法的时间复杂度应为O(n)，最好不使用额外的内存空间   难度：中等   分析： 与#136类似，都是考察位运算。不过出现两次的可以使用异或运算的特性 n XOR n = 0, n XOR 0 = n，即某一
  • 《JavaScript语言精粹》笔记 aijuans JavaScript
    0、JavaScript的简单数据类型包括数字、字符创、布尔值（true/false）、null和undefined值，其它值都是对象。 1、JavaScript只有一个数字类型，它在内部被表示为64位的浮点数。没有分离出整数，所以1和1.0的值相同。 2、NaN是一个数值，表示一个不能产生正常结果的运算结果。NaN不等于任何值，包括它本身。可以用函数isNaN(number)检测NaN,但是
  • 你应该更新的Java知识之常用程序库 Kai_Ge java
    在很多人眼中，Java 已经是一门垂垂老矣的语言，但并不妨碍 Java 世界依然在前进。如果你曾离开 Java，云游于其它世界，或是每日只在遗留代码中挣扎，或许是时候抬起头，看看老 Java 中的新东西。 Guava Guava[gwɑ:və]，一句话，只要你做Java项目，就应该用Guava（Github）。 guava 是 Google 出品的一套 Java 核心库，在我看来，它甚至应该
  • HttpClient 120153216 httpclient
    /** * 可以传对象的请求转发，对象已流形式放入HTTP中 */ public static Object doPost(Map<String,Object> parmMap,String url) { Object object = null; HttpClient hc = new HttpClient(); String fullURL
  • Django model字段类型清单 2002wmj django
    Django 通过 models 实现数据库的创建、修改、删除等操作，本文为模型中一般常用的类型的清单，便于查询和使用： AutoField：一个自动递增的整型字段，添加记录时它会自动增长。你通常不需要直接使用这个字段；如果你不指定主键的话，系统会自动添加一个主键字段到你的model。(参阅自动主键字段) BooleanField：布尔字段,管理工具里会自动将其描述为checkbox。 Cha
  • 在SQLSERVER中查找消耗CPU最多的SQL 357029540 SQL Server
    返回消耗CPU数目最多的10条语句 SELECT TOP 10    total_worker_time/execution_count AS avg_cpu_cost, plan_handle,    execution_count,    (SELECT SUBSTRING(text, statement_start_of
  • Myeclipse项目无法部署，Undefined exploded archive location 7454103 eclipseMyEclipse
    做个备忘！ 错误信息为：       Undefined exploded archive location 原因：           在工程转移过程中，导致工程的配置文件出错； 解决方法：    
  • GMT时间格式转换 adminjun GMT时间转换
    普通的时间转换问题我这里就不再罗嗦了，我想大家应该都会那种低级的转换问题吧，现在我向大家总结一下如何转换GMT时间格式，这种格式的转换方法网上还不是很多，所以有必要总结一下，也算给有需要的朋友一个小小的帮助啦。 1、可以使用 SimpleDateFormat SimpleDateFormat    EEE-三位星期 d-天 MMM-月 yyyy-四位年
  • Oracle数据库新装连接串问题 aijuans oracle数据库
    割接新装了数据库，客户端登陆无问题，apache/cgi-bin程序有问题，sqlnet.log日志如下： Fatal NI connect error 12170.   VERSION INFORMATION:         TNS for Linux: Version 10.2.0.4.0 - Product
  • 回顾java数组复制 ayaoxinchao java数组
    在写这篇文章之前，也看了一些别人写的，基本上都是大同小异。文章是对java数组复制基础知识的回顾，算是作为学习笔记，供以后自己翻阅。首先，简单想一下这个问题：为什么要复制数组？我的个人理解：在我们在利用一个数组时，在每一次使用，我们都希望它的值是初始值。这时我们就要对数组进行复制，以达到原始数组值的安全性。java数组复制大致分为3种方式：①for循环方式 ②clone方式 ③arrayCopy方
  • java web会话监听并使用spring注入 bewithme Java Web
            在java web应用中，当你想在建立会话或移除会话时，让系统做某些事情，比如说，统计在线用户，每当有用户登录时，或退出时，那么可以用下面这个监听器来监听。        import java.util.ArrayList; import java.ut
  • NoSQL数据库之Redis数据库管理(Redis的常用命令及高级应用) bijian1013 redis数据库NoSQL
    一 .Redis常用命令         Redis提供了丰富的命令对数据库和各种数据库类型进行操作，这些命令可以在Linux终端使用。         a.键值相关命令         b.服务器相关命令 1.键值相关命令       &
  • java枚举序列化问题 bingyingao java枚举序列化
    对象在网络中传输离不开序列化和反序列化。而如果序列化的对象中有枚举值就要特别注意一些发布兼容问题: 1.加一个枚举值 新机器代码读分布式缓存中老对象，没有问题，不会抛异常。 老机器代码读分布式缓存中新对像，反序列化会中断，所以在所有机器发布完成之前要避免出现新对象，或者提前让老机器拥有新增枚举的jar。 2.删一个枚举值 新机器代码读分布式缓存中老对象，反序列
  • 【Spark七十八】Spark Kyro序列化 bit1129 spark
    当使用SparkContext的saveAsObjectFile方法将对象序列化到文件，以及通过objectFile方法将对象从文件反序列出来的时候，Spark默认使用Java的序列化以及反序列化机制，通常情况下，这种序列化机制是很低效的，Spark支持使用Kyro作为对象的序列化和反序列化机制，序列化的速度比java更快，但是使用Kyro时要注意，Kyro目前还是有些bug。 Spark
  • Hybridizing OO and Functional Design bookjovi erlanghaskell
      推荐博文： Tell Above, and Ask Below - Hybridizing OO and Functional Design 文章中把OO和FP讲的深入透彻，里面把smalltalk和haskell作为典型的两种编程范式代表语言，此点本人极为同意，smalltalk可以说是最能体现OO设计的面向对象语言，smalltalk的作者Alan kay也是OO的最早先驱，
  • Java-Collections Framework学习与总结-HashMap BrokenDreams Collections
            开发中常常会用到这样一种数据结构，根据一个关键字，找到所需的信息。这个过程有点像查字典，拿到一个key，去字典表中查找对应的value。Java1.0版本提供了这样的类java.util.Dictionary(抽象类)，基本上支持字典表的操作。后来引入了Map接口，更好的描述的这种数据结构。  &nb
  • 读《研磨设计模式》-代码笔记-职责链模式-Chain Of Responsibility bylijinnan java设计模式
    声明： 本文只为方便我个人查阅和理解，详细的分析以及源代码请移步 原作者的博客http://chjavach.iteye.com/ /** * 业务逻辑：项目经理只能处理500以下的费用申请，部门经理是1000，总经理不设限。简单起见，只同意“Tom”的申请 * bylijinnan */ abstract class Handler { /*
  • Android中启动外部程序 cherishLC android
    1、启动外部程序 引用自： http://blog.csdn.net/linxcool/article/details/7692374 //方法一 Intent intent=new Intent(); //包名 包名+类名（全路径） intent.setClassName("com.linxcool", "com.linxcool.PlaneActi
  • summary_keep_rate coollyj SUM
    BEGIN /*DECLARE minDate varchar(20) ; DECLARE maxDate varchar(20) ;*/ DECLARE stkDate varchar(20) ; DECLARE done int default -1; /* 游标中 注册服务器地址 */ DE
  • hadoop hdfs 添加数据目录出错 daizj hadoophdfs扩容
    由于原来配置的hadoop data目录快要用满了，故准备修改配置文件增加数据目录，以便扩容，但由于疏忽，把core-site.xml, hdfs-site.xml配置文件dfs.datanode.data.dir 配置项增加了配置目录，但未创建实际目录，重启datanode服务时，报如下错误： 2014-11-18 08:51:39,128 WARN org.apache.hadoop.h
  • grep 目录级联查找 dongwei_6688 grep
           在Mac或者Linux下使用grep进行文件内容查找时，如果给定的目标搜索路径是当前目录，那么它默认只搜索当前目录下的文件，而不会搜索其下面子目录中的文件内容，如果想级联搜索下级目录，需要使用一个“-r”参数： grep -n -r "GET" .   上面的命令将会找出当前目录“.”及当前目录中所有下级目录
  • yii 修改模块使用的布局文件 dcj3sjt126com yiilayouts
    方法一：yii模块默认使用系统当前的主题布局文件，如果在主配置文件中配置了主题比如: 'theme'=>'mythm', 那么yii的模块就使用 protected/themes/mythm/views/layouts 下的布局文件； 如果未配置主题，那么 yii的模块就使用  protected/views/layouts 下的布局文件， 总之默认不是使用自身目录 pr
  • 设计模式之单例模式 come_for_dream 设计模式单例模式懒汉式饿汉式双重检验锁失败无序写入
                    今天该来的面试还没来，这个店估计不会来电话了，安静下来写写博客也不错，没事翻了翻小易哥的博客甚至与大牛们之间的差距，基础知识不扎实建起来的楼再高也只能是危楼罢了，陈下心回归基础把以前学过的东西总结一下。   *********************************
  • 8、数组 豆豆咖啡 二维数组数组一维数组
      一、概念       数组是同一种类型数据的集合。其实数组就是一个容器。   二、好处       可以自动给数组中的元素从0开始编号，方便操作这些元素   三、格式   //一维数组 1,元素类型[] 变量名 = new 元素类型[元素的个数] int[] arr =
  • Decode Ways hcx2013 decode
    A message containing letters from A-Z is being encoded to numbers using the following mapping: 'A' -> 1 'B' -> 2 ... 'Z' -> 26 Given an encoded message containing digits, det
  • Spring4.1新特性——异步调度和事件机制的异常处理 jinnianshilongnian spring 4.1
    目录 Spring4.1新特性——综述 Spring4.1新特性——Spring核心部分及其他 Spring4.1新特性——Spring缓存框架增强 Spring4.1新特性——异步调用和事件机制的异常处理 Spring4.1新特性——数据库集成测试脚本初始化 Spring4.1新特性——Spring MVC增强 Spring4.1新特性——页面自动化测试框架Spring MVC T
  • squid3(高命中率)缓存服务器配置 liyonghui160com
        系统:centos 5.x   需要的软件:squid-3.0.STABLE25.tar.gz 1.下载squid wget http://www.squid-cache.org/Versions/v3/3.0/squid-3.0.STABLE25.tar.gz tar zxf squid-3.0.STABLE25.tar.gz &&
  • 避免Java应用中NullPointerException的技巧和最佳实践 pda158 java
    1) 从已知的String对象中调用equals()和equalsIgnoreCase()方法，而非未知对象。 　　总是从已知的非空String对象中调用equals()方法。因为equals()方法是对称的，调用a.equals(b)和调用b.equals(a)是完全相同的，这也是为什么程序员对于对象a和b这么不上心。如果调用者是空指针，这种调用可能导致一个空指针异常 Object unk
  • 如何在Swift语言中创建http请求 shoothao httpswift
    概述：本文通过实例从同步和异步两种方式上回答了”如何在Swift语言中创建http请求“的问题。 如果你对Objective-C比较了解的话，对于如何创建http请求你一定驾轻就熟了，而新语言Swift与其相比只有语法上的区别。但是，对才接触到这个崭新平台的初学者来说，他们仍然想知道“如何在Swift语言中创建http请求？”。 在这里,我将作出一些建议来回答上述问题。常见的
  • Spring事务的传播方式 uule spring事务
    传播方式：        新建事务       required       required_new   - 挂起当前         非事务方式运行       supports   &nbs
按字母分类： ABCDEFGHIJKLMNOPQRSTUVWXYZ其他