基因数据处理112之运行gcdss的avocado编译识别报错getRecordGroupSample空指针异常解决办法

前面一片已经讲了遇到的问题，之前也遇到过：

基因数据处理31之avocado运行avocado-cli中的avocado问题3-变异识别找不到RecordGroupSample（null）
http://blog.csdn.net/xubo245/article/details/51525241

解决办法：

在读入的sam/bam/adam数据进行判定，如果RecordGroup等数据为空，则加上默认值

在gcdss/avocado中的run方法后

val reads: RDD[AlignmentRecord] = LoadReads.time {
      Input(sc, args.readInput, reference, config)
    }

修改并加入：

    val readsTmp: RDD[AlignmentRecord] = LoadReads.time {
      Input(sc, args.readInput, reference, config)
    }
    val reads = readsTmp.map { each =>
      if (each.getRecordGroupSample == null) {
        each.setRecordGroupSample("sample")
      }
      if (each.getRecordGroupName == null) {
        each.setRecordGroupName("read_group_id")
      }
      if (each.getRecordGroupPlatform == null) {
        each.setRecordGroupPlatform("illumina")
      }
      if (each.getRecordGroupPlatformUnit == null) {
        each.setRecordGroupPlatformUnit("platform_unit")
      }
      if (each.getRecordGroupLibrary == null) {
        each.setRecordGroupLibrary("library")
      }
      each
    }

然后就可以运行SparkBWA或者bwamem处理的sam数据了。

处理代码：

hadoop@Master:~/xubo/project/callVariant/sparkBWAResult$ cat runBBg38L50c10000Nhs20Paired12YarnT201606252236LocalNopartition.sh 
    #!/usr/bin/env bash  
    spark-submit   \
--class  org.gcdss.cli.callVariant.callVariantByAvocado \
--master spark://219.219.220.149:7077 \
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
--conf spark.kryo.registrator=org.bdgenomics.adam.serialization.ADAMKryoRegistrator \
--jars /home/hadoop/cloud/adam/lib/adam-apis_2.10-0.18.3-SNAPSHOT.jar,/home/hadoop/cloud/adam/lib/adam-cli_2.10-0.18.3-SNAPSHOT.jar,/home/hadoop/cloud/adam/lib/adam-core_2.10-0.18.3-SNAPSHOT.jar,/home/hadoop/cloud/adam/xubo/data/GRCH38Sub/cs-bwamem/BWAMEMSparkAll/gcdss-cli-0.0.3-SNAPSHOT.jar \
--executor-memory 4096M \
--total-executor-cores 20 BWAMEMSparkAll.jar \
/xubo/alignment/output/sparkBWA/BBg38L50c10000Nhs20Paired12YarnT201606252236LocalNopartition/SparkBWA_g38L50c10000Nhs20Paired1.fastq-0-NoSort-local-1466932122385-0WithRG.sam \
/xubo/ref/GRCH38L1Index/GRCH38chr1L3556522.fasta \
/xubo/callVariant/avocado/output/sparkBWA/BBg38L50c10000Nhs20Paired12YarnT201606252236LocalNopartition/SparkBWA_g38L50c10000Nhs20Paired1.fastq-0-NoSort-local-1466932122385-0.samVar \
/home/hadoop/xubo/data/testTools/basic.properties

运行结果：

hadoop@Master:~/xubo/project/callVariant/sparkBWAResult$ ./runBBg38L50c10000Nhs20Paired12YarnT201606252236LocalNopartition.sh 
start main:
start run:
fqFile:/xubo/alignment/output/sparkBWA/BBg38L50c10000Nhs20Paired12YarnT201606252236LocalNopartition/SparkBWA_g38L50c10000Nhs20Paired1.fastq-0-NoSort-local-1466932122385-0WithRG.sam
faFile:/xubo/ref/GRCH38L1Index/GRCH38chr1L3556522.fasta
configFile:/home/hadoop/xubo/data/testTools/basic.properties
output:/xubo/callVariant/avocado/output/sparkBWA/BBg38L50c10000Nhs20Paired12YarnT201606252236LocalNopartition/SparkBWA_g38L50c10000Nhs20Paired1.fastq-0-NoSort-local-1466932122385-0.samVar20160627101927816
fqLoad.count:18480
faLoad.count:24896                                                              
[Stage 11:======================================>                   (2 + 1) / 3]16/06/27 10:30:28 ERROR TaskSchedulerImpl: Lost executor 3 on 219.219.220.149: Executor heartbeat timed out after 157049 ms
[Stage 11:======================================>                   (2 + 1) / 3]16/06/27 10:35:28 ERROR TaskSchedulerImpl: Lost executor 3 on 219.219.220.149: remote Rpc client disassociated
[Stage 12:==============>                                           (1 + 3) / 4]16/06/27 10:57:26 ERROR TaskSchedulerImpl: Lost executor 154 on 219.219.220.149: Executor heartbeat timed out after 154828 ms
[Stage 12:==============>                                           (1 + 3) / 4]16/06/27 10:58:39 ERROR TaskSchedulerImpl: Lost executor 154 on 219.219.220.149: remote Rpc client disassociated
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".                
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
read parquet:5677                                                               
*************end*************       

数据查看：

package org.gcdss.cli.load

import org.apache.spark.{SparkConf, SparkContext}
import org.bdgenomics.adam.rdd.ADAMContext._

//import org.bdgenomics.avocado.AvocadoFunSuite

object loadVcfFromAdamSparkBWAResult {
  //  def resourcePath(path: String) = ClassLoader.getSystemClassLoader.getResource(path).getFile

  //  def tmpFile(path: String) = Files.createTempDirectory("").toAbsolutePath.toString + "/" + path

  //  def apply(local: Boolean, fqFile: String, faFile: String, configFile: String, output: String) {
  def main(args: Array[String]) {
    println("start:")
    //    var conf = new SparkConf().setAppName(this.getClass().getSimpleName().filter(!_.equals('$'))).setMaster("spark://219.219.220.149:7077")
    var conf = new SparkConf().setAppName("AvocadoSuite").setMaster("local[4]")
    val sc = new SparkContext(conf)
    val startTime = System.currentTimeMillis()
    //    val path = "hdfs://219.219.220.149:9000/xubo/callVariant/vcf/All_20160407.vcf"
    //    val path = "hdfs://219.219.220.149:9000/xubo/callVariant/vcf/small.vcf"
//    val path = "file/callVariant/small.adam"
    val path = "hdfs://219.219.220.149:9000/xubo/callVariant/avocado/output/sparkBWA/BBg38L50c10000Nhs20Paired12YarnT201606252236LocalNopartition/SparkBWA_g38L50c10000Nhs20Paired1.fastq-0-NoSort-local-1466932122385-0.samVarNoRG20160627113153282/"

    val rdd = sc.loadVariantAnnotations(path)
    println(rdd.count())
    val loadTime = System.currentTimeMillis()
    println("load time:" + (loadTime - startTime) + " ms")
    //    rdd.take(10).foreach(println)
    rdd.foreach(println)

    val saveTime = System.currentTimeMillis()
    println("rdd.count:"+rdd.count())
    println("run time:" + (saveTime - startTime) + " ms")
    println("*************end*************")
    sc.stop()

  }

}

部分结果：

{"variant": {"variantErrorProbability": null, "contig": {"contigName": "chr1", "contigLength": 248956422, "contigMD5": null, "referenceURL": null, "assembly": null, "species": null, "referenceIndex": null}, "start": 248855372, "end": 248855373, "referenceAllele": "C", "alternateAllele": "G", "svAllele": null, "isSomatic": false}, "variantCallingAnnotations": {"variantIsPassing": null, "variantFilters": [], "downsampled": null, "baseQRankSum": null, "fisherStrandBiasPValue": "Infinity", "rmsMapQ": 60.0, "mapq0Reads": null, "mqRankSum": null, "readPositionRankSum": null, "genotypePriors": [], "genotypePosteriors": [], "vqslod": null, "culprit": null, "attributes": {}}, "sampleId": "sample", "sampleDescription": null, "processingDescription": null, "alleles": ["Ref", "Alt"], "expectedAlleleDosage": null, "referenceReadDepth": 0, "alternateReadDepth": 1, "readDepth": 1, "minReadDepth": null, "genotypeQuality": 5, "genotypeLikelihoods": [-3.9818644E-4, -0.6931472, -7.828789], "nonReferenceLikelihoods": [-3.9818644E-4, -0.6931472, -7.828789], "strandBiasComponents": [], "splitFromMultiAllelic": false, "isPhased": false, "phaseSetId": null, "phaseQuality": null}
{"variant": {"variantErrorProbability": null, "contig": {"contigName": "chr1", "contigLength": 248956422, "contigMD5": null, "referenceURL": null, "assembly": null, "species": null, "referenceIndex": null}, "start": 248934548, "end": 248934549, "referenceAllele": "G", "alternateAllele": "A", "svAllele": null, "isSomatic": false}, "variantCallingAnnotations": {"variantIsPassing": null, "variantFilters": [], "downsampled": null, "baseQRankSum": null, "fisherStrandBiasPValue": "Infinity", "rmsMapQ": 60.0, "mapq0Reads": null, "mqRankSum": null, "readPositionRankSum": null, "genotypePriors": [], "genotypePosteriors": [], "vqslod": null, "culprit": null, "attributes": {}}, "sampleId": "sample", "sampleDescription": null, "processingDescription": null, "alleles": ["Ref", "Alt"], "expectedAlleleDosage": null, "referenceReadDepth": 0, "alternateReadDepth": 1, "readDepth": 1, "minReadDepth": null, "genotypeQuality": 3, "genotypeLikelihoods": [-7.946439E-4, -0.6931472, -7.138014], "nonReferenceLikelihoods": [-7.946439E-4, -0.6931472, -7.138014], "strandBiasComponents": [], "splitFromMultiAllelic": false, "isPhased": false, "phaseSetId": null, "phaseQuality": null}
{"variant": {"variantErrorProbability": null, "contig": {"contigName": "chr1", "contigLength": 248956422, "contigMD5": null, "referenceURL": null, "assembly": null, "species": null, "referenceIndex": null}, "start": 248934572, "end": 248934573, "referenceAllele": "C", "alternateAllele": "A", "svAllele": null, "isSomatic": false}, "variantCallingAnnotations": {"variantIsPassing": null, "variantFilters": [], "downsampled": null, "baseQRankSum": null, "fisherStrandBiasPValue": "Infinity", "rmsMapQ": 60.0, "mapq0Reads": null, "mqRankSum": null, "readPositionRankSum": null, "genotypePriors": [], "genotypePosteriors": [], "vqslod": null, "culprit": null, "attributes": {}}, "sampleId": "sample", "sampleDescription": null, "processingDescription": null, "alleles": ["Ref", "Alt"], "expectedAlleleDosage": null, "referenceReadDepth": 0, "alternateReadDepth": 1, "readDepth": 1, "minReadDepth": null, "genotypeQuality": 15, "genotypeLikelihoods": [-2.511918E-5, -0.6931472, -10.591891], "nonReferenceLikelihoods": [-2.511918E-5, -0.6931472, -10.591891], "strandBiasComponents": [], "splitFromMultiAllelic": false, "isPhased": false, "phaseSetId": null, "phaseQuality": null}
{"variant": {"variantErrorProbability": null, "contig": {"contigName": "chr1", "contigLength": 248956422, "contigMD5": null, "referenceURL": null, "assembly": null, "species": null, "referenceIndex": null}, "start": 248943770, "end": 248943771, "referenceAllele": "A", "alternateAllele": "T", "svAllele": null, "isSomatic": false}, "variantCallingAnnotations": {"variantIsPassing": null, "variantFilters": [], "downsampled": null, "baseQRankSum": null, "fisherStrandBiasPValue": "Infinity", "rmsMapQ": 60.0, "mapq0Reads": null, "mqRankSum": null, "readPositionRankSum": null, "genotypePriors": [], "genotypePosteriors": [], "vqslod": null, "culprit": null, "attributes": {}}, "sampleId": "sample", "sampleDescription": null, "processingDescription": null, "alleles": ["Ref", "Alt"], "expectedAlleleDosage": null, "referenceReadDepth": 0, "alternateReadDepth": 1, "readDepth": 1, "minReadDepth": null, "genotypeQuality": 3, "genotypeLikelihoods": [-7.946439E-4, -0.6931472, -7.138014], "nonReferenceLikelihoods": [-7.946439E-4, -0.6931472, -7.138014], "strandBiasComponents": [], "splitFromMultiAllelic": false, "isPhased": false, "phaseSetId": null, "phaseQuality": null}
{"variant": {"variantErrorProbability": null, "contig": {"contigName": "chr1", "contigLength": 248956422, "contigMD5": null, "referenceURL": null, "assembly": null, "species": null, "referenceIndex": null}, "start": 248943800, "end": 248943801, "referenceAllele": "A", "alternateAllele": "G", "svAllele": null, "isSomatic": false}, "variantCallingAnnotations": {"variantIsPassing": null, "variantFilters": [], "downsampled": null, "baseQRankSum": null, "fisherStrandBiasPValue": "Infinity", "rmsMapQ": 60.0, "mapq0Reads": null, "mqRankSum": null, "readPositionRankSum": null, "genotypePriors": [], "genotypePosteriors": [], "vqslod": null, "culprit": null, "attributes": {}}, "sampleId": "sample", "sampleDescription": null, "processingDescription": null, "alleles": ["Ref", "Alt"], "expectedAlleleDosage": null, "referenceReadDepth": 0, "alternateReadDepth": 1, "readDepth": 1, "minReadDepth": null, "genotypeQuality": 3, "genotypeLikelihoods": [-7.946439E-4, -0.6931472, -7.138014], "nonReferenceLikelihoods": [-7.946439E-4, -0.6931472, -7.138014], "strandBiasComponents": [], "splitFromMultiAllelic": false, "isPhased": false, "phaseSetId": null, "phaseQuality": null}
rdd.count:5677
run time:8708 ms
*************end*************

参考

【1】https://github.com/xubo245/AdamLearning
【2】https://github.com/bigdatagenomics/adam/ 
【3】https://github.com/xubo245/SparkLearning
【4】http://spark.apache.org

研究成果：

【1】 [BIBM] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Chao Wang, and Xuehai Zhou, "Distributed Gene Clinical Decision Support System Based on Cloud Computing", in IEEE International Conference on Bioinformatics and Biomedicine. (BIBM 2017, CCF B)
【2】 [IEEE CLOUD] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Xuehai Zhou. Efficient Distributed Smith-Waterman Algorithm Based on Apache Spark (CLOUD 2017, CCF-C).
【3】 [CCGrid] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Jinhong Zhou, Xuehai Zhou. DSA: Scalable Distributed Sequence Alignment System Using SIMD Instructions. (CCGrid 2017, CCF-C).
【4】more: https://github.com/xubo245/Publications

Help

If you have any questions or suggestions, please write it in the issue of this project or send an e-mail to me: [email protected]
Wechat: xu601450868
QQ: 601450868